Comparison of Transfer Learning Performance in Lung and Colon Classification with Knowledge Distillation
DOI:
https://doi.org/10.59247/jahir.v2i2.289Keywords:
Lung, Colon, Knowledge, Distillation, Transfer LearningAbstract
This research aims to apply the knowledge distillation method to medical image classification, specifically in the case of lung and colon image classification using various transfer learning models. Knowledge distillation allows the transfer of knowledge from a larger model (teacher) to a smaller model (student), which enables more efficient model building without sacrificing accuracy. In this research, the DenseNet169 model is used as the teacher model. The student model uses several alternative transfer learning architectures such as DenseNet121, MobileNet, ResNet50, InceptionV3, and Xception. The data used consists of 25,000 histopathology images that have been processed and divided into training, validation, and test data. Data augmentation was performed to enlarge the dataset from 750 to 25,000 images, which helped improve the performance of the model. Model performance evaluation was performed by measuring the accuracy and loss value of each student model compared to the teacher model. The results showed that the student models generated through the knowledge distillation process performed close to or even exceeded the teacher model in some cases, with the Xception model showing the highest accuracy of 96.95%. In conclusion, knowledge distillation is effective in reducing model complexity without compromising performance, which is particularly beneficial for implementation on resource-constrained devices.
References
A. Hosny, C. Parmar, J. Quackenbush, L. H. Schwartz, and H. J. W. L. Aerts, “Artificial intelligence in radiology,” Nat Rev Cancer, vol. 18, no. 8, pp. 500–510, 2018, doi: 10.1038/s41568-018-0016-5.
S. Tresker, “A typology of clinical conditions,” Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences, vol. 83, p. 101291, 2020, doi: https://doi.org/10.1016/j.shpsc.2020.101291.
B. Jiang, D. Xie, S. Wang, X. Li, and G. Wu, “Advances in early detection methods for solid tumors,” Front Genet, vol. 14, 2023, [Online]. Available: https://www.frontiersin.org/journals/genetics/articles/10.3389/fgene.2023.1091223
M. Mamun, M. I. Mahmud, M. Meherin, and A. Abdelgawad, “LCDctCNN: Lung Cancer Diagnosis of CT scan Images Using CNN Based Model,” in 2023 10th International Conference on Signal Processing and Integrated Networks (SPIN), 2023, pp. 205–212. doi: 10.1109/SPIN57001.2023.10116075.
J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge Distillation: A Survey,” Int J Comput Vis, vol. 129, no. 6, pp. 1789–1819, 2021, doi: 10.1007/s11263-021-01453-z.
F. Sarfraz, E. Arani, and B. Zonooz, “Knowledge Distillation Beyond Model Compression,” in 2020 25th International Conference on Pattern Recognition (ICPR), 2021, pp. 6136–6143. doi: 10.1109/ICPR48806.2021.9413016.
J. Nayem et al., “Few Shot Learning for Medical Imaging: A Comparative Analysis of Methodologies and Formal Mathematical Framework,” in Data Driven Approaches on Medical Imaging, B. Zheng, S. Andrei, M. K. Sarker, and K. D. Gupta, Eds., Cham: Springer Nature Switzerland, 2023, pp. 69–90. doi: 10.1007/978-3-031-47772-0_4.
S. Byeon, J. Park, Y. A. Cho, and B.-J. Cho, “Automated histological classification for digital pathology images of colonoscopy specimen via deep learning,” Sci Rep, vol. 12, no. 1, p. 12804, 2022, doi: 10.1038/s41598-022-16885-x.
Md. A. Talukder, Md. M. Islam, M. A. Uddin, A. Akhter, K. F. Hasan, and M. A. Moni, “Machine learning-based lung and colon cancer detection using deep feature extraction and ensemble learning,” Expert Syst Appl, vol. 205, p. 117695, 2022, doi: https://doi.org/10.1016/j.eswa.2022.117695.
M. A. Balcı, L. M. Batrancea, Ö. Akgüller, and A. Nichita, “A Series-Based Deep Learning Approach to Lung Nodule Image Classification,” Cancers (Basel), vol. 15, no. 3, 2023, doi: 10.3390/cancers15030843.
G. M. M. Alshmrani, Q. Ni, R. Jiang, H. Pervaiz, and N. M. Elshennawy, “A deep learning architecture for multi-class lung diseases classification using chest X-ray (CXR) images,” Alexandria Engineering Journal, vol. 64, pp. 923–935, 2023, doi: https://doi.org/10.1016/j.aej.2022.10.053.
I. Naseer, S. Akram, T. Masood, M. Rashid, and A. Jaffar, “Lung Cancer Classification Using Modified U-Net Based Lobe Segmentation and Nodule Detection,” IEEE Access, vol. 11, pp. 60279–60291, 2023, doi: 10.1109/ACCESS.2023.3285821.
B. R. Pandit et al., “Deep learning neural network for lung cancer classification: enhanced optimization function,” Multimed Tools Appl, vol. 82, no. 5, pp. 6605–6624, 2023, doi: 10.1007/s11042-022-13566-9.
J. Gou, X. Xiong, B. Yu, L. Du, Y. Zhan, and D. Tao, “Multi-target Knowledge Distillation via Student Self-reflection,” Int J Comput Vis, vol. 131, no. 7, pp. 1857–1874, 2023, doi: 10.1007/s11263-023-01792-z.
Z. Tao, Q. Xia, S. Cheng, and Q. Li, “An Efficient and Robust Cloud-Based Deep Learning with Knowledge Distillation,” IEEE Transactions on Cloud Computing, vol. 11, no. 02, pp. 1733–1745, 2023, doi: 10.1109/TCC.2022.3160129.
A. Hoyle, P. Goel, and P. Resnik, Improving Neural Topic Models using Knowledge Distillation. 2020. doi: 10.18653/v1/2020.emnlp-main.137.
C. Yang et al., “Learning to Distill Graph Neural Networks,” in Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, in WSDM ’23. New York, NY, USA: Association for Computing Machinery, 2023, pp. 123–131. doi: 10.1145/3539597.3570480.
H.-J. Ye, S. Lu, and D.-C. Zhan, “Generalized Knowledge Distillation via Relationship Matching,” IEEE Trans Pattern Anal Mach Intell, vol. 45, no. 2, pp. 1817–1834, 2023, doi: 10.1109/TPAMI.2022.3160328.
A. H. Ali, M. G. Yaseen, M. Aljanabi, S. A. Abed, and C. GPT, “Transfer Learning: A New Promising Techniques,” Mesopotamian Journal of Big Data, pp. 29–30, Feb. 2023, doi: 10.58496/MJBD/2023/004.
T. Zhou, X. Ye, H. Lu, X. Zheng, S. Qiu, and Y. Liu, “Dense Convolutional Network and Its Application in Medical Image Analysis,” Biomed Res Int, vol. 2022, pp. 1–22, Apr. 2022, doi: 10.1155/2022/2384830.
G. Huang, Z. Liu, G. Pleiss, L. van der Maaten, and K. Q. Weinberger, “Convolutional Networks with Dense Connectivity,” IEEE Trans Pattern Anal Mach Intell, vol. 44, no. 12, pp. 8704–8716, Dec. 2022, doi: 10.1109/TPAMI.2019.2918284.
G. Huang, Z. Liu, G. Pleiss, L. van der Maaten, and K. Q. Weinberger, “Convolutional Networks with Dense Connectivity,” IEEE Trans Pattern Anal Mach Intell, vol. 44, no. 12, pp. 8704–8716, Dec. 2022, doi: 10.1109/TPAMI.2019.2918284.
G. Huang, Z. Liu, G. Pleiss, L. van der Maaten, and K. Q. Weinberger, “Convolutional Networks with Dense Connectivity,” IEEE Trans Pattern Anal Mach Intell, vol. 44, no. 12, pp. 8704–8716, Dec. 2022, doi: 10.1109/TPAMI.2019.2918284.
Y. Lin, Y. Zhang, and X. Yang, “A Low Memory Requirement MobileNets Accelerator Based on FPGA for Auxiliary Medical Tasks,” Bioengineering, vol. 10, no. 1, p. 28, Dec. 2022, doi: 10.3390/bioengineering10010028.
T. N. V. S. Praveen, D. Sivathmika, G. Jahnavi, and J. Bolledu, “An In-depth Exploration of ResNet-50 for Complex Emotion Recognition to Unraveling Emotional States,” in 2023 International Conference on Advancement in Computation & Computer Technologies (InCACCT), IEEE, May 2023, pp. 1–5. doi: 10.1109/InCACCT57535.2023.10141774.
A. A. Mahmood, S. Sadeq, Y. I. Aljanabi, and A. H. Sabry, “Developing a convolutional neural network for classifying tumor images using Inception v3,” Eastern-European Journal of Enterprise Technologies, vol. 3, no. 9 (123), pp. 86–93, Jun. 2023, doi: 10.15587/1729-4061.2023.281227.
L. Luo et al., “A Reconfigurable Spatial Architecture for Energy-Efficient Inception Neural Networks,” IEEE J Emerg Sel Top Circuits Syst, vol. 13, no. 1, pp. 7–20, Mar. 2023, doi: 10.1109/JETCAS.2023.3243619.
A. R. Kusumastuti, Y. Kristian, and E. Setyati, “Klasifikasi Ketertarikan Belajar Anak PAUD Melalui Video Ekspresi Wajah Dan Gestur Menggunakan Convolutional Neural Network,” Jurnal Sisfokom (Sistem Informasi dan Komputer), vol. 10, no. 2, pp. 182–188, Aug. 2021, doi: 10.32736/sisfokom.v10i2.1146.
H. Zhang, D. Chen, and C. Wang, “Confidence-Aware Multi-Teacher Knowledge Distillation,” in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, May 2022, pp. 4498–4502. doi: 10.1109/ICASSP43922.2022.9747534.
J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge Distillation: A Survey,” Int J Comput Vis, vol. 129, no. 6, pp. 1789–1819, Jun. 2021, doi: 10.1007/s11263-021-01453-z.
J. H. Cho and B. Hariharan, “On the Efficacy of Knowledge Distillation,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, Oct. 2019, pp. 4793–4801. doi: 10.1109/ICCV.2019.00489.
T. Andrei-Alexandru and D. E. Henrietta, “Low Cost Defect Detection Using a Deep Convolutional Neural Network,” in 2020 IEEE International Conference on Automation, Quality and Testing, Robotics (AQTR), IEEE, May 2020, pp. 1–5. doi: 10.1109/AQTR49680.2020.9130004.
F. Yuan et al., “Reinforced Multi-Teacher Selection for Knowledge Distillation,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 16, pp. 14284–14291, May 2021, doi: 10.1609/aaai.v35i16.17680.
M. Cesarini et al., “Usage of the Kullback–Leibler divergence on posterior Dirichlet distributions to create a training dataset for a learning algorithm to classify driving behaviour events,” Journal of Computational Mathematics and Data Science, vol. 8, p. 100081, Aug. 2023, doi: 10.1016/j.jcmds.2023.100081.
E. U. Haq, H. Jianjun, X. Huarong, K. Li, and L. Weng, “A Hybrid Approach Based on Deep CNN and Machine Learning Classifiers for the Tumor Segmentation and Classification in Brain MRI,” Comput Math Methods Med, vol. 2022, pp. 1–18, Aug. 2022, doi: 10.1155/2022/6446680.
H. Jang, J. Jung, J. Song, J. Yu, Y. Kim, and J. Lee, “Pipe-BD: Pipelined Parallel Blockwise Distillation,” in 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE), IEEE, Apr. 2023, pp. 1–6. doi: 10.23919/DATE56975.2023.10137044.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Annastasya Nabila Elsa Wulandari, Aimar Yudhistira , Purwono, Abdel-Nasser Sharkawy
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All articles published in the JAHIR Journal are licensed under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license. This license grants the following permissions and obligations:
1. Permitted Uses:
- Sharing – You may copy and redistribute the material in any medium or format.
- Adaptation – You may remix, transform, and build upon the material for any purpose, including commercial use.
2. Conditions of Use:
- Attribution – You must give appropriate credit to the original author(s), provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in a way that suggests the licensor endorses you or your use.
- ShareAlike – If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original (CC BY-SA 4.0).
- No Additional Restrictions – You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
3. Disclaimer:
- The JAHIR Journal and the authors are not responsible for any modifications, interpretations, or derivative works made by third parties using the published content.
- This license does not affect the ownership of copyrights, and authors retain full rights to their work.
For further details, please refer to the official Creative Commons Attribution-ShareAlike 4.0 International License.
