Explainability and Feature Fusion for Colon CT Image Classification

Abstract

Accurate classification of colon computed tomography (CT) images using deep learning has shown promising results. However, the contribution of handcrafted features, reliability of explainability methods, and potential influence of data leakage remain important concerns. This study evaluates six XGBoost-based classification strategies using 11,340 CT images from the ColonCancerCT-2025 dataset comprising 40 patients. The dataset was divided at the image level into training (7,937), validation (1,701), and testing (1,702) sets without verified patient separation. Deep features extracted using EfficientNet-B0 were compared with whole-image and Grad-CAM++ region-of-interest (ROI)-associated handcrafted features and their combinations. Four explainability methods were evaluated using deletion and insertion metrics. Additionally, duplicate detection, nearest-neighbor analysis, and body masking were performed. Experimental results showed that EfficientNet-B0 achieved 99.82% accuracy, while feature fusion provided no improvement. ROI-associated handcrafted features achieved 85.61% accuracy compared with 79.44% for whole-image features. LIME and Integrated Gradients produced the lowest, overlapping deletion AUCs. Duplicate analysis identified 134 test images (7.9%) with identical copies, while nearest-neighbor classification achieved 99.94% accuracy. Body masking reduced accuracy to 95.06%, whereas background-only classification achieved 0.74 ROC-AUC. These findings highlight the importance of patient-independent evaluation and leakage assessment before interpreting high classification performance as diagnostic reliability.

Keywords

Funding

This research received no external funding.

References

1. Huang, Y.Q.; Liang, C.H.; He, L.; Tian, J.; Liang, C.S.; Chen, X.; Ma, Z.L.; Liu, Z.Y. Development and Validation of a Radiomics Nomogram for Preoperative Prediction of Lymph Node Metastasis in Colorectal Cancer. Journal of Clinical Oncology 2016, 34, 2157–2164. https://doi.org/10.1200/JCO.2015.65.9128
2. Tan, M.; Le, Q. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA, 9–15 June 2019; Proceedings of Machine Learning Research 2019, 97, 6105–6114. Available online: https://proceedings.mlr.press/v97/tan19a.html
3. Chattopadhay, A.; Sarkar, A.; Howlader, P.; Balasubramanian, V.N. Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks. In Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018; pp. 839–847. https://doi.org/10.1109/WACV.2018.00097
4. Sundararajan, M.; Taly, A.; Yan, Q. Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 6–11 August 2017; Proceedings of Machine Learning Research 2017, 70, 3319–3328. Available online: https://proceedings.mlr.press/v70/sundararajan17a.html
5. Lundberg, S.M.; Lee, S.-I. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30; Curran Associates: Red Hook, NY, USA, 2017; pp. 4765–4774. https://doi.org/10.48550/arXiv.1705.07874
6. Kokhlikyan, N.; Miglani, V.; Martin, M.; Wang, E.; Alsallakh, B.; Reynolds, J.; Melnikov, A.; Kliushkina, N.; Araya, C.; Yan, S.; Reblitz-Richardson, O. Captum: A Unified and Generic Model Interpretability Library for PyTorch. arXiv 2020, arXiv:2009.07896. https://doi.org/10.48550/arXiv.2009.07896 (Captum version 0.8.0 was used in this study.)
7. Ribeiro, M.T.; Singh, S.; Guestrin, C. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016; pp. 1135–1144. https://doi.org/10.1145/2939672.2939778
8. Petsiuk, V.; Das, A.; Saenko, K. RISE: Randomized Input Sampling for Explanation of Black-Box Models. In Proceedings of the British Machine Vision Conference (BMVC), Newcastle, UK, 3–6 September 2018. https://doi.org/10.48550/arXiv.1806.07421
9. Adebayo, J.; Gilmer, J.; Muelly, M.; Goodfellow, I.; Hardt, M.; Kim, B. Sanity Checks for Saliency Maps. In Advances in Neural Information Processing Systems 31; Curran Associates: Red Hook, NY, USA, 2018; pp. 9505–9515. https://doi.org/10.48550/arXiv.1810.03292
10. Orvile. ColonCancerCT-2025: Abdominal CT Scans [Dataset]. Kaggle. Available online: https://www.kaggle.com/datasets/orvile/coloncancerct-2025-abdominal-ct-scans/data
11. Ziegelmayer, S.; Reischl, S.; Havrda, H.; Gawlitza, J.; Graf, M.; Lenhart, N.; Nehls, N.; Lemke, T.; Wilhelm, D.; Lohöfer, F.; et al. Development and Validation of a Deep Learning Algorithm to Differentiate Colon Carcinoma From Acute Diverticulitis in Computed Tomography Images. JAMA Network Open 2023, 6, e2253370. https://doi.org/10.1001/jamanetworkopen.2022.53370
12. Yao, L.; Li, S.; Tao, Q.; Mao, Y.; Dong, J.; Lu, C.; Han, C.; Qiu, B.; Huang, Y.; Huang, X.; et al. Deep Learning for Colorectal Cancer Detection in Contrast-Enhanced CT without Bowel Preparation: A Retrospective, Multicentre Study. EBioMedicine 2024, 104, 105183. https://doi.org/10.1016/j.ebiom.2024.105183
13. Attallah, O. Multi-Domain Feature Incorporation of Lightweight Convolutional Neural Networks and Handcrafted Features for Lung and Colon Cancer Diagnosis. Technologies 2025, 13, 173. https://doi.org/10.3390/technologies13050173
14. Alsubai, S. Transfer Learning Based Approach for Lung and Colon Cancer Detection Using Local Binary Pattern Features and Explainable Artificial Intelligence (AI) Techniques. PeerJ Computer Science 2024, 10, e1996. https://doi.org/10.7717/peerj-cs.1996
15. Zhang, D.; Zheng, B.; Xu, L.; Wu, Y.; Shen, C.; Bao, S.; Tan, Z.; Sun, C. A Radiomics-Boosted Deep-Learning for Risk Assessment of Synchronous Peritoneal Metastasis in Colorectal Cancer. Insights into Imaging 2024, 15, 150. https://doi.org/10.1186/s13244-024-01733-5
16. Wani, S.A.; Khan, S.A.; Quadri, S.M.K. scJVAE: A Novel Method for Integrative Analysis of Multimodal Single-Cell Data. Computers in Biology and Medicine 2023, 158, 106865. https://doi.org/10.1016/j.compbiomed.2023.106865
17. Di Giammarco, M.; Martinelli, F.; Santone, A.; Cesarelli, M.; Mercaldo, F. Colon Cancer Diagnosis by Means of Explainable Deep Learning. Scientific Reports 2024, 14. https://doi.org/10.1038/s41598-024-63659-8
18. Wightman, R. PyTorch Image Models. GitHub Repository, 2019. https://doi.org/10.5281/zenodo.4414861 (timm version 1.0.14 was used in this study.)
19. Chen, T.; Guestrin, C. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016; pp. 785–794. https://doi.org/10.1145/2939672.2939785
20. Wilson, E.B. Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association 1927, 22, 209–212. https://doi.org/10.1080/01621459.1927.10502953
21. Hanley, J.A.; McNeil, B.J. The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve. Radiology 1982, 143, 29–36. https://doi.org/10.1148/radiology.143.1.7063747
22. McNemar, Q. Note on the Sampling Error of the Difference between Correlated Proportions or Percentages. Psychometrika 1947, 12, 153–157. https://doi.org/10.1007/BF02295996
23. Tejani, A.S.; Klontzas, M.E.; Gatti, A.A.; Mongan, J.T.; Moy, L.; Park, S.H.; Kahn, C.E., Jr.; CLAIM 2024 Update Panel. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update. Radiology: Artificial Intelligence 2024, 6, e240300. https://doi.org/10.1148/ryai.240300
24. Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing 2004, 13, 600–612. https://doi.org/10.1109/TIP.2003.819861