Accuracy Is Not Enough: An Efficiency- and Statistics-Aware Benchmark of ResNet50 and Vision Transformer for Explainable Multi-Class Gastrointestinal Diseases Classification

Background

• CNNs learn hierarchical spatial features suited to texture and shape, whereas ViTs use self-attention to model relationships across image regions.The introduction notes that ViTs may capture global context but are more computationally intensive and generally require larger datasets.Source: §1 · p. 2

Study scope

• The study compares ResNet50, ViT-B/16, and MobileViT-S on Kvasir’s 4,000 images across eight classes and evaluates class-specific performance as well as multiple deployment-relevant dimensions.The evaluation includes computational efficiency, generalization, calibration, explainability, and robustness to an acquisition artefact.Source: §1 · p. 2

Contributions

• 92.12% vs. 91.70% cross-validated accuracy: ResNet50 and ViT-B/16 were equivalent within a ±3-point margin under the reproducible benchmark.The comparison used a fixed stratified split and stratified group 5-fold cross-validation; the equivalence test used TOST.Source: §1 · p. 3
• 3.6× fewer parameters and 4.3× fewer operations, 2.5× faster CPU inference, and 3.8× higher GPU throughput favored ResNet50 over ViT-B/16.The benchmark also compared latency, throughput, memory, and training time; ResNet50 reached its best checkpoint about four times sooner.Source: §1 · p. 3
• The benchmark combines accuracy and efficiency comparisons with calibration, multi-method explainability, and lesion-level localization against expert polyp masks.The interpretability assessment includes Grad-CAM, LIME, SHAP, and attention rollout.Source: §1 · p. 3
• MobileViT-S lost 9.4 accuracy points when the ScopeGuide inset was occluded, while ResNet50 was not significantly affected; external evaluation showed a 13–17-point drop for all models.The authors use these tests to assess reliance on an acquisition artefact and performance beyond the curated benchmark.Source: §1 · p. 3

2 Related Work

• Prior GI image-analysis work spans handcrafted-feature methods, CNNs, and transformer-based approaches.The review frames these as three broad directions informing the study.Source: §2 · p. 3

2.1 Early Approaches: Handcrafted Features and Classical Methods

• 93.7% test accuracy and 0.747 F1-score were reported for Pogorelov et al.’s Logistic Model Tree using handcrafted global features.The review notes that handcrafted approaches struggled with visual variability and generalization.Source: §2.1 · p. 4

2.2 Deep Learning and CNN-Based Methods

• CNN studies applied learned image features to GI classification and segmentation, while reported results span different datasets and tasks.Examples include ResNet50 classification on Kvasir and ResUNet segmentation on Kvasir-SEG; the latter was not evaluated on multi-class classification.Source: §2.2 · p. 4

2.3 Vision Transformers and Attention-Based Models

• Transformer research covers GI classification, segmentation, capsule endoscopy, and edge deployment, with reported results tied to differing datasets and tasks.Examples include ViT comparisons across medical datasets, capsule-image classification, polyp segmentation, and quantized on-edge ViTs.Source: §2.3 · p. 5

2.4 Beyond White-Light RGB Imaging

• Spectral imaging and software enhancement seek tissue information beyond white-light RGB, but whether this study’s accuracy–efficiency conclusions transfer to enhanced inputs remains open.The present study restricts itself to RGB images, the modality of standard endoscopy systems and public benchmarks such as Kvasir.Source: §2.4 · p. 6

2.5 Research Gap and Positioning of This Study

• Prior multi-class CNN–transformer comparisons typically emphasize raw accuracy or new architectures rather than jointly assessing deployment-relevant trade-offs.This study positions its contribution as a controlled benchmark that adds equivalence testing, computational-cost measures, attribution analysis, an artefact test, and external evaluation.Source: §2.5 · p. 6

3.1 Dataset

• Kvasir v1 contributes 4,000 images across eight classes; near-duplicates were grouped for cross-validation, while the ScopeGuide inset was identified as a class-dependent candidate shortcut.The dataset lacks patient, examination, and video identifiers, so patient- or procedure-level splitting was not possible.Source: §3.1 · pp. 7–8

3.2 Image Preprocessing and Augmentation and Data Splitting

• A fixed stratified split used 3,200 training, 400 validation, and 400 test images, supplemented by group-aware stratified five-fold cross-validation over all 4,000 images.Both models received identical resizing, normalization, and training-only augmentation; near-duplicate groups were kept within folds.Source: §3.2 · p. 11

3.3 Model Architectures and Training Configuration

• ResNet50 and ViT-B/16 were fine-tuned under an identical protocol, with MobileViT-S added as an edge-oriented transformer and label smoothing disabled in an ablation.The standard architectures received ImageNet-pretrained weights; early stopping selected checkpoints by validation macro-F1.Source: §3.3 · pp. 11–13

3.4 Evaluation Metrics

• Evaluation covered held-out-test accuracy, per-class and macro-averaged classification metrics, ROC/AUC, confusion matrices, confidence intervals, and multiple calibration measures.Macro-F1 weights each of the eight classes equally; calibration was assessed with alternative ECE binning schemes, Brier score, and NLL.Source: §3.4 · pp. 13–14

3.5 Statistical Comparison

• Paired McNemar testing compared classifiers on shared test images, while TOST tested cross-validation accuracy equivalence using a prespecified ±3-percentage-point margin.Equivalence required the 90% confidence interval to lie entirely within the margin; a nonsignificant difference alone was not treated as evidence of equivalence.Source: §3.5 · pp. 14–15

3.6 Computational Efficiency Profiling

• Deployment cost was profiled using parameters, MACs, latency, throughput, and memory on GPU and CPU, alongside training time.Inference measurements used standardized hardware and protocols, with single-image CPU latency and batch-32 throughput among the reported measures.Source: §3.6 · p. 15

3.7 Interpretability Analysis

• Interpretability analysis used Grad-CAM and exploratory attribution methods, then assessed lesion localization against expert masks on 500 unseen polyp images.Localization measures included pointing-game hit rate, energy in the lesion, and IoU; ScopeGuide occlusion tested sensitivity to the inset.Source: §3.7 · pp. 15–16

3.8 External Evaluation and Prevalence Analysis

• External evaluation applied the primary-split models unchanged to 1,998 de-duplicated GastroVision images, while prevalence analysis estimated PPV across 1–20% prevalence.GastroVision was treated as partially independent and naturally imbalanced; calibration was also examined under an illustrative screening-like class prior.Source: §3.8 · p. 16

4.1 Classification Performance

• 93.75% vs. 92.25% held-out accuracy favored ResNet50, while MobileViT-S reached 95.50%.Macro-F1 was 0.937 for ResNet50, 0.922 for ViT-B/16, and 0.955 for MobileViT-S.Source: §4.1 · p. 17

4.2 Confusion Matrices and ROC Analysis

• 84% of ResNet50 and 74% of ViT test errors came from esophagitis/normal-Z-line and dyed-polyp class confusions; micro-AUCs were 0.990 and 0.991.The same adjacent class pairs accounted for 70% of both models’ cross-validation errors.Source: §4.2 · p. 19, Figure 8 · p. 19, Figure 9 · p. 19

4.3 Statistical Significance Analysis

• The primary split did not establish equivalence within ±3 points (TOST p = 0.066), unlike the confirmatory cross-validation.Per-class F1 intervals overlapped, but 50 test images per class were insufficient to detect or exclude class-level differences.Source: §4.1 · p. 17, §4.3 · p. 21
• 92.12% vs. 91.70% cross-validated accuracy; the +0.43-point difference was equivalent within ±3 points (TOST p < 0.001).The 90% CI was −0.33 to +1.18 points across 4,000 out-of-fold predictions.Source: §4.3 · p. 21

4.4 Computational Efficiency

• ResNet50 used 3.6× fewer parameters and 4.3× fewer operations than ViT-B/16, with 69 vs. 177 ms CPU latency and 337 vs. 88 images/s GPU throughput.MobileViT-S had the lowest CPU latency (53.5 ms) and highest FP32 GPU throughput (444 images/s), but used 711 MB CPU memory at batch size 32.Source: §4.4 · pp. 21, 23, Table 8 · p. 23
• ResNet50 reached its best checkpoint after 3.8 min on average in cross-validation, versus 15.8 min for ViT-B/16.Across folds, ResNet50 reached 99% of its best macro-F1 in 2.9 ± 0.9 min, versus 11.0 ± 8.2 min for ViT-B/16.Source: §4.4 · p. 23, Figure 11 · p. 24

4.5 Calibration

• Cross-validated ECE was similar: 0.021 for ResNet50 and 0.023 for ViT-B/16.On the primary split, the ECE ranking changed with the binning scheme, and the reliability diagrams had sparse lower-confidence bins.Source: §4.5 · pp. 23, 25, Figure 12 · p. 25, Figure 13 · p. 25
• Without label smoothing, both models reached 93.75% accuracy, while ECE increased from 0.025 to 0.045 for ResNet50 and 0.039 to 0.050 for ViT-B/16.The calibration gap between architectures remained small and nonsignificant with or without smoothing.Source: §4.5 · pp. 25–26
• At 2% prevalence, estimated PPV was 0.48 vs. 0.54 for esophagitis, 0.82 vs. 0.63 for polyps, and 0.83 vs. 0.59 for ulcerative colitis (ResNet50 vs. ViT-B/16).The estimates used cross-validated sensitivities and false-positive rates.Source: §4.5 · p. 27, Table 12 · p. 27, Figure 14 · p. 27

Interpretability

• For most classes, attribution maps concentrated on the central mucosal field, although ResNet50 Grad-CAM highlighted the ScopeGuide inset in the normal-cecum example.The inset is in the lower-left corner of that example.Source: §4.6 · pp. 27–28

Lesion-level localization

• 86.8% of ResNet50 and 72.8% of ViT Grad-CAM maxima fell inside lesions; ResNet50's peak was more often on the polyp.There were 102 versus 32 discordant images (exact p < 0.001).Source: §4.6 · p. 28
• 0.553 (ResNet50) and 0.575 (ViT-B/16) Grad-CAM energy-in-lesion exceeded the 0.154 chance level on 500 unseen polyp images.The chance level equals mean lesion area.Source: §4.6 · p. 28, Table 14 · p. 29

Reliance on an Acquisition Artefact

• 9.4 points: MobileViT-S accuracy fell from 96.6% to 87.2% after ScopeGuide occlusion, while ResNet50's change was not significant.ResNet50 changed from 95.3% to 94.0% (McNemar p = 0.69); the MobileViT-S reduction had p < 0.001.Source: §4.6 · pp. 28–29, Table 15 · p. 29
• Attribution mass did not identify artefact dependence: ResNet50 maps highlighted the inset, yet its predictions were least affected by removing it.ResNet50 attribution on the inset was 1.19 times chance for Grad-CAM, 1.32 for LIME, and 1.36 for SHAP.Source: §4.6 · pp. 29–30, Table 16 · p. 30

4.7 External Evaluation on GastroVision

• 0.57–0.68: esophagitis was the weakest class for all models, while per-class sensitivities varied more than overall accuracy.The external set was imbalanced, with polyps at 37% and esophagitis at 5%.Source: §4.7 · p. 30
• 78.08% (ResNet50), 79.63% (ViT), and 78.33% (MobileViT-S) accuracy on GastroVision were 13–17 points below internal results.The evaluation used 1,998 de-duplicated images; ViT's 1.55-point lead over ResNet50 was not significant (McNemar p = 0.12).Source: §4.7 · p. 30, Table 17 · p. 34

4.8 Discussion

• A 0.43-point cross-validated accuracy difference between ResNet50 and ViT-B/16 was within the ±3-point equivalence margin, while ResNet50 used 3.6 times fewer parameters and 4.3 times fewer operations.The discussion favors ResNet50 as a defensible default for resource-constrained settings, while noting that ViT remains reasonable with larger and more diverse training data.Source: §4.8 · p. 35
• MobileViT-S lost 9.4 accuracy points after ScopeGuide occlusion, whereas ResNet50 was not significantly affected; all models lost 13–17 points on external data.The authors caution against selecting an edge-optimized model on benchmark accuracy alone and recommend explicit robustness checks and local validation.Source: §4.8 · p. 35

4.9 Limitations

• GastroVision shares a hospital with Kvasir and lacks image-centre labels, so it cannot replace genuinely independent, ideally prospective, multicentre validation.Other limitations include unavailable patient-level identifiers and lesion annotations only for polyps.Source: §4.9 · p. 36
• Efficiency was measured on one GPU and one cloud CPU, so it should be re-measured on the intended deployment device.The study also used standard matched training configurations rather than exhaustive tuning.Source: §4.9 · p. 36

Conclusion

• 92.12% (ResNet50) and 91.70% (ViT-B/16) cross-validated accuracy were equivalent within ±3 points (90% CI for difference −0.33 to +1.18; TOST p < 0.001).The fixed test split showed no significant difference (93.75% vs. 92.25%; McNemar p = 0.21).Source: §5 · p. 36
• ResNet50 used 3.6 times fewer parameters and 4.3 times fewer operations; all models lost 13–17 points on external data, despite similar internal accuracy.Both models' Grad-CAM maps localized polyps on unseen images at about 3.6 times chance.Source: §5 · p. 36
• Future work prioritizes prospective multicentre validation with patient-level data and realistic prevalence, expert review of attribution maps, and artefact-shortcut mitigation.The authors also propose target-hardware evaluation and prior-shift recalibration for local prevalence.Source: §5 · p. 36

Ethical Statement

• Kvasir was publicly available and fully de-identified, so no additional institutional ethical approval was required.All images were anonymized before release in accordance with data privacy and ethical guidelines.Source: Ethical Statement · p. 37

• The study received no external funding and used institutional resources and the authors’ computational facilities.Source: Funding · p. 37

• Patient consent was not required for this study using a publicly available, de-identified dataset of anonymized endoscopic images.Source: Patient Consent Statement · p. 37

• No copyrighted materials from external sources were reproduced in the manuscript.Source: Permission to Reproduce Material from… · p. 38

• The study does not involve a clinical trial.Source: Clinical Trial Registration · p. 38