PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiAttenGastro: Multi-Dimensional Attention Augmentation for Gastrointestinal Endoscopy Classification

Sadhana Devarajan, Praveen Kumar Chandaliya, Dhruvin Jashvant Kumar Shah, Kishor Upla, Kiran Raja

arXiv:2609.05070v1cs.CV

TL;DR

GI endoscopy classification must generalize across modalities and class distributions that can differ substantially from natural-image pretraining. This paper introduces MultiAttenGastro and evaluates it across five datasets and eight backbones, finding that attention effectiveness tracks the representational gap rather than being universally beneficial. The authors present the gains and losses as evidence about when attention helps, while emphasizing that the tested explanation is correlational and not statistically decisive at the single-dataset level.

  • Problem

    GI endoscopy classification requires models that generalize across diverse modalities and class distributions, but systematic evidence on multi-dimensional attention across such conditions remains limited.

  • Method

    MultiAttenGastro uses parallel 1-D channel, 2-D spatial, and 3-D contextual attention, evaluated across eight CNN and transformer backbones on five GI datasets.

  • Results

    Attention effectiveness tracks the representational gap: it helps under large gaps, hurts uniformly on Kvasir-v2, and is mixed at intermediate gaps.

  • Takeaways & Limitations

    MultiAttenGastro is best treated as a probe for when attention augmentation helps or hurts GI classification, not as a universally superior architecture.

  • Takeaways & Limitations

    The domain-gap account is correlational, and the single-dataset multi-seed tests do not establish statistically decisive effects.

Abstract

from arXiv · show

Automated gastrointestinal (GI) endoscopy classification requires models that generalize across diverse modalities and class distributions, often far from natural-image pretraining. We propose MultiAttenGastro, a plug-and-play attention framework with parallel 1-D channel, 2-D spatial, and 3-D contextual heads, and present the first systematic cross-dataset evaluation across eight CNN and transformer backbones on five public GI datasets (80 backbone--dataset runs). We find that attention effectiveness is not universal but tracks the representational gap between ImageNet features and the target distribution: MultiAttenGastro improves 6 of 8 backbones on Kvasir-Capsule (14-class WCE, large gap; best macro F1 98.33\%), is uniformly negative on the small-gap Kvasir-v2 benchmark (0/8), and shows mixed outcomes on datasets with intermediate gap. Five-seed ablation on the strongest case (Kvasir-Capsule, ConvNeXt-Tiny) shows this improvement is directionally consistent, but not statistically decisive (paired $t$: $p=0.47$; Wilcoxon: $p=0.63$), and that individual attention heads are not uniformly beneficial in isolation only their combination yields a positive mean effect. Centered Kernel Alignment (CKA) analysis links this pattern to representational redundancy: low inter-head CKA under large domain gaps coincides with the framework's only consistent gains, while high redundancy under small gaps coincides with its losses. We report these results, including the non-significant margins, as evidence for when and why multi-dimensional attention helps GI endoscopy classification, rather than as a claim that MultiAttenGastro is a strictly superior architectural choice.

1 Introduction

MultiAttenGastro is introduced as a plug-and-play multi-dimensional attention framework and evaluated systematically across GI datasets and backbone architectures. Its reported effectiveness varies with the representational gap between ImageNet pretraining and target data, with qualitative activation patterns supporting this account.

  • Motivation: 47,238 labelled images across 14 categories and a 3,434:1 imbalance make Kvasir-Capsule a demanding WCE classification benchmark.Rare pathologies may appear in fewer than 0.1% of frames.
  • Framework: MultiAttenGastro combines parallel 1-D channel, 2-D spatial, and 3-D contextual attention for fine-grained lesion discrimination.The framework is designed as a plug-and-play module.
  • Evaluation: 80 runs cover eight CNN and transformer backbones across five GI datasets, constituting a systematic cross-dataset evaluation.The study examines diverse imaging conditions and target distributions rather than a single dataset.
  • Findings: Attention effectiveness correlates with the ImageNet-to-target representational gap: gains appear under large gaps, uniform losses under small gaps, and mixed outcomes at intermediate gaps.The authors frame this as an empirical association rather than a proven causal mechanism.
  • Findings: Five-seed ablation shows directionally consistent gains on the strongest case, but the result is not statistically significant and individual heads are not independently beneficial.Only the combined attention configuration yields a positive mean effect; CKA links the pattern to inter-head redundancy.
  • Interpretability: Grad-CAM shows lesion-focused activations on the high-gap dataset but dispersed, non-diagnostic shifts on the low-gap dataset.These qualitative patterns align with the framework’s contrasting quantitative behavior across domain gaps.

2 Related Work

Prior GI classification work often emphasizes curated subsets, single metrics, or specialized architectures, leaving systematic evidence about multi-dimensional attention across datasets and domain gaps limited. MultiAttenGastro addresses this gap through cross-dataset evaluation while distinguishing comparable classification metrics from detection results.

  • GI Endoscopy Classification: MultiAttenGastro is evaluated end-to-end on the complete 19-class HyperKvasir taxonomy under natural imbalance, unlike prior results on balanced 10-class subsets.This broader evaluation produces a more clinically realistic comparison but changes task difficulty.
  • GI Endoscopy Classification: Prior GI studies commonly report accuracy on curated single-split subsets, whereas this work evaluates natural class distributions with macro F1 and stratified multiseed analysis.The study explicitly reports accuracy and macro F1 rather than treating them as equivalent.
  • GI Endoscopy Classification: On overlapping datasets, MultiAttenGastro reaches 94.3% accuracy on Kvasir-v2 and 89.9% accuracy with 86.3% macro F1 on HyperKvasir.On Kvasir-Capsule, ConvNeXt-Tiny reaches 98.8% accuracy and 98.3% macro F1.
  • GI Endoscopy Classification: MultiAttenGastro trails prior accuracy on Kvasir-v2 and HyperKvasir, partly because comparisons use curated subsets rather than the complete imbalanced taxonomy.The reported gaps are 94.33% versus 97.8–98.94% on Kvasir-v2 and 89.93% versus 95.30–98.4% on HyperKvasir.
  • Attention Mechanisms in Medical Imaging: The paper’s central contribution is systematic cross-dataset and cross-architecture evaluation identifying when multi-dimensional attention helps or hurts, rather than maximizing accuracy on one curated split.The study frames its explanation of these outcomes as correlational and not statistically decisive at the single-dataset level.
  • Attention Mechanisms in Medical Imaging: Unlike sequential or channel-focused modules such as CBAM, Triplet Attention, Coordinate Attention, and ECA-Net, MultiAttenGastro uses parallel 1-D, 2-D, and 3-D branches fused by addition.Its contribution is the evaluation of this design across GI datasets and representational domain gaps, not merely a new attention mechanism.

3 Proposed Architecture

MultiAttenGastro attaches to pretrained feature extractors and processes projected features through parallel channel, spatial, and contextual attention branches. Their outputs are added, pooled, and classified to integrate complementary representations without modifying the host backbone.

  • Proposed Architecture: MultiAttenGastro is plug-and-play and backbone-agnostic, attaching to pretrained CNN or transformer feature extractors before supervised fine-tuning.It is not training-free or zero-shot; the top backbone layers are optimized during the second training phase.
  • Proposed Architecture: The three attention outputs are fused by element-wise addition, then global average pooling produces a 256-dimensional representation for classification.Dropout with rate 0.4 precedes a linear classifier with K output classes.
  • Proposed Architecture: A 1 × 1 convolution with BatchNorm and ReLU projects backbone features to C = 256 channels before attention processing.The resulting representation is Xin ∈ RH×W×256.
  • Proposed Architecture: Three parallel branches model channel-wise, spatial, and contextual dependencies using 1-D, 2-D, and 3-D attention mechanisms.The contextual branch uses depthwise and dilated depthwise convolutions to capture local and long-range spatial context in a static image.
  • Proposed Architecture: The framework supports direct head-level ablation because its channel, spatial, and contextual branches are computed independently before fusion.Backbone interactions remain non-uniform, with some architectures benefiting consistently and others showing mixed or near-null effects.

4 Datasets and Experimental Setup

The experiments span five GI datasets and 80 backbone–dataset runs with preprocessing for severe Kvasir-Capsule imbalance. Training uses standardized image preparation, dataset-specific splits, a two-phase schedule, and a single fixed seed for the main benchmark.

  • Datasets: The benchmark covers five public GI datasets spanning two imaging modalities, varied class distributions, and different visual complexity.Kvasir-Capsule has 14 rare pathological classes and the largest ImageNet domain gap, while SEE-AI has four classes and a smaller effective gap.
  • Preprocessing: Under-sampling plus targeted augmentation raises EfficientNet-B3 F1 from 88.69% on raw data to 97.65%, an 8.96-point gain adopted for Kvasir-Capsule experiments.Under-sampling alone reaches 95.88% F1 and caps majority classes at 3,000 samples, reducing imbalance to 300:1.
  • Experimental Setup: Images are resized to 224×224, or 299×299 for Inception-v3, and normalized with ImageNet statistics.Kvasir-Capsule uses its official split with 15% of training data for validation, while other datasets use predefined or stratified splits.
  • Experimental Setup: The main benchmark comprises 80 runs from eight backbones across five datasets and two conditions, trained for 92 epochs with Adam and weight decay 10^-4.A fixed random seed of 42 is used unless otherwise noted.
  • Training: Training uses a two-phase schedule: two warm-up epochs with the backbone frozen, followed by 90 fine-tuning epochs with the top 30% of layers unfrozen.The learning rates are 10^-3 during warm-up and 10^-4 during fine-tuning, with ReduceLROnPlateau scheduling.

5 Quantitative Results

Across five GI datasets, MultiAttenGastro helps most under large representational gaps, degrades uniformly under small gaps, and shows mixed intermediate outcomes. Five-seed ablations indicate that gains depend on combining attention heads, while observed margins remain statistically non-decisive.

  • Backbone benchmark: 6/8 backbones improve on Kvasir-Capsule, led by ConvNeXt-Tiny at 98.33% macro F1 (+1.05%).Kvasir-Capsule combines a large domain gap with extreme class imbalance.
  • Backbone benchmark: 0/8 backbones improve on Kvasir-v2, with performance reductions ranging from −0.14% to −2.79%.The uniform degradation is consistent with redundancy under a small domain gap.
  • Backbone benchmark: 3/8 and 2/8 backbones improve on HyperKvasir and GastroVision, respectively, while SEE-AI shows 4/8 wins.These outcomes support dataset-specific effects rather than fixed backbone behavior or modality-only effects.
  • Attention-head ablation: Only the full three-head combination yields a consistent positive mean effect (+0.166%), whereas 1D and 2D heads fall below baseline.The 1D and 2D heads reach −0.402% and −0.202%, while the 3D head is roughly on par at +0.036%.
  • Alternative mechanisms: MultiAttenGastro achieves the highest five-seed mean macro F1 among tested alternatives at 98.262 ± 0.337%, but no margin reaches conventional significance.The MAG-versus-baseline margin is +0.166%, with p = 0.472 for the paired t-test and p = 0.625 for Wilcoxon.
  • Training cost: +4.3% to +10.7% training-time overhead is introduced across backbones.The lowest overhead is for ConvNeXt-Tiny and the highest for MobileNet-v3-Large.
  • Multiseed extension: Five-seed ConvNeXt-Tiny results show −0.17% on Kvasir-v2, +0.48% on HyperKvasir, and −0.77% on SEE-AI.SEE-AI also has the largest variance, with standard deviation 2.622.

6 Qualitative Results

Grad-CAM visualizations show the clearest qualitative benefit on Kvasir-Capsule, where MultiAttenGastro localizes diagnostically relevant regions more sharply than the baseline. Other datasets exhibit diffuse or mixed activation patterns consistent with weaker quantitative gains.

  • Kvasir-Capsule: On Kvasir-Capsule, MultiAttenGastro sharply localizes activations over polyps, bleeding sites, and angiectasia, while baseline activations remain diffuse.The qualitative pattern is consistent with the dataset’s strong 6/8 backbone improvement.
  • Kvasir-Capsule: MultiAttenGastro sharpens activation for Bleeding, while normal and structural classes show little gain.This indicates that qualitative benefits vary with per-class representational difficulty.
  • Extension datasets: HyperKvasir and GastroVision show diffuse, mixed patterns consistent with their small-to-medium gaps and 3/8 and 2/8 win rates.The visualizations add no qualitatively new insight beyond the presented cases.

7 Discussion

The discussion frames MultiAttenGastro’s gains as dependent on representational distance and head complementarity rather than attention or modality alone. CKA supports this interpretation, while multiseed and proxy analyses identify important uncertainty and scope boundaries.

  • Domain-gap framework: MultiAttenGastro is most effective when the target domain has a substantial representational gap from ImageNet pretraining.When baselines already generalize well, additional attention can introduce redundancy or conflicting signals, and mean-level trends are not individually significant at n = 5.
  • Representational redundancy: Low inter-head CKA co-occurs with large gaps and positive faithfulness gains, whereas high CKA aligns with small gaps and negative gains.Across five datasets, CKA is approximately 0.58 in Kvasir-Capsule versus at least 0.85 in small-gap datasets.
  • Representational redundancy: Kvasir-Capsule has the lowest inter-head CKA and the only strongly positive faithfulness gain.This supports the interpretation that combined heads capture complementary signal even when individual heads contribute little in isolation.
  • Backbone adaptation: Linear CKA between frozen ImageNet and fine-tuned features measures backbone shift, with lower CKA indicating greater adaptation during fine-tuning.The metric provides a complementary proxy for domain gap rather than a complete replacement for the qualitative construct.
  • Backbone adaptation: SEE-AI diverges from the qualitative gap labels, showing the largest feature shift despite a “Small” label and mixed 4/8 F1 outcomes.The paper attributes this divergence to factors including class count, imbalance, and constrained training data, and treats it as a limitation of any single proxy.
  • Backbone dependence: Multiseed evaluation shows heterogeneous backbone behavior: ConvNeXt-Tiny and MobileNet-v2 improve in 5/5 seeds, while DenseNet121 has a −0.05% mean effect.ConvNeXt-Tiny and MobileNet-v2 are recommended for their consistent performance–efficiency trade-offs; other backbones require deployment-specific validation.

8 Conclusion

Across 80 backbone–dataset runs, MultiAttenGastro’s effectiveness tracks the representational gap between ImageNet pretraining and the target domain rather than being universally beneficial. Gains are clearest on high-gap Kvasir-Capsule, while low-gap Kvasir-v2 shows uniform declines and intermediate-gap datasets show mixed outcomes.

  • 98.33% macro F1 was achieved with ConvNeXt-Tiny on high-domain-gap Kvasir-Capsule, the clearest gain.Performance declined uniformly on low-gap Kvasir-v2, while HyperKvasir, GastroVision, and SEE-AI fell between these extremes.
  • Five-seed evaluation was directionally consistent but not statistically decisive, with p = 0.47 for the paired t-test and p = 0.63 for Wilcoxon.The reported strongest case was Kvasir-Capsule with ConvNeXt-Tiny.
  • The positive effect arose from combining three attention heads, because individual heads were not beneficial in isolation.
  • Low inter-head redundancy under large domain gaps coincided with consistent gains, whereas high redundancy under small gaps coincided with losses.CKA analysis provided this mechanistic account of the observed pattern.
Loading 2609.05070v1…