PaperlayerView exact revision on arXiv

Source-linked AI summary

MedViT: A Robust Vision Transformer for Generalized Medical Image Classification

Omid Nejati Manzari, Hamid Ahmadabadi, Hossein Kashiani, Shahriar B. Shokouhi, Ahmad Ayatollahi

arXiv:2302.09462v1cs.CV

TL;DR

Medical image classifiers face concerns about adversarial reliability and the challenge of modeling long-range dependencies across varied medical data. MedViT combines convolutional and Transformer components with feature-statistic augmentation, and achieves state-of-the-art accuracy and robustness on standard large-scale 2D biomedical datasets.

  • Problem

    Medical image analysis needs robust representations that model long-range dependencies across varied data, while adversarial attacks pose security concerns for deep diagnosis systems.

    Source: §1 · pp. 1–2

  • Method

    MedViT is a hybrid CNN-Transformer architecture that combines local and global features and uses patch-moment augmentation to diversify training data.

    Source: §1 · p. 2, §6 · p. 12

  • Results

    MedViT achieves state-of-the-art accuracy and robustness on the standard large-scale collection of 2D biomedical datasets.

    Source: §6 · p. 12

  • Takeaways & Limitations

    The authors present MedViT as a potential starting point for future research on realistic medical deployment.

    Source: §6 · p. 12

Abstract

from arXiv · show

Convolutional Neural Networks (CNNs) have advanced existing medical systems for automatic disease diagnosis. However, there are still concerns about the reliability of deep medical diagnosis systems against the potential threats of adversarial attacks since inaccurate diagnosis could lead to disastrous consequences in the safety realm. In this study, we propose a highly robust yet efficient CNN-Transformer hybrid model which is equipped with the locality of CNNs as well as the global connectivity of vision Transformers. To mitigate the high quadratic complexity of the self-attention mechanism while jointly attending to information in various representation subspaces, we construct our attention mechanism by means of an efficient convolution operation. Moreover, to alleviate the fragility of our Transformer model against adversarial attacks, we attempt to learn smoother decision boundaries. To this end, we augment the shape information of an image in the high-level feature space by permuting the feature mean and variance within mini-batches. With less computational complexity, our proposed hybrid model demonstrates its high robustness and generalization ability compared to the state-of-the-art studies on a large-scale collection of standardized MedMNIST-2D datasets.

1. Introduction

MedViT is a hierarchical CNN–Transformer hybrid designed for efficient medical image analysis across domains and greater adversarial robustness. It combines convolutional and attention mechanisms with feature-statistic interpolation to model local and global structure.

1. Introduction

  • CNNs have supported medical diagnosis but struggle with long-range dependencies, while Transformers model them [7] [8] yet require substantial data and incur quadratic complexity [9].The paper notes that this complexity limits Transformer use in realistic clinical settings and high-resolution tasks.Source: §1 · p. 1
  • Clinical deployment also faces domain shifts across sites due to scanner and protocol differences and variation in population label distributions.The paper identifies these shifts as especially pronounced in healthcare.Source: §1 · pp. 1–2
  • MedViT targets CT, X-ray, ultrasound, and OCT through a hierarchical hybrid architecture with efficient convolutional and Transformer blocks.Its multi-head convolutional attention learns token affinities in representation subspaces while reducing the computational burden of conventional attention.Source: §1 · p. 2
  • MedViT supplements attention with a depth-wise-convolution local feed-forward network and fuses low-frequency self-attention with high-frequency convolutional attention.These components are intended to model nearby-pixel dependencies and strengthen global and local representations.Source: §1 · p. 2
  • The authors report MedViTs’ superiority over baseline ResNets in accuracy–complexity trade-offs across 2D datasets.Figure 1 compares average accuracy- and AUC-to-parameter trade-offs; the supplied passages provide no numerical values.Source: Figure 1 · p. 1, §1 · p. 2
  • To improve robustness and generalization, MedViT interpolates feature-space means and variances to emphasize global structure over texture and learn smoother decision boundaries.The authors motivate this approach by noting that networks often rely on texture and that interpolation explores feature-space regions relevant to global structure.Source: §1 · p. 2

2. Related Works

Related work spans CNNs, vision Transformers, and hybrid architectures, while prior adversarial-robustness findings for Transformers motivate MedViT’s robust hybrid design and added augmentation.

Convolutional networks

  • CNN designs address deep feature extraction and efficiency through residual connections, depthwise separable convolutions [21], dense skip connections, 7 × 7 depthwise convolutions, and channel shuffling.ConvNext reintroduces Vision Transformer design elements and is described as achieving results comparable with Transformers.Source: §2 · p. 2

Vision Transformers

  • ViT models patches as tokens to capture global relations; variants refine tokenization, local and cross-window attention, or pyramid feature maps [7] [28].The passage also notes growing interest in efficient methods such as pyramidal designs, training strategies, and efficient self-attention.Source: §2 · pp. 2–3

Hybrid Models

  • Hybrid models combine convolutional local perception with Transformer attention through modified ResNet blocks, lightweight Transformer blocks, or convolutions preceding attention, [14],.Other designs use convolutional stems, series-connected Transformer and MobileNetV2 blocks [32], or bidirectional CNN–Transformer bridges [34].Source: §2 · p. 3

Robustness Study

  • CNN robustness methods include model design [35, 36], data augmentation [38], architecture search [40], training strategies [41] [42], pruning, quantization [45], activation functions [46], and pooling [48].The passage notes that these methods perform well on CNNs, but there is no evidence they also improve ViT adversarial robustness.Source: §2 · pp. 3–4
  • Prior work reports greater adversarial robustness for Transformers than CNNs, but MedViT goes beyond comparison by designing a robust hybrid family and a novel augmentation technique.The cited studies also report unexpectedly low adversarial transferability between CNNs and Transformers and ViT superiority over CNNs under natural distribution shift and common image corruption.Source: §2 · p. 4

3. Method

MedViT combines convolution and Transformer components in a hierarchical hybrid architecture, using blocks for local and global feature modeling and feature-level augmentation to target robustness and generalization.

3.1. Overview

  • MedViT reduces spatial resolution by 32× across four stages while doubling channels after each stage’s convolution blocks, and uses ECB, LTB, and PMC to model context and target robustness.The resolution reductions are [4×, 2×, 2×, and 2×].Source: §3.1 · p. 4

3.2. Efficient Convolution Block

  • ECB combines a Locally Feed Forward Network with Multi-Head Convolutional Attention to capture local structure and mix tokens in a robust hybrid block.The authors describe ECB as retaining the deployment advantage of a Residual block while achieving Transformer-based performance.Source: §3.2 · p. 5

3.2.1. Locally Feed-Forward Network

  • The Locally Feed-Forward Network reshapes tokens into a 2D feature map, replaces fully connected layers with 1 × 1 convolutions, then converts the features back into a token sequence.This restores token proximity to introduce locality while retaining a sequence for the next attention layer.Source: §3.2.1 · p. 5

3.3. Local Transformer Block

  • LTB mixes low-frequency global information from Efficient Self Attention with local token interactions from multi-head convolutional attention to capture multi-frequency signals efficiently.Efficient Self Attention downsamples spatial dimensions before attention, while point-wise convolutions reduce the channel dimension to accelerate inference.Source: §3.3 · pp. 5–6

3.4. Transformer Augmentation Block

  • PMC asymmetrically combines one sample’s normalized intermediate features with another’s feature moments and labels to smooth decision boundaries and target robustness and generalization.The operation can be applied at each stage but is more effective at the first; the authors report consistent accuracy gains and enhanced adversarial robustness when used in MedViT.Source: §3.4 · pp. 7–8

4. Experiments

Experiments evaluate three MedViT sizes across 12 standardized MedMNIST datasets under benchmark training settings, using ACC and AUC for comparison.

4.1. Datasets

  • The MedMNIST collection spans 12 pre-processed datasets covering CT, X-ray, ultrasound, and OCT images and classification tasks of varied types and dataset sizes [63].The collection includes multi-label, ordinal, multi-class, regression, and binary tasks, with dataset sizes ranging from at least 100 to over 100,000.Source: §4.1 · p. 8

4.2. Implementation Details

  • MedViT-T, MedViT-S, and MedViT-L were trained separately for 100 epochs on 12 standardized MedMNIST datasets using the MedMNISTv2 settings [63].Training used batch size 128, 224 x 224 images, AdamW [73], and learning-rate decay at epochs 50 and 75; MedViT∗ adds training augmentation and mixture feature normalization.Source: §4.1 · p. 8, §4.2 · p. 9

4.3. Evaluation metric

  • ACC and AUC assess performance across datasets, with average scores reported over MedMNIST2D for comparison with benchmarks [63].ACC is threshold-based and more sensitive to class discrepancy, whereas AUC evaluates continuous prediction scores without a fixed threshold.Source: §4.3 · p. 9

5. Evaluation Results

MedViT performs strongly across MedMNIST-2D and comparisons with similarly sized models, while its augmentation and architectural blocks support adversarial robustness and focused image localization. Ablations identify the Efficient Convolution Block as the strongest tested replacement and favor applying PMC after the first stage.

5.1. Results on Each Dataset

  • MedViT improves classification across MedMNIST-2D, with MedViT-S gaining 2.3% AUC and 3.0% ACC on RetinaMNIST and 1.1% AUC and 2.8% ACC on TissueMNIST over the cited AutoML comparators.The text also reports steady performance improvements on PathMNIST, ChestMNIST, DermaMNIST, PneumoniaMNIST, and BreastMNIST, and describes generalization across varied medical-image types.Source: §5.1 · pp. 9–10, Table 3 · p. 10

5.2. Comparison with State-of-the-art Models

  • Against similarly sized models, MedViT-T reaches 70.3% Top-1 accuracy, while MedViT-S reaches 73.1% and exceeds ResNet-50 by 5.1%, EfficientNet-B4 by 2.6%, and ConvNext-T by 0.5%.MedViT-L exceeds ConvNext-B by 0.8% with approximately half its parameters; MedViT-T exceeds PVT-T by 6.9%, and MedViT-S exceeds Twins-SVT-S by 1%.Source: §5.2 · p. 10

5.3. Average performance

  • MedViT-S reports the highest averages among the three MedViT models—94.2% average AUC and 85.1% average ACC—and outperforms baseline ResNets and AutoML methods on both metrics.The reported average AUCs for MedViT-T, MedViT-S, and MedViT-L are 93.6%, 94.2%, and 93.5%, respectively; their average ACCs are 84%, 85.1%, and 84.2%. Model complexity is described as on par with baseline ResNets.Source: §5.3 · pp. 10–11, Table 5 · p. 11

5.4. Visual inspection of MedViT

  • Grad-CAM visualizations show MedViT-T localizing relevant image regions more precisely than ResNet-18, attending less to background, and handling lesions at different scales.Examples include retinal lesion direction and area, a focal bacterial-infection region in Chest, and multifocal peripheral lesions in Pneumonia; the authors attribute these observations to convolutional locality and scale-invariance.Source: §5.4 · p. 11

5.5. Augmentation and Robustness Evaluation

  • On TissueMNIST, MedViT-T* gains 38.4% and 6.1% under FGSM, and 30.2% and 7.5% under PGD, versus ResNet-18 and MedViT-T, respectively.FGSM and PGD use adversarial noise magnitude ε = 8/255 = 0.031; PGD uses step size 4/255 = 0.015 and 5 steps. The Transformer Augmentation Block improves ACC by 1.5%, 3.1%, 1.1%, and 0.7% on Oct, Tissue, Retina, and Path MNIST, respectively; MedViT-T* is reported to yield the best accuracy/robustness tradeoff.Source: §5.5 · p. 11

5.6. Ablation Study

  • On TissueMNIST, the Efficient Convolution Block achieves the best tested clean and robust accuracy, exceeding the ConvNext block by 0.6% and 4.4%, respectively, with lower model complexity.The ablation compares ECB with ConvNext, ResNet residual, PoolFormer, and Twins LSA blocks while keeping other components unchanged and using similar complexity.Source: §5.6 · p. 12
  • In MedViT-T ablations on TissueMNIST, PMC works best after the first of four stages, and the study adopts that placement.The authors suggest PMC has greater advantage in early stages, whereas later stages already contain substantial information and are less affected by PMC.Source: §5.6 · p. 12

6. Conclusion

MedViT is a hybrid CNN-Transformer architecture for medical image classification that combines local representations and global features. It uses patch moment changer augmentation, and experiments report state-of-the-art accuracy and robustness on large-scale 2D biomedical datasets.

6. Conclusion

  • MedViT combines local representations and global features in a hybrid CNN-Transformer architecture for medical image classification.The conclusion describes the architecture as using robust components.Source: §6 · p. 12
  • MedViT achieves state-of-the-art accuracy and robustness on the standard large-scale collection of 2D biomedical datasets.The authors suggest the model may encourage future work on realistic medical deployment.Source: §6 · p. 12
  • Patch moment changer augmentation adds rich diversity and affinity to the training data.Source: §6 · p. 12

Related papers

Loading 2302.09462v1…