PaperlayerView exact revision on arXiv

Source-linked AI summary

TransMed: Transformers Advance Multi-modal Medical Image Classification

Yin Dai, Yifan Gao

arXiv:2103.05940v1cs.CV

TL;DR

Multimodal medical images contain important long-range dependencies, but small medical datasets make pure-transformer approaches difficult and existing fusion methods have limitations. TransMed combines CNNs for low-level features with transformers for cross-modality relationships, and experiments on parotid tumor classification report competitive, efficient performance with fewer parameters and computational costs.

  • Problem

    Multimodal medical-image fusion needs to model informative cross-modality relationships efficiently, while medical datasets are small and existing methods have difficulty establishing long-range connections.

  • Method

    TransMed serializes multimodal images, uses CNNs to extract low-level features, and applies transformers to learn relationships between sequences and make predictions.

  • Results

    TransFuse-S improves average accuracy by about 12.8% over P3D, while TransMed achieves state-of-the-art performance with fewer parameters and computational costs.

  • Takeaways & Limitations

    The CNN–transformer combination is reported as a flexible, low-resource-cost design for multimodal medical image classification.

Abstract

from arXiv · show

Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it can not deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependencies, and effective multi-modal fusion strategies can greatly improve the performance of deep models. This prompts us to study transformer-based structures and apply them to multi-modal medical images. Existing transformer-based network architectures require large-scale datasets to achieve better performance. However, medical imaging datasets are relatively small, which makes it difficult to apply pure transformers to medical image analysis. Therefore, we propose TransMed for multi-modal medical image classification. TransMed combines the advantages of CNN and transformer to efficiently extract low-level features of images and establish long-range dependencies between modalities. We evaluated our model for the challenging problem of preoperative diagnosis of parotid gland tumors, and the experimental results show the advantages of our proposed method. We argue that the combination of CNN and transformer has tremendous potential in a large number of medical image analysis tasks. To our best knowledge, this is the first work to apply transformers to medical image classification.

I. INTRODUCTION

TransMed addresses limitations in multimodal medical-image fusion by combining CNN feature extraction with transformer modeling of long-range relationships. The paper motivates this design through explicit sequence dependencies in medical images and evaluates it for parotid tumor classification.

  • Motivation: Existing fusion strategies can struggle with cross-modality relationships, scalability, or computational cost.Input- and decision-level methods may not establish internal modality relationships, while feature-level fusion can require multiple networks.
  • Motivation: Medical-image modalities contain explicit, informative sequences and long-range dependencies that existing fusion methods often fail to model.These sequences include modality, slice, and patch relationships relevant to diagnosis.
  • Proposed method: TransMed combines CNNs and transformers to extract low-level features and learn cross-modality relationships from serialized multimodal images.The images are processed as sequences by CNNs before transformers model relationships and make predictions.
  • Contributions: The proposed approach is reported to improve accuracy and efficiency while modeling long-range relationships between modalities.The paper evaluates these claims through experiments comparing TransMed with existing multimodal fusion methods.
  • Contributions: The paper reports the first application of transformers to medical image classification and proposes a multimodal fusion strategy for capturing mutual information.The stated evaluation target is parotid gland tumor classification.

II. RELATED WORK

Prior multimodal medical-image studies use input-, feature-, and decision-level fusion, but these approaches have reported limitations in scalability, computational complexity, and long-range connections. TransMed is presented as a CNN–transformer alternative for this setting.

  • Multimodal medical image analysis: Multimodal medical-image analysis combines modalities to capture more pathological information and improve diagnosis quality.The related work frames multimodal analysis as a fundamental and challenging part of medical image analysis.
  • Multimodal medical image analysis: Prior studies use input-level, feature-level, and decision-level fusion strategies for multimodal medical images.Examples include modality-specific networks, linked features, and final-feature fusion for decisions.
  • TransMed: TransMed differs from prior fusion approaches by combining CNN and transformer branches in a hybrid architecture.The CNN branch generates patch embeddings, while the transformer branch processes relationships between sequences.
  • Limitations: Existing fusion methods improve performance to some extent but may have poor scalability, high computational complexity, and difficulty establishing long-range connections.These limitations motivate architectures that model modality relationships more directly and efficiently.

B. TRANSFORMERS

Transformers have shown promise in vision but remain difficult to apply directly to multi-modal medical images. TransMed addresses this gap with a transformer-based fusion framework that combines self-attention with CNN-style processing.

  • TransMed is presented as the first transformer-based framework for multi-modal medical image classification.
  • Transformer-based methods have produced encouraging computer-vision results, but direct application to multi-modal medical images is described as ineffective and computationally expensive.
  • The framework introduces self-attention into multi-modal fusion to aggregate feature representations from decomposed image patches.
  • The authors follow the original DeiT implementation to reduce the influence of additional performance-enhancing techniques.This setting also enables use of the extensible DeiT model and its pre-trained weights.
  • The transformer components discussed include self-attention, multi-head self-attention, and a multi-layer perceptron.

1) Self-Attention

Self-attention transforms each input into query, key, and value representations, then uses query–key similarity to compute a weighted sum of value vectors. Scaling by the key dimension stabilizes gradients.

  • Self-attention transforms each input into query, key, and value matrices and produces a weighted sum of value vectors.
  • The weight assigned to each value is determined by the dot product between its corresponding query and key.
  • Scaling by √d_k provides normalization intended to make gradients more stable.

2) Multi-head Self-Attention

Multi-head self-attention splits inputs into multiple parts, computes scaled dot-product attention in parallel, and concatenates the outputs. This lets the model learn sequence and location information across representation subspaces.

  • Multi-head self-attention splits the input into multiple parts, computes scaled dot-product attention in parallel, and concatenates the resulting outputs.
  • The multi-head mechanism allows the model to learn sequence and location information in different representation subspaces.
  • The projections W_Q and W_O are trainable parameter matrices in the multi-head attention formulation.

3) Multi-Layer Perceptron

The transformer input combines patch and class-related embeddings and tokens before passing through transformer layers. A multilayer perceptron follows multi-head attention and uses normalized residual processing before classification.

  • The MLP uses linear layers separated by a GeLU activation and, like MSA, includes layer normalization and skip-connections.
  • The transformer input contains patch embedding, position embedding, class embedding, patch token, and class token.
  • Patch embeddings represent CNN outputs for image patches, while position embeddings encode spatial and location information in patch tokens.
  • The class token is attached to patch tokens before the transformer layers and is then passed to a fully connected layer for class prediction.

B. TRANSMED

TransMed combines CNN-based low-level feature extraction with transformer-based modeling of relationships among multi-modal image sequences. Its pipeline constructs image sequences, embeds CNN-derived patches, and produces transformer-compatible representations.

  • TransMed uses a hybrid CNN-transformer encoder rather than a pure transformer.CNN extracts low-level features and generates patch embeddings for the transformer.
  • Three adjacent 2D slices are superimposed to form three-channel images before sequence construction.Each image is then divided into K × K patches, with larger K producing smaller patches.
  • The constructed image sequence is processed by a 2D CNN whose final fully connected layer is replaced by a linear projection.The projection maps vector-patch features into the transformer’s embedding space.
  • The CNN preliminarily encodes the image sequence and outputs a representation shaped to fit the transformer input.The output representation is described as having an embedding dimension P adapted to the transformer.

IV. RESULTS

The study evaluates TransMed on preoperative classification of parotid gland tumors using paired T1 and T2 MRI from 344 patients. Preprocessing extracts and aligns foreground regions, standardizes image dimensions, and applies augmentation.

  • The dataset contains T1 and T2 MRI from 344 patients, with biopsy-derived ground-truth labels.
  • Parotid gland tumors are classified into five categories: PA, WT, MT, BCA, and OBL.These correspond to Pleomorphic Adenoma, Warthin Tumor, Malignant Tumor, Basal Cell Adenoma, and Other Benign Lesions.
  • Preprocessing applies OTSU foreground extraction, registers modalities within each patient, and resamples images to standardized dimensions.The final data comprise 344 paired-modality 3D MRI stacks, each sized (36, 448, 448).
  • Data augmentation uses random flipping with 50% probability and Gaussian noise with mean 0 and variance 0.1.

B. EXPERIMENTAL SETTINGS AND EVALUATION CRITERIA

The experiments train and evaluate models using accuracy and per-category precision on randomly partitioned parotid tumor data. Training uses SGD with specified optimization settings and a 100-round maximum.

  • Patients are split randomly into 275 training cases and 69 independent test cases using a 4:1 ratio.The training group is used to optimize model parameters.
  • Training uses SGD with learning rate 10^-3, momentum 0.7, and a maximum of 100 training rounds.
  • Overall accuracy ACC(i) and per-category precision P(i) are used as evaluation criteria.
  • Overall accuracy is defined as the total number of correct predictions divided by the total number of samples.T denotes total samples and Tc denotes correctly predicted samples.
  • Per-category precision uses correctly predicted samples of class i relative to all samples predicted as class i.The paper states that P(i) describes model stability and robustness on small datasets.

C. BASELINE METHODS

The study compares TransMed with input-, feature-, and decision-level fusion baselines and evaluates multiple TransMed backbone variants. TransMed outperforms prior fusion strategies while using fewer parameters and computational costs, although larger variants show overfitting.

  • Baseline methods: Baseline experiments include 2D and 3D CNN fusion models, including ResNet34, ResNet152, 3D ResNet34, P3D, and C3D.Feature-level and decision-level fusion methods are also adapted for classification.
  • Baseline methods: Table 1 reports average accuracy and per-disease precision on the parotid gland tumor dataset.IF, FF, and DF denote input-level, feature-level, and decision-level fusion, respectively.
  • Results: TransMed consistently outperforms previous multi-modal fusion strategies and achieves state-of-the-art performance with fewer parameters and computational costs.TransFuse-S improves average accuracy by about 12.8% relative to P3D, while TransMed-B and TransMed-L slightly overfit the dataset.
  • Ablation experiments: Increasing the backbone from TransMed-T to TransMed-S improves average accuracy by 1.9% but substantially increases computational cost.Consequently, comparisons use TransMed-T to demonstrate effectiveness while considering computation cost.
  • Ablation experiments: Removing the CNN or transformer shows that the transformer improves modality-relationship modeling with little additional parameter and computation cost.The pure transformer structure performs poorly on the small dataset.
  • Ablation experiments: Performance is poor when K is large, possibly because smaller image patches destroy semantic information.

V. CONCLUSION

TransMed achieves competitive parotid tumor classification results while combining CNN and transformer components, but its evaluation and broader applicability remain limited. The authors identify extending it beyond classification and improving pure-transformer performance on small medical datasets as open challenges.

  • TransMed achieves very competitive results for challenging parotid tumor classification.
  • Its flexible structure can extend to multiple medical image modalities with low resource cost.
  • Applying TransMed to tumor segmentation and lesion detection remains an unresolved challenge.
  • Preliminary experiments show a substantial performance gap between pure transformers and typical CNNs on small medical image datasets.
Loading 2103.05940v1…