Source-linked AI summary
Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification
Aysan Ghayouri Pirsoltan, Shima Babakordi, Mohammad Reza Mohammadi
TL;DR
Existing mammography models often inadequately integrate complementary CC and MLO views. This paper introduces token-based multi-depth fusion within a frozen vision transformer, achieving consistent gains across VinDr-Mammo and CMMD, including 50.40% F1-score and 0.8090 AUC on VinDr-Mammo.
Problem
Existing multi-view mammography methods typically use feature aggregation or single-stage cross-attention, limiting structured interaction between complementary CC and MLO views.
Method
The framework combines shared prompt adaptation with bidirectional cross-attention fusion tokens inserted at multiple depths of a frozen vision transformer.
Results
Consistent improvements over linear probing, prompt-only adaptation, and conventional fusion baselines were observed across VinDr-Mammo and CMMD, with 50.40% F1-score and 0.8090 AUC on VinDr-Mammo.
Takeaways & Limitations
Token-centric adaptation and progressive cross-view interaction provide an effective framework for integrating complementary mammographic views while maintaining a frozen backbone.
Takeaways & Limitations
Patient-level dataset partitioning is required to prevent information leakage between training, validation, and test sets.
Abstract
from arXiv · showhide
Accurate breast cancer classification from mammography requires effective integration of complementary information from craniocaudal (CC) and mediolateral oblique (MLO) views, which provide a more complete characterization of breast abnormalities. However, existing multi-view learning approaches typically rely on feature-level aggregation or single-stage cross-attention, which can entangle view-specific and shared representations and restrict interaction to limited network depths. To address these limitations, we propose a token-centric dual-view learning framework that unifies prompt-based adaptation and cross-view fusion within a frozen vision transformer backbone. The framework reformulates inter-view interaction as structured token-level communication, where dedicated fusion tokens explicitly encode bidirectional information exchange between CC and MLO views via cross-attention, serving as intermediate carriers of cross-view dependencies rather than relying on direct feature fusion. Unlike conventional methods that apply fusion at a single layer, fusion modules are inserted at multiple transformer depths, enabling progressive and repeated interaction across the encoder hierarchy. Fusion tokens are reintegrated into the token sequence and refined by subsequent transformer layers, facilitating hierarchical propagation of complementary information while preserving view-specific structure. Experiments on VinDr-Mammo and CMMD datasets demonstrate consistent improvements over linear probing, prompt-only adaptation, and conventional fusion baselines. On the VinDr-Mammo BI-RADS classification task, the framework achieves 50.40% F1-score and 0.8090 AUC, including a 0.10 AUC improvement over a dual-view fusion baseline in the binary setting. Ablation studies further validate the effectiveness of token-based fusion and multi-depth interaction design.
1. Introduction
Mammogram classification is challenged by subtle lesions and the need to combine complementary CC and MLO information. The proposed framework adapts a frozen vision transformer with shared prompts and token-based, bidirectional cross-view fusion at multiple depths.
- Mammogram interpretation remains difficult because lesions have low contrast, tissue structures are complex, and class differences can be subtle.
- Single-view models discard complementary cross-view information, motivating multi-view strategies for combining CC and MLO representations.
- The framework integrates parameter-efficient representation adaptation and cross-view interaction within a frozen vision transformer backbone.
- Shared prompt tokens adapt pretrained vision models across CC and MLO views, aligning representations while preserving pretrained knowledge.
- Bidirectional cross-attention encodes inter-view dependencies into dedicated fusion tokens, which are reintegrated and refined across multiple transformer depths for hierarchical integration.The design also complements intermediate fusion with final representation-level fusion.
2. Related Work
Related work spans parameter-efficient adaptation of vision-language models and increasingly sophisticated multi-view mammography fusion strategies. However, existing methods commonly entangle view-specific and cross-view representations and restrict fusion to a single or limited number of network depths.
- Vision-language adaptation: Parameter-efficient methods adapt pretrained models by optimizing only a small set of learnable parameters, reducing computational cost while preserving backbone generalization.Examples include VPT, MaPLe, Tip-Adapter, AdaptFormer, and LoRA.
- Vision-language adaptation: Our work extends prompt-learning methods, typically used for single-image classification, to multi-view mammography classification.The extension targets complementary information from CC and MLO mammographic views.
- Multi-view fusion: Multi-view mammography methods use image-level, prediction-level, feature-level, weakly supervised, graph-based, consistency-learning, and attention-based fusion strategies.These approaches include voting or image combination, variable-instance pooling, anatomical correspondence graphs, global-local consistency modules, and cross-attention between views.
- Attention-based interaction: Cross-view attention has enabled bidirectional interaction across intermediate feature maps, region proposals, patches, and token-like embeddings without requiring explicit registration.These designs span mammography classification, detection, Swin Transformer processing, and MRI surgical planning.
- Limitations of prior work: Most existing methods fuse views through residual addition, feature concatenation, or aggregated embeddings at a single depth or limited layers, potentially entangling view-specific and cross-view representations.This limitation motivates multi-depth interaction for hierarchical exchange of complementary information.
3. Method
The method is a two-stage dual-view framework built on a frozen MedSigLIP encoder. It first learns shared deep prompts for CC and MLO views, then performs bidirectional token-based fusion at intermediate transformer layers.
- Stage 1: Shared-view prompt learning: Stage 1 jointly learns shared deep prompt tokens across CC and MLO mammograms to establish view-consistent adaptation while preserving pretrained knowledge.Interaction between views remains indirect because information is exchanged only through shared prompt parameters.
- Stage 2: Cross-view fusion: Stage 2 introduces cross-view fusion tokens that explicitly exchange complementary information between CC and MLO representations at intermediate transformer levels.The approach summarizes one view into a compact fusion token rather than directly combining feature maps through summation or averaging.
- Stage 2: Cross-view fusion: The Cross-View Fusion Block performs bidirectional cross-attention, mean-pools its outputs into fusion tokens, and inserts them into the corresponding token sequences.These tokens are jointly processed with visual tokens and propagated to subsequent transformer layers.
- Stage 2: Cross-view fusion: Fusion modules are inserted at designated transformer layers, enabling hierarchical cross-view interactions and progressive refinement throughout the encoder.This extends cross-view interaction beyond a single fusion depth while retaining the transformer’s subsequent processing.
- Optimization and prediction: During Stage 2, only the fusion modules, attention pooling head, and classifier are updated, while the pretrained backbone and Stage 1 prompt parameters remain frozen.After the final transformer layer, CC and MLO representations are aggregated, concatenated, and passed to a linear classifier.
4. Datasets
Experiments used the VinDr-Mammo and CMMD mammography datasets, with task-specific splits and patient-level partitioning to prevent information leakage. Mammograms were standardized through breast-region cropping, CLAHE enhancement, and resizing to 448 × 448 pixels.
- Experiments used two publicly available mammography datasets: VinDr-Mammo and the Chinese Mammography Database (CMMD).
- VinDr-Mammo: VinDr-Mammo contains 5,000 four-view examinations (20,000 images) with breast-level BI-RADS annotations for five-class and binary classification.The official train/test partition was used for the multi-class task, with 400 training cases randomly selected for validation.
- CMMD: CMMD contains 5,202 mammography images from 1,775 patients with pathology-confirmed diagnoses and was stratified into training, validation, and test sets for binary classification.No official split was provided.
- All dataset partitions were performed at the patient level to prevent information leakage between training, validation, and test sets.
- Preprocessing: Images were cropped to the breast region, enhanced with CLAHE, and resized to 448 × 448 pixels to match the MedSigLIP input resolution.Foreground contour detection removed most irrelevant background area before contrast enhancement.
5. Experiments
Experiments show that the proposed prompt-adaptive, token-based cross-view framework consistently improves mammography classification over baselines across CMMD and VinDr-Mammo. Ablations support fusion tokens, moderate multi-depth fusion, prompt learning, and encoder-head fine-tuning as important design choices.
- Overall Performance: 64.96% F1-score and 0.7161 AUC on CMMD improve over linear probing values of 57.57% and 0.7045, respectively.On VinDr-Mammo, the proposed method achieves 50.40% F1-score and 0.8090 AUC, outperforming all baselines.
- Overall Performance: Prompt learning improves over linear probing, while fusion without prompt initialization remains competitive, demonstrating complementary benefits from adaptation and explicit CC–MLO interaction.The full framework achieves the best overall trade-off, with the highest AUC on both datasets and the best F1-score on VinDr-Mammo.
- Fusion-Token Ablation: The proposed method consistently outperforms the residual fusion baseline, with fusion tokens increasing VinDr-5 performance by 5.22 percentage points in F1-score and 0.0521 in AUC.These gains indicate that dedicated fusion-token representations are more effective than direct residual cross-attention fusion under the same training configuration.
- Additional Ablations: Mean pooling achieves the best VinDr-Mammo Binary performance among fusion-token aggregation strategies, while fine-tuning the vision encoder head improves performance with the best results at learning rate 1e-5.Competitive performance remains when the encoder head is frozen, indicating substantial contributions from fusion and prompt-learning components.
- Fusion-Depth Ablation: Two fusion blocks at layers 12 and 23 achieve the best AUC of 0.8593, exceeding single-block AUC of 0.8441 and four-block AUC of 0.8313.The results suggest that moderate multi-depth fusion is preferable to overly dense fusion across deeper layers.
6. Conclusion
The work unifies prompt-based adaptation and progressive cross-view integration within a token-centric framework for frozen vision transformers in multi-view mammography classification. It further presents token-based adaptation and fusion as a flexible, scalable alternative to conventional feature-level fusion in multi-view medical imaging.
- Conclusion: The framework adapts frozen pretrained vision transformers by combining task-specific prompt tokens with view-specific cross-view fusion tokens.This unifies adaptation and cross-view integration within a token-centric framework while introducing task-specific and view-specific information into pretrained representations.
- Conclusion: Token-based adaptation and fusion offer a flexible, scalable paradigm for leveraging vision foundation models in multi-view medical imaging beyond mammography.The approach is presented as a promising alternative to conventional feature-level fusion strategies.