MagViT: Interpretable Multi-Magnification Transformers with Patient-Level Model Selection for Breast Histopathology

Nabil Ashab; Soumit Kumar Kundu; Saif Mahmud Parvez; Shahadat Hossain Sohag; Bidhan Biswas; Nazmus Subha

Source-linked AI summary

MagViT: Interpretable Multi-Magnification Transformers with Patient-Level Model Selection for Breast Histopathology

Nabil Ashab, Soumit Kumar Kundu, Saif Mahmud Parvez, Shahadat Hossain Sohag, Bidhan Biswas, Nazmus Subha

arXiv:2608.16959v1Aug 16, 2026eess.IVcs.CVcs.LG

TL;DR

Breast histopathology models may appear reliable at the image level while remaining less reliable across patients, datasets, and magnifications. MagViT addresses this with mask-aware multi-scale fusion and patient-level branch selection, showing strong BreakHis patient-level performance alongside transfer evidence on BUSI and IDC. Its findings support patient-level evaluation and transfer-aware testing as practical priorities for histopathology classification.

Problem
Image-level evaluation can obscure patient-level reliability, while BreakHis performance may not transfer across datasets and interpretability consistency remains unclear.
Method
MagViT fuses multi-magnification ViT representations with mask-aware scale gating and selects among architectural branches using patient-level five-fold cross-validation.
Results
MagViT delivers strong BreakHis patient-level performance and shows transfer evidence on BUSI and IDC under controlled evaluation settings.
Takeaways & Limitations
Patient-level model selection and transfer-aware evaluation provide a practical path toward more reliable histopathology classification than image-level optimization alone.
Takeaways & Limitations
The evaluation is restricted to binary source training, two external datasets, limited data, and five validation folds, constraining generalizability.

Abstract

from arXiv · show

Breast cancer is one of the most common types of cancer among women around the world. Rapid detection and early treatment can hinder its progress to more complex stages and can impede its spread to other parts of the body. Histopathological image classification is the most common task in cancer detection due to its robustness in analyzing cellular data. Breast histopathology classification requires handling both multi-scale tissue morphology and clinically relevant generalization beyond the source domain. This paper presents MagViT, an interpretable multi-magnification transformer framework with scale-gated fusion and patient-level model selection. The model uses four BreakHis magnifications (40X, 100X, 200X, 400X) and extracts per-scale representations with a ViT backbone, and combines them via a learnable gate that masks missing scales. Patient-level five-fold cross-validation with a fixed seed has been run and compared with three architectural branches. The most accurate branch is then selected as the final model due to the strongest patient-level accuracy while retaining the simplest fusion pathway. On BreakHis, our architecture achieves a mean image accuracy of 0.9191, a mean patient accuracy of 0.9643, and a mean macro-F1 of 0.9042. External transfer experiments provide preliminary evidence of cross-dataset generalization under controlled adaptation settings on BUSI (image accuracy 0.8306, macro-F1 0.7480, patient accuracy 0.8291) and IDC (image accuracy 0.8577, macro-F1 0.8191, patient accuracy 0.8372). Grad-CAM visualization indicates that the model focuses on diagnostically significant and meaningful regions across magnifications. Relative to prior ViT-centered BreakHis work, this study emphasizes patient-level selection and cross-dataset robustness under a reproducible protocol.

1 Introduction

Breast histopathology requires multi-scale analysis and clinically reliable patient-level decisions, while BreakHis performance may not generalize across distribution shifts. MagViT addresses these gaps through mask-aware multi-magnification fusion, patient-level model selection, transfer evaluation, and visual diagnostics.

Motivation: BreakHis provides diverse patient samples across 40X, 100X, 200X, and 400X magnifications, exposing multi-scale breast tissue morphology.The dataset is presented as an established benchmark for machine-learning and deep-learning research in breast pathology.
Motivation: Image-level performance may obscure patient-level reliability, while BreakHis results may not transfer to BUSI or invasive ductal carcinoma patches.The introduction also identifies inconsistent interpretability beyond isolated examples as a practical gap.
Contributions: MagViT introduces a mask-aware multi-magnification transformer fusion framework over 40X, 100X, 200X, and 400X views.The framework is designed to address multi-scale morphology while accommodating the listed magnification views.
Contributions: A patient-level model-selection protocol evaluates three branch variants using five-fold cross-validation.This protocol targets reliability at the patient level rather than relying only on image-level aggregation.
Contributions: The study provides cross-dataset transfer evidence on BUSI and IDC with linear-probe and few-shot adaptation, alongside comprehensive visual diagnostics.Diagnostics include training curves, ROC curves, confusion matrices, magnification importance, and Grad-CAM.

2 Related Work

Prior breast histopathology studies have used CNNs, attention mechanisms, and more recently transformers, but often prioritize image-level accuracy over patient-level evaluation. Interpretability and multi-domain robustness also remain limited, motivating the proposed framework.

CNN-Based Methods: CNN-based models, including ResNet, DenseNet, and EfficientNet, have shown strong breast histopathology classification performance, often with multi-scale fusion strategies.Despite high accuracy, these models are described as computationally expensive and difficult to deploy in practice.
CNN-Based Methods: Pretrained CNN feature extractors paired with lightweight classifiers, along with attention modules such as CBAM and dual squeeze-and-excitation blocks, have improved BreakHis pipelines.These attention mechanisms emphasize diagnostically relevant regions.
Transformer-Based Methods: Transformer-based and CNN-transformer hybrid architectures have been explored, yet many studies emphasize image-level accuracy without clinically relevant patient-level evaluation.Interpretability and multi-domain robustness are often limited to qualitative examples, hindering assessment of decision consistency.

3 Data and Experimental Protocol

The study used BreakHis as the primary training dataset and BUSI and IDC for transfer learning. Experiments followed patient-level five-fold cross-validation with safeguards against patient overlap and data leakage.

Datasets: Three datasets were used: BreakHis for training and evaluation, and BUSI and IDC for transfer learning.BreakHis served as the source dataset, while BUSI and IDC supported transfer experiments.
Datasets: 9,109 BreakHis images from 82 patients span 40X, 100X, 200X, and 400X magnifications, with 2,480 benign and 5,429 malignant samples.BreakHis was the main training dataset.
Datasets: 780 BUSI ultrasound images were collected from 600 female patients aged 25–75 years, with an average image size of 500*500 pixels.BUSI was used for transfer learning.
Datasets: IDC transfer data comprised diagnosed invasive ductal carcinoma cases from 162 women at the Hospital of the University of Pennsylvania.IDC was used as a transfer dataset.
Experimental protocol: 5-fold cross-validation used official fold metadata and defined patient-level splits, with execution aborted if patient overlap or leakage was detected.The protocol prevented the same image from appearing in training, validation, and test sets.

4 MagViT Method … 4.4 Methodological and Fair-Comparison Rationale

MagViT combines synchronized multi-magnification ViT features with scale-gated fusion and masking, using controlled, patient-level evaluation to support reliability under realistic data constraints. The methodology standardizes augmentation, preprocessing, training settings, and reproducibility safeguards across experiments.

4 MagViT Method: MagViT extracts features across 40X, 100X, 200X, and 400X magnifications, fuses CLS tokens with scale-gated fusion, and performs binary classification.The pipeline was selected from multiple experimental pipelines to address real-world diagnosis and maximize performance gains.
4.1 Dataset Augmentation: Training augmentation applies one randomly selected transformation combination consistently across every zoom level in an image group.Transformations include flips, rotations, color adjustments, and small positional or scale changes.
4.2 Image Preprocessing: Preprocessing preserves corresponding tissue structure across 40X, 100X, 200X, and 400X views while improving resilience to rotation, color, and positional changes.Images are colored, resized, normalized, tensor-converted, and augmented with color jitter; missing zooms are represented by a mask.
4.2 Image Preprocessing: The architecture explicitly masks missing magnification levels so multi-scale processing remains robust when one or more zooms are unavailable.The preprocessing pipeline marks missing zooms with a mask before model input preparation.
4.3 Reproducibility Preserved Core Settings: Experiments use fixed configurations and random seed 42 to ensure performance differences reflect architectural design rather than hyperparameter optimization.The source task is binary BreakHis classification with patient-level 5-fold cross-validation, official Folds.csv, and explicit patient-overlap checks.
4.3 Reproducibility Preserved Core Settings: The core evaluation tracks image accuracy, patient accuracy, and macro-F1 to assess both image-level and clinically relevant patient-level outcomes.A ViT-B/16 backbone, fixed training defaults, and class-weighted cross-entropy with label smoothing define the preserved experimental configuration.
4.4 Methodological and Fair-Comparison Rationale: MagViT aligns its design with clinical decision-making by emphasizing patient-level reliability, complementary magnification information, adaptive scale weighting, and controlled experimentation.Fixed hyperparameters isolate structural effects, while masking supports robustness to missing data and clarifies design influences on image- and patient-level outcomes.

4.5 Multi-Magnification Representation · 4.6 Scale-Gated Fusion

MagViT represents four BreakHis magnifications with a shared ViT encoder and fuses available scales using sample-specific, mask-constrained gates. This design preserves common feature geometry while excluding missing scales and adapting weights to heterogeneous morphology.

4.5 Multi-Magnification Representation: 4 magnifications—40X, 100X, 200X, and 400X—define the multi-scale input for each patient region.
4.5 Multi-Magnification Representation: An availability mask accommodates patient regions missing one or more magnification scales.
4.5 Multi-Magnification Representation: A single ViT backbone operates across all magnifications, reducing branch-specific drift and enforcing common feature geometry.
4.5 Multi-Magnification Representation: Unavailable scales are strictly excluded from the fused representation through the binary fusion mask.
4.6 Scale-Gated Fusion: High magnification captures cellular detail whereas low magnification captures global tissue layout, motivating adaptive scale weighting.
4.6 Scale-Gated Fusion: Per-scale features are concatenated and mapped to gate logits by a learnable gate network.
4.6 Scale-Gated Fusion: Masked softmax produces valid weights for available scales, yielding adaptive sample-specific fusion while excluding unavailable inputs.

4.7 Objective and Optimization · 4.8 Branch Definitions and Best Branch Selection · 4.9 Algorithm Description

Sections 4.7–4.9 define the class-imbalance-aware optimization objective, compare three controlled architectural branches, select Branch 1 using patient-centric criteria, and summarize the end-to-end pipeline.

4.7 Objective and Optimization: Class-weighted cross-entropy with label smoothing defines the binary classification objective using predictive probabilities and class weights.The class label is specified as y_i ∈ {1, ..., C} with C = 2, and the objective uses w_c, ỹ_i,c, and p_i,c.
4.7 Objective and Optimization: AdamW, cosine learning-rate decay, and warmup support stable transformer fine-tuning while accounting for class imbalance.The setup is described as keeping the optimization objective compact through Eqs. 8 and 9.
4.8 Branch Definitions and Best Branch Selection: Three branches isolate fusion complexity and optimization policy while preserving hyperparameters, data splits, and core settings for controlled comparison.Training metrics and checkpoints also reveal frequent patient-level decision flipping, motivating targeted adjustments categorized as critical, high, and optional.
4.8 Branch Definitions and Best Branch Selection: Branch 1 handles missing scales with masks, applies consistent augmentation across magnifications, and aggregates patient predictions by majority vote.These components define the branch’s baseline fusion and patient-level decision pathway.
4.8 Branch Definitions and Best Branch Selection: Branch 2 adds residual fusion, normalization, and enhanced gating, whereas Branch 3 further adds confidence-weighted voting, partial freezing, and entropy regularization.The branches therefore increase fusion and optimization complexity progressively from Branch 1 through Branch 3.
4.8 Branch Definitions and Best Branch Selection: Branch 1 is selected as the final configuration because patient-level consistency, lower complexity, stability, interpretability, and computational efficiency outweigh isolated image-level gains.The selection criterion prioritizes minimizing patient-level misclassification; Branch 3 has maximum complexity with the lowest gain among the alternatives.
4.9 Algorithm Description: The full architecture is summarized as an end-to-end algorithm that creates grouped multi-magnification samples, enforces patient-wise disjoint folds, and initializes a shared-encoder MagViT model.These are the first listed pipeline steps before per-fold processing.

5 Results on BreakHis

MagViT’s selected Branch 1 achieves the strongest patient-level BreakHis performance under five-fold evaluation, with supporting evidence from ablations and stable fold-wise behavior. Controlled transfer experiments on BUSI and IDC show improved or comparable few-shot performance, but provide preliminary rather than definitive domain-generalization evidence.

Branch comparison: 0.9643 patient-level mean accuracy makes Branch 1 the strongest branch, so it is selected for integration despite Branch 2’s slightly higher image-level accuracy.Branch 3 has the greatest architectural complexity and reduced overall accuracy.
Ablation analysis: Removing augmentation, color jitter, or class weighting consistently lowers patient-level accuracy, supporting their contribution to robustness and generalization.More fine-grained comparisons of alternative fusion strategies remain limited and are deferred as future work.
BreakHis performance: 0.9191 image accuracy, 0.9643 patient accuracy, and 0.9042 macro-F1 are reported for Branch 1 across five folds.The corresponding variability estimates are ±0.0416, ±0.0443, and ±0.0470, respectively.
Magnification and fold analysis: 200X and 100X are consistently strong as single magnifications, whereas 400X shows more variance, motivating multi-scale fusion.Fold-wise performance is also affected by missing magnifications for some patients, with folds 2 and 4 suffering most.
Cross-dataset transfer: Few-shot learning consistently improves patient-level performance on BUSI and IDC, but the controlled adaptation results are preliminary rather than definitive domain-adaptation evidence.The evaluation excludes adversarial alignment, discrepancy minimization, test-time adaptation, and multi-center external cohorts.
Cross-dataset transfer: 0.861 versus 0.691 AUC on BUSI shows that few-shot learning improves class separation over linear probing, with fewer confusion-matrix misclassifications.On IDC, AUC values remain close, while confusion matrices indicate high prediction accuracy with minimal errors.

6 Grad-CAM Visualization

The paper uses Grad-CAM for explainability by highlighting image regions associated with predictions. It applies Grad-CAM to the ViT projection layer and overlays upsampled heatmaps on BreakHis images.

Grad-CAM Visualization: Grad-CAM highlights image regions that match the model’s prediction for explainability.It is computed on the convolutional ViT projection layer where patch embedding occurs.
Grad-CAM Visualization: Upsampled Grad-CAM heatmaps are overlaid on BreakHis histopathology images.The figure shows original images, heatmaps, and overlays for one correct benign and one correct malignant prediction.

7 Discussion

The discussion favors patient-level stability over isolated image-level optimization, while highlighting multi-scale resilience, aggregation benefits, and adaptation-dependent transfer performance. Few-shot adaptation is especially useful for BUSI, whereas IDC shows smaller gains consistent with a strong source representation.

Branch comparison: Branch 2 slightly outperforms Branch 1 in image accuracy and macro-F1, but Branch 1 remains better in patient accuracy.The discussion favors Branch 1 when patient-level consistency is prioritized.
Multi-scale fusion: Multi-scale fusion improves resilience when one magnification carries weak signal.
Patient-level evaluation: Patient-level aggregation reduces overconfidence caused by isolated image-level mistakes.
Transfer analysis: Few-shot adaptation improves BUSI discrimination and macro-F1, while IDC shows smaller positive gains in patient accuracy and ROC-AUC.The discussion attributes BUSI’s stronger benefit to substantial modality shift and describes IDC fine-tuning as calibration.

8 Limitations and future directions

The study is limited by binary source labels, restricted external datasets, and incomplete calibration and uncertainty analysis. Future work should expand validation, patient-level calibration, weakly supervised whole-slide learning, and domain adaptation.

Limitations: Source training remains binary and excludes subtype-level pathology labels, while external transfer is limited to BUSI and IDC.
Limitations: Calibration, uncertainty decomposition, and prospective study design are not yet included in the evaluation.
Limitations: The available data scale and number of validation folds may limit the generalizability of the findings.
Future directions: Future work can add patient-level confidence calibration and threshold analysis, extend to weakly supervised whole-slide settings, and address staining and scanner variation through domain adaptation.

9 Conclusion

MagViT combines multi-zoom ViT representations with mask-aware scale-gated fusion and patient-level branch selection. The selected Branch 1 achieves strong BreakHis patient-level performance and shows generalization to BUSI and IDC.

Architecture: MagViT combines different zoom-level ViT representations with mask-aware scale-gated fusion and patient-level branch selection.These components define the proposed architecture.
Results: The selected Branch 1 configuration delivers strong patient-level performance on BreakHis.The conclusion identifies Branch 1 as the selected configuration.
Generalization: The architecture shows generalization to BUSI ultrasound images and the IDC histopathological dataset.The transfer claim covers both external datasets named in the conclusion.
Practical implications: The paper concludes that patient-level model selection and transfer support a practical use of the architecture.This conclusion is based on the detailed tables and diagnostic figures.