Source-linked AI summary
Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images
Ming Y. Lu, Bowen Chen, Andrew Zhang, Drew F. K. Williamson, Richard J. Chen, Tong Ding, Long Phi Le, Yung-Sung Chuang, Faisal Mahmood
TL;DR
Pathology lacks large paired image-caption datasets, and whole-slide images create major computational challenges for zero-shot transfer. MI-Zero addresses these challenges with multiple-instance learning for gigapixel WSIs, achieving validation across three real-world cancer subtyping tasks and improving average performance by 11.6% when trained on a larger dataset.
Problem
Pathology lacks large publicly available paired image-caption datasets, while whole-slide images can reach 100,000 × 100,000 pixels and are computationally intractable to embed directly.
Method
MI-Zero uses multiple-instance learning to apply pretrained visual-language encoders to zero-shot classification of gigapixel histopathology whole-slide images.
Results
11.6% overall average performance improvement was achieved across all tasks when training on the larger image-caption dataset rather than ARCH alone.
Takeaways & Limitations
MI-Zero provides a zero-shot transfer framework for pathology and its relevant-ROI retrieval suggests potential usefulness for semi-supervised histopathology workflows.
Takeaways & Limitations
Current results are constrained by the difficulty of curating larger datasets of high-quality image-caption pairs.
Abstract
from arXiv · showhide
Contrastive visual language pretraining has emerged as a powerful method for either training new language-aware image encoders or augmenting existing pretrained models with zero-shot visual recognition capabilities. However, existing works typically train on large datasets of image-text pairs and have been designed to perform downstream tasks involving only small to medium sized-images, neither of which are applicable to the emerging field of computational pathology where there are limited publicly available paired image-text datasets and each image can span up to 100,000 x 100,000 pixels. In this paper we present MI-Zero, a simple and intuitive framework for unleashing the zero-shot transfer capabilities of contrastively aligned image and text models on gigapixel histopathology whole slide images, enabling multiple downstream diagnostic tasks to be carried out by pretrained encoders without requiring any additional labels. MI-Zero reformulates zero-shot transfer under the framework of multiple instance learning to overcome the computational challenge of inference on extremely large images. We used over 550k pathology reports and other available in-domain text corpora to pre-train our text encoder. By effectively leveraging strong pre-trained encoders, our best model pretrained on over 33k histopathology image-caption pairs achieves an average median zero-shot accuracy of 70.2% across three different real-world cancer subtyping tasks. Our code is available at: https://github.com/mahmoodlab/MI-Zero.
1. Introduction
Histopathology zero-shot transfer is constrained by scarce paired image-text data and the enormous size of whole slide images. MI-Zero addresses these challenges with a multiple-instance framework for slide-level classification using pretrained visual-language encoders.
- Task-specific WSI model development typically requires large labeled cohorts, patch feature extraction, supervised slide classification, and downstream transfer.
- Scaling task-specific models across hundreds of tumor types is difficult because many types are under-represented or lack adequate public samples.
- Zero-shot transfer has been demonstrated for natural images and some medical modalities, but pathology lacks large public paired image-caption datasets.
- Whole slide images can reach 100,000 × 100,000 pixels and often lack textual descriptions, bounding boxes, or region-of-interest labels.
- MI-Zero is introduced as a multiple-instance zero-shot framework for gigapixel histopathology slides and is evaluated on three real-world cancer subtyping tasks.
2. Related Work
Related work establishes contrastive visual-language pretraining as a route to aligned image-text representations and zero-shot recognition, while multiple instance learning provides a framework for weakly supervised whole-slide analysis.
- Contrastive visual representation learning: Contrastive learning aligns similar samples while separating dissimilar samples in a shared representation space.
- Visual language pretraining: Visual-language pretraining uses paired images and text to learn cross-modal representations that support downstream visual recognition.
- Visual language pretraining: Prior pathology-related visual-language work did not study zero-shot transfer for gigapixel whole slide images.
- Visual language pretraining: CLIP and ALIGN demonstrated zero-shot transfer by prompting image encoders with text labels after large-scale image-text pretraining.
- Multiple instance learning: Multiple instance learning represents weakly annotated data as bags of instances, aggregating instance information for WSI-level predictions.
3. Methods
The method builds aligned visual-language encoders from pathology image-caption and text corpora, then extends prompt-based zero-shot classification to gigapixel slides through multiple-instance aggregation.
- Image-caption dataset: The curated pathology dataset contains 33,480 diverse image-caption pairs gathered from public educational resources and the ARCH dataset.
- Unsupervised pretraining: The text encoder is pretrained on over 550k surgical pathology reports and over 400k relevant PubMed abstracts before image-text alignment.
- Embedding alignment: Visual and language encoders are aligned with a temperature-scaled cross-modal contrastive loss using paired samples as positives and other batch members as negatives.
- Image zero-shot transfer: Prompt-based zero-shot classification embeds class-specific text templates and selects the class whose prompt embedding has the highest similarity to the image embedding.
- WSI zero-shot transfer: MI-Zero divides each WSI into patches, embeds them independently, and aggregates patch-prompt scores into slide-level predictions using parameter-free permutation-invariant pooling.
- WSI zero-shot transfer: Graph-based aggregation additionally smooths patch scores over spatially connected neighbors before applying slide-level pooling.
4. Experiments and results
Experiments evaluate MI-Zero on independent histopathology WSI datasets using prompt-based zero-shot transfer, compare pooling and encoder configurations, and assess training-data and pretraining effects. The larger image-caption dataset and pretrained encoders improve performance, while spatial smoothing has little effect and locked-image tuning offers limited benefit.
- Downstream datasets: MI-Zero is evaluated on independent BRCA, NSCLC, and RCC WSI datasets for three cancer-subtyping tasks, with supervised ABMIL baselines trained on TCGA cohorts.The evaluation uses in-house independent datasets because public-data pretraining could create information leakage.
- Evaluation methodology: Zero-shot evaluation samples 50 prompts per task and reports median balanced accuracy across prompt sets.Prompt templates and class names are curated for each task before evaluation.
- Training data comparison: Training on the larger 33,480-pair dataset raises overall average performance by 11.6% compared with training on ARCH alone.The comparison uses the same CTP plus HistPathGPT configuration and ARCH contains 7,562 pathology pairs.
- Ablation study: Pretraining both image and text encoders performs best across tasks, whereas locked-image text tuning gives only marginal improvement for CTP with in-domain HistPathGPT and lowers performance in other configurations.The authors conjecture that in-domain pretraining makes image and text features easier to align for the locked-image setting.
5. Conclusion
The paper introduces MI-Zero for zero-shot transfer on gigapixel histopathology whole slide images and shows that visual-language models can retrieve relevant regions for class labels. The authors identify limited high-quality image-caption data as a current constraint and propose broader data collection and benchmark evaluation as future directions.
- Conclusion: MI-Zero is introduced as the first pathology zero-shot transfer method applied to gigapixel-scale whole slide images.The framework’s region retrieval may support semi-supervised histopathology workflows such as pseudolabeling.
- Conclusion: Relevant regions retrieved for class labels suggest potential usefulness for semi-supervised learning workflows in histopathology.The paper gives pseudolabeling as an example application.
- Conclusion: Current results are constrained by the difficulty of curating larger datasets of high-quality image-caption pairs.Future directions include collecting additional datasets, improving sample efficiency, and evaluating diverse computational pathology benchmarks.
A.1. Image-caption dataset preprocessing
The preprocessing workflow filters invalid and non-histopathology pairs, crops multipanel figures, and examines staining variation without applying stain normalization.
- Invalid image-caption pairs were removed when captions were empty or nonsensical, or corresponding images were missing.
- Nonhistopathology images, including gross, cytology, radiology, electron microscopy, fluorescence, and schematic images, were removed.
- Multipanel figures were cropped into individual images, with captions cleaned accordingly.
- Stain normalization was omitted to avoid computational overhead and choosing among normalization algorithms.The authors examined staining variation and report broad coverage across TCGA and their dataset.
B. Additional training details
Additional training details cover encoder and baseline configurations, cross-validation, and prompt construction for zero-shot evaluation.
- HistPathGPT pretraining used de-identified in-house pathology reports, four 80GB NVIDIA A100 GPUs, and sequences up to length 512.
- MI-Zero and ABMIL embed 256 × 256 patches at 20×-equivalent magnification using the CTransPath self-supervised encoder.
- Visual-language pretraining and weakly supervised baselines used separate GPU configurations documented in their respective hyperparameter tables.
- TCGA experiments used 5-fold patient-level Monte Carlo cross-validation with 80% training, 10% validation, and 10% testing splits.Splits were stratified by class label, with validation used for early stopping and model selection.
- Zero-shot evaluation sampled 50 prompt sets per task from task-specific class names and shared templates, then ensembled templates in embedding space.
C.2. Additional results to ablation studies
Ablations show benefits from broader image-caption data and in-domain encoder pretraining, while locked-image tuning depends on text-domain alignment; findings extend consistently to TCGA.
- Training on the full 33,480-pair dataset outperformed training on the 7,562-pair ARCH subset across pooling methods.
- In-domain pretraining of both image and text encoders performed best across all three tasks and pooling methods.
- Locked-image tuning slightly improved performance with HistPathGPT but significantly degraded it without in-domain text pretraining.
- The authors hypothesize that cross-domain image-text latent-space gaps make alignment through a linear projection difficult.
- A pathologist confirmed that high-similarity patches contained relevant morphological patterns driving topK slide-level predictions.
- TCGA results showed consistent trends with the independent test sets across BRCA, NSCLC, and RCC subtyping tasks.
D.1. Comparing runtime of MI-Zero against AB-
MI-Zero uses efficient patch-prompt similarity computation for WSI inference, with runtime advantages over ABMIL and consistent zero-shot evaluation patterns across datasets and pooling choices.
- MI-Zero requires one linear projection and cosine similarities after patch embeddings, avoiding a learned multilayer attention operator during inference.The similarity computation can be implemented efficiently with matrix multiplication.
- 0.70 to 0.75 ms per WSI was measured for MI-Zero inference, nearly twice as fast as ABMIL at 1.4 ms per WSI.Measurements excluded input/output time and used topK pooling on the in-house BRCA dataset.
- 20 ms per WSI for input/output remained the primary bottleneck despite a fast SSD.Text-based classifier construction added around 70 ms per prompt, amortized to 0.35 ms per WSI for 200 test slides.
- TopK pooling performed better than mean pooling for all models, while spatial smoothing did not yield consistent improvement on TCGA.