PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiple Instance Learning for Digital Pathology: A Review on the State-of-the-Art, Limitations & Future Potential

Michael Gadermayr, Maximilian Tschuchnig

arXiv:2206.04425v2cs.CV

TL;DR

Digital pathology needs automated analysis for information-dense whole-slide images, but deep learning is limited by annotation requirements and hardware constraints. This review synthesizes recent deep MIL architectures and finds growing use of attention and combined instance- and embedding-based learning, while identifying remaining practical limitations.

  • Problem

    Whole-slide images contain extensive information, while patch-, region-, and pixel-level annotations are often unavailable and gigapixel processing exceeds practical neural-network input limits.

  • Method

    The paper provides an unstructured literature analysis and structured mathematical and textual synthesis of state-of-the-art MIL building blocks and recent technical approaches.

  • Results

    The review finds no clear dominant architecture, but identifies trends toward attention mechanisms and combinations of instance- and embedding-based learning.

  • Takeaways & Limitations

    MIL approaches are generally applicable across histological fields and provide a basis for analyzing digital pathology with whole-slide labels rather than complete local annotations.

  • Takeaways & Limitations

    The review is non-exhaustive, emphasizes recent technical contributions, and does not focus on application-oriented studies or large clinical evaluations.

Abstract

from arXiv · show

Digital whole slides images contain an enormous amount of information providing a strong motivation for the development of automated image analysis tools. Particularly deep neural networks show high potential with respect to various tasks in the field of digital pathology. However, a limitation is given by the fact that typical deep learning algorithms require (manual) annotations in addition to the large amounts of image data, to enable effective training. Multiple instance learning exhibits a powerful tool for learning deep neural networks in a scenario without fully annotated data. These methods are particularly effective in this domain, due to the fact that labels for a complete whole slide image are often captured routinely, whereas labels for patches, regions or pixels are not. This potential already resulted in a considerable number of publications, with the majority published in the last three years. Besides the availability of data and a high motivation from the medical perspective, the availability of powerful graphics processing units exhibits an accelerator in this field. In this paper, we provide an overview of widely and effectively used concepts of used deep multiple instance learning approaches, recent advances and also critically discuss remaining challenges and future potential.

1. Motivation

Digital pathology combines clinically valuable but information-dense whole-slide images with limited annotation and constrained clinical workflows. This motivates automated analysis and MIL-focused review of recent technical developments and limitations.

  • Clinical motivation: Microscopic biopsy evaluation remains the clinical gold standard, while rising prevalence and fewer pathologists increase the importance of automated assistance.Digitization enables storage, transmission, and digital processing of slides.
  • Clinical motivation: Digital workflows do not substantially differ from analog workflows, and time pressure plus large information volumes can cause relevant findings to be missed.Routine examination typically moves from low-resolution screening to detailed inspection of selected regions.
  • Technical limitations: Gigapixel whole-slide images cannot be processed holistically by modern deep convolutional networks because GPU memory remains limiting.The first convolutional layer can exceed current GPU limits during training even when the image itself fits in GPU memory.
  • Motivation for MIL: MIL relaxes whole-slide processing and local-annotation requirements by using whole-slide labels instead of manually collected region labels.Fully annotated segmentation maps can require roughly an hour per image.
  • Recent literature growth: 260 PubMed results were identified, including 196 published between 2020 and 2023 and 137 since 2022.The search combined multiple instance learning with digital pathology and related terms.
  • Review scope: The review analyzes state-of-the-art MIL building blocks, recent technical advances, and practical limitations related to computational pathology and hardware.Its structure covers general architectures, pooling, feature extraction, further aspects, critical discussion, and conclusion.

2. Overview of Multiple Instance Learning in Digital Pathology

Digital pathology MIL separates patch processing from aggregation and collective slide-level processing, enabling whole-slide classification from weak labels. Deep learning extends this pipeline while preserving alternative instance- and embedding-based representations.

  • Deep-learning context: Deep learning integrates learned feature extraction with classification and can be separated into pretrained feature extraction plus efficient classifiers when training data are scarce.This separation is particularly relevant because digital pathology often has few available whole-slide images.
  • MIL pipeline: MIL extracts patches from a whole-slide image, processes them individually, aggregates their outputs, and produces a bag-level label.In this abstraction, the whole-slide image is the bag and the patches are its instances.
  • MIL variants: Instance-based MIL outputs a scalar for each patch, whereas embedding-based MIL outputs a feature vector for each patch.The distinction concerns the representation produced after patch-level processing.
  • MIL variants: Instance-based methods provide patch-level decisions that can generate whole-slide relevance maps, while embedding-based methods are typically more powerful for slide-level classification.The scalar restriction can limit instance-based model capacity.
  • Scope: The same abstract MIL pipeline applies to conventional and deep-learning models, with the review focusing on state-of-the-art deep architectures.This establishes the generic pipeline as a baseline for later architectural developments.

3. General Architectures

Deep MIL architectures process patches with convolutional networks, aggregate patch features into slide descriptors, and then produce bag-level predictions. Their design varies by aggregation strategy, feature representation, attention mechanisms, and memory constraints.

  • Input and feature extraction: A deep MIL network receives samples of size P × X × Y, where P is the number of patches and X and Y are patch dimensions.Although P is theoretically flexible, memory demands impose practical restrictions.
  • Input and feature extraction: Convolutional layers process each patch independently, after which patch outputs are flattened into per-patch representations.Two-dimensional filters are used because patch order in the third dimension is arbitrary.
  • Aggregation: A pooling function maps the P × F patch-feature matrix to an F-length vector representing the complete slide.Pooling converts features per patch into features per whole-slide image.
  • Bag-level prediction: Instance-based MIL uses the pooled output directly for the bag-level label, whereas embedding-based MIL applies additional fully connected layers.This implements the two representation choices described for deep MIL.
  • Training: Differentiable MIL pipelines can be trained end-to-end from patches paired with whole-slide labels, although feature extraction is often separated to reduce GPU memory and accelerate training.End-to-end optimization updates all model parameters together using backpropagation-based optimization.
  • Attention-based architectures: Transformers and attention mechanisms are used mainly for pooling or later patch-feature processing because whole-slide size makes direct patch feature extraction impractical.Transformer modules can encode inter-patch connections and self-attention interactions.

4. Pooling Formulations

MIL pooling aggregates patch-level feature vectors into a bag representation for whole-slide prediction. The review covers fixed pooling rules, trainable attention, distributional and transformer-based aggregation, plus sequence and graph alternatives.

  • 4.1. Basic Pooling Formulations: Max-pooling selects the largest value for each feature across all patches.
  • 4.1. Basic Pooling Formulations: Mean-pooling computes the arithmetic mean of each feature across patches.
  • 4.1. Basic Pooling Formulations: Log-sum-exp pooling continuously relaxes max-pooling through an adjustable hyperparameter r > 0.
  • 4.2. Attention Mechanism: Attention-based pooling learns patch weights ap, providing an importance measure for each patch’s contribution to the final decision.
  • 4.2. Attention Mechanism: Self-attention mechanisms model dependencies between instances by transforming feature vectors into spaces whose inner products measure pairwise similarity.
  • 4.2. Attention Mechanism: Distribution pooling estimates marginal feature distributions under a Gaussian assumption rather than single summary scores such as means or maxima.
  • 4.3. Pooling with Transformer Architectures: Transformer pooling encodes patch interactions with self-attention while applying multilayer perceptrons to individual feature vectors.
  • 4.4. Alternative Pooling Formulations: Structured state space models compress long patch sequences into memory units, while graph neural networks predict bag labels from feature-space connectivity.

5. Patch & Feature Extraction

Patch and feature extraction convert enormous whole-slide images into manageable representations for MIL. The review discusses sampling, memory constraints, selection, pretrained and adapted CNNs, autoencoders, contrastive learning, and multiscale fusion.

  • The pipeline begins with patch extraction and feature extraction before MIL aggregation.
  • Patches are sampled randomly or on grids, with overlapping regular-grid extraction supporting higher-resolution confidence maps.
  • Increasing patch count raises feature-map memory linearly, while changing patch size can require architectural changes.
  • Preprocessing can select tissue-containing patches, whereas implicit attention-based selection avoids a separate stage but increases end-to-end memory use.
  • CNN features may come from training from scratch, direct pretraining, or further optimization of pretrained weights on the target dataset.
  • Pretrained CNNs provide efficient generic representations when patch annotations, labeled data, or computing infrastructure are limited, but are not optimized for each application.
  • Autoencoder-based approaches compress patches into lower-dimensional representations, including VAE-GAN feature representations combined with feature selection for WSI classification.
  • Contrastive learning trains patch representations by maximizing agreement among patches from the same WSI before downstream MIL use.

6. Further Aspects

Further MIL aspects address the trade-off between bag-level classification and patch-level interpretability, along with patch selection, pseudo-labeling, survival prediction, domain adaptation, and confidence estimation.

  • Embedding-based MIL can be stronger for complete-WSI classification, but does not directly provide single-patch scores for confidence maps.
  • Dual-stream MIL jointly trains instance-based and embedding-based classifiers to combine patch identification with bag representation.
  • Other hybrid approaches iteratively couple instance and embedding pathways or combine them with transformer backbones.
  • Embedding-based confidence maps can be produced by classifying local bags on a raster grid and assigning each prediction to its region center.
  • Clustering selects a fixed number of similar-feature patches per WSI cluster to approximate relevant slide information.
  • Proxy labels assign positive and negative classes to the highest- and lowest-scoring patches while discarding intermediate patches from loss computation.
  • Attention-derived pseudo-labels can support joint optimization, while Siamese attention-based MIL can perform survival prediction using ranked pairs of WSIs.
  • Domain-adversarial training seeks stain-invariant features by treating each patient as an individual domain, and repeated dropout predictions yield bag-level confidence scores.

7. Discussion

The review identifies recurring technical constraints in MIL for digital pathology, including limited WSI data, hardware memory, patch-sampling coverage, and dataset-pretraining gaps. It also surveys alternative bag-label approaches and limits its scope primarily to technical methods rather than clinical applications.

  • Hardware and sampling: MIL reduces hardware and annotation demands but often processes only a randomly sampled subset of each WSI, potentially excluding tissue regions.The review notes that sampled inputs may represent about one percent of gigapixel WSI data, although white areas reduce the practical loss.
  • Hardware and sampling: Patch size and patch count impose a memory trade-off, but the literature lacks clear evaluations or justifications for the settings used.Increasing patch size decreases the number of patches when memory consumption remains stable, motivating further analysis.
  • Data limitations: Training data are often limited to tens or hundreds of WSIs, while each WSI yields many patches, creating a data challenge for end-to-end MIL training.Data augmentation is explicitly considered in relatively few works, although random sampling and feature-level MixUp have been proposed.
  • Data limitations: Stain normalization and related augmentation strategies address variability across real-world sites and may improve robustness in MIL classification.The review describes end-to-end image normalization and adversarial stain normalization as examples.
  • Transfer learning: The review found little use of aggregated medical datasets or pretrained models based on histological content, despite common generic pretraining on datasets such as ImageNet.The authors note that digital pathology differs from natural images and that cancer tissue can be similar across organs.
  • Alternatives to MIL: Count-based and bag-of-words alternatives use patch-level predictions or pretrained features followed by shallow supervised classifiers, and the latter worked well with few WSIs.The reviewed approaches aggregate patch labels or clustered features into histograms before classification, often with support vector machines.
  • Review scope: The review is non-exhaustive, emphasizes technically novel conference and journal contributions, and does not focus on application-oriented clinical studies.The authors refer readers to separate clinical studies for large-scale application assessment.
  • Related work: Recent related reviews cover weakly supervised MIL, deep MIL, general MIL, experimental comparisons, and high-level digital pathology surveys, but differ in technical depth or recency.The review positions itself as a technical synthesis of recent state-of-the-art methods and their limitations.

8. Conclusion

The review finds many recent MIL approaches for WSI classification, with attention and combined instance- and embedding-based learning emerging as trends but no clear dominant architecture. It highlights limited training data and GPU memory as continuing constraints and identifies augmentation and transfer learning as research needs.

  • Findings: Numerous MIL approaches for WSI classification were developed in recent years, but the review found no clear trend toward one architecture or methodology.The review instead identifies recurring design patterns and specific architectural modifications.
  • Findings: Attention mechanisms and combinations of instance-based and embedding-based learning were identified as emerging trends.The approaches are generally applicable across histological fields rather than being handcrafted for one domain.
  • Limitations: Insufficient training data and GPU memory remain the main limitations identified for MIL in digital pathology.Hardware improvements may allow more information from large histological images to be incorporated.
  • Future directions: The review identifies data augmentation and transfer learning as research needs for dealing more effectively with small datasets.These directions are presented as future opportunities within the review’s supported scope.
Loading 2206.04425v2…