PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiple Instance Learning Framework with Masked Hard Instance Mining for Whole Slide Image Classification

Wenhao Tang, Sheng Huang, Xiaoxian Zhang, Fengtao Zhou, Yi Zhang, Bo Liu

arXiv:2307.15254v3cs.CVcs.AI

TL;DR

Attention-based MIL tends to focus on easy salient instances, leaving hard instances underused despite their value for learning a discriminative boundary. MHIM-MIL uses attention-guided masking and a teacher–student consistency framework to train on hard instances. Across two WSI datasets, it reportedly outperforms recent methods in performance and training cost.

  • Problem

    Attention-based MIL prioritizes salient, easy-to-classify instances and neglects hard instances that can help define a discriminative boundary.

    Source: §1 · p. 2, §3.2 · p. 3

  • Method

    MHIM-MIL uses a momentum teacher to score instances, masks salient patches to retain hard instances for an attention-based student, and applies teacher–student consistency optimization.

    Source: §3.2 · p. 3, §3.4 · p. 5

  • Results

    Across two WSI datasets, MHIM boosts different MIL models and outperforms recent methods in performance and training cost.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Its hybrid masking strategies address reliance on instance-level supervision while enhancing training efficiency and mitigating over-fitting risk.

    Source: §1 · p. 2

    Without complete instance supervision, difficulty assessment and selection of the most helpful hard instances remain rough, affecting convergence speed and model discriminability.

    Source: F. Limitation · p. 15

Abstract

from arXiv · show

The whole slide image (WSI) classification is often formulated as a multiple instance learning (MIL) problem. Since the positive tissue is only a small fraction of the gigapixel WSI, existing MIL methods intuitively focus on identifying salient instances via attention mechanisms. However, this leads to a bias towards easy-to-classify instances while neglecting hard-to-classify instances. Some literature has revealed that hard examples are beneficial for modeling a discriminative boundary accurately. By applying such an idea at the instance level, we elaborate a novel MIL framework with masked hard instance mining (MHIM-MIL), which uses a Siamese structure (Teacher-Student) with a consistency constraint to explore the potential hard instances. With several instance masking strategies based on attention scores, MHIM-MIL employs a momentum teacher to implicitly mine hard instances for training the student model, which can be any attention-based MIL model. This counter-intuitive strategy essentially enables the student to learn a better discriminating boundary. Moreover, the student is used to update the teacher with an exponential moving average (EMA), which in turn identifies new hard instances for subsequent training iterations and stabilizes the optimization. Experimental results on the CAMELYON-16 and TCGA Lung Cancer datasets demonstrate that MHIM-MIL outperforms other latest methods in terms of performance and training cost. The code is available at: https://github.com/DearCaat/MHIM-MIL.

1. Introduction

MHIM-MIL addresses the tendency of attention-based MIL to prioritize easy salient instances by masking them to expose hard instances for training. Its teacher–student framework and hybrid masking strategies are reported to improve performance and training cost across WSI datasets.

1. Introduction

  • WSI classification is commonly framed as MIL, where a slide is a bag of thousands of unlabeled patches and is positive if at least one patch is disease-positive.Source: §1 · p. 1
  • Attention-based MIL often emphasizes salient, easy-to-classify instances, while hard instances can better depict the classification boundary and are difficult to mine without instance labels.Source: §1 · pp. 1–2
  • MHIM boosts different MIL models and outperforms recent methods on two WSI datasets in performance and training cost.The introduction reports this as the result of extensive experiments.Source: §1 · p. 2
  • MHIM-MIL masks high-attention instances and adds hybrid masking strategies that address reliance on instance supervision, improve training efficiency, and mitigate over-fitting.Source: §1 · p. 2
  • A Siamese teacher–student structure uses a momentum teacher updated by EMA and a consistency constraint to guide instance mining and progressively improve discriminability.The teacher obtains instance attention scores efficiently and stably.Source: §1 · p. 2

2. Related Work

The related work situates MHIM-MIL among embedding- and instance-level WSI methods that often prioritize salient instances, and among hard-sample mining approaches that typically rely on complete sample supervision.

2.1. Multiple Instance Learning in WSI Analysis

  • Prior WSI MIL spans instance-level and embedding-level methods, but salient-instance-focused approaches can overlook hard instances important for training.Embedding-level methods aggregate features into a bag embedding; the related work also describes instance-label aggregation and salient-instance selection.Source: §2.1 · p. 2

2.2. Hard Sample Mining in Computer Vision

  • Hard-sample mining selects difficult positives and negatives to address class imbalance and facilitate training, with loss-, similarity-, and learnable-weight-based approaches described.Source: §2.2 · pp. 2–3
  • Unlike directly locating hard negatives, MHIM-MIL indirectly mines hard instances by masking salient ones, without complete instance labels.Source: §2.2 · p. 3

3. Proposed Method

MHIM-MIL addresses attention-based MIL’s focus on salient, easy-to-classify instances by masking them to retain harder instances for training. A teacher-student framework and consistency-based optimization iteratively support instance mining and student learning.

3.1. Background: MIL Formulation

  • A WSI is modeled as a bag of patch instances whose bag label is predicted from an aggregated representation, using attention-based or multi-head self-attention aggregation.Source: §3.1 · p. 3

3.2. MHIM-MIL for WSI Classification

  • MHIM-MIL uses a momentum teacher to score instances and mask salient ones, preserving hard instances for a general attention-based MIL student.The teacher shares the student’s network structure and is updated without gradient-based updates.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3. Masked Hard Instance Mining Strategy

  • Hybrid mining combines high-attention masking with low-attention masking to filter redundant instances or random masking to reduce overfitting risk.LR-HAM unions the high-, low-, and random-attention mask flags.Source: §3.3 · p. 4

3.4. Consistency-based Iterative Optimization

  • Student optimization combines bag-label loss with consistency between student and teacher bag representations; EMA updates teacher parameters from the student for subsequent mining.The consistency loss constrains the models’ classification results, and the teacher is updated without gradient updates.Source: §3.4 · p. 5

4. Experiments and Results

MHIM-MIL improves WSI classification performance on CAMELYON-16 and TCGA while reducing computational costs. Ablations, teacher comparisons, and patch visualizations examine how masked hard instance mining contributes across models and settings.

4.1. Datasets and Evaluation Metrics

  • Evaluations use CAMELYON-16 and TCGA Lung Cancer, with cross-validation and Accuracy, AUC, and F1-score reported as appropriate.CAMELYON-16 uses three-times three-fold cross-validation; TCGA uses patient-level splits and four-fold cross-validation. Network and training details are in the Supplementary Material.Source: §4.1 · p. 5, §4.2 · p. 5

4.3. Performance Comparison with Exiting Works

  • Max- and mean-pooling perform poorly on both datasets, while DTFD-MIL is second-best on both but overemphasizes salient instances.The authors attribute the pooling results to insufficient modeling of key instance information.Source: §4.3 · pp. 5–6
  • MHIM-MIL reports +1.34% AUC on CAMELYON-16 and +1.70% AUC on TCGA, achieving significant improvement on both datasets by mining hard instances during training.Source: §4.3 · p. 6

4.4. Computational Cost Analysis

  • -24% training time and -48% memory usage accompany MHIM-MIL’s computational-cost reduction, while its C16 AUC standard deviation is 0.48%.The comparison reports computational requirements on a 3090 GPU; the cited stability figure is for C16.Source: §4.4 · p. 6, Table 2 · p. 6

4.5.1 Importance of the Different Components

  • Naive masked hard instance mining improves CAMELYON-16 AUC by 1.86% with AB-MIL and 2.55% with TransMIL; adding consistency loss yields the best reported AUCs.The full framework reaches 96.49% AUC on CAMELYON-16 and 94.97% on TCGA.Source: §4.5.1 · pp. 6–7, Table 3 · p. 7

4.5.2 Impact of the Different MHIM Strategies

  • High Attention Masking already improves CAMELYON-16 AUC, while hybrid masking strategies show different performance patterns across datasets and MIL models.AB-MIL gains with randomness, TransMIL shows the reverse pattern on C16, and LR-HAM performs best on TCGA.Source: §4.5.2 · p. 7, Table 4 · p. 7

4.5.3 Impact of the Choice of Teacher Network

  • The EMA momentum teacher is more stable than the non-batch-gradient-updated student, and proper initialization gives the momentum teacher the best performance.A fixed-initialization teacher fails to learn new knowledge, underscoring the role of iterative optimization.Source: Table 5 · p. 7, §4.5.3 · pp. 7–8

4.6. Visualization

  • Patch visualizations show MHIM-MIL generalizing better than AB-MIL on noise robustness and subtle tumor areas, with broader predictions supporting more complete judgments.The authors note that attention scores indicate regions of interest but do not reflect tumor probabilities.Source: §4.6 · p. 8, Figure 4 · p. 8

5. Conclusion

The paper argues that masking salient patches encourages hard-instance learning and improves discriminative learning in MIL-based WSI classification. MHIM-MIL combines these strategies with a momentum teacher and consistency loss; the authors identify more precise hard-instance localization as future work.

5. Conclusion

  • MHIM-MIL combines masked hard-instance mining with a momentum teacher and consistency loss to enhance hard-instance mining.The framework masks salient patches to encourage attention to informative regions.Source: §5 · p. 8
  • The authors report that MHIM-MIL is superior and generalizes across other latest methods.The supplied conclusion does not specify the metrics or datasets for this comparison.Source: §5 · pp. 8–9
  • The authors leave devising a more precise hard-instance localization scheme for future work.They state that such a scheme could facilitate model training and convergence.Source: §5 · p. 9

A. Additional Visualization

The visualizations compare attention, tumor predictions, and masked instances before and after Masked HIM training. The authors report that masked training improves use of key-region features, despite attention shifting toward less salient regions.

A. Additional Visualization

  • Masked HIM masks salient regions to indirectly mine hard instances, while random masking mitigates over-fitting.Figure 5 marks tumor regions with blue outlines, higher attention with brighter patches, and masked instances with dark patches.Source: A. Additional Visualization · p. 11, Figure 5 · p. 11
  • Masked HIM training improves gradually in discriminatory power, and models retain or improve discriminative power for key regions after salient instances are masked.Figure 6 visualizes patches during the iteration process.Source: A. Additional Visualization · p. 11
  • Masked HIM-trained models attend more to “irrelevant regions” yet use key-region features better to produce higher-quality bag features and improve performance.The authors contrast this with traditional MIL models that focus on individual features and damage model discriminativeness.Source: A. Additional Visualization · p. 11

B.1. More on Masked Hard Instance Mining

The supplementary analyses examine mask ratios, student initialization, transformer attention choices, and loss scaling. They report that these choices affect performance, training cost, and the quality of hard-instance mining in dataset- and model-dependent ways.

B.1. More on Masked Hard Instance Mining

  • Low high-attention mask ratios reduce mined-instance difficulty and overall performance, while training remains stable at high ratios with randomization.TransMIL requires a larger high-attention mask ratio than AB-MIL; performance is less sensitive to random- and low-attention ratios, though appropriate choices remain important.Source: §B.1 · pp. 11–12, Figure 7 · p. 12
  • Large-scale low-attention and random-attention masking reduce student input, memory use, and training time, while mixing strategies adds computation.The momentum teacher adds little extra training cost; additional computation from mixing strategies is more evident with AB-MIL.Source: §B.1 · p. 12, Table 7 · p. 13
  • Mask-ratio decay significantly boosts performance, while randomly high-attention masking performs well on CAMELYON-16 but impairs discriminability on TCGA.The authors attribute the TCGA result to low training difficulty where tumor area is typically over 40%; random masking preserves key information on CAMELYON-16.Source: §B.1 · pp. 12–13, Table 8 · p. 13, Table 9 · p. 13

B.2. Initialization of Student Network

  • Initializing the student’s first fully connected layer facilitates MHIM-MIL optimization but does not generally improve mainstream MIL models.Without initialization, slow early student training lowers teacher performance and harms iterative optimization; the comparison is described in Table 10.Source: §B.2 · p. 13, Table 10 · p. 13

B.3. Transformer Attention

  • TransMIL’s first-layer attention scores outperform its final-layer scores for identifying hard instances.The authors attribute this to the first layer using original instance features, before self-attention modifies them.Source: §B.3 · p. 14, Table 11 · p. 14
  • Voting uses a majority rule to suppress nondiscriminative attention heads, avoiding the localization dilution that averaging can cause.Some TransMIL heads produce identical attention scores; the lower part of Table 11 is reported to show voting’s effectiveness.Source: §B.3 · p. 14, Figure 11 · p. 14, Figure 10 · p. 14

B.4. Discussion on Hyperparameter

  • Loss scaling factor α affects both models consistently, and values that are too high or too low bias training.A high α weakens the positive effect of slide labels, especially on CAMELYON-16, where challenging slides require slide-label supervision.Source: §B.4 · pp. 14–15, Figure 12 · p. 15

C. Data Pre-processing

The preprocessing crops each WSI into non-overlapping patches at 20X magnification and removes background regions. The processed datasets contain millions of patches, averaging roughly 9,000 to 10,300 per bag.

C. Data Pre-processing

D. Implementation Details

The implementation uses pretrained ResNet-50 patch features and specifies training settings; the following subsection presents MHIM-MIL training pseudocode.

D. Implementation Details

  • ResNet-50 pretrained on ImageNet extracts 1024-dimensional patch features, which are reduced to 512 dimensions by a fully connected layer.The final convolutional module is removed and global average pooling is applied.Source: D. Implementation Details · p. 15
  • Training uses an EMA momentum rate of 0.9999, consistency-loss temperature of 0.1, Adam learning rate of 2 × 10−4, and weight decay of 1 × 10−5.Models train for up to 200 epochs with early stopping, batch size 1, and an NVIDIA RTX3090 GPU.Source: D. Implementation Details · p. 15

E. Pseudocode

F. Limitation

MHIM-MIL mines hard instances without instance-level supervision, but its rough strategy has limitations in estimating instance difficulty and selecting useful examples.

F. Limitation

  • Without instance supervision, MHIM-MIL cannot accurately assess instance difficulty or reliably identify the most helpful hard instances.Source: F. Limitation · p. 15
  • The authors characterize their approach as sub-optimal and rough compared with supervised hard-sample mining strategies.Source: F. Limitation · p. 15
  • The authors identify convergence speed and model discriminability as affected by this rough mining strategy.Future work will focus on more accurate difficulty assessment and selecting beneficial instances without complete supervision.Source: F. Limitation · p. 15

G. Code and Data Availability

This section lists code and dataset resources and provides training pseudocode, alongside additional patch-visualization comparisons between AB-MIL and MHIM-MIL.

G. Code and Data Availability

Related papers

Loading 2307.15254v3…