PaperlayerView exact revision on arXiv

Source-linked AI summary

Additive MIL: Intrinsically Interpretable Multiple Instance Learning for Pathology

Syed Ashar Javed, Dinkar Juyal, Harshith Padigela, Amaro Taylor-Weiner, Limin Yu, Aaditya Prakash

arXiv:2206.01794v2cs.CVcs.LG

TL;DR

Pathology models need to be inspected to identify failures and maintain trust, but attention-based patch scores do not directly express each patch’s contribution to a prediction. Additive MIL makes those contributions exactly computable and visualizable, with reported alignment to expert regions and predictive performance maintained.

  • Problem

    Pathology MIL models need interpretability for debugging, validation, and physician trust, while patch contributions to predictions are not directly specified by attention scores.

    Source: §1 · p. 2

  • Method

    Additive MIL constrains the final predictor to be additive over instances, allowing exact patch contributions to be visualized as heatmaps.

    Source: §1 · pp. 2–3

  • Results

    Additive MIL retains predictive performance without loss and its heatmaps align better with expert pathologist region annotations than Attention MIL heatmaps.

    Source: §3.1 · p. 5

  • Takeaways & Limitations

    The formulation can make existing pooling-based MIL models additive and supports patch-level interpretation for model debugging and decision-support settings.

    Source: §1 · p. 3, §4 · p. 10

    Heatmaps are limited to the patch resolution used in training, and the additive constraint may reduce model expressivity, though no practical example was found in this study.

    Source: §3.3.6 · p. 9

Abstract

from arXiv · show

Multiple Instance Learning (MIL) has been widely applied in pathology towards solving critical problems such as automating cancer diagnosis and grading, predicting patient prognosis, and therapy response. Deploying these models in a clinical setting requires careful inspection of these black boxes during development and deployment to identify failures and maintain physician trust. In this work, we propose a simple formulation of MIL models, which enables interpretability while maintaining similar predictive performance. Our Additive MIL models enable spatial credit assignment such that the contribution of each region in the image can be exactly computed and visualized. We show that our spatial credit assignment coincides with regions used by pathologists during diagnosis and improves upon classical attention heatmaps from attention MIL models. We show that any existing MIL model can be made additive with a simple change in function composition. We also show how these models can debug model failures, identify spurious features, and highlight class-wise regions of interest, enabling their use in high-stakes environments such as clinical decision-making.

1 Introduction

Pathology’s large, information-dense whole-slide images make manual analysis challenging, while slide-level prediction is important for clinical applications. Additive MIL is proposed to provide exact spatial credit assignment without sacrificing predictive performance.

1 Introduction

  • Whole-slide images contain billions of pixels and thousands of cells, making manual analysis challenging despite their value for diagnosis and disease assessment.Histologic examination is regarded as medicine’s ground truth for various diseases and supports severity assessment, treatment-effect measurement, and biomarker scoring.Source: §1 · p. 1
  • Existing pathology approaches use hand-engineered tissue representations or patch-level training followed by aggregation; these are not end-to-end and may perform sub-optimally.The patch-based approach also assumes every patch shares the slide label, which the authors describe as incorrect.Source: §1 · p. 2
  • MIL learns a slide-level label from a bag of patches without patch-level labels, and attention pooling has become a successful, efficient end-to-end approach.Attention heatmaps offer spatial interpretability, but their scores relate nonlinearly to the final prediction, making visual interpretation inexact and incomplete.Source: §1 · p. 2
  • Interpretable pathology models are needed to help identify spurious shortcuts, support debugging and validation, build trust, and highlight regions for human review.A predicted score alone is insufficient for decision-support use cases that require a visual cue.Source: §1 · p. 2
  • Additive MIL constrains the final predictor to be additive over instances, enabling exact patch-level credit assignment and heatmaps.The authors report similar predictive performance and say existing MIL models can be made additive by changing function composition at the final layer.Source: §1 · pp. 2–3

2 MIL Models & Interpretability

Attention weights do not directly specify each patch’s contribution to a prediction. Additive MIL makes patch contributions traceable, and its scores are proportional to Shapley values, providing exact instance-level decomposition and class-wise, signed heatmaps.

2.1 Interpretability in Attention MIL

  • Attention scores can under- or overestimate patch contributions because they do not capture classification-stage interactions between patches.A high score may indicate a patch is needed downstream, but is not sufficient to attribute the prediction to it; attention also cannot distinguish positive from negative evidence or identify class-specific contributions.Source: §2.1 · pp. 3–4

2.2 Additive MIL Models

  • Additive MIL reframes the final predictor as additive over instances, yielding class-wise patch scores whose sum gives the final logits.Positive contributions raise a class logit and negative contributions lower it; patch representations can remain arbitrarily complex.Source: §2.2 · pp. 4–5

2.3 Proof of equivalence between Additive MIL and Shapley Values

  • Theorem 1 states that an Additive MIL instance contribution is proportional to its Shapley value, enabling exact decomposition of bag predictions into marginal instance contributions.The authors characterize this as optimal credit assignment across instances of an MIL bag.Source: §2.3 · p. 5

2.4 Features of Additive MIL Models

  • Additive MIL scores provide exact patch contributions that track model sensitivities and invariances, with separate class-wise and excitatory or inhibitory contributions.Class-wise scores help distinguish signals for multiple classes in one slide; signed scores distinguish evidence for a class from evidence against it.Source: §2.4 · p. 5

3 Experiments & Results

Across pathology experiments, Additive MIL is reported to retain predictive performance while providing spatial interpretability. Its heatmaps align better with expert region annotations and supply class-wise and signed patch information absent from attention heatmaps.

3 Experiments & Results

  • Additive MIL heatmaps align better with expert pathologist region annotations than Attention MIL heatmaps.The experiments also report intrinsic spatial interpretability without loss of predictive performance and similar predictive results after making pooling-based MIL models additive.Source: §3.1 · pp. 5–6
  • Additive MIL heatmaps provide class-wise spatial assignment and distinguish excitatory from inhibitory patches, information missing from attention heatmaps.The authors note that this additional information can be useful for model debugging.Source: §3.1 · p. 6
  • Experiments cover cancer-subtype prediction in TCGA RCC and NSCLC and breast-cancer metastasis detection in Camelyon16.The datasets comprise 966 RCC slides, 1,002 NSCLC slides, and 267 Camelyon16 training plus 129 test slides.Source: §3.1 · p. 6

3.2 Implementation Details

Implementation used dataset-specific splits, patch and bag sampling, a pretrained feature extractor, and slide-level aggregation for evaluation.

3.2 Implementation Details

  • TCGA datasets used 60/15/25 train/validation/test splits with case-level leakage prevention, while Camelyon16 used its original splits.Bag sizes of 48–1600 patches and batch sizes of 16–64 were tested, with the best selected by cross-validation.Source: §3.2 · p. 6
  • Patches came from non-background regions at 1 micron per pixel without overlap, and an ImageNet-pretrained Shufflenet served as feature extractor.Models were trained with ADAM at a learning rate of 1e-4.Source: §3.2 · p. 6
  • Multiple bag predictions were aggregated by majority vote for slide-level prediction, with AUROC computed from majority-label proportions.TCGA-RCC used macro-averaged one-versus-rest AUROC across three classes; additive patch contributions were sigmoid-bounded to form excitatory and inhibitory heatmap scores.Source: §3.2 · p. 6

3.3 Experimental Results

Additive MIL performs comparably or better than standard Attention MIL while providing patch-level interpretability. Its heatmaps align more closely with annotated cancer regions, show class-specific contributions, and help inspect model failures, with limitations in patch resolution and expressivity.

3.3.1 Predictive Performance of Additive MIL Models & Applicability to Previous Methods

  • Additive MIL achieves comparable or superior predictive performance to standard Attention MIL across three datasets, and changing classifier composition can make previous MIL methods additive.The comparison covers Camelyon16, TCGA NSCLC, and RCC.Source: §3.3.1 · p. 6, Table 1 · p. 7

3.3.2 Region-level Alignment of Additive Heatmaps with Expert Pathologist

  • 0.42 AUPRC for Additive heatmaps versus 0.36 for Attention, and 0.47 best F1 versus 0.43, on Camelyon16 cancer-region localization.The heatmaps were compared against expert annotations, with both generated from the same model.Source: §3.3.2 · p. 7, Figure 2 · p. 7

3.3.3 Faithful Representation of Patch-level Contributions to Slide-level Predictions

  • Additive patch contributions vary linearly with slide-level predictions, whereas Attention scores for the top 10% of patches show no relationship with final predictions.Figure 3 contrasts the sum of Additive patch contributions with the median Attention score of the top 10% of patches.Source: §3.3.3 · pp. 7–8, Figure 3 · p. 8

3.3.4 Qualitative Assessment of Multi-Class & Excitatory-Inhibitory Heatmaps

  • Class-wise Additive heatmaps show regions voting for or against cancer subtypes; in RCC, patches were excitatory for KIRC or KIRP and inhibitory for other classes, which Attention heatmaps cannot show.Red denotes excitatory and blue inhibitory contributions.Source: §3.3.4 · pp. 8–9, Figure 4 · p. 8, Figure 5 · p. 9

3.3.5 Model Debugging Using Additive MIL Heatmaps

  • Additive heatmaps localized sources of two errors: adrenal glands confused for KICH in a KIRP slide and germinal centers behind a benign-to-malignant false positive.The authors report finding the germinal-center pattern in multiple other slides, supporting model debugging.Source: Figure 6 · p. 9, §3.3.5 · p. 9

3.3.6 Limitations

  • Interpretability is coupled to the model, restricting heatmaps to the trained patch resolution; the additive constraint may reduce expressivity, though no practical example appeared in this study.Source: §3.3.6 · p. 9

4 Conclusion & Broader Impact

Additive MIL reformulates Attention MIL with an additive function, enabling exact attribution of slide predictions to individual patches while preserving predictive performance.

  • Additive MIL reformulates Attention MIL through an additive function to make models intrinsically interpretable.Source: §4 · p. 10
  • The model exactly attributes each pathology-slide prediction to individual patch contributions.Source: §4 · p. 10
  • The resulting spatial interpretability is achieved without loss of predictive performance and supports model debugging and highlighting regions of interest.Source: §4 · p. 10

A Appendix

The appendix states that Additive MIL instance contributions are proportional to Shapley values and describes the proof linking them. It also notes supplementary heatmaps and an expert evaluation.

  • Additive MIL’s marginal instance contribution g(x_i) is proportional to the instance’s Shapley value φ_i.Source: A Appendix · p. 14
  • The proof expresses the additive model over instances in S and its complement, then relates the value function with instance i included to the Shapley equivalence.The appendix states that the second term is the expected model output before establishing the equivalence.Source: A Appendix · p. 14
  • Supplementary material provides TCGA and Camelyon16 heatmap comparisons and reports an expert evaluation of heatmap utility for a clinical-grade tool.Source: A Appendix · p. 15

B Heatmaps from TCGA-RCC Dataset

The RCC examples show Additive MIL heatmaps distinguishing class-specific regions and revealing patches that attention heatmaps miss.

C Heatmaps from Camelyon16 Dataset

Camelyon examples show Additive MIL indicating contribution direction and localizing a false positive and additional cancer foci that attention maps miss.

D Heatmaps from TCGA-NSCLC Dataset

Additive MIL heatmaps distinguish class-predictive regions and localized uncertainty in three NSCLC subtype cases, while attention heatmaps can be ambiguous about patch contributions.

D Heatmaps from TCGA-NSCLC Dataset

  • In case (a), Additive MIL identifies regions predictive of both subtypes despite a correct Squamous Cell Carcinoma prediction, while attention does not reveal this information.Attention assigns high importance to regions from both subtypes.Source: D Heatmaps from TCGA-NSCLC Dataset · p. 15
  • In case (b), Additive MIL localizes uncertainty between the two classes to a specific region despite a correct Adenocarcinoma prediction.Attention's most-attended patches correspond only to Adenocarcinoma, despite regions predictive of both subtypes.Source: D Heatmaps from TCGA-NSCLC Dataset · p. 15
  • In case (c), Additive MIL delineates regions predictive of each subtype, whereas attention highlights both without distinguishing them.Source: D Heatmaps from TCGA-NSCLC Dataset · p. 15

E Expert Evaluation of Additive Heatmaps & Applicability in Decision Support Tool

A board-certified pathologist evaluated Additive MIL and attention heatmaps for highlighting regions of interest in TCGA-RCC and Camelyon16. Additive MIL heatmaps were almost always preferred, with feedback citing class-specific cues and fewer false positives and false negatives.

E Expert Evaluation of Additive Heatmaps & Applicability in Decision Support Tool

  • A board-certified pathologist compared heatmaps for regions of interest using random slide samples and a preference question framed around AI-plus-human decision support.The evaluation included 50 Camelyon16 test slides sampled at a 1:4 benign-to-malignant ratio and 39 TCGA-RCC slides equally representing all three classes.Source: E Expert Evaluation of Additive… · pp. 15–16
  • Table 2 reports that Additive MIL heatmaps were almost always preferred over attention heatmaps for TCGA-RCC and Camelyon16.For TCGA-RCC, the pathologist cited class-specific patches as reminders to consider differential diagnoses; for Camelyon16, feedback cited more false positives and false negatives in attention heatmaps except one slide.Source: Table 2 · p. 16, E Expert Evaluation of Additive… · p. 16
  • Figure 7 shows Additive MIL heatmaps highlighting different regions from attention heatmaps, with greater interpretive granularity.Source: Figure 7 · p. 17
  • Figure 8 notes that attention heatmaps neither distinguish excitatory from inhibitory patch contributions nor often highlight false-positive patches relevant to debugging.Source: Figure 8 · p. 18
  • Figure 9 notes that attention heatmaps lack class-dependent patch attribution and can differ from Additive MIL in patch contribution values.Source: Figure 9 · p. 19

Related papers

Loading 2206.01794v2…