PaperlayerView exact revision on arXiv

Source-linked AI summary

CAR-MIL: Counterfactual Attention Regularization for Multiple Instance Learning

Imane Chraki, Pierre Marza, Stergios Christodoulidis, Maria Vakalopoulou

arXiv:2609.08419v1cs.CV

TL;DR

Attention in MIL may fail to reflect instance importance or may emphasize spuriously correlated regions, while offering limited insight into evidence against a prediction. CAR-MIL trains a lightweight counterfactual attention branch alongside the factual branch, encouraging alternative predictions through close, structured attention changes. It maintains competitive classification across the evaluated tasks while improving attention reliability, with larger performance gains on more challenging tasks.

  • Problem

    Attention in MIL may highlight irrelevant or spuriously correlated regions and provide little insight into evidence that contradicts a prediction.

    Source: §1 · pp. 1–2

  • Method

    CAR-MIL trains a lightweight counterfactual attention branch to produce a different prediction while remaining close to factual attention, yielding complementary evidence maps.

    Source: §1 · pp. 2–3, §3.2 · p. 5

  • Results

    Across five histopathology tasks, CAR-MIL performs comparably to or better than strong MIL baselines, with larger gains on more challenging tasks and improved attention alignment with instance-level evidence.

    Source: §4.2 · p. 10, §6 · p. 13

  • Takeaways & Limitations

    Factual attention highlights regions supporting a prediction, while counterfactual attention reveals evidence whose reweighting would challenge it.

    Source: §1 · p. 3

Abstract

from arXiv · show

Multiple Instance Learning (MIL) is widely used for weakly supervised learning, particularly in digital pathology, where fine-grained annotations are costly. Most MIL methods aggregate instance features via attention mechanisms. However, attention weights do not always faithfully reflect instance importance and may focus on spuriously correlated regions. In this work, we propose CAR-MIL, a framework that explicitly guides attention learning through a counterfactual attention regularization objective inspired by counterfactual explanations. Built on a standard attention-based MIL architecture, our approach introduces a lightweight counterfactual attention branch trained to produce an alternative prediction while remaining close to the factual attention distribution. This encourages prediction changes to arise from minimal, structured redistributions of attention, leading to more informative evidence allocation. The resulting factual and counterfactual attention maps capture complementary evidence: the former highlights regions supporting the prediction, while the latter reveals regions whose reweighting would challenge it. We evaluate our method on synthetic MIL benchmarks with instance-level ground truth enabling controlled analysis of attention behavior and on five digital pathology datasets across four tasks. CAR-MIL maintains competitive classification performance, with the largest gains observed on more challenging tasks, while improving attention reliability, demonstrating the benefits of integrating counterfactual explainability reasoning into attention learning. Code is available at: https://github.com/ImaneCR/CAR-MIL/.

1 Introduction

Attention-based MIL can achieve strong prediction performance, yet its attention may highlight irrelevant regions and offer little evidence that challenges a prediction. CAR-MIL applies counterfactual reasoning during training to learn complementary factual and counterfactual attention, maintaining competitive classification while improving attention reliability.

1 Introduction

  • Attention may highlight irrelevant or spuriously correlated regions and provide little insight into evidence contradicting a decision, limiting reliability and explanatory capacity.This is especially problematic in medical applications where attention maps are used to justify automated decisions.Source: §1 · pp. 1–2
  • CAR-MIL adapts counterfactual explanations to train attention directly, perturbing attention rather than inputs and learning perturbations instead of relying on random ones.The authors distinguish this training-time approach from post-hoc counterfactual explanations and random attention perturbations.Source: §1 · p. 2
  • A lightweight counterfactual branch stays close to factual attention while changing the prediction, producing complementary maps of supporting and refuting evidence.The branch shares the encoder and classifier, requiring no additional supervision and adding minimal computational overhead.Source: §1 · p. 3
  • CAR-MIL maintains or improves predictive accuracy and interpretability, with the largest performance gains on more challenging tasks.Evaluations cover synthetic MIL benchmarks and five digital pathology datasets across four tasks.Source: §1 · p. 3

2 Related Work

Prior MIL work has modified attention mechanisms and developed post-hoc explanations, but these approaches leave gaps in prediction-aware attention guidance and counterfactual evidence. CAR-MIL places learned counterfactual reasoning within attention training to structure evidence redistribution.

Attention-based embedding-level MIL

  • Attention-based embedding-level MIL aggregates instance embeddings into a global bag representation through attention; standard ABMIL infers weights from instance features to perform that aggregation.Variants incorporate self-attention, enforce consistency between attention weights and instance-feature representations, constrain attention through negative-bag supervision for binary classification, or introduce masking modules.Source: §2 · p. 3

Interpretability of MIL Models

  • Changing attention without evaluating its effect on predictions may not achieve the intended predictive outcome.Raw attention maps also do not guarantee reliable localization.Source: §2 · p. 3

Counterfactual Reasoning for Explanation and Attention Learning

  • Post-hoc explanations often reveal supporting evidence but not evidence challenging a decision, while counterfactual training approaches lack control over perturbations.CAR-MIL instead learns perturbations that structure how evidence is redistributed during training.Source: §2 · p. 4

3 Methods

The methods define attention-based MIL over bags with bag-level labels, then add a factual-counterfactual attention architecture sharing instance embeddings and a classifier. Its training objective combines factual classification, label-consistent differences between branch predictions, and proximity between attention logits.

3.1 Preliminaries

  • MIL represents each sample as a bag of instances with a bag-level label, while instance labels remain unobserved.An attention module assigns logits to instances, normalizes them into weights, and aggregates embeddings for bag-level classification.Source: §3.1 · pp. 4–5

3.2 Counterfactual Attention Regularization

  • CAR-MIL adds a counterfactual attention branch with its own attention weights while sharing the encoder and downstream classifier with the factual branch.The additional branch is lightweight, and prediction differences arise from differences in attention weighting.Source: §3.2 · p. 5
  • The evidence differential subtracts counterfactual logits from factual logits, and its class-specific values indicate relative class evidence between branches.A larger value for class y means factual attention provides stronger class-y evidence than counterfactual attention.Source: §3.2 · p. 5

3.3 Training Objective

  • The proximity loss can use normalized L1 distance or cosine dissimilarity to shape how counterfactual attention differs from factual attention.L1 penalizes absolute deviations and can encourage changes to a small subset of influential instances; cosine captures directional disagreement and is scale-invariant.Source: §3.3 · pp. 6–7
  • The joint objective combines factual-branch classification, a label-consistent evidence differential, and attention-logit proximity to encourage prediction shifts from small, structured changes.The objective is L = L_cls + α L_diff + λ L_div, with α and λ controlling the loss contributions.Source: §3.3 · pp. 6–7

4 Experiments and Results

The experiments test CAR-MIL on synthetic MIL tasks with instance-level evidence labels and on five histopathology datasets. They report strong instance-evidence alignment on synthetic data and competitive or better slide-level performance across histopathology tasks.

4.1 Experiments on MNIST-bags Synthetic Data

  • The synthetic benchmarks comprise Four Bags, where labels depend on the presence of 8s and 9s, and Adjacent Pairs, where evidence depends on consecutive digits from 0 to 4.The study uses ResNet18 features, repeats experiments 10 times, and evaluates bag AUC plus instance-level AUPRC metrics.Source: §4.1 · pp. 7–8
  • On Adjacent Pairs, CAR-MIL reaches 94.1 vs. 94.6 AUC for TransMIL and 85.7 AUPRC+ and 84.6 AUPRC±; it also improves bag performance and attention reliability on Four Bags.The synthetic tasks provide instance-level ground truth for evaluating whether attention aligns with supporting or refuting evidence.Source: Table 1 · p. 8, §4.1 · p. 9

4.2 Experiments on Histopathology Data

  • Histopathology evaluation covers five public WSI datasets: BRCA and NSCLC subtyping, LUAD TP53 mutation prediction, BRACS tissue classification, and CAMELYON16 metastasis detection.The experiments use UNI-V1 features, compare against non-attention and attention-based MIL methods, and report AUC or balanced accuracy and F1; CAMELYON16 also uses AUPRC+.Source: §4.2 · p. 9
  • CAR-MIL matches or improves strong MIL baselines across five histopathology tasks, with larger gains on the more challenging LUAD and BRACS tasks.On CAMELYON16, it also maintains competitive patch-level AUPRC+ for tumor annotations.Source: §4.2 · p. 10, Table 2 · p. 10

5 Analysis of Counterfactual Attention Regularization

The analysis examines sensitivity to regularization weights, compatibility with different MIL architectures, and performance with another feature encoder. It also analyzes how counterfactual predictions redistribute across classes in synthetic and histopathology settings.

Hyperparameter Sensitivity

  • 95.3–96.0 AUC is reported on BRCA versus the 94.5 baseline, and 84.2–84.4 AUC on LUAD versus the 81.3 baseline across configurations.Moderate counterfactual regularization gives the best BRCA performance; weight choices have more visible effects on LUAD, and cosine trends are smoother than L1 trends.Source: §5 · pp. 10–11, Figure 3 · p. 12

MIL Setting

  • CAR generally improves or maintains performance when integrated into ABMIL, DSMIL, and TransMIL, with gains especially visible for ABMIL and on LUAD and BRACS.The compared architecture variants use the best-performing L1 or cosine distance for each setting.Source: §5 · p. 11, Table 3 · p. 11

Feature Extractor

  • With ResNet50 features, CAR-MIL consistently improves AUC and F1 over ABMIL across three TCGA datasets, although absolute performance remains below that with UNI features.The reported gains are more pronounced with out-of-domain ResNet50 features than with UNI features.Source: §5 · p. 12

Multi-class Setting

  • On Four Bags, the counterfactual logit shifts from class 1 toward class 3 in 100% of bags and from class 3 toward class 1 in 62%; on BRACS, increases are diffuse across seven classes.The BRACS pattern is described as consistent with higher semantic ambiguity.Source: §5 · p. 12

6 Attention Analysis

On synthetic MIL tasks, CAR-MIL attention aligns better with instance-level evidence than ABMIL and perturbation-based attribution. Perturbation tests and BRACS heatmaps further assess attention faithfulness and show complementary factual and counterfactual patterns.

6 Attention Analysis

  • CAR-MIL achieves the highest AUPRC+ and AUPRC± scores on both synthetic tasks, improving alignment between attention and instance-level evidence over ABMIL and perturbation-based attribution.The evaluation uses synthetic MIL datasets with ground-truth evidence labels.Source: §6 · p. 13, Table 5 · p. 13
  • Removing instances in decreasing attention order produces a smooth, monotonic confidence drop for CAR-MIL, whereas ABMIL shows non-monotonic behavior.CAR-MIL's removal ranking uses the discrepancy between factual and counterfactual attention scores.Source: §6 · p. 13, Figure 4 · p. 13
  • On a representative BRACS DCIS slide, CAR-MIL factual attention is more localized than ABMIL, while counterfactual attention highlights benign or non-malignant areas.The factual and counterfactual branches provide complementary supporting and refuting evidence.Source: Figure 5 · p. 14, §6 · p. 14
  • Across BRACS slides, factual attention is moderately correlated with ABMIL (L1: r=0.69±0.05; Cosine: r=0.69±0.03), while factual and counterfactual attention are anti-correlated under L1 (r=−0.67±0.02).On the representative example, factual attention correlates with ABMIL at Spearman r=0.67, and factual-counterfactual attention at r=0.29.Source: §6 · p. 15

7 Conclusion

CAR-MIL uses learned counterfactual attention perturbations in a double-branch architecture to guide attention-based MIL. Experiments on synthetic datasets and whole-slide image benchmarks show maintained or improved downstream performance alongside better alignment with instance-level evidence.

7 Conclusion

  • Across synthetic datasets and multiple whole-slide image benchmarks, CAR-MIL maintains or improves downstream performance while producing attention maps that better align with instance-level evidence.The paper presents counterfactual explainability reasoning within attention optimization as a promising direction for more accurate and interpretable MIL models.Source: §7 · p. 15
  • CAR-MIL guides attention through learned counterfactual perturbations that induce prediction changes while remaining close to the main attention distribution.Its double-branch architecture produces factual and counterfactual attention.Source: §7 · p. 15
  • Future work could use learned counterfactual attentions as auxiliary supervision or for weak supervision, active learning, model debugging, and analysis of model behavior.Source: §7 · p. 15

A.1 Background on Counterfactual Explanations

Counterfactual explanations characterize a prediction by identifying a minimal input change that alters the output. This work adapts that reasoning to attention over instances in MIL bags.

A.1 Background on Counterfactual Explanations

  • A counterfactual explanation describes the smallest input change that shifts a prediction to a predefined output.The account draws on counterfactual explanations as a framework for machine-learning interpretability.Source: §A.1 · p. 19
  • The Wachter-style cost seeks a counterfactual prediction close to a desired target while keeping the counterfactual close to the original input.The formulation combines a loss for deviation from the desired output with a distance between original and counterfactual inputs.Source: §A.1 · p. 19
  • Rather than searching in input space, such as image pixels, CAR-MIL targets the attention module.Source: §A.1 · p. 19

A.2 Comparison with Weakly Supervised Knowledge Distillation

WENO uses attention scores as soft instance-level pseudo-labels to train an instance classifier, whereas CAR-MIL contrasts evidence for and against bag-level predictions without assigning patch labels. Their comparison spans a binary metastasis task and LUAD mutation prediction.

A.2 Comparison with Weakly Supervised Knowledge Distillation

  • WENO interprets bag-classifier attention scores as soft pseudo-labels for an instance classifier, which performs hard positive-instance mining to refine the bag classifier.Source: §A.2 · p. 19
  • WENO's patch-level pseudo-label assumption fits binary tasks with localized tumor regions, but may introduce noisy supervision for mutation tasks with spatially diffuse signals.Source: §A.2 · pp. 19–20
  • Unlike WENO, CAR-MIL evaluates how instances support or refute bag-level predictions through counterfactual attention contrast, without assigning explicit instance class labels.Source: §A.2 · pp. 19–20
  • WENO performs strongly on binary CAMELYON16 metastasis detection but substantially lower on LUAD mutation prediction, while CAR-MIL is described as more stable across tasks.The comparison reports AUC and F1 for LUAD, and AUC with AUPRC+ for CAMELYON16; the supplied table caption identifies representative datasets but gives no cell values.Source: §A.2 · p. 20, Table A.1 · p. 20

A.3 Visualisations and Attention Study

The attention study compares factual and counterfactual maps across L1 and cosine distance settings on a BRACS slide with annotated DCIS. Factual attention resembles ABMIL, while counterfactual attention shows complementary behavior; cosine produces a more concentrated factual focus around DCIS.

Attention Distance Metric Choice

  • Under both L1 and cosine, factual attention resembles ABMIL while counterfactual attention exhibits inverse, complementary behavior.The maps are compared on a representative BRACS slide with annotated DCIS regions.Source: §A.3 · p. 20, Figure A.1 · p. 21
  • Cosine yields a more concentrated factual focus around DCIS, while L1 attention spreads more smoothly into surrounding tissue.Counterfactual attention shows the opposite trend under each metric.Source: §A.3 · pp. 20–21, Figure A.1 · p. 21

Baselines Comparison

  • The baseline comparison displays CAR-MIL factual attention alongside ABMIL, CLAM, AddMIL, and DSMIL maps for a DCIS-labelled BRACS slide.The figure includes zooms around annotated DCIS and a benign area; the supplied text does not state which baseline performs best.Source: §A.3 · p. 21

A.4 Theoretical Analysis

This section analyzes how attention-logit perturbations and constrained redistributions shape counterfactual evidence, and illustrates attention allocation on a BRACS slide labeled DCIS.

A.4.1 Local Analysis of the Evidence Margin

  • For small perturbations, the pairwise evidence-margin change is approximately the negative dot product of the margin-sensitivity vector and the perturbation; alignment with its negative increases the margin.The first-order analysis assumes class logits are differentiable with respect to attention logits.Source: §A.4.1 · pp. 22–23

A.4.2 Attention Sensitivity under Linear Classifiers

  • With a linear classifier, an instance’s influence on a class logit depends on its attention weight and how its instance-level evidence deviates from the attention-weighted bag average.The derivative is a_j(s_c,j − µ_c), where µ_c is the attention-weighted average class alignment.Source: §A.4.2 · p. 23

A.4.3 Margin Sensitivity

  • An instance most affects the evidence margin when its relative support for the ground-truth class over a competitor differs strongly from the bag-average evidence difference.The magnitude of this unusual discriminativeness determines the instance’s influence on the margin.Source: §A.4.3 · p. 24

A.4.4 Optimal Perturbations under L1 Distance

  • Under an L1 constraint, margin-maximizing perturbations target instances with maximal absolute margin sensitivity, motivating small, structured attention redistributions in CAR-MIL.Instances supporting the ground-truth class receive negative perturbations, while instances with negative margin sensitivity receive positive perturbations; the analysis also motivates the cosine-based proximity used in the main paper.Source: §A.4.4 · p. 24

A.5.1 WSI Datasets

  • The WSI dataset overview reports each cohort’s data source, task, slide count, and label distribution; synthetic Four Bags and Adjacent Pairs settings assign instance-level evidence for controlled experiments.Four Bags uses digits 8 and 9 as opposite evidence for classes, while Adjacent Pairs makes evidence depend on specific co-occurring digits.Source: §A.5.1 · p. 25, §A.5.2 · p. 25, Table A.2 · p. 26

A.6.1 Evaluation Metrics: AUPRC+ and AUPRC±

  • For CAMELYON16 metastasis detection, tumor-patch labels derived from pixel annotations serve as positive-evidence proxies, and explanation quality is evaluated with positive, negative, and averaged AUPRC measures.Scores use sigmoid-transformed raw attention values; negative-evidence AUPRC evaluates the negated scores.Source: §A.6.1 · p. 27

A.6.2 Perturbation Curves

  • The perturbation-curve procedure ranks patches by attention and progressively removes successive 1% groups, starting with the highest-attention patches.The sequence runs from the original bag to an empty bag after all groups are removed.Source: §A.6.2 · pp. 27–28

A.7 Implementation Details Across Different MIL Settings

  • CAR-MIL shares ABMIL’s encoder and classifier, adds a parallel attention branch, duplicates DSMIL’s bag-classifier query projection, and uses a parallel final TransMIL block whose maps are approximate proxies.For a 1×10000×1024 dummy bag, added cost is small for attention-based MIL models and moderate for transformer-based models.Source: §A.7 · pp. 28–29
  • CAR-MIL concentrates attention on the expert-annotated DCIS lesion and suppresses attention in benign tissue, while ABMIL, CLAM, AddMIL, and DSMIL distribute attention more broadly.The comparison uses Min–Max scaled unnormalised attention maps for one BRACS whole-slide image and zoomed views of DCIS and benign regions.Source: Figure A.2 · p. 30

Related papers

Loading 2609.08419v1…