PaperlayerView exact revision on arXiv

Source-linked AI summary

Masked Feature Encoding for Large-Scale Whole Slide Image Representation

Haoyu He, Basile Tessier-Cloutier, Yang Wang, Mahdi S. Hosseini

arXiv:2610.10225v1cs.CV

TL;DR

In WSI classification, within-slide variation in independently extracted patch embeddings can obscure diagnostic signal. MFE-MIL adapts frozen features using MIL classification and window-based masked reconstruction. It improves ACC/F1 in nearly all reported settings and improves average survival C-index for every tested aggregator.

  • Problem

    In standard WSI MIL, staining, scanner, and local-texture variation in independently computed patch embeddings can mask diagnostic signal.

    Source: §1 · p. 1

  • Method

    MFE-MIL jointly trains a lightweight adapter and MIL classification head with window-based masked reconstruction as an auxiliary feature regularizer.

    Source: §1 · p. 2

  • Results

    Across CAMELYON16/17, PANDA, and TCGA-BRCA, MFE-MIL improves ACC/F1 in nearly all settings, and it improves average C-index for every tested aggregator on five TCGA survival cohorts.

    Source: §5 · p. 14

  • Takeaways & Limitations

    Analyses indicate classification drives most feature compression, while reconstruction adds a modest regularizing effect.

    Source: §5 · p. 14

    The decoder adds a one-time training-time cost and is discarded at inference.

    Source: §4.4 · p. 10

Abstract

from arXiv · show

Whole slide image (WSI) analysis in computational pathology follows a multiple instance learning (MIL) pipeline where patch embeddings are extracted independently and aggregated for slide-level prediction, but within-slide variance from staining, scanner, and local texture can overwhelm the discriminative signal. We propose Masked Feature Encoding for Multiple Instance Learning (MFE-MIL), a feature-space masking framework that trains a lightweight MLP adapter jointly with a window-based masked reconstruction branch and a MIL classification head. The two objectives are complementary. Classification guides the adapter to suppress within-slide patch variance, while window-based masked reconstruction provides an auxiliary regularizer for the adapted features without using patch coordinates, coordinate graphs, or segmentation preprocessing. The raster patch-extraction order is used only as a weak implicit prior. At inference, the decoder is removed, leaving only the adapter and MIL head. Across CAMELYON16/17, PANDA, and TCGA-BRCA with four diverse encoders, MFE-MIL improves ACC/F1 for nearly all tested aggregator-encoder settings and AUC in most, outperforms coordinate-based spatial methods (CAMIL), and achieves higher AUC than 2DMamba on three of four datasets (UNI). On five TCGA survival cohorts it improves the average concordance index for every aggregator tested, its most consistent gain. Code is available at https://github.com/AtlasAnalyticsLab/MFE-MIL.

1 Introduction

Standard MIL can let staining, scanner, and texture variation obscure diagnostic signal in independently embedded patches. MFE-MIL adapts frozen features with classification and masked reconstruction, improving performance across the reported datasets and settings.

1 Introduction

  • Standard MIL aggregates independently embedded patches whose staining, scanner, and local-texture variation can mask diagnostic signal.The figure contrasts this with MFE-MIL’s adapted features and masked reconstruction over neighbouring packed features.Source: §1 · p. 1, Figure 1 · p. 2
  • MFE-MIL combines a trainable feature adapter and MIL classification with window-based masked reconstruction as an auxiliary regularizer.Features are packed in tissue-extraction order; raster order is treated only as a weak implicit prior, without patch coordinates at training or inference.Source: §1 · p. 2
  • Classification suppresses within-slide patch variance, while reconstruction further regularizes adapted features.Controlled experiments show coherence improvements do not consistently depend on mask arrangement, although standard window masking achieves the highest AUC on CAMELYON16.Source: §1 · p. 2
  • Across six aggregators, MFE-MIL improves ACC/F1 in nearly all settings and outperforms CAMIL without coordinate graphs, while measurably sharpening tumour localization as assessed by ABMIL attention-localization AUROC.ABMIL attention-localization AUROC increases from 0.879 to 0.942 against CAMELYON16 annotations.Source: §1 · pp. 2–3

2 Related Work

Prior MIL approaches commonly treat patches as an unordered set, while spatially aware alternatives add explicit priors or preprocessing. MFE-MIL instead regularizes adapted features within frozen-encoder pipelines.

2 Related Work

  • Most MIL methods treat patches primarily as an unordered set, although diagnostically relevant tissue patterns can depend on spatial organization and local context.Methods include pooling, attention-based aggregation, and interaction-aware variants.Source: §2 · p. 3
  • Spatially aware MIL uses coordinate, topology, or segmentation priors, but preprocessing and architectural coupling can reduce compatibility with frozen-feature pipelines.MFE-MIL regularizes adapted features instead, leaving the aggregator interface unchanged.Source: §2 · p. 3
  • Pathology foundation encoders are commonly frozen while task-specific MIL aggregators train on their patch embeddings, avoiding backbone retraining.MFE-MIL is compatible with this frozen-encoder setup.Source: §2 · p. 3
  • Unlike MAE-style pretraining, MFE-MIL applies masked reconstruction jointly with MIL classification at fine-tuning time.MHIM-MIL masks patches by attention score, whereas MFE-MIL masks contiguous blocks; coherence effects do not reliably depend on mask arrangement or grouping.Source: §2 · p. 3

3 Methodology

MFE-MIL adapts frozen patch features through joint MIL classification and masked reconstruction. It packs features in tissue-extraction order for reconstruction without patch coordinates, then removes the decoder at inference.

3 Methodology

  • A two-block MLP adapter maps frozen-encoder embeddings into same-dimensional task-specific features, leaving downstream aggregators unchanged.The adapter preserves the encoder feature dimension, making it a drop-in replacement for the original embeddings.Source: §3 · p. 4
  • Classification shapes adapted features toward class-discriminative directions, while masked reconstruction requires retaining information to predict normalized frozen embeddings.The two losses act on the same representation; reconstruction targets are fixed functions of the frozen encoder.Source: §3 · pp. 4–5
  • Reconstruction packs adapted features in tissue-extraction raster order into a square grid without coordinates, preserving only partial locality.Within-row coordinate precision is 0.89–0.97, while across-row precision is below 0.05.Source: §3 · p. 5
  • A lightweight Transformer decoder reconstructs normalized frozen embeddings at masked positions using mean squared error, excluding zero-padded cells.The decoder has four Transformer blocks and is trained only alongside the adapter and MIL head; the frozen encoder is unchanged.Source: §3 · pp. 5–6
  • Random 3×3 or 5×5 blocks fill about 80% of the masking budget before individual cells complete it, and this two-phase strategy outperforms fixed window sizes.The full budget is set by the masking ratio; the block-based task empirically gives the best AUC in Tab. 2.Source: §3 · p. 6
  • Training balances MIL and reconstruction losses with λrec = 0.3, and inference discards the decoder while retaining the adapter and MIL head.The MIL branch does not use decoder outputs, so removing the decoder causes no train–test mismatch.Source: §3 · pp. 4, 6

4 Experiments

Across the tested datasets and encoders, MFE-MIL generally improves classification performance and consistently raises average survival C-index across aggregators. The experiments also examine its computational costs, attention localization, ablations, and feature-space effects.

4.2 Survival Prediction

  • Average C-index improves for all six aggregators, and every MFE-MIL variant exceeds 2DMamba’s average of 0.695, without patch coordinates.The paired tests across cohort-aggregator pairs find the overall improvement significant (p < 0.05); the largest gains occur on UCEC and KIRP.Source: §4.2 · p. 8, Table 1 · p. 9

4.3 Classification Results

  • MFE-MIL improves ACC and F1 in nearly all tested aggregator-encoder settings, beats CAMIL on all four datasets, and leads 2DMamba in AUC on three of four UNI datasets.Gains are largest for weaker baselines; AUC regressions are rare and small, especially on near-ceiling TCGA-BRCA. Improvements are larger with ViT-L and ViT-S than with UNI and CONCH.Source: §4.3 · pp. 8–9

4.4 Computational Cost

  • Inference for MFE-MEAN uses 0.60 GB and 0.049 s per slide, versus 0.48 GB and 0.010 s for MEAN; training time rises about 21×.The decoder adds 13.7M parameters during training but is discarded at inference, making that decoder cost one-time.Source: §4.4 · pp. 9–10

4.5 Ablation Studies

  • Adaptive multi-scale window masking gives the best average AUC on PANDA, while depth-2 MFE-MIL is best overall among tested adapter-depth settings.Single-token and 3×3 masking reach 93.56 average AUC, versus 93.88 for fixed 5×5 windows; λ_rec = 0.3 is best among tested weights. Per-slide z-score normalization does not replicate the learned adapter’s attention entropy (0.982 versus 0.985).Source: §4.5 · pp. 10–11, Table 2 · p. 11

4.6 Qualitative Evaluation

  • On CAMELYON16 test annotations, MFE-MIL improves localization AUROC for all three aggregators; paired slide tests find significant gains for ABMIL (p = 4.1×10−5) and CLAM-SB (p = 0.003), but not CLAM-MB (p = 0.056).One unblinded board-certified pathologist reported better visual agreement overall, clearest on TCGA-LUAD and UCEC; KIRC and STAD were exceptions, while KIRP maps were similar. Localization used 48 of 49 tumor test slides with usable pixel-level annotations; significance testing used n=43.Source: §4.6 · pp. 11–12, Table 3 · p. 12
  • The pathologist judged MFE-MIL attention to focus more on lesions and align better with identified tissue features, most clearly on LUAD and UCEC.KIRC and STAD were exceptions, while KIRP maps were similar; the review covered 10 randomly selected WSIs per cohort and was qualitative, not quantitative validation.Source: §4.6 · pp. 11–12

4.7 Geometric Analysis

  • Classification alone drives semantic compression: the within-slide feature-variance ratio falls from 1.952× to 1.07–1.10× across masking conditions; coherence gaps show no reliable ordering by spatial arrangement or grouping.On CAMELYON16, Win MFE’s larger gap mainly reflects lower random-pair similarity (0.874 versus 0.904 for Baseline), not closer neighbours; the effect is modest and dataset-dependent.Source: §4.7 · p. 13
  • Attention entropy sharpens selectively by class: tumor entropy falls on CAMELYON16 while normal remains high, and PANDA Grades 1–5 sharpen while Grade 0 remains higher.Top-50 attention coherence also widens on CAMELYON16 but is essentially unchanged on PANDA; CONCH shows the same overall coherence-ratio reduction as UNI.Source: §4.7 · p. 14, Table 5 · p. 14

5 Conclusion

MFE-MIL combines classification-driven feature compression with masked reconstruction, improving benchmark performance across several datasets and aggregators. Its most consistent gain is higher average C-index across five TCGA survival cohorts for every aggregator tested.

5 Conclusion

  • MFE-MIL improves ACC/F1 in nearly all settings, outperforms CAMIL without coordinate graphs at inference, and achieves higher AUC than 2DMamba on three of four datasets (UNI).Across CAMELYON16/17, PANDA, and TCGA-BRCA.Source: §5 · p. 14
  • On five TCGA survival cohorts, MFE-MIL improves the average C-index for every aggregator tested.This is its most consistent gain.Source: §5 · p. 14
  • Classification drives most feature compression, while reconstruction adds a modest regularizing effect, according to coherence, attention-entropy, and effective-rank analyses.Source: §5 · p. 14

Related papers

Loading 2610.10225v1…