PaperlayerView exact revision on arXiv

Source-linked AI summary

Whole-Slide Image Analysis under Realistic Few-Shot Annotation Protocols

Tiffanie Godelaine, Maxime Zanella, Karim El Khoury, Benoit Macq, Christophe De Vleeschouwer

arXiv:2608.30420v1cs.CVcs.AI

TL;DR

Whole-slide zero-shot predictions are noisy, while existing transductive evaluations overlook WSI spatial organization, class imbalance, absent classes, and realistic annotation behavior. The paper introduces SlideCRF, which combines spatial and biological relations with a class-presence term, alongside localized click and scribble protocols. Across four datasets, SlideCRF outperforms strong baselines and improves over zero-shot predictions by +24.2% in macro F1 with one annotation per present class and up to +37.5% with larger budgets.

  • Problem

    Existing transductive WSI evaluations overlook spatial organization, severe class imbalance, absent classes, and how pathologists provide limited annotations.

  • Method

    SlideCRF refines patch-level VLM predictions with a CRF combining spatial and biological relations, a class-presence term, and realistic localized click and scribble protocols.

  • Results

    +24.2% macro F1 over zero-shot predictions with one annotation per present class, increasing to +37.5% with larger annotation budgets; SlideCRF also outperforms strong baselines across four datasets.

  • Takeaways & Limitations

    Iterative error correction yields the largest gains, supporting pathologist-in-the-loop refinement for clinically realistic WSI analysis.

  • Takeaways & Limitations

    When a present class is left unannotated, sufficiently strong class-presence penalties can suppress it severely or erase it entirely.

Abstract

from arXiv · show

Automating the analysis of whole-slide images has high clinical value, since characterizing cancers requires examining them in detail. Such analysis increasingly relies on vision-language models that provide patch-level zero-shot predictions. However, these predictions remain noisy and must be refined with a few annotations. A promising paradigm for this refinement is few-shot transduction. Rather than treating each patch independently, these methods leverage the relations between patches, together with a few annotations, to refine all predictions jointly. However, current transductive methods are evaluated under conditions that overlook key properties of whole-slide images: (i) datasets consist of independent patches extracted from multiple slides, ignoring the complex tissue organization; (ii) datasets are mostly balanced, whereas a single whole-slide image exhibits severe class imbalance, with several classes absent; and (iii) annotations are sampled at random, without reflecting how a pathologist annotates a limited number of regions. To align the transduction paradigm to realistic whole-slide settings, we introduce the following contributions. First, we propose SlideCRF, which adapts conditional random fields for whole-slide images by combining spatial and biological cues while accounting for classes that may be absent from a given slide. Second, we provide a set of realistic annotation protocols, based on spatially localized clicks and scribbles, modeling different pathologist interactions, such as the iterative correction of model errors. Across four datasets, we show that SlideCRF outperforms current transductive methods in macro F1, improving over the zero-shot predictions by +24.2% and +37.5% with one and 16 clicks per present class, respectively.

1 Introduction

Whole-slide analysis has clinical value, but noisy patch-level zero-shot predictions and unrealistic transductive evaluations limit refinement in clinical settings. The paper introduces SlideCRF and localized annotation protocols to address these gaps.

  • Motivation: Whole-slide image analysis supports detailed cancer characterization, but gigapixel slides require processing as patches.This patch-based representation makes automation computationally feasible while preserving the clinical motivation.
  • Motivation: Vision-language models provide patch-level zero-shot predictions, yet these predictions often remain noisy and require expert refinement.The refinement uses a limited number of pathologist annotations.
  • Motivation: Transductive methods jointly refine predictions by exploiting relationships between patches and expert annotations, unlike independent patch classification.Conditional random fields enforce consistency across neighboring predictions while refining all patches simultaneously.
  • Research gap: Current evaluations overlook WSI spatial organization, severe intra-slide class imbalance, absent classes, and realistic annotation behavior.Standard benchmarks pool independent patches, use mostly balanced datasets, and sample annotations randomly.
  • Contributions: SlideCRF adapts CRFs to refine WSI predictions using spatial and biological patch relations while remaining robust to absent classes.The method progressively refines zero-shot predictions as a few regions are iteratively annotated.
  • Contributions: The annotation protocols use localized clicks and scribbles to model representative-region selection, erroneous-region focus, and iterative error correction.They are designed to reflect pathologist interactions rather than widespread annotations.

2 Related Work

Prior transductive methods refine patch predictions jointly, but histology benchmarks and annotation tools do not fully model WSI structure or pathologist interactions. The paper addresses these gaps with a unified realistic annotation benchmark.

  • Transductive refinement: Few-shot transductive methods jointly refine unlabeled predictions using relationships between samples and a few annotations.This contrasts with inductive methods that classify each patch independently.
  • Histology methods: Existing histology methods primarily connect patches by feature similarity, which may link anatomically distant regions and miss local tissue organization.The paper motivates adding spatial and biological information alongside model features.
  • Histology methods: SlideCRF illustrates progressive refinement from a few localized clicks or scribbles, recovering spatially coherent regions.The figure depicts this process on two whole-slide images.
  • Annotation protocols: Existing annotation tools support points, scribbles, random clicks, erroneous-region clicks, or iterative correction, but none combines these facets into one realistic benchmark.The tools capture different parts of annotation behavior and often rely on masks or trained models.
  • Annotation protocols: The proposed benchmark spans clicks and scribbles, multiple annotation budgets, and three pathologist interactions including error correction.It is intended to support comparisons of interactive WSI annotation methods.

3.1 CRF Objective function

The CRF objective represents a WSI as a graph of patch variables and optimizes their joint class assignment using clique-based energies and unary and pairwise potentials.

  • Graph representation: The WSI is represented as a graph whose vertices are patches and whose undirected edges encode relations between patches.The objective assigns one class label to every patch vertex.
  • Objective: CRF inference seeks a joint assignment y of class labels to all patch variables.The labels define the predicted class for each patch.
  • Objective: The CRF joint distribution is expressed as a Gibbs distribution over energies defined on maximal fully connected subgraphs.These maximal subgraphs are called cliques.
  • Objective: A Bayesian formulation yields an objective function after taking the negative logarithm of the posterior.This converts posterior estimation into energy minimization.
  • Potential terms: Unary potentials score each patch from its observations, while pairwise potentials encourage compatible labels for neighboring patches.Together they provide the local evidence and relational consistency terms of the CRF.

3.2 Unary potential

SlideCRF builds unary potentials from VLM image-text similarities and adds a class-presence term that discourages predicting classes absent from expert annotations.

  • Class presence: SlideCRF augments the preliminary unary potential with a term designed to handle classes that may be absent from a slide.This modification targets the realistic partial class presence of whole-slide images.
  • Initial prediction: VLM unary potentials use visual patch embeddings and text embeddings averaged across prompts for each class.The method obtains these quantities without training a task-specific model.
  • Initial prediction: The unary potential is computed from the cosine similarity between each patch embedding and each class text embedding.Lower unary potential values favor the corresponding class prediction.
  • Class presence: Expert annotations identify present classes, and a constant λ is added to every unannotated class's unary potential at every patch.This discourages predictions for classes absent from the annotation set.
  • Class presence: The indicator function in the class-presence expression activates the penalty only for classes not included among annotated classes.The indicator is denoted by ⊮[·].

3.3 Pairwise potentials

SlideCRF combines four pairwise terms to refine patch labels using expert annotations, patch similarity, spatial neighborhoods, and biological similarity, while adapting annotation weighting for partially present classes.

  • Annotated patches connect to highly similar patches in embedding space, encouraging labels consistent with expert input.Similarities use cosine similarity between histology-oriented vision-model embeddings.
  • The diversity term links visually dissimilar patches so shared labels can be adjusted toward greater label diversity.Its connections are drawn from each patch’s most dissimilar neighbors.
  • Spatial connections favor identical labels for neighboring patches, imposing local smoothing across the slide.The spatial term has the same functional form as the annotation term but operates on spatial-neighbor pairs.
  • Biological cues connect visually different but biologically similar patches using texture and color features.Texture uses gray-level co-occurrence and local-binary-pattern features, while color uses hematoxylin and eosin statistics in HED space.
  • The pairwise potential combines annotation, diversity, spatial, and biological terms, weighted by α, β, γ, and η.The four terms respectively model annotated-patch similarity, label diversity, spatial smoothing, and biological cues.
  • The annotation weight follows a step schedule, becoming zero after iteration τ to handle classes that may be absent from a slide.Similar and dissimilar neighbor sets are randomly resampled from restricted candidate subsets to increase connections without increasing memory load.

3.4 Objective function optimization

SlideCRF optimizes its CRF objective with mean-field inference, implemented as iterative message passing that updates patch distributions and unary potentials after expert annotations are incorporated.

  • The CRF objective combines unary and pairwise potentials, but its exact solution is intractable.The objective is expressed as the sum of unary terms and the combined pairwise potential.
  • Mean-field approximation replaces the joint distribution with a product of independent marginal distributions Q.The approximation minimizes KL-divergence between P(Y|X) and Q.
  • Iterative updates implement message passing over the patch graph to refine the marginal distributions.The update equation operates on Q_v and Q_w label probabilities.
  • Algorithm 1 initializes unary potentials, obtains expert annotations, computes pairwise potentials, and iteratively updates Q and the unary potential.These operations are repeated across expert-annotation steps.

4 Annotation protocols

The annotation protocols model localized pathologist interactions through clicks or scribbles and representative, error-driven, or iterative human-in-the-loop selection strategies.

  • An annotation protocol combines an annotation type, which specifies shape, with an interaction type, which specifies patch selection.The study uses clicks and scribbles as annotation types.
  • A click covers five patches: a center patch and its four neighbors.It is defined by one control point and a radius of one patch.
  • A scribble interpolates a smooth curve between several control points, capped at 15 patches.
  • Annotation centers and scribble control points are generated from dense segmentation masks to mimic pathologist annotation of tissue regions.The generation process selects connected components, samples representative interior points, and repels successive annotations from already annotated patches.
  • The protocols include representative, error-driven, and human-in-the-loop interactions.Representative annotations target prominent regions, whereas error-driven annotations target mispredicted patches.
  • HITL adds one annotation per class after each inference round, repeatedly targeting currently mispredicted patches.This progressively focuses the annotation budget on remaining model errors.

5 Experimental Setup

The experiments evaluate SlideCRF across four heterogeneous histology WSI datasets using slide-level patch segmentation, realistic annotation budgets, multiple baselines, and balanced accuracy and macro F1.

  • Four datasets—BACH, CATCH, SKINCANCER, and TIGER—cover diverse cancer types, species, class counts, slide sizes, and spatial complexity.Dense segmentation masks support patch-based evaluation across these datasets.
  • Slides are tiled into non-overlapping patches, filtered for tissue, and assigned one label using polygon intersection or pixel-majority rules.Retained slides contain approximately 950 to 35,000 labeled patches per slide.
  • Each slide is segmented independently with all unlabeled patches and protocol-sampled annotations, using budgets b ∈ {1, 2, 4, 8, 16}.Budgets count annotation actions so protocols are compared at equal action counts.
  • SlideCRF is compared with two inductive and four transductive baselines, plus zero-shot CONCH predictions.ECALP is the closest baseline; SlideCRF additionally models spatial and biological pairwise potentials.
  • Performance uses per-slide balanced accuracy and macro F1 averaged across evaluated patients, with five random seeds per experiment.Balanced accuracy averages recall over present classes, while macro F1 also incorporates precision and absent-class predictions.

6 Experiments

Across four datasets, SlideCRF consistently improves WSI patch classification over zero-shot predictions and performs competitively against transductive baselines. Its spatial and biological terms improve coherence and boundaries, while realistic annotation protocols show benefits from scribbles and iterative error correction.

  • SlideCRF matches or outperforms baselines: SlideCRF and ECALP are the top-performing approaches, with SlideCRF achieving the highest average score across balanced accuracy and macro F1.SlideCRF is nearly on par with ECALP in balanced accuracy and leads more clearly in macro F1.
  • SlideCRF matches or outperforms baselines: SlideCRF processes approximately 10^5 patches in about 38 seconds, remaining 13 times faster than ECALP.Runtime is measured for a slide of approximately 10^5 patches.
  • SlideCRF matches or outperforms baselines: Adding spatial and biological information improves SlideCRF over HistoCRF by +10.3% in balanced accuracy and +12.4% in macro F1 at one annotation.The comparison includes the class-presence prior.
  • Spatial and biological terms: The spatial term contributes +5.9% balanced accuracy and +8.1% macro F1, while the biological term further improves macro F1 by +5.4%.Spatial information promotes neighboring label consistency; biological texture and color cues refine class boundaries.
  • Diversity weighting factor and class-presence term: On CATCH, the diversity step schedule reduces predicted classes while slightly increasing both metrics; on BACH, performance is essentially unaffected.CATCH averages four present classes out of twelve, whereas nearly all classes are present in BACH.
  • Diversity weighting factor and class-presence term: The class-presence term improves balanced accuracy and macro F1 on CATCH by +12.3% and +3.3%, respectively, while reducing predicted classes.The term complements non-uniform diversity weighting and acts on unary potentials using annotations.
  • HITL interaction yields the best performance: Scribbles match or outperform clicks on most datasets because they span more tissue structures and provide richer supervision at equal action budgets.The broader coverage improves propagation, especially in macro F1 at high budgets.

7 Conclusion

SlideCRF adapts CRF-based transductive refinement to realistic whole-slide image analysis, combining patch relations with sparse expert annotations. Across four datasets, it outperforms strong baselines, while iterative error correction produces the largest annotation gains; clinical validation and efficiency improvements remain open directions.

  • Contributions: SlideCRF combines spatial and biological information with a class-presence penalty to refine patch-level predictions under whole-slide class imbalance.The formulation accounts for classes that are absent from a slide and exploits relations between patches alongside a few expert annotations.
  • Results: Across four diverse datasets, SlideCRF outperforms strong baselines in balanced accuracy and macro F1.
  • Annotation study: Iterative error correction yields the largest gains among the studied annotation interactions.The annotation study supports keeping the pathologist in the loop during refinement.
  • Future work: Direct validation with pathologists is needed to confirm the clinical realism of the proposed annotation protocols.Future work also identifies multi-resolution patches and extension to 3D medical images as possible directions.
Loading 2608.30420v1…