Source-linked AI summary
Refining Cytology Predictions with Conditional Random Fields
Manon Dausort, Tiffanie Godelaine, Karim El Khoury, Maxime Zanella, Christophe De Vleeschouwer, Benoît Macq
TL;DR
Histology-oriented VLMs make noisy zero-shot predictions on cytology, and existing CRFs were designed for histopathology rather than independent, multi-stain cytology patches. CytoCRF adapts CRF cues and neighborhoods to this setting, outperforming existing CRF baselines across annotation budgets on ten datasets. Combining backbones to expand neighborhoods brings further gains.
Problem
Histology-trained VLMs provide noisy zero-shot cytology predictions, and existing CRF frameworks do not transfer directly to independent cytology patch pools spanning multiple stains.
Source: §1 · p. 1Method
CytoCRF adapts CRF pairwise cues to chromatin and cytology-specific staining and combines multiple backbones to enrich patch neighborhoods.
Source: §1 · p. 1, §2.3 · p. 2Results
Across ten cytology datasets, CytoCRF beats existing CRF baselines at every tested annotation budget, with a +13.6pp balanced-accuracy margin over HistoCRF-style at budget 50.
Source: §3.2 · p. 3, §4 · p. 4Takeaways & Limitations
Adding neighbors identified by additional foundation models improves results even when the pairwise potential itself remains computed from DinoBloom embeddings.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
Vision-language models (VLMs) achieve strong zero-shot (ZS) classification on histology images but do not perform as well on cytology, whose stains and cell morphology differ markedly compared to histology. Conditional random fields (CRFs) can refine noisy VLM predictions by propagating information across patches, but existing CRF frameworks were designed for histopathology and do not transfer to cytology datasets, released as independent patch pools spanning multiple staining protocols. We introduce CytoCRF, which adapts the pairwise terms to cytology by targeting chromatin and cytology-specific staining, and further enrich the neighborhood of each potential term by combining multiple backbones. Across ten cytology datasets, CytoCRF outperforms existing CRF frameworks at every annotation budget, reaching +13.6 percentage points over the best baseline and +33.7 over ZS with only 50 annotations. Combining information from multiple backbones brings further gains, showing that the neighborhood topology matters more than the pairwise potential computed over it.
1. INTRODUCTION
Histology-trained VLMs provide noisy zero-shot predictions on cytology, while cytology-specific foundation models require expert annotations for fine-tuning. CytoCRF adapts transductive CRF refinement to independent, multi-stain cytology patches and reports gains across ten datasets.
1. INTRODUCTION
- Histology-oriented VLMs such as PLIP, CONCH and Quilt make zero-shot patch predictions, but their cytology predictions remain noisy and can be close to random on cytomorphology tasks.Source: §1 · p. 1
- Cytology-specific foundation models close part of the gap but are vision-only, so using them for zero-shot classification still requires expert annotations, for example for fine-tuning.Transductive methods can instead leverage the latent distribution of all samples; the authors pursue this training-free direction with a probabilistic graph formulation.Source: §1 · p. 1
- Existing CRF frameworks were developed for histology, and neither HistoCRF nor SlideCRF had been applied to cytology.SlideCRF uses shared whole-slide coordinates and biological cues based on texture and H&E color.Source: §1 · p. 1
- Across 10 cytology datasets, CytoCRF exceeds the best baseline by +13.6pp balanced accuracy with 50 annotations, and combining multiple models adds +2pp.The introduction also reports an average increase of +33.7pp balanced accuracy; the supplied passage does not specify its comparator in that sentence.Source: §1 · p. 1
- CytoCRF drops the spatial term and adapts biological cues and neighborhood construction for patch pools with multiple staining protocols.Its cues target chromatin and cytology-specific staining, while multiple models contribute to the neighborhood graph.Source: §1 · p. 1
2. METHOD
The method represents cytology patches as a graph and minimizes a CRF objective combining patch-level label fit with pairwise compatibility. CytoCRF keeps the unary term training-free and adapts pairwise cues and neighborhoods to cytology.
2.1. Problem formulation
- The CRF objective combines unary costs for patch-label mismatch with pairwise costs for incompatible labels on connected patches.Mean-field inference approximates its minimization by iteratively updating each patch’s marginal from its unary potential and neighboring marginals.Source: §2.1 · p. 2
2.2. Unary potential
- A VLM embeds each patch and each class description, and cosine similarity defines a unary potential without task-specific training.Class text embeddings are averaged over a set of prompts.Source: §2.2 · p. 2
2.3. Pairwise potential
- The pairwise potential retains HistoCRF’s diversity and annotation terms: dissimilar patches are discouraged from sharing labels, while annotations propagate to similar patches.The diversity term links each patch to its most dissimilar neighbors; the annotation term links annotated patches to their most similar neighbors and encourages matching labels.Source: §2.3 · p. 2
- CytoCRF normalizes each biological cue by its mean magnitude, then combines cues additively and links patches by embedding similarity.The final cues are May-Grünwald-Giemsa staining and chromatin texture; the similar-neighbor graph is based on the embedding described in Eq. (3).Source: §2.3 · p. 2
3. EXPERIMENTS
Experiments evaluate CytoCRF across ten cytology datasets and compare its performance with existing CRF frameworks and ablations of its models, cues, graphs, and annotation strategies.
3.1. Experimental setup
- Experiments cover 10 patch-level cytology datasets spanning cervical, breast, blood, body-cavity-fluid, and bone-marrow cells.The evaluation also tests annotation selection by random sampling, current model errors, or prediction flips across mean-field steps.Source: §3.1 · pp. 2–3
3.2. CytoCRF outperforms existing CRF frameworks
- 41.2–60.0% balanced accuracy across annotation budgets, with CytoCRF leading every budget and a +13.6pp margin over HistoCRF-style at budget 50.The comparison uses KEEP and DinoBloom with mgg+chrom; the reported means are averaged across ten datasets.Source: Table 1 · p. 3, §3.2 · p. 3
3.3. Ablation experiments
- DinoBloom is the best pairwise encoder for every unary VLM, while Quilt-B16 is the worst, with a gap exceeding 14pp regardless of unary choice.Vision-only encoders consistently outperform the VLMs’ own vision encoders as pairwise candidates; KEEP with DinoBloom gives the best overall balanced accuracy.Source: §3.3 · p. 3, Table 2 · p. 3
- chrom is the strongest solo biological cue, and mgg+chrom is the strongest pair among the tested cues.The comparison covers five cues individually and in pairwise combinations.Source: §3.3 · p. 3, Table 3 · p. 4
- Switching from SlideCRF’s dissimilar graph to the similar graph raises balanced accuracy by 20pp, from 34.6% to 54.7%.For mgg+chrom, the similar graph reaches 55.3%, versus 43.9% and 43.8% on the mgg and chrom raw-feature graphs, respectively.Source: §3.3 · p. 4
- Adding neighbors from additional foundation-model embedding spaces brings further gains, from +1.35pp for KEEP to +3.10pp for KEEP and GigaPath, with no saturation across three tested configurations.The potential remains computed from DinoBloom; only the candidate-neighbor set expands.Source: §3.3 · p. 4, Table 4 · p. 4
- At 5 annotations the three selection strategies are within 1.2pp; from 50 to 100 annotations, random and oscillation perform similarly.Oscillation gains do not increase until 125 annotations, suggesting its signal requires more propagation rounds to become reliable.Source: §3.3 · p. 4, Figure 2 · p. 4
4. CONCLUSION
CytoCRF adapts CRF refinement to independent, multi-stain cytology patch collections. Across ten datasets, it outperforms existing CRF-based baselines at every tested annotation budget, and the results indicate that neighborhood topology matters more than the pairwise potentials computed over it.
4. CONCLUSION
- Across ten datasets, CytoCRF beats both existing CRF-based baselines at every tested annotation budget, with margins widening as annotations increase.Combining neighborhoods across additional foundation models brings further gains, with no saturation across the tested configurations.Source: §4 · p. 4
- Across the tested ablations, neighborhood topology mattered more than changes to the pairwise potentials computed over it.Topology includes choices of pairwise foundation model, similar or dissimilar graph, and multi-model ensemble.Source: §4 · p. 4
- CytoCRF replaces the spatial-neighborhood assumption with a systematic search over foundation-model pairs and cytology-suited biological and color cues.The approach is tailored to independent patch collections.Source: §4 · p. 4
- The authors identify learning or adapting neighborhood structure as a direction for future work.Their current neighborhoods use hand-picked foundation models and similarity thresholds.Source: §4 · p. 4
5. COMPLIANCE WITH ETHICAL STANDARDS
The study did not require ethical approval.
5. COMPLIANCE WITH ETHICAL STANDARDS
- The study did not require ethical approval.Source: §5 · p. 4