PaperlayerView exact revision on arXiv

Source-linked AI summary

PathLang: A Language-Centered Benchmark for Vision-Language Models in Computational Pathology

Fanqi Cheng, Kuo Gong, Shangke Liu, Beidi Zhao, Junchao Zhu, Zheyu Zhu, Leiyue Zhao, Fengbei Liu, John Cannon, Gang Wang, Zu-hua Gao, Kenji Ikemura, Yihe Yang, Yaohong Wang, Yuankai Huo, Xiaoxiao Li, Mert R. Sabuncu, Ruining Deng

arXiv:2610.11329v1cs.CV

TL;DR

Pathology VLM benchmarks often fix prompts or vary only generic templates, leaving robustness to diagnostic-language changes poorly characterized. PathLang varies clinically grounded phrasing while holding slides, labels, and evaluation direction fixed; across models and datasets, performance is sensitive to equivalent paraphrases and alignment does not necessarily imply class separability.

  • Problem

    Existing pathology VLM benchmarks largely fix prompts or test generic template edits, leaving behavior under clinically realistic diagnostic-language variation poorly characterized.

    Source: §1 · p. 2

  • Method

    PathLang holds slides, labels, and image–text evaluation direction fixed while varying diagnostic language across four task families using clinically validated prompts and candidate pools.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across nine VLMs and five datasets, performance is highly sensitive to clinically equivalent paraphrases, varies across semantic-competition settings, and alignment quality does not necessarily translate into inter-class separability.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Canonical-prompt closed-set evaluation substantially overestimates pathology VLM language capability, and linguistic fragility remains an unresolved bottleneck.

    Source: §5 · p. 10

    PathLang covers patch-level dual-encoder models, five datasets, and a finite set of expert-curated variations rather than the natural frequency or distribution of diagnostic phrasing.

    Source: §5 · p. 10

Abstract

from arXiv · show

Pathology vision-language models (VLMs) have shown strong visual perception ability, but their robustness in the language domain remains poorly characterized. Existing pathology VLM benchmarks largely rely on canonical closed-set prompts or perturb only generic templates, treating language as a fixed evaluation component rather than a variable axis of model behavior. In clinical practice, however, diagnostic language varies across reports, institutions, and candidate diagnoses. We introduce PathLang, a language-centered and clinically grounded zero-shot benchmark. PathLang holds the underlying slides, ground-truth labels, and image-text evaluation direction fixed while systematically varying only the diagnostic language, so that performance differences reflect how a diagnosis is phrased rather than what is imaged. The language variation follows how pathologists actually rephrase diagnoses (terminology, specificity, and reporting style), and all prompts and candidate pools are validated by six board-certified pathologists. PathLang covers four task families: (1) zero-shot classification with image-text alignment analysis, (2) cross-modal retrieval, (3) paraphrase robustness, including semantic-equivalence paraphrases, length and reporting-style variation, and prompt ensembling, and (4) open-vocabulary diagnosis retrieval over four candidate pools with distinct forms of semantic competition. Across nine VLMs and five public datasets spanning four organs, we find that performance is highly sensitive to clinically equivalent paraphrases, varies substantially across forms of semantic competition, and that image-text alignment quality does not necessarily translate into inter-class separability. We release the prompt corpus, candidate pools, pre-computed text embeddings, and evaluation code at https://anonymous.4open.science/r/PathLang.

1. Introduction

Pathology VLM benchmarks have largely fixed prompts or tested generic template edits, leaving diagnostic-language robustness poorly characterized. PathLang instead varies clinically grounded diagnostic phrasing while holding slides, labels, and evaluation direction fixed.

1. Introduction

  • Most pathology VLM benchmarks fix language, while limited text-variation studies test generic templates rather than the diagnostic rephrasing used in practice.The resulting behavior under changed diagnostic wording remains largely uncharacterized, even when images and labels are unchanged.Source: §1 · p. 2
  • Pathologists vary equivalent diagnoses by terminology, specificity, and reporting style, making language robustness relevant to natural-language clinical queries.A model that succeeds with one canonical prompt may fail on a clinically equivalent paraphrase.Source: §1 · p. 2
  • PathLang isolates language-side behavior by varying diagnostic text while holding slides, labels, and image–text evaluation direction fixed.Its diagnostic-level variations are clinically grounded and prompts and candidate pools were validated by six board-certified pathologists.Source: §1 · p. 2
  • Across nine models and five datasets, PathLang identifies instability under equivalent paraphrases, cross-organ distractor degradation, and no consistent winner across language-robustness settings.The benchmark complements visual-axis evaluation and releases its prompt corpus, candidate pools, text embeddings, and evaluation code.Source: §1 · p. 2

2. Related Works

Prior pathology VLM benchmarks emphasize fixed-prompt or visual evaluation, while language-variation studies often use generic template edits. PathLang centers diagnostic phrasing, a distinct axis from taxonomic difficulty in answer choices.

2.1. Computational Pathology VLM Benchmarks

  • Most dual-encoder pathology benchmarks emphasize classification, retrieval, or visual robustness with fixed prompts; Histo-VL varies text mainly through template-level edits.PathLang makes diagnostic-language variation its primary axis while holding slides and labels fixed within an image–text direction.Source: §2.1 · pp. 2–3
  • OmniPathoVQA varies answer-space difficulty among taxonomically neighboring diseases, whereas PathLang varies the surface expression of a fixed diagnostic referent.The designs are orthogonal: success separating adjacent entities may not ensure stability under synonym substitution.Source: §2.1 · p. 3

2.2. Paraphrase Robustness in Vision–Language Models

  • Prompt-robustness research documents wording and context sensitivity, but conventional studies often change carrier templates while preserving the diagnostic term.PathLang separately evaluates clinically equivalent paraphrases, synonyms, prompt length, and reporting style.Source: §2.2 · p. 3
  • Table 1 positions representative studies across classification, retrieval, language sensitivity, and clinical validation axes, with rows ordered roughly chronologically.PathLang does not include a controlled comparison between diagnostic-level and template-level perturbations.Source: Table 1 · p. 4, §2.2 · p. 4

3. PathLang: Language-Centered Benchmarking

PathLang combines four complementary language-robustness task families with clinically validated prompts and candidate pools, using a unified patch-to-slide protocol for comparing patch-based models.

3.2. Four Task Families

  • Four orthogonal tasks isolate distinct aspects of language robustness while controlling visual content: canonical classification, retrieval, paraphrase robustness, and open-vocabulary diagnosis retrieval.T2 uses its own bank of four positive descriptions per class; its retrieval directions are reported descriptively rather than compared by absolute score.Source: §3.2 · p. 4

3.3. Clinically Validated Prompt and Candidate Pool Construction

  • Six board-certified pathologists from multiple institutions across two countries reviewed prompt and candidate text for diagnostic accuracy, terminological correctness, and clinical plausibility.Review used established pathology reference standards and current diagnostic guidelines.Source: §3.3 · p. 4
  • T3 varies diagnostic language through five semantic-equivalence rephrasings and four length or reporting-style levels; T4 uses four pools with distinct forms of semantic competition.The four T4 pools differ in construction and composition rather than along one controlled axis.Source: §3.3 · p. 5

3.4. Unified Patch-to-Slide Aggregation

  • A single fixed top-K patch-to-slide aggregation rule is applied across all nine patch-level VLMs, datasets, prompts, and candidate pools.The main-text configuration is selected by maximizing the primary metric using evaluation labels, so it is an oracle reference rather than a validation-selected deployment estimate.Source: §3.4 · pp. 5–6

4. Experiments

Across four task families, results show that model performance depends on diagnostic phrasing and candidate-pool semantics, while image–text alignment alone does not ensure class separation.

4.1. Experimental Setup

  • Experiments evaluated nine VLMs on five public datasets spanning four organs, using dual-encoder embeddings and cosine-similarity inference across the four task families.The models performed zero-shot inference without task-specific fine-tuning.Source: §4.1 · p. 6

4.2. T1 – Zero-shot Classification

  • No model led consistently across datasets and metrics, and high image–text alignment did not guarantee inter-class separability; KEEP had a negative similarity gap on CAMELYON17.Patho-CLIP maintained positive similarity gaps across all datasets.Source: §4.2 · p. 7

4.3. T2 – Cross-Modal Retrieval

  • Across datasets, cross-modal retrieval was weaker than image-to-image retrieval for all nine models, but several models’ text-to-image MRR was higher on CAMELYON16.The authors caution that retrieval directions are not on a common scale; they treat this asymmetry as descriptive rather than evidence of weak language grounding.Source: §4.3 · pp. 7, 9

4.4. T3 – Paraphrase Robustness

  • Clinically equivalent paraphrases induced large, model- and dataset-dependent accuracy fluctuations; ensembling ranged from +120.4% for MUSK to −71.8% for PathGen-CLIP on CAMELYON17 at n = 9.Instability persisted under query-independent mean pooling, and ensembling reduced but did not eliminate the observed fragility.Source: §4.4 · pp. 9–10

4.5. T4 – Open Vocabulary Retrieval

  • Set 1 was harder than Set 3 despite twice as many candidates; on CAMELYON16, PLIP’s Set 1 Recall@1 was 0.005 versus Set 1 chance Recall@1 of 0.050.The authors identify abstraction away from organ-specific language, rather than candidate count, as the dominant difficulty for evaluated models.Source: §4.5 · p. 10

5. Conclusion

PathLang evaluates pathology VLMs by varying diagnostic language while holding slides, labels, and evaluation direction fixed. Across the evaluated models and datasets, results show sensitivity to clinically equivalent paraphrases, variation across semantic competition, and a mismatch between alignment quality and inter-class separability.

5. Conclusion

  • PathLang isolates diagnostic-language effects by comparing language conditions against identical slides and labels within a fixed image–text evaluation direction.It covers zero-shot classification, cross-modal retrieval, paraphrase robustness, and open-vocabulary diagnosis retrieval.Source: §5 · p. 10
  • Across nine VLMs and five datasets, PathLang finds instability under clinically equivalent paraphrases, substantial variation across semantic competition, and a mismatch between alignment quality and inter-class separability.The authors conclude that conventional closed-set, canonical-prompt evaluation substantially overestimates pathology VLM language capability.Source: §5 · p. 10
  • The benchmark is limited to patch-level dual-encoder VLMs, five datasets, and a finite set of linguistic perturbations.Its expert-curated prompt corpus is not intended to estimate naturally occurring diagnostic-phrasing frequencies across institutions, reporting systems, or clinical populations.Source: §5 · p. 10
  • The authors state that pathology VLMs remain unsuitable for autonomous diagnostic use without expert oversight and external validation.PathLang aims to expose failure modes hidden by canonical closed-set prompts and may help reduce overestimation of model capability in deployment.Source: §5 · p. 10

Appendix A. Additional Experimental Details

The appendix details the benchmark’s aggregation rule, task implementations, prompt and candidate-pool construction, and metrics. Its sensitivity analyses report broadly stable model rankings across many aggregation settings, while paraphrase instability persists under query-independent mean pooling.

Appendix A.1. Hybrid Top-K Aggregation

  • Hybrid Top-K selects the larger of a minimum tile count and a slide-size-proportional count, then averages the top-K tiles ranked by cosine similarity to the text query.The minimum guards against undersampling focal signals, while the proportional term scales selection for diffuse patterns; the benchmark uses Kmin = 10.Source: Appendix A · pp. 11–12

Appendix A.1.1. Aggregation Sensitivity Analysis

  • Model rankings remain stable across aggregation for zero-shot classification (ρ=0.883–0.967) and semantic-equivalence paraphrases (ρ=0.817–0.983), while image-to-image retrieval is more sensitive (ρ=0.633–0.950).Length and reporting-style rankings are perfectly preserved across settings (ρ=1.000); the appendix also reports that open-vocabulary results are not driven by one aggregation setting.Source: T2 – Cross-Modal Retrieval · p. 12, Table A.1 · p. 13, Table A.2 · p. 13

Appendix A.1.2. Query-Independent Baseline

  • Under query-independent mean pooling, negative PSS persists in 35/45 semantic-paraphrase and 34/45 length/style model–dataset pairs, versus 31/45 and 33/45 under Hybrid Top-1%.This comparison tests whether instability depends on text-conditioned patch selection; the appendix reports closely matched per-model magnitudes across the two aggregation schemes.Source: T2 – Cross-Modal Retrieval · p. 12

Appendix A.2. Detailed Implementation of Tasks

  • Task 1 classifies slides by cosine similarity to canonical class prompts, while Task 2 evaluates image-to-text, text-to-image, and image-to-image retrieval.Task 2 uses positive descriptions and hard negatives, including semantically related descriptions from other datasets; retrieval directions differ in candidate-set size and relevance definition, so their absolute scores are not compared as a common scale.Source: T2 – Cross-Modal Retrieval · pp. 12–13

Task 3

  • Task 3 holds slides and labels fixed while testing semantic-equivalence paraphrases, length and reporting-style variation, and prompt ensembling.For ensemble sizes below nine, prompt indices are sampled without replacement and evaluated over 100 draws; the nine-prompt result is deterministic.Source: T2 – Cross-Modal Retrieval · p. 13

Task 4

  • Task 4 retrieves diagnoses from four label-specific candidate-pool conditions that impose different semantic competition, including abstract terms, mixed distractors, cross-organ diagnoses, and labels across datasets.Set 2 is not an isolated measure of within-organ discrimination, and raw scores across candidate sets are treated as separate conditions rather than directly compared against a common baseline.Source: §3 · pp. 14–15, T2 – Cross-Modal Retrieval · p. 14

Appendix A.4. Prompt Development and Clinical Review

  • The prompt corpus was iteratively reviewed and refined by domain experts for diagnostic terminology, specificity, and clinical plausibility while preserving intended diagnostic concepts.Review and verification were performed on individual prompt entries rather than through a corpus-wide list of prohibited terms.Source: §3 · pp. 15–16

Appendix B. Additional Discussion

PathLang shows that pathology VLM performance is fragile under clinical language variation, which canonical-prompt results do not reliably predict. The discussion connects this instability to prompt wording and calls for language-aware evaluation, pretraining, and deployment strategies.

Main findings

  • Canonical-prompt evaluation overestimates language robustness: equivalent paraphrases can collapse consistency, and open-vocabulary performance varies with candidate pools.On CAMELYON16, BiomedCLIP and QuiltNet reached prediction consistency of 0.000 under equivalent paraphrases; several models fell to or below chance on Set 1.Source: Appendix B · p. 16

Interpretation

  • With slides and labels fixed, KEEP’s competitive alignment score but most negative CAMELYON17 similarity gap shows alignment and inter-class separability can diverge.Prompt-length effects also vary by model: CONCH degrades on longer CAMELYON16 prompts (PSS = −0.291), while KEEP shows the opposite trend.Source: Appendix B · p. 16

Implications

  • The authors recommend clinically varied evaluation and report-based pretraining; prompt ensembling is a partial mitigation, while architectural or pretraining interventions may be needed.They argue closed-set canonical-prompt benchmarks are insufficient predictors of clinical readiness.Source: Appendix B · p. 17

Appendix C. Result Details

Appendix results show that alignment, retrieval, and paraphrase robustness vary across models and datasets. In particular, embedding visualizations require caution, while clinically equivalent and stylistically varied prompts expose instability.

Zero-shot classification

  • PathGen-CLIP and KEEP have the strongest alignment on most datasets, while Patho-CLIP has the largest similarity gaps in several settings.The CAMELYON17 UMAP panels show text embeddings separated from most slide embeddings, but this reflects the modality gap rather than class separability.Source: Appendix C · p. 17
  • UMAP distances and cluster shapes cannot establish cross-modal alignment or class separability; language fragility is instead tested against identical slides in a fixed image–text direction.This design holds the modality gap constant when comparing language conditions.Source: Appendix C · p. 17

Cross-modal retrieval

  • No model dominates retrieval across directions: leaders vary by dataset, and CAMELYON16’s higher T2I than I2I MRR for some models reflects only 8–24 text queries.KEEP and Patho-CLIP lead in multiple I2I settings, CONCH is strongest overall for I2T, and T2I results are more variable.Source: Appendix C · p. 17

Paraphrase robustness

  • Across five datasets, no model is stable on every paraphrase dataset; length sensitivity is model-specific, and ensembling improves average accuracy but usually fails to restore semantic agreement.For CONCH, the long-prompt PSS on CAMELYON16 is −0.291; 85 of 90 PSS bootstrap intervals exclude zero.Source: Appendix C · pp. 17–18

Appendix C.4. Open Vocabulary Retrieval

Open-vocabulary retrieval results differ across candidate pools and datasets, with model leadership shifting by condition. KEEP and PathGen-CLIP are strongest overall, while BiomedCLIP and MI-Zero struggle across pools.

Open-vocabulary diagnosis retrieval

  • On Set 1, Patho-CLIP led CAMELYON16 (mean recall 0.891), KEEP led PANDA and CAMELYON17, and PathGen-CLIP led UniToPatho; MI-Zero and BiomedCLIP neared zero on some datasets.Set 1 is the Open Vocabulary pool; its caption identifies chance as expected performance under a uniform random ranking.Source: Appendix C.4 · p. 18, Table C.12 · p. 25
  • In Set 2, KEEP ranked among the top models on CAMELYON16, CAMELYON17, and PANDA, while PLIP led UniToPatho.Set 2 contains label-specific mixed diagnostic candidates.Source: Appendix C.4 · p. 18, Table C.13 · p. 26
  • In Set 3 (Cross-Organ), KEEP dominates CAMELYON16, CAMELYON17, and PANDA; PathGen-CLIP achieves near-perfect recall on TCGA-GBMLGG, while PLIP leads UniToPatho, where QuiltNet, PathGen-CLIP, and KEEP reach perfect recall.In Set 4 (All Dataset Labels), KEEP achieves the best mean recall on CAMELYON17 and PANDA, while PathGen-CLIP leads on TCGA-GBMLGG. Across all four sets, KEEP and PathGen-CLIP are the most robust models; BiomedCLIP and MI-Zero struggle consistently.Source: Appendix C.4 · p. 18

Appendix D. Prompt Lists

Appendix D documents the prompt collections used for zero-shot classification and cross-modal retrieval, including clinically phrased diagnostic variants that vary in paraphrase, length, and reporting style. It also lists open-vocabulary retrieval candidates and defines one of four candidate-pool settings.

Appendix D.1. Zero-Shot Classification Prompts

  • Zero-shot classification prompts describe the visual appearance of a whole-slide image corresponding to a class label.Table D.1 lists the prompts across all datasets.Source: Appendix D.1 · p. 29, Table D.1 · p. 29

Appendix D.2. Cross-Modal Retrieval Prompts

  • For cross-modal retrieval, each dataset label has a positive prompt bank, while negatives come from other classes in that dataset and cross-organ distractors.The distractors are drawn from the remaining datasets.Source: Table D.2 · p. 30
  • Prostate prompts span Grades 1–5, expressing grade through glandular morphology and, in alternate formulations, ISUP grades or Gleason scores.Examples progress from well-formed glands in Grade 1 to loss of gland formation or solid growth in Grade 5.Source: Appendix D.2 · p. 30
  • Colorectal prompts cover hyperplastic polyps and tubular or tubulovillous adenomas, with adenoma descriptions distinguishing low- and high-grade dysplasia.The examples include architectural and cytologic descriptions as well as shorter diagnostic formulations.Source: Appendix D.2 · p. 31
  • Lymph-node prompts distinguish negative nodes, isolated tumor cells, micrometastases, and macrometastases using tumor burden and architectural disruption.Examples describe ITCs as tiny clusters or scattered cells and macrometastases as deposits exceeding 2 mm.Source: Appendix D.2 · p. 32
  • Glioma prompts contrast lower-grade glioma lacking glioblastoma features with glioblastoma descriptions featuring necrosis or microvascular proliferation.The prompt examples also use histologic and WHO-grade terminology.Source: Appendix D.2 · p. 32

Appendix D.3. Paraphrase Robustness Prompts

  • The semantic-equivalence variant represents each label with five lexically varied prompts across datasets.Table D.3 lists these paraphrases; examples include alternate formulations for prostate grades and colorectal lesions.Source: Appendix D.2 · p. 32, Table D.3 · p. 32
  • The length and reporting-style variant gives each label four prompts: short, medium, long, and clinical, with increasing length and clinical detail.The appendix identifies Table D.4 as the listing of these prompts.Source: Appendix D.2 · p. 35

Appendix D.2. Cross-Modal Retrieval Prompts

  • Four prompt variants per label cover short, medium, long, and clinical reporting styles across datasets.Examples include colorectal polyp diagnoses phrased by architecture and dysplasia, alongside shorter diagnostic labels.Source: Table D.4 · p. 35, Appendix D.2 · p. 36
  • Colorectal prompts vary in specificity, from brief adenoma labels to descriptions of glandular architecture and dysplasia.Tubular and tubulovillous adenomas are represented with both low- and high-grade dysplasia formulations.Source: Appendix D.2 · p. 36
  • Lymph-node prompts distinguish benign or negative nodes with preserved architecture and no metastatic carcinoma from metastatic deposits; micrometastases measure 0.2–2 mm, whereas macrometastases exceed 2 mm.Micrometastases cause only minor architectural disturbance; macrometastases markedly distort or replace normal lymphoid tissue.Source: Appendix D.2 · pp. 36–37
  • Glioma prompts contrast LGG without necrosis or microvascular proliferation with GBM descriptions that include those features.Source: Appendix D.2 · p. 37

Appendix D.4. Open Vocabulary Retrieval Prompts

  • Open-vocabulary retrieval ranks each slide against a label-specific candidate pool, requiring retrieval of the ground-truth text among distractors.The appendix lists ground-truth and candidate texts for every dataset and label across four pool settings.Source: Appendix D.2 · p. 37
  • Set 1 uses 20 generic, organ-agnostic pathology candidates per label, with dataset-specific ground-truth mappings.Examples of generic candidates include benign tissue and high-grade malignancy.Source: Appendix D.2 · p. 37, Table D.5 · p. 37

Related papers

Loading 2610.11329v1…