PaperlayerView exact revision on arXiv

Source-linked AI summary

PathJev: Decision-Native Vision-Language Modelling for Histopathology

Cristian Tommasino, Antonio M. Rinaldi

arXiv:2610.32478v1cs.CVcs.LG

TL;DR

Pathology vision–language models often lack calibrated probabilities over the candidates needed for decisions, particularly for unseen organs. PathJev-P uses runtime typed candidate questions and a decision head; it outperforms the compared models in accuracy, while calibration comparisons depend on the generative model’s probability read-out.

  • Problem

    Pathology decision tasks need trustworthy probabilities over user-relevant candidates, including “none of these,” but contrastive and generative VLM outputs do not directly provide such decision probabilities.

    Source: §1 · p. 1

  • Method

    PathJev-P turns expert labels from three public collections into typed, organ-specific candidate questions and trains a decision-native model to return candidate distributions.

    Source: §1 · p. 2

  • Results

    PathJev-P is more accurate than the compared generative, zero-shot pathology, and general medical decision models, but its calibration advantage depends on the generative model’s read-out.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The demonstration supports cluster-level labelling with abstention for coarse whole-slide questions in an open-vocabulary, label-free setting.

    Source: §6 · p. 11

    The whole-slide pipeline was not systematically evaluated against pathologists, only two organs were unseen, and “clean” test status means no documented overlap with partly undisclosed backbone training data.

    Source: §6 · p. 11

Abstract

from arXiv · show

Vision-language models used for decisions in histopathology rarely return calibrated probabilities over the candidates a user cares about. Decision-native models write a question and a candidate set chosen at run time into the input and return a normalised distribution from a small decision head in one forward pass. We build PathJev-P, a decision-native model for histopathology: MedGemma-1.5-4B with LoRA adapters and a decision head, trained on 40,000 patch states from three public collections whose expert labels are turned into typed questions with organ-specific candidate sets and a "none of these" option. In a comparison over 3 seeds with an analysis plan fixed before testing, on test sets without documented overlap with the training data of the backbone, PathJev-P is more accurate than a generative model trained with the same backbone, data and selection rule, the zero-shot pathology model CONCH and a general medical decision model (balanced accuracy 0.541 against 0.492, 0.345 and 0.213); its advantage over the generative model comes from two organs absent from training. The calibration advantage depends on how the generative model's probabilities are read out. With the summed candidate log-likelihood used in previous work, PathJev-P has the lower calibration error (ECE 0.204 against 0.279); with a length-normalised read-out of the same generative model, examined post hoc, the two have similar mean calibration error (0.201) and the generative model is better calibrated on the unseen organs, where PathJev-P remains over-confident. A supervised linear probe on a pathology foundation model is better calibrated in every setting. In a qualitative whole-slide demonstration, PathJev-P labels unsupervised tissue clusters with abstention and separates carcinoma from other tissue while running the model on a fraction of the tiles.

1 Introduction

Pathology decisions need trustworthy probabilities over runtime candidate sets, including unseen-organ cases. PathJev-P addresses this with typed decisions and is more accurate than the compared models, while calibration depends on how the generative baseline is read out.

1 Introduction

  • Pathology decisions need trusted probabilities over user-chosen candidates, including “none of these,” especially for tissue from unseen organs or laboratories.Contrastive models yield arbitrarily scaled similarities and generative assistants use token probabilities; pathology calibration has been reported as particularly problematic.Source: §1 · pp. 1–2
  • PathJev-P turns expert annotations from three public collections into 40,000 patch states with typed questions, organ-specific candidates, and a “none of these” option.Its evaluation includes breast and prostate sets and two organs absent from training, with no documented overlap with the backbone’s training data.Source: §1 · p. 2
  • PathJev-P is more accurate than the generative, zero-shot pathology, and general medical decision models, with its advantage over the generative model concentrated on new organs.Its calibration advantage holds with summed candidate log-likelihoods but not with length-normalised probabilities; it remains over-confident on new organs.Source: §1 · p. 2
  • PathJev returns a normalised distribution over runtime candidates in one forward pass for class choice, yes/no, and ordered-level questions.The paper also compares decision-native training with a matched generative model, zero-shot pathology models, a general medical decision model, and a supervised pathology probe.Source: §1 · p. 2
  • The whole-slide demonstration labels unsupervised tissue clusters with abstention while processing only a fraction of the tiles.This targets a setting where clusters are otherwise named by hand or with uncalibrated keyword scores.Source: §1 · pp. 1–2

2 Related work

Related work covers decision-native models, pathology vision–language systems, tissue-cluster labelling, and calibration. PathJev is positioned against the limits of existing outputs and the still-open evaluation of calibrated, abstaining cluster labels.

Decision-native (“System One”) models

  • Decision-native models return typed, normalised distributions over runtime candidate sets rather than generated text, but histopathology training with contamination-aware test sets had not been reported.Prior medical work reported calibration gains with histopathology coverage limited to PathMNIST.Source: §2 · p. 3

Pathology vision–language models

  • Contrastive pathology models support zero-shot classification and related tasks, but their arbitrary-scale similarity scores lack an explicit “none of these” option.Generative assistants answer in free text, making token probability different from a decision probability; pathology VLM calibration has been reported as particularly severe [3] [4].Source: §2 · p. 3

Labelling unsupervised tissue clusters

  • Unsupervised tile clustering can recover histomorphological phenotypes, but cluster naming remains manual, similarity-based [6], or delegated to generative descriptions.The cited approaches do not provide calibrated, abstaining cluster labels, and systematic evaluation of PathJev’s demonstration remains future work.Source: §2 · p. 3

Calibration

  • Calibration work uses ECE, adaptive ECE, proper scoring rules, and temperature scaling [30], while uncertainty thresholds for confident histopathology predictions have also been advocated [32].Source: §2 · p. 3

3 Method

The method defines runtime-candidate decisions, scores them with a shared head over backbone hidden states, and trains the model in two stages. It also controls prompt construction so test answers to the main question can vary with the image.

3.1 Typed decisions over runtime candidate sets

  • A decision pairs a question and type with a finite candidate set supplied as model input, supporting Choice, yes/no Noul, and ordered-level Score questions.The model returns a normalised distribution over candidates and can answer all decisions attached to a state in one forward pass.Source: §3.1 · pp. 3–4

3.2 Model

  • A shared head reads question and candidate marker hidden states and combines per-candidate scores, question–candidate interaction, and candidate-set attention into logits and a softmax distribution.The common objective is cross-entropy against annotation-induced targets; this avoids projecting onto the language-model vocabulary.Source: §3.2 · p. 4

3.3 Training recipe

  • Training first fits the decision head with the backbone frozen, then jointly trains the head and LoRA adapters.Choice candidates are shuffled to prevent reliance on position, whereas Score candidates retain their level order; test prompts keep the question and candidates fixed within each set, with the main answer varying by image.Source: §3.3 · p. 5

4 Data

This section describes how datasets are assigned to training and test roles, how expert labels become typed decisions, and which test sets assess in-domain performance and generalisation to unseen organs.

4.1 Contamination-aware roles

  • Datasets are assigned to train, clean test, or seen-source test roles, with clean meaning no publicly documented overlap with the backbone’s training data.Undisclosed backbone data cannot be excluded, so clean sets favour datasets released after or independently of the backbone.Source: §4.1 · p. 5

4.2 Training and validation data

  • PathJev-P training uses 40,000 patch states from BRACS, SICAPv2, and four SPIDER organ subsets, with questions derived from expert labels.Each state includes source-specific and organ Choice questions, presence questions, and an ordinal Score question; no LLM-generated labels are used.Source: §4.2 · pp. 5–6

4.3 Test data

  • Test states use a fixed class-balanced draw, while cluster-bootstrap groups are patients for BRACS, SICAPv2, and HiESD, slides for SPIDER, and images for HepatoBench.Because HepatoBench has no patient or slide identifiers, treating images as independent units understates uncertainty; the duplicate audit found no training or validation image copies in clean test sets.Source: Table 1 · p. 6, §4.3 · p. 6
  • Liver and stomach test sets probe unseen organs, while BRACS and SICAPv2 provide patient-disjoint in-domain tests and SPIDER is reported separately as a seen-source test.The unseen-organ sets test new organs and partly new vocabularies, not entirely new concepts; SPIDER is only slide-disjoint and may overlap with foundation-model training data.Source: §4.3 · p. 6

5 Experiments

The experiments compare PathJev-P with generative and zero-shot baselines on clean test sets, then examine calibration read-outs and a whole-slide use case. PathJev-P leads the clean-set accuracy comparison, but its calibration advantage depends on the generative read-out, and the slide demonstration is limited in scope.

5.1 Setup

  • All methods answer the same questions on the same images and candidate sets; trained methods were selected on validation NLL, with no test data used for selection.The comparison uses three seeds and an analysis plan fixed before test evaluation.Source: §5.1 · p. 7

5.2 Comparison with the baselines

  • 0.541 balanced accuracy for PathJev-P compares with 0.492 for the generative baseline, 0.345 for CONCH, 0.288 for Inner-Jev and 0.213 for OmniMed-Jev on clean test sets.Its ECE is 0.204 against 0.279 for the generative baseline and 0.403 for CONCH under the planned read-outs.Source: §5.2 · pp. 7–8, Table 2 · p. 8
  • On the two organs absent from training, PathJev-P reaches a balanced accuracy of 0.390 against 0.277 for the generative baseline and 0.328 for CONCH.On in-domain sets, the linear probe is better calibrated (ECE 0.074). On unseen organs, it is fitted on labeled patches from the test domain itself and reaches 0.772.Source: §5.2 · pp. 8–9

5.3 Calibration

  • Length-normalising candidate log-likelihoods makes the generative baseline’s clean ECE 0.201, similar to PathJev-P’s 0.204, and better on unseen organs, where ECE is 0.110 against 0.301.PathJev-P remains over-confident on unseen organs after training-domain temperature scaling; fitting temperature on one unseen organ lowered its NLL on the unseen organs from 2.079 to 1.781, but this is an observation from two organs.Source: §5.3 · p. 9

5.4 Whole-slide demonstration

  • In four unseen-organ whole slides, most fine-grained gastric clusters are not recognised and most clusters abstain, while adenocarcinoma presence separates carcinoma from non-carcinoma tiles on the two slides containing carcinoma.Only tiles annotated at least 90% as one class were evaluated; the metrics were added after inspecting the first slide, and four slides support no claim about diagnostic sensitivity or abstention.Source: §5.4 · p. 10, Figure 3 · p. 10, Table 4 · p. 11

6 Discussion and limitations

PathJev-P is more accurate than the compared generative model, with gains concentrated on organs absent from training, but calibration depends on the generative model’s probability read-out. Its strongest case is open-vocabulary, label-free tissue-cluster labelling; accuracy on fine-grained new-organ classes remains insufficient.

What the comparison shows

  • The accuracy gain over the generative model lies on unseen organs, but differences in objectives, supervised targets, and optimizer steps prevent attributing it to the decision head alone.The authors offer direct training on candidate-set distributions as one untested explanation.Source: §6 · p. 11
  • Calibration superiority depends on read-out: PathJev-P is better under summed log-likelihoods but not under length-normalised probabilities.PathJev-P uses one forward pass; the generative read-out requires one pass per candidate.Source: §6 · p. 11

When to use it

  • For fixed questions with labels, a pathology foundation-model linear probe is better calibrated in every setting and far more accurate with target-domain labels.On new organs, PathJev-P remains over-confident; a training-domain-fitted temperature does not correct this, while labelled data from one new organ corrected most over-confidence on the other in post-hoc analysis.Source: §6 · p. 11
  • Open-vocabulary, label-free whole-slide cluster labelling with abstention is feasible for coarse questions, but fine-grained new-organ accuracy is not yet sufficient.The reported fine-grained new-organ result is 0.390 balanced accuracy.Source: §6 · p. 11

Limitations

  • The study uses single-patch states and only two unseen organs, and it does not systematically evaluate the whole-slide pipeline against pathologists.The authors identify training on mosaics of cluster tiles and validation of the whole-slide pipeline with pathologists as next steps.Source: §6 · p. 11

Data statement

The work uses public datasets under their published licences for research only and does not redistribute them. Some datasets carry non-commercial restrictions, and undocumented backbone training data cannot be ruled out.

Data statement

  • All public datasets were used under their published licences for research only, and the authors do not redistribute them.Figures reproduce example patches and slide maps from SICAPv2 and HiESD, both CC BY 4.0, with attribution.Source: Data statement · p. 12
  • No dataset is documented to contain the backbone’s known histopathology sources, but undisclosed backbone training data cannot be excluded.SPIDER is derived from HISTAI slides and is reported only as a seen-source test.Source: Data statement · p. 12
  • Several datasets have non-commercial licences, including SPIDER and BRACS under CC BY-NC 4.0 and HepatoBench under CC BY-NC 4.0.The table records the most restrictive licence when dataset pages state different licences.Source: Data statement · p. 12

A Analysis protocol

The analysis plan specified hypotheses, endpoints, test sets, tests, and a decision rule before test-set evaluation, with later amendments reported as post hoc where applicable. Statistical comparisons use paired cluster-bootstrap intervals and adjusted one-sided tests, while intervals remain conditional on trained models.

A Analysis protocol

  • The study resamples test sets by their patient, slide, or image clusters and reports marginal 95% percentile intervals of paired differences.The same 10,000 replicates are used for each method and seed.Source: A Analysis protocol · p. 15
  • One-sided bootstrap p-values are Holm-adjusted within comparison families, and claims require a seed-mean effect plus the effect in at least two of three seeds.The intervals are conditional on trained models and do not include retraining variance.Source: A Analysis protocol · p. 15
  • The validation mix used to fit temperature was also used for model selection.Selecting on the main Choice question alone would have selected the same checkpoint and configuration in every run.Source: A Analysis protocol · p. 16
  • The plan’s hypotheses, endpoints, test sets, statistical tests, and decision rule were fixed and hashed before test-set evaluation.Amendments A2 and A3, covering later analyses, were specified after interim or full results and are reported as post hoc.Source: A Analysis protocol · p. 16

B Secondary analysis on the full test files

The appendix repeats the main analysis on full test files, excluding the generative baseline for compute reasons, and separately reports the main Choice question by test set rather than averaging across sets.

B Secondary analysis on the full test files

C Results per test set

D Implementation details

The appendix documents the compared methods’ implementation, training, image preprocessing, and data-overlap provenance. It also specifies how predictions and validation folds were constructed.

D Implementation details

  • 170,390 Gen-SFT examples come from Choice, Noul, and Score decisions; its selected checkpoint saw each decision about once, versus two passes over every PathJev-P state.The compared checkpoints use different numbers of optimiser steps; Gen-SFT candidates are scored by summed token log-likelihood without length normalisation.Source: D Implementation details · p. 17
  • The linear probe selects its scaler, regularisation, and temperature by grouped cross-validation; HepatoBench folds cannot be grouped by slide, so patches from a slide may cross folds.HiESD folds are grouped by patient, and regrouping HepatoBench in consecutive-image blocks changes probe balanced accuracy only slightly.Source: D Implementation details · p. 17
  • PathJev-P and Gen-SFT use the same LoRA-adapted decoder and vision encoder, but differ in loss averaging: per decision for PathJev-P and per answer token for Gen-SFT.The multimodal projector and marker-token embeddings are not adapted; no input was truncated.Source: D Implementation details · pp. 17–18
  • MedGemma-based methods receive whole images resized square to 896 × 896 pixels (aspect ratio not preserved); OmniMed-Jev is first resized to 448 pixels, while CONCH and the probe receive 224 × 224 images; native sizes and fields of view vary by source.Source: D Implementation details · p. 18
  • No overlap with model training data is documented, but for most dataset-model pairs it cannot be excluded.Table 8 distinguishes documented overlap, no documented overlap with listed fine-tuning data, undisclosed training data, and release-date information.Source: D Implementation details · p. 18, Table 8 · p. 18

E Post-hoc robustness analyses (amendment A3)

The A3 appendix reports post-hoc sensitivity analyses whose plan was fixed before they were run. It examines HepatoBench grouping and alternative generative-model read-outs.

E Post-hoc robustness analyses (amendment A3)

  • The A3 analyses were specified after full results and external review, then frozen before being run; Table 10 compares summed, end-token-added, and length-normalised generative read-outs.Table 10 includes PathJev-P as a reference, but the supplied caption gives no result values.Source: E Post-hoc robustness analyses… · p. 18, Table 10 · p. 19
  • HepatoBench probe balanced accuracy was 0.967 with blocks of 50 and 0.959 with blocks of 200, versus 0.971 with image groups; these blocks are not slide- or patient-level groups.Blocks formed within each class can split a slide at a block boundary.Source: E Post-hoc robustness analyses… · p. 19, Table 9 · p. 19
  • GPU reruns changed individual probabilities by up to 0.12, but predicted classes stayed unchanged for at least 97.7% of states per set and macro metrics moved by at most 0.002.Adding the end-of-answer token changed candidate probabilities by at most 2.4 × 10^-6.Source: E Post-hoc robustness analyses… · p. 19

F Secondary and exploratory metrics

Appendix F covers secondary Choice metrics, exploratory Noul and Score decisions, and robustness checks. Appendix G shows the other two slides in the whole-slide demonstration.

F Secondary and exploratory metrics

  • The secondary metrics cover the main Choice question, while Noul and Score results are exploratory and not pre-registered; the organ question is omitted because its answer is in the question text.Tables 16 and 17 report these respective metrics and decisions.Source: F Secondary and exploratory metrics · p. 19, Table 16 · p. 23, Table 17 · p. 24
  • Table 11 tracks prediction-distribution shifts, candidate flips, and “other” predictions across random candidate orders; Table 12 compares OmniMed-Jev at 448- and 896-pixel input sizes.The supplied captions describe these comparisons but provide no result values.Source: Table 11 · p. 20, Table 12 · p. 20
  • The appendix reports Brier scores on unseen organs after temperature scaling and per-test-set calibration before and after scaling.Table 15 covers calibration across clean test sets; Table 14 focuses on unseen organs.Source: Table 14 · p. 21, Table 15 · p. 21

G Whole-slide demonstration: remaining slides

Loading 2610.32478v1…