Source-linked AI summary
Pathology foundation models for training-free case retrieval in salivary gland tumors
Maximilian J. Roesel, Niklas Kiermeyer, Philipp Keyl, Frederick Klauschen
TL;DR
Rare tumors may fall outside an individual pathologist's experience, while departmental archives can hold relevant precedents that are difficult to search. The study uses pathology foundation models to retrieve similar cases from an archive, with manually or automatically selected regions of interest. On salivary gland tumors, retrieval found the correct diagnosis among the top five cases in up to 81.3% of cases without task-specific training.
Problem
Individual pathologists encounter rare entities only occasionally, and searching institutional WSI archives for relevant precedents is challenging because of their diverse morphology and size.
Source: Introduction · p. 2Method
The approach uses pathology foundation model representations to retrieve archive cases matching a manually selected or ABMIL-selected region of interest.
Source: Results · p. 6, Discussion · p. 9Results
Up to 81.3% of cases had the correct diagnosis among the top five retrieved candidates, with no statistically significant model differences at any depth.
Source: Results · p. 8, Discussion · p. 9Takeaways & Limitations
Pretrained foundation models can extend an individual pathologist's frame of reference to that of the department.
Source: Discussion · p. 10All cases came from one institution and were digitized on the same scanners, limiting external validity.
Source: Discussion · p. 10
Abstract
from arXiv · showhide
In pathology, pattern recognition relies on a frame of reference built through extensive training and experience, which may still fall short when encountering rare cancer subtypes. Archives of previously diagnosed tumors often contain relevant precedents, yet identifying the most relevant cases remains challenging. Here we explore whether content-based retrieval with current pathology foundation models facilitates rare cancer diagnostics without any task-specific training. Pathologists can query a digitized archive using a region of interest (ROI) from a whole slide image (WSI) and retrieve morphologically similar cases, along with their ICD-O codes and corresponding WSIs. The ROI can be selected manually or automatically using a gated attention-based multiple instance learning model. We evaluate our approach by leave-one-case-out on an imbalanced dataset of 137 salivary gland tumor cases spanning 12 ICD-O morphology categories, comparing four pathology foundation models (Atlas2, Atlas2-B, Atlas2-S, and UNI2h). Top-1/3/5 retrieval was 62.5%/76.7%/81.3% for Atlas2, 50.2/66.6/72.9% for UNI2h, 52.3%/71.5%/77.6% for Atlas2-B and 54.4/65.2/73.7% for Atlas2-S. Macro-averaged F1 ranged from 69.0% for the smallest model Atlas2-S to 76.6% for the largest model Atlas2 across the five classes with at least ten cases. Our modified Chamfer similarity separated correct from incorrect matches among the Top-5 more reliably for Atlas2 (AUC 0.754) than for Atlas2-B (0.679, p = 0.020) and Atlas2-S (0.653, p = 0.003), but not UNI2h (0.697, p = 0.196). Requiring no additional training or manual annotation, this approach makes departmental archives searchable with pathology foundation models, providing a low-barrier route to case-based differential diagnosis and extending the individual frame of reference to the collective experience of the department.
Introduction
Individual pathologists encounter rare entities only occasionally, while departmental archives accumulate precedents that are difficult to search. This study explores training-free content-based retrieval with pathology foundation models as a way to search those archives.
Introduction
- Departmental WSI archives can provide an external frame of reference, but their diverse morphology and size make relevant cases difficult to find.Rare entities are seldom seen by any one pathologist, even though they collectively form a substantial part of the diagnostic spectrum.Source: Introduction · p. 2
- Foundation models learn general-purpose representations from large slide corpora, while content-based retrieval returns cases rather than classifier-generated labels.Unlike classifiers, this retrieval approach does not require enough labeled cases per class to learn a decision boundary.Source: Introduction · p. 2
- The proof-of-concept evaluates four pathology foundation models on 137 salivary gland tumor cases across 12 ICD-O morphology categories, using pathologist- or ABMIL-selected queries.The study focuses on a rare, morphologically challenging tumor group; initial histological assessment can guide ancillary testing even when morphology cannot reliably separate some entities.Source: Introduction · p. 2
Methods
The methods combine frozen pathology foundation models, an attention-based model for selecting tumor regions, and approximate patch retrieval followed by case-level ranking. Leave-one-out and bootstrap procedures assess retrieval performance and model differences.
Methods
- The cohort comprised 137 tumor cases and 974 H&E WSIs; four frozen DINOv2-pretrained pathology models encoded patches, with no stain normalization applied.Slides were tiled into nonoverlapping 512×512-pixel patches at 20×, each covering 256×256 µm; patches required at least 50% artifact-free tissue.Source: Methods · p. 2, Feature extraction · pp. 2–3
- The gated ABMIL model formed bag representations with softmax-normalized attention weights, then used bag-level binary cross-entropy and two auxiliary losses to align attention with classifier-ranked tumor patches.The second auxiliary loss compared top-k positive-bag patches with sampled confirmed negatives; hyperparameters were tuned jointly by Bayesian optimization before five-fold cross-validation.Source: ABMIL model · pp. 3–4
- Tumor scores and attention weights were smoothed over 3×3 neighborhoods to select spatially coherent 2×2 and 4×4 ROIs; query regions were scale-matched and pooled according to descriptor similarity.Homogeneous queries were averaged into one vector, while heterogeneous queries were greedily clustered using a cosine-similarity threshold of 0.8.Source: ABMIL model · p. 4
- Each model's FAISS index used L2-normalized embeddings and IVFPQ, partitioning vectors into Voronoi cells and quantizing residual sub-vectors for 64× compression.Vectors were inserted in batches after training the IVF centroids and PQ codebooks on a random corpus subsample.Source: ABMIL model · p. 4
- Each query vector searched nearby FAISS cells, with the number of probed cells set dynamically and a default patch-level candidate pool of 150,000.The search balances recall against latency by limiting the cells scanned rather than comparing against every database vector.Source: ABMIL model · p. 5
- Cases were ranked with a sparse Chamfer score combining query coverage and support across retrieved patches, with the backward term adjusted to favor cases containing more retrieved patches.The score was computed only over retrieved patches; its saturating adjustment approaches its plateau at around 15 patches.Source: ABMIL model · p. 5
- Leave-one-case-out evaluation measured macro-averaged recall at Top-1/3/5 and, for five classes with at least ten cases, precision, recall, and F1 at Top-1.Model comparisons used paired t-tests with Holm correction; Chamfer AUC and Top-1 metrics used case-level bootstrap comparisons, while class-level hits were tested against chance for seven classes with at least five cases.Source: ABMIL model · p. 5
Results
The search engine retrieves morphologically similar archive cases from manually or automatically selected ROIs, and reports diagnoses alongside the matched slides. Across the evaluated settings, retrieval performance varied by indexing model and morphology, while Chamfer scores distinguished correct from incorrect matches.
Results
- The engine accepts a manual or ABMIL-selected ROI, adaptively pools patch embeddings, and ranks archive cases by modified Chamfer similarity, showing each diagnosis and WSI.Users can inspect the patches driving a match in their original slide locations.Source: Results · p. 6, Figure 1 · p. 6
- Neither the FM used to source automated ABMIL ROIs nor query size produced a relevant retrieval-accuracy difference; 4×4 queries were selected for later analyses because their field of view better matched pathologist assessment.Across 16 configurations, Top-1 recall was 50.3% for 4×4 versus 48.3% for 2×2 (p = 0.110); Top-3 and Top-5 were also similar.Source: Results · p. 7, Figure 2 · p. 7
- 53.5% Top-1 recall for Atlas2 exceeded Atlas2-S (47.4%, p = 0.029) and UNI2h (47.9%, p = 0.029), but not Atlas2-B (48.5%, p = 0.074), pooled across query sizes.There was no significant difference at Top-3 or Top-5 among the four models.Source: Results · p. 7, Figure 2 · p. 7
- 62.5%/76.7%/81.3% Top-1/3/5 macro-averaged recall for Atlas2 led the pathologist-selected query results, with no statistically significant differences between models at any depth.UNI2h achieved 50.2%/66.6%/72.9%, Atlas2-B 52.3%/71.5%/77.6%, and Atlas2-S 54.4%/65.2%/73.7%.Source: Results · p. 8, Figure 3 · p. 8
- 76.6% macro-averaged F1 for Atlas2 was highest across the five classes with at least ten cases, followed by Atlas2-B (74.0%), UNI2h (71.8%), and Atlas2-S (69.0%), without significant differences.Adenocarcinoma was under-retrieved rather than over-called; its F1 ranged from 0.36 for Atlas2-S to 0.62 for Atlas2.Source: Results · p. 8, Figure 3 · p. 8
- Four of seven classes with at least five cases exceeded chance at Top-1 and Top-3 across all four models (p < 0.05), while three only tended to exceed chance.The three were adenocarcinoma, malignant myoepithelioma, and infiltrating duct carcinoma.Source: Results · p. 8, Figure 3 · p. 8
- Atlas2's Top-5 Chamfer-score AUC (0.754) exceeded Atlas2-B (0.679, p = 0.020) and Atlas2-S (0.653, p = 0.003), but not UNI2h (0.697, p = 0.196).All models' Chamfer scores significantly discriminated correct from incorrect retrievals; within the Atlas family, Atlas2-S had the highest mean score (1.49) but weakest retrieval, while Atlas2 had the lowest (1.34) and strongest retrieval.Source: Figure 4 · p. 9, Results · p. 9
Discussion
Off-the-shelf pathology foundation models retrieved the correct diagnosis among the top five cases in up to 81.3% of cases, without task-specific training or manual annotation. The discussion presents retrieval as a way to extend departmental case experience while noting constraints from diagnostic labels, cohort scope, and automated query selection.
Discussion
- Up to 81.3% of cases had the correct diagnosis among the top five candidates, without task-specific training or manual annotation.The evaluation used salivary gland tumors, a rare and diagnostically challenging group.Source: Discussion · p. 9
- Top-1 recall varied across models from 50–63% and narrowed at Top-5 to 73–81%, with no significant differences at any depth.Macro-averaged F1 ranged from 68.8% for Atlas2-S to 76.6% for Atlas2; Atlas2 ranked better than Atlas2-B and Atlas2-S, but not UNI2h, on Chamfer ranking.Source: Discussion · p. 9
- The retrieval design limits query-side computation with selected regions while indexing all artifact-free archive patches, balancing speed and accuracy through modified Chamfer similarity.The authors note that this approach still leaves room for intra-tumor heterogeneity.Source: Discussion · p. 10
- A shortlist may reduce confirmatory anchoring compared with a single-label aid and guide ancillary testing, while prior oral-pathology retrieval was reported to improve accuracy most in underrepresented categories (34).Salivary gland neoplasms can be ambiguous; a Dutch expert review reported 8.3% discordance (33).Source: Discussion · p. 10
- The cohort did not cover the full morphological spectrum and came from one institution using the same scanners, limiting external validity.The authors also note that approximate nearest-neighbor search does not guarantee the mathematically most similar case, and automatic WSI-to-WSI retrieval performed below pathologist-selected queries, likely due to limited training data.Source: Discussion · p. 10
- Future work will test whether the tool measurably improves pathologists’ performance and how it scales to larger cohorts.The authors conclude that pretrained models can extend an individual pathologist’s frame of reference to the department’s experience.Source: Discussion · p. 10
Funding
The project received support from the Bavarian Cancer Research Center (BZKF).
Funding
- The project received support from the Bavarian Cancer Research Center (BZKF).Source: Funding · p. 11