Source-linked AI summary
CellPath-Bench: A Multidimensional Benchmark for Whole-Slide Cellular Representations in Pathology Foundation Models
Bokai Zhao, Yiyang Zhang, Hanqing Chao, Yawei Ma, Long Bai, Tai Ma, Minfeng Xu, Ming Song, Tianzi Jiang
TL;DR
Existing benchmarks do not systematically measure how cell-type information is encoded and transferred in frozen pathology foundation-model representations. CellPath-Bench addresses this gap with a coordinate-aligned cellular benchmark and standardized linear probes across a multi-organ reference. Its evaluation reveals model-dependent, multidimensional differences in cell-type decodability and cross-domain generalization.
Problem
Existing evaluations do not directly quantify cell-level representation capability in frozen PFMs, because downstream outcomes also depend on aggregation, prediction, adaptation, and optimization.
Method
CellPath-Bench freezes encoders, samples WSI features at registered nuclear coordinates, and evaluates them with standardized multiclass linear probes across hierarchical transfer protocols.
Results
Model-dependent differences in cell-type decodability and cross-domain transferability produce distinct multidimensional capability profiles across 30 pathology-specific and general-purpose foundation models.
Takeaways & Limitations
Separating absolute decodability, spatial-readout sensitivity, and cross-domain generalization provides a more detailed diagnosis of frozen PFM representations than a single downstream task score.
Takeaways & Limitations
CellPath-Bench is limited to frozen linear probing and discrete cell-type classification, and CRT depends on the evaluated model panel and statistical power.
Abstract
from arXiv · showhide
Pathology foundation models (PFMs) are increasingly used as general-purpose backbones, yet existing benchmarks cannot systematically diagnose their whole-slide cellular representation capabilities, including the decodability of cell-type information and the transferability of such information across tissue sections, datasets, and anatomical organs. We introduce CellPath-Bench, a cellular-resolution benchmark that evaluates frozen PFMs themselves. Following quality control of 52 candidate Xenium datasets, we construct a panel of 25 spatially aligned H\&E--Xenium tissue sections spanning 11 organs and 7,079,283 cells, harmonized into fine- and coarse-grained taxonomies. CellPath-Bench samples frozen WSI feature maps at registered nuclear coordinates and evaluates them using standardized multiclass linear probes. Cell Representation Advantage (CRA) measures the within-section advantage of nucleus-anchored representations over patch-level mean pooling, while Cell Representation Transferability (CRT) characterizes the generalization of cell-type decodability across tissue sections, datasets, and organs. We benchmark 30 pathology-specific and general-purpose foundation models through 304,920 runs across spatial readouts, magnifications, taxonomic granularities, and evaluation protocols. The results reveal substantial model-dependent differences in cell-type decodability and its cross-domain generalization, yielding distinct multidimensional capability profiles. CellPath-Bench provides a standardized framework for auditing cellular information in frozen PFM representations.
Introduction
CellPath-Bench addresses the limited direct measurement of cellular information in frozen pathology foundation-model representations. It introduces a coordinate-aligned, multi-organ benchmark that separates cell-type decodability, spatial-readout sensitivity, and cross-domain transferability.
- Motivation: Existing PFM evaluations emphasize patch-, region-, and whole-slide-level tasks, whose outcomes do not directly quantify frozen cell-level representation capability.Such outcomes can depend on spatial aggregation, prediction architecture, adaptation, and optimization, and may exploit aggregated tissue evidence.
- Benchmark design: CellPath-Bench keeps encoders frozen, samples WSI feature fields at registered nuclear coordinates, and applies a unified multiclass linear probe.This design separates frozen representation quality from task-specific encoder adaptation.
- Benchmark design: The benchmark panel contains 25 spatially registered H&E–Xenium sections spanning 11 organs and 7,079,283 cells, constructed from 52 candidate datasets.Cell annotations use harmonized fine- and coarse-grained taxonomies with a standardized molecularly informed cellular reference.
- Evaluation dimensions: CRT evaluates how reliably nucleus-anchored cell-type information generalizes across tissue sections, datasets, and anatomical organs.The cross-domain protocols include intra-organ cross-dataset transfer, multi-organ cross-validation, and leave-one-organ-out transfer.
- Findings: 30 pathology-specific and general-purpose foundation models show substantial variation in cell-type decodability and cross-domain transferability.The joint CRA–CRT landscape indicates that within-section advantage and cross-domain transfer are complementary rather than fully aligned properties.
- Evaluation dimensions: The study benchmarks models through 304,920 controlled linear-probe runs across spatial readouts, magnifications, taxonomic granularities, and evaluation protocols.This multidimensional evaluation is intended to diagnose frozen representations beyond a single downstream task score.
CellPath-Bench
CellPath-Bench audits frozen pathology foundation models through a registered cellular reference, coordinate-aligned feature extraction, and standardized linear probing. It evaluates nucleus-anchored and patch-level representations across spatial, section, dataset, and organ transfer settings.
- Benchmark workflow: CellPath-Bench comprises a registered cellular reference, a coordinate-aligned representation interface, and protocol-specific linear probing.Each tissue section supplies an H&E image, registered nuclear coordinates, and harmonized Xenium-derived cell labels.
- Cellular reference: Annotation uses 14 labeled scRNA-seq references to transfer 68 harmonized molecular cell types to Xenium sections, with fine- and coarse-grained taxonomies for evaluation.Reference matching prioritizes same-patient or tissue-matched data, with organ and disease-state matching otherwise; annotations undergo manual review.
- Linear probing: A new multiclass linear probe is fitted for each model and evaluation condition while the encoder remains frozen, operationalizing linearly decodable cell-type information without encoder adaptation.Conditions vary by magnification, readout, taxonomy, and protocol, with model selection on validation partitions and one evaluation on each test partition.
- Coordinate-aligned readouts: For each frozen encoder, registered nuclear coordinates identify cells within assigned patches, whose spatial token grids yield Nuc, Mean, Cls, and their direct fusions when available.Nuc samples the feature grid at the nuclear center, whereas Mean and Cls summarize patch-level context; class-token readouts are omitted when unsupported.
- Transfer protocols: Four hierarchical protocols measure transfer across spatial regions, tissue sections, datasets, and organs while excluding cells from held-out spatial or anatomical units during probe fitting.The protocols include IS, IOCD, MOCV, and LOOO; protocol summaries balance anatomical or cross-validation units rather than pooling all cells.
- Metrics and analysis: Macro-F1 is the primary metric, Macro-AUROC is secondary, and reported variation reflects organs or folds rather than pooled cells.Model comparisons use paired Wilcoxon signed-rank tests over common valid evaluation units, with significant wins requiring p < 0.05 and a favorable median paired difference.
Experiments and Results
CellPath-Bench evaluates frozen foundation models through cell-level readouts and cross-domain transfer protocols. Nucleus-anchored representations generally preserve more cell-type information than patch-level averaging, while CRA and CRT reveal complementary, model-dependent capabilities.
- Experimental Setup: 30 foundation models were evaluated across four protocols, with runs spanning 3 magnifications, 2 taxonomies, and 7 representation modes.The benchmark’s run calculations use these constant multipliers.
- Experimental Setup: At 20×, Nuc consistently outperformed Mean across both taxonomies and all four protocols, while 10× showed clearer performance decline.Under IS, retention relative to 40× was 99.6% at 20× and 92.9% at 10×.
- Within-Section Evaluation of Cell Representation Advantage: CRA was positive for all 30 models, and H-Optimus-1 achieved the highest CRA at 10.14±6.05%.CRA quantifies the advantage of nucleus-anchored representations over patch-level mean pooling.
- Cross-Domain Evaluation of Cell Representation Transferability: IOCD, MOCV, and LOOO measure transfer to held-out sections within known organs, after multi-organ training, and to unseen organs, respectively.Their absolute scores are interpreted within rather than directly across protocols.
- Cross-Domain Evaluation of Cell Representation Transferability: CRT ranges from 0 to 29 and summarizes panel-relative significant-win counts across the three cross-domain protocols, rather than absolute transfer performance.Its value depends on the evaluated model panel, matched evaluation units, and statistical power.
- Cross-Domain Evaluation of Cell Representation Transferability: SEAL(UNI2) and UNI2 achieved the highest CRT at 26.33, followed by H-Optimus-1 at 25.33 and H-Optimus-0 at 24.33.CRA and CRT were not fully aligned: Virchow2 ranked second in CRA but had CRT 16.33, whereas H-Optimus-1 performed strongly on both dimensions.
Discussion and Conclusion
CellPath-Bench evaluates cellular information in frozen pathology foundation models through standardized, coordinate-aligned linear probing. Its multidimensional measures distinguish local cell-type decodability, nucleus-anchored readout advantage, and cross-domain generalization, while remaining limited in task scope.
- CellPath-Bench aligns WSI feature maps with registered nuclear locations and applies standardized linear probes across taxonomies, readouts, resolutions, and evaluation settings.
- Cell-level representation capability is multidimensional and cannot be adequately summarized by a single score.Nuc Macro-F1 measures linearly accessible cell-type information, whereas CRA and CRT characterize readout advantage and cross-domain competitiveness.
- Together, Nuc, CRA, and CRT provide complementary views of local cellular information and its generalization.
- CellPath-Bench remains limited to frozen linear probing and discrete cell-type classification, while CRT depends on the evaluated model panel and statistical power.
- CellPath-Bench establishes a scalable framework for characterizing where cell-type information is encoded and how reliably it transfers across biological domains.
A. Evaluated Foundation Models
The benchmark evaluates 30 foundation models, as listed in Table 6.
- Table 6 lists the 30 foundation models evaluated in CellPath-Bench.
B. Dataset Composition and Cell-Type Annotation
The dataset composition comprises a quality-controlled collection of Xenium slides, organized through ranked panel selection and hierarchical cell-type annotation. The benchmark uses a three-level taxonomy spanning biological and quality-control labels.
- The full collection comprises 52 Xenium slides, with 51 retaining at least two Coarse classes after label mapping.The acute lymphoid leukemia section is excluded from ranking because its labels collapse to a single Coarse class.
- The annotation taxonomy contains 72 labels organized into three levels, including 68 biological cell types and four state or quality-control labels.
- The taxonomy includes epithelial, immune, stromal, blood-vascular, and other quality-control categories with marker definitions for biological cell types.
C. Complete Experimental Results
The complete experimental results are organized across panels, magnifications, metrics, taxonomies, readouts, and CRA-based comparisons. Tables use organ-balanced aggregation and report extensive intra-section results across the benchmark conditions.
- The complete IS results cover panels S25, S20, S15, and S10, magnifications 40×, 20×, and 10×, three metrics, and two taxonomies.
- Tfine is a 9-class taxonomy and Tcoarse its 3-class coarsening, with class-eligibility gates that vary the number of scored classes by evaluation unit.
- Nuc samples the registered nuclear centre, Mean averages patch tokens, CLS uses the native class token, and fusion readouts combine these representations.
- Every value is an organ-balanced mean, while reported dispersion is the sample standard deviation across sections and therefore reflects between-section variability.
- CRA is the paired nucleus-minus-mean-pooling difference computed across model, taxonomy, section, and fold units.
- Significant wins count models significantly exceeded in CRA using paired two-sided Wilcoxon signed-rank tests at p < 0.05 with a positive median difference.