PaperlayerView exact revision on arXiv

Source-linked AI summary

VISTA-PATH: An interactive foundation model for pathology image segmentation and quantitative analysis in computational pathology

Peixian Liang, Songhao Li, Shunsuke Koga, Yutong Li, Zahra Alipour, Yucheng Tang, Daguang Xu, Zhi Huang

arXiv:2601.16451v1cs.CV

TL;DR

Pathology segmentation models remain limited in their ability to generalize across heterogeneous tissues and support interactive expert refinement. VISTA-PATH addresses this with joint semantic–spatial conditioning and achieves stronger segmentation benchmarks while producing clinically meaningful tumor-interaction signals associated with survival.

  • Problem

    Existing pathology segmentation pipelines are often organ-, tissue-, or dataset-specific, limiting generalization across heterogeneous pathology images and increasing annotation burden.

  • Method

    VISTA-PATH jointly conditions segmentation on class-aware semantic and spatial prompts and is trained on over 1.6 million masks spanning 93 tissue classes across 9 organs.

  • Results

    VISTA-PATH achieves the highest average Dice scores across held-out internal datasets and produces a Tumor Interaction Score strongly associated with patient survival.

  • Takeaways & Limitations

    VISTA-PATH supports interactive pathology segmentation and translates biologically faithful pixel-level tumor maps into clinically interpretable survival-related signals.

  • Takeaways & Limitations

    VISTA-PATH currently supports patch-level corrections but not richer expert interactions such as region marking, uncertainty annotation, or iterative hypothesis testing across slides.

Abstract

from arXiv · show

Accurate semantic segmentation for histopathology image is crucial for quantitative tissue analysis and downstream clinical modeling. Recent segmentation foundation models have improved generalization through large-scale pretraining, yet remain poorly aligned with pathology because they treat segmentation as a static visual prediction task. Here we present VISTA-PATH, an interactive, class-aware pathology segmentation foundation model designed to resolve heterogeneous structures, incorporate expert feedback, and produce pixel-level segmentation that are directly meaningful for clinical interpretation. VISTA-PATH jointly conditions segmentation on visual context, semantic tissue descriptions, and optional expert-provided spatial prompts, enabling precise multi-class segmentation across heterogeneous pathology images. To support this paradigm, we curate VISTA-PATH Data, a large-scale pathology segmentation corpus comprising over 1.6 million image-mask-text triplets spanning 9 organs and 93 tissue classes. Across extensive held-out and external benchmarks, VISTA-PATH consistently outperforms existing segmentation foundation models. Importantly, VISTA-PATH supports dynamic human-in-the-loop refinement by propagating sparse, patch-level bounding-box annotation feedback into whole-slide segmentation. Finally, we show that the high-fidelity, class-aware segmentation produced by VISTA-PATH is a preferred model for computational pathology. It improve tissue microenvironment analysis through proposed Tumor Interaction Score (TIS), which exhibits strong and significant associations with patient survival. Together, these results establish VISTA-PATH as a foundation model that elevates pathology image segmentation from a static prediction to an interactive and clinically grounded representation for digital pathology. Source code and demo can be found at https://github.com/zhihuanglab/VISTA-PATH.

1 Main

VISTA-PATH is an interactive, class-aware pathology segmentation foundation model designed to address heterogeneous tissue segmentation, expert-guided refinement, and clinically grounded analysis. It is supported by a 1.6-million-triplet corpus spanning 9 organs and 93 tissue classes, and yields improvements over existing foundation models while enabling prognostically associated Tumor Interaction Scores.

  • Pathology segmentation supports tumor delineation, microenvironment profiling, biomarker discovery, and prognostic modeling, but existing models remain narrowly focused on organs, tissues, or datasets.
  • Radiology- and natural-image-oriented foundation models improve generalization but do not fully address heterogeneous patterns in H&E-stained histopathology images.
  • VISTA-PATH unifies heterogeneous pathology segmentation, expert-guided refinement, and clinically grounded tissue analysis within an interactive, class-aware framework.
  • VISTA-PATH Data: Over 1.6 million image–mask–text triplets span 9 organs and 93 tissue classes in VISTA-PATH Data, capturing intra-organ and inter-organ morphological diversity.
  • VISTA-PATH shows robust and consistent improvements over existing segmentation foundation models, while Tumor Interaction Score derived from its tissue maps has strong and significant associations with colon cancer patient survival.

2 Results

VISTA-PATH addresses pathology segmentation’s heterogeneity through class-aware semantic–spatial conditioning and demonstrates strong performance across internal, external, and interactive evaluations. Its high-fidelity segmentations also support clinically interpretable tumor morphology analysis through the Tumor Interaction Score.

  • Dataset construction: VISTA-PATH Data aggregates pathology segmentation resources from 22 publicly available datasets to support segmentation across diverse organs and tissue types.The corpus addresses the scarcity and fragmentation of pathology datasets, whose challenges include ultra-high-resolution whole-slide images, morphological diversity, and variable semantic definitions.
  • Model design: VISTA-PATH jointly conditions segmentation on class-aware text prompts and spatial prompts to resolve class ambiguity in overlapping heterogeneous tissue regions.This unified vision–language architecture addresses fragmented predictions and misclassification that arise when class-isolated masks or semantic prompts alone are insufficient.
  • Held-out evaluation: 0.006–0.414 Dice: VISTA-PATH’s average gains over MedSAM across held-out datasets accompanied its highest average Dice scores and reduced variance versus MedSAM and BiomedParse.The held-out evaluation measures in-distribution multi-class segmentation under controlled but heterogeneous conditions.
  • External evaluation: 0.495 Dice: VISTA-PATH on LungHP25 exceeded MedSAM’s 0.121 and BiomedParse’s 0.078, while reaching 0.802 on OCDC26 versus 0.528 and 0.203, respectively.These external results demonstrate generalization on low-sample, low-class-count pathology data.
  • Human-in-the-loop refinement: 0.745 Dice: VISTA-PATH on Lung-Immuno exceeded the patch classifier’s 0.716 and MedSAM’s 0.626; on Breast-5K, it achieved 0.693 versus 0.610 and 0.578.Across unseen Visium HD and Xenium datasets, iterative refinement rapidly increased patch-level accuracy, with performance usually saturating after 4–5 iterations using about 1,000 annotated patches.
  • Clinical morphology analysis: The Tumor Interaction Score quantifies tumor coherence versus infiltration from pixel-level and patch-level tumor predictions, with high values indicating cohesive architecture and low values capturing fragmentation or invasive growth.TIS translates tumor spatial organization into a single explainable, clinically interpretable morphological index.

3 Discussion

The discussion presents VISTA-PATH as an interactive, clinically grounded pathology segmentation foundation model that integrates semantic–spatial reasoning, expert feedback, and downstream clinical inference. It also identifies richer pathologist interaction cues as an important direction for future development.

  • Core contribution: VISTA-PATH is presented as an interactive, clinically grounded segmentation foundation model supporting expert reasoning, iterative refinement, and downstream clinical inference.The authors contrast it with models treating segmentation as static visual prediction and emphasize heterogeneous tissue resolution, expert feedback, and interpretable outputs.
  • Semantic–spatial modeling: Its joint semantic–spatial modeling represents both the tissue present and its location, addressing densely interwoven, visually similar structures with distinct biological meanings.The discussion highlights tumor epithelium, stroma, immune infiltrates, necrosis, and normal structures as compositional pathology elements.
  • Interactive refinement: Sparse corrections on the order of 10–1,000 patches can be propagated into high-fidelity, whole-slide, pixel-level outputs.VISTA-PATH interprets bounding boxes as class-conditioned spatial signals rather than merely geometric constraints.
  • Clinical inference: The Tumor Interaction Score operationalizes segmentation outputs as clinically meaningful biomarkers for downstream inference.It combines pixel-level tumor maps with patch-level tumor localization to quantify tumor coherence versus infiltration.
  • Limitations and future directions: The discussion identifies richer interaction cues—including region marking, uncertainty annotation, and iterative hypothesis testing across multiple tissues—as remaining challenges.These cues extend beyond patch-level corrections and motivate future directions for interactive digital pathology.

4 Methods · 4.1 VISTA-PATH Data: dataset for pathology segmentation · 4.2 External Datasets

The methods establish VISTA-PATH Data through stringent curation and standardization of heterogeneous pathology segmentation resources, then evaluate generalization on independent pathology and spatial-transcriptomic datasets. External annotations are converted into semantically labeled pixel-level masks through tissue-aware clustering, expert review, and spatial registration.

  • 4.1 VISTA-PATH Data: dataset for pathology segmentation: VISTA-PATH Data integrates 22 publicly available segmentation resources from multiple platforms into a unified pathology segmentation corpus.Sources include TCIA, Kaggle, Grand Challenge, Scientific Data, CodaLab, and MICCAI-hosted challenges.
  • 4.1 VISTA-PATH Data: dataset for pathology segmentation: Images were included only when manually or semi-manually segmented at pixel level and annotated objects had clearly defined names.These criteria were used to enforce quality and consistency across diverse data sources.
  • 4.1 VISTA-PATH Data: dataset for pathology segmentation: The preprocessing pipeline standardized heterogeneous image formats, resolutions, masks, and class labels for consistent model training and downstream processing.Images were converted to a standardized three-channel representation; polygon annotations were rasterized into spatially aligned masks, and labels were manually verified and standardized.
  • 4.1 VISTA-PATH Data: dataset for pathology segmentation: Samples were split 95% for training and 5% for testing at the sample level, while preserving multiple magnifications and cropping images into 1024×1024-pixel patches.The split and patching strategy addressed variable acquisition magnifications and image sizes.
  • 4.2 External Datasets: External evaluation used unseen datasets from independent sources spanning different imaging conditions, organs, and tissue types to assess robustness and transferability under domain shifts.Two additional pathology segmentation datasets, LungHP and OCDC, were processed with the same preprocessing pipeline as VISTA-PATH Data.
  • 4.2.2 Visium HD Datasets: Five Visium HD datasets from 10x Genomics were clustered by molecular features, manually assigned semantic tissue labels by pathologists, and converted to pixel-level masks using 8 µm×8 µm bins.The datasets included Colon, Kidne, Lung, Pancreas, and Prostate.
  • 4.2.3 Xenium Datasets: Twenty-two Xenium datasets were modeled at single-cell resolution using CellCharter neighborhood embeddings, MiniBatch K-means clustering, and projection onto co-registered H&E images.CellCharter embeddings used L = 3 aggregation layers; pathologists labeled clusters, and labels were assigned to pixels within cell contours.

4.3 Details of VISTA-PATH model architecture

VISTA-PATH combines image, text, spatial-prompt, and mask-decoding modules to fuse pathology context with semantic descriptions and optional bounding-box guidance. Its decoder progressively upsamples fused features to produce dense pixel-level class predictions.

  • Architecture: VISTA-PATH comprises an image encoder, text encoder, bounding-box prompt encoder, and mask decoder.The model supports interactive visual prompts and text prompts.
  • Image encoder: The image encoder uses pretrained PLIP features, projecting 49 patch embeddings from dimension 768 into a shared dimension of 512.The 49 patches correspond to a 7×7 grid after removing the class token.
  • Text encoder: The PLIP text encoder processes 77-token class descriptions and projects their 768-dimensional embeddings into the shared dimension of 512.The resulting text representations align with image embeddings for cross-modal interaction.
  • Cross-modal fusion and context modeling: Cross-attention first fuses visual tokens with textual cues, then injects localized spatial information from the SAM-derived bounding-box prompt embedding.The fused representation is refined by four Transformer encoder layers into Ffinal ∈R512×7×7.
  • Mask decoder: The mask decoder upsamples fused features from 7×7 to 224×224 through four bilinear-upsampling and Conv(3×3)–BN–ReLU stages, producing dense per-pixel predictions.Channel widths progress as 512→256→128→64→32 before a final 1×1 convolution generates K class logits.

4.4 Details of VISTA-PATH model training

VISTA-PATH is trained for grounded binary segmentation from text prompts, optionally augmented with bounding-box spatial priors. Its training uses frozen-text mixed-precision optimization initialized from PLIP, with specified computational settings.

  • Grounded segmentation objective: Training defines each prompted class’s ground-truth mask as binary pixels and predicts foreground/background logits at every pixel.Pixels belonging to the specified class are labeled 1, while all others are labeled 0; the model outputs two logits per pixel.
  • Grounded segmentation objective: The model minimizes softmax cross-entropy over per-pixel foreground and background class probabilities.The probabilities are denoted pfg,ij and pbg,ij for foreground and background, respectively.
  • Spatial prompting: Bounding-box prompts provide an additional spatial prior during forward propagation without changing the training objective.The same objective is used whether or not a bounding-box prompt is available.
  • Optimization and implementation: The image and text encoders are initialized from PLIP, with the text encoder frozen during mixed-precision FP16 training.Optimization uses a learning rate of 5×10−5 for 10 epochs and a batch size of 512.
  • Optimization and implementation: Training completes in approximately 25 hours on one NVIDIA H200 GPU, while inference uses about 2.4 GB of memory on one H200 GPU.These are the reported training and evaluation hardware settings.

4.5 Evaluation metrics and statistical analysis

The section defines Dice-based evaluation for binary and multi-class segmentation and describes bootstrap confidence intervals, hypothesis testing, significance thresholds, and survival-plot analysis.

  • Evaluation metrics: Segmentation is assessed with the Dice coefficient.For binary tasks, Dice is computed per image and averaged over the test set; for multi-class tasks, Dice is computed per class and averaged across classes.
  • Evaluation metrics: Empty-mask cases are assigned Dice_n=1 when both prediction and ground truth are empty, and Dice_n=0 when only one is empty.
  • Statistical testing: Performance is summarized by mean per-image Dice with 95% confidence intervals from percentile bootstrap using 10,000 resamples.
  • Statistical testing: Significance uses a two-sided Student’s t-test on per-image Dice differences, with thresholds of ∗P < 0.05, ∗∗P < 1×10^-2, and ∗∗∗P < 1×10^-3.
  • Statistical testing: Kaplan–Meier survival-plot P values are computed with the log-rank test using a chi-square approximation.

4.6 Details of experiments on internal segmentation evaluation

The internal segmentation evaluation reports results for every benchmark dataset and summarizes macro means across datasets. It also aggregates performance by organ using mean Dice scores with bootstrap confidence intervals to assess anatomical generalization.

  • Results are reported separately for every dataset in the internal benchmark, with macro means summarized across datasets.The macro mean is averaged across datasets and presented in Extended Data Table 3.
  • Performance is aggregated by organ to assess how well the model handles different anatomical contexts.Organ-level aggregation covers all images from datasets associated with each organ.
  • Organ-level performance is measured using mean Dice scores with 95% percentile-bootstrap confidence intervals.The intervals are estimated from 10,000 resamples over images belonging to datasets of each organ.

4.7 Details of experiments on external segmentation evaluation · 4.8 Details of experiments on Human-in-the-loop study · 4.9 Details of experiments on TCGA-COAD cohort study

The experiments assess VISTA-PATH on unseen external datasets, iterative whole-slide human refinement, and TCGA-COAD survival analysis using the Tumor Interaction Score (TIS). Together, these protocols examine generalization, interactive correction, and clinically interpretable tumor–microenvironment quantification.

  • 4.7 Details of experiments on external segmentation evaluation: External evaluation uses zero-shot inference on datasets unseen during training, with identical prompts and inference protocols and per-image Dice reporting.No fine-tuning or dataset-specific adaptation is performed.
  • 4.7 Details of experiments on external segmentation evaluation: External results are additionally summarized by organ-wise mean Dice and class-wise mean Dice across tumor-related, microenvironment-related, and normal anatomical tissues.Labels are standardized into the three stated categories before class-wise evaluation.
  • 4.8 Details of experiments on Human-in-the-loop study: The human-in-the-loop framework treats segmentation as iterative review, enabling pathologists to correct errors while supporting refinement at the whole-slide image level.The framework is motivated by the need for reliable downstream analysis from whole-slide images.
  • 4.8 Details of experiments on Human-in-the-loop study: Whole-slide images are tiled into fixed-size patches, MUSK generates patch embeddings and initial classification-based segmentation, and pathologists provide localized corrections for selected subregions.Corrected patch-level annotations are collected for subsequent lightweight model training.
  • 4.8 Details of experiments on Human-in-the-loop study: Refinement ends when pathologists judge the segmentation satisfactory, with iterative rounds and per-round patch-level annotations summarized in the cited figures and tables.The reported summaries are Fig. 4b–c and Extended Data Tables 9–10.
  • 4.9 Details of experiments on TCGA-COAD cohort study: The TCGA-COAD study motivates precise segmentation and morphological measurement as prerequisites for meaningful, interpretable survival signals from pathological structures.The stated focus includes tumor-region identification, morphological assessment, and quantitative analysis.
  • 4.9 Details of experiments on TCGA-COAD cohort study: TIS quantifies tumor solidity and interaction with other microenvironments through patch-level tumor prediction followed by pixel-level tumor segmentation.It is designed to measure the density and cohesiveness of the tumor component.
  • 4.9 Details of experiments on TCGA-COAD cohort study: Higher TIS values indicate more well-demarcated solid tumor regions associated with reduced invasiveness and improved patient outcomes, and are evaluated against ABMIL and MedSAM baselines.ABMIL uses MUSK patch embeddings, while MedSAM produces segmentation-based TIS scores for survival prediction.

Data Availability

The study uses DROID data from AIDA under a permissive research license, with access information provided online. DROV is available for lawful and ethical medical diagnostics research, subject to access requirements.

  • Data access: DROID data were obtained from the Analytic Imaging Diagnostics Arena (AIDA) and are available through the AIDA Data Hub.The source specifies https://datahub.aida.scilifelab.se/10.23698/aida/drsk.
  • Data access: The DROID license permits use, modification, and redistribution for medical diagnostics research when original copyright and permission notices are retained.
  • Data access: DROV is free to use for legal and ethical medical diagnostics research, although accessing the dataset requires additional authorization.The supplied passage states that access requires further conditions but does not specify them.

Ethics declarations

The ethics declaration reports that Yucheng Tang and Daguang Xu are employees of Nvidia Corporation in the United States.

  • Ethics declarations: Yucheng Tang and Daguang Xu are employees at Nvidia Corporation, USA.

Author contributions statement

The authors contributed across conceptualization, methodology, experiments, data curation, pathology annotation, writing, and supervision, and all authors approved the final manuscript.

  • Contributions: Peixian Liang and Zhi Huang led conceptualization, manuscript writing, and supervision, with Huang also supervising the work.Liang and Huang handled conceptualization and writing; Huang provided supervision.
  • Contributions: Methodology involved Peixian Liang, Songhao Li, Shunsuke koga, Yutong Li, and Zhi Huang.These authors were specifically credited for methodological development.
  • Contributions: Experiments were conducted by Peixian Liang, Songhao Li, Shunsuke koga, Yucheng Tang, Daguang Xu, and Zhi Huang.The experimental contribution included six authors.
  • Contributions: Peixian Liang, Yutong Li, and Zhi Huang handled data curation and acquisition, while Shunsuke Koga and Zahra Alipour provided expert pathology annotation, interpretation, and feedback.The statement distinguishes data-related work from expert pathology contributions.
  • Contributions: All authors reviewed and approved the final manuscript.Final approval was collective across the author group.

Survial time and event: 5.65, Death

Extended Data Figure 4 illustrates how low, intermediate, and high Tumor Interation Score (TIS) cases differ in tumor segmentation overlays from TCGA colorectal cancer patches. It also clarifies that risk scores rank relative survival risk independently of sign, with higher values indicating worse outcomes.

  • Extended Data Figure 4 presents three representative H&E patches with corresponding tumor segmentation overlays spanning low, intermediate, and high Tumor Interation Score (TIS).
  • Risk score represents a relative risk ranking derived from the survival model and is independent of sign.Higher risk values indicate worse survival outcomes.
  • The low TIS case is characterized by pronounced tumor budding and infiltrative growth.
  • TNM denotes Tumor–Node–Metastasis staging in the illustrated TCGA colorectal cancer cases.
Loading 2601.16451v1…