PaperlayerView exact revision on arXiv

Source-linked AI summary

FTU-Seek: Foundation Model-Guided Hard-Negative Learning for Sparse Functional Tissue Unit Segmentation

Zonghao Liu, Lei Su, Jiguang Yu, Xuqing Geng, Louis Shuo Wang, Jianmin Wang, Jingfeng Liu

arXiv:2609.00704v1cs.CVmath.NA

TL;DR

Sparse, heterogeneous FTUs are difficult to segment because target-containing patches are rare and target-absent tissue contains both redundant and confusing background. FTU-Seek uses frozen pathology foundation-model features to rank target-like negatives and select compact TopK training sets; across three FTU tasks and a held-out TLS cohort, it maintained competitive segmentation while reducing retained workload, though broader validation and direct computational benchmarking remain needed.

  • Problem

    Sparse FTUs are surrounded by abundant background containing redundant easy negatives and rare morphologically confusing regions, complicating efficient WSI segmentation.

  • Method

    FTU-Seek trains a patch-level classifier on frozen multi-depth pathology foundation-model features, ranks annotated FTU-absent patches, and selects static per-WSI TopK hard negatives.

  • Results

    Across TLS, blood-vessel, and gland tasks, FTU-Seek maintained competitive segmentation performance while reducing redundant background sampling and retained training workload; in 30 held-out TLS WSIs, it exceeded matched random TopK at all four budgets.

  • Takeaways & Limitations

    Pre-segmentation, morphology-aware negative selection extends pathology foundation models toward compact training-set construction for sparse and imbalanced FTU segmentation.

  • Takeaways & Limitations

    Annotated WSI numbers were relatively limited, the internal test cohorts contained only two WSIs per task, and the held-out TLS cohort was not multi-center external validation.

Abstract

from arXiv · show

Functional tissue units (FTUs), including tertiary lymphoid structures (TLSs), blood vessels, and glands, encode localized immune, vascular, and epithelial organization in histopathology. Accurate quantification of these structures is important for studying tissue architecture and disease-associated tissue organization. However, FTUs are frequently sparse, heterogeneous, and surrounded by large amounts of morphologically similar background tissue, making automated segmentation in whole-slide images (WSIs) challenging. We therefore developed FTU-Seek, a pathology foundation model-guided framework that treats morphology-aware negative-patch selection as a key component of sparse FTU segmentation. FTU-Seek uses frozen multi-depth features from the UNI pathology foundation model to train a patch-level classifier that distinguishes FTU-containing from FTU-absent tissue. Target-absent patches are subsequently ranked according to their predicted target-containing probabilities, and the highest-scoring hard negatives are selected through a static Top$K$ strategy to construct compact segmentation training sets. The framework was evaluated using five-fold cross-validation and internal test cohorts across TLS, blood-vessel, and gland segmentation tasks, with an additional independent 30-WSI held-out cohort for TLS. Positive-only, all-tissue, random-negative, and matched random Top$K$ sampling strategies served as comparators. Segmentation-derived phenotypes were further explored in external TCGA cohorts.

1 Introduction

FTUs are localized, biologically meaningful structures whose sparse, heterogeneous, and uneven distribution makes WSI segmentation difficult. FTU-Seek addresses this challenge by selecting morphology-aware hard negatives before segmentation training.

  • Motivation: FTUs encode epithelial, vascular, or immune functions and can yield interpretable image-derived phenotypes from routine histopathology slides.The study defines FTUs as histologically identifiable multicellular structures whose morphology, abundance, and spatial organization relate to these functions.
  • Motivation: TLSs and blood vessels occupy small fractions of tissue, while glands are more abundant but morphologically variable across cancer types.FTU distributions are also highly uneven across prostate, liver, and pancreatic cancer WSIs.
  • Segmentation challenge: Target-containing patches are rare, whereas target-absent tissue includes redundant easy negatives and morphologically target-like regions that can cause false positives.All-tissue training is inefficient, and random negative sampling may miss rare confusing background regions.
  • Proposed framework: FTU-Seek uses frozen multi-depth pathology foundation-model features to rank annotated FTU-absent patches by target-like morphology.A patch-level classifier performs the ranking before segmentation training.
  • Proposed framework: Static per-WSI TopK selection constructs compact, reproducible negative pools while reducing retained training workload and data-transfer burden.The benefit is expected rather than directly measured for wall-clock time, GPU memory, or energy consumption.
  • Study scope: The framework is evaluated across TLS, blood-vessel, and gland tasks against positive-only, all-tissue, random-negative, and matched random TopK strategies.External TCGA analyses further explore segmentation-derived phenotypes describing immune, vascular, and glandular organization.

2.1 Study Design and Datasets

The study retrospectively assembled three task-specific H&E-stained WSI cohorts at Fujian Cancer Hospital to capture diverse tissue architectures and staining profiles. Development data supported five-fold cross-validation, while internal and additional held-out cohorts were reserved for evaluation and downstream analyses used public TCGA data.

  • Cohort construction: Three task-specific H&E-stained WSI cohorts were retrospectively constructed from Fujian Cancer Hospital with deliberately broad morphological and staining heterogeneity.The cohorts targeted TLS, blood-vessel, and gland segmentation.
  • Evaluation design: Development cohorts underwent five-fold cross-validation for hyperparameter optimization, while independent test sets were held out for final segmentation evaluation.An additional 30-WSI TLS cohort from the same institution was assembled to assess robustness beyond the original internal test set.
  • External analyses: Diagnostic WSIs and matched clinicopathological data for downstream analyses were obtained from public TCGA cohorts through the NIH Genomic Data Commons.These analyses investigated segmentation-derived FTU phenotypes.
  • External analyses: TCGA analyses included TLS phenotyping in READ, ESCA, and STAD, with 146, 148, and 326 patients, respectively.The external analyses assessed immune-structure abundance, size composition, and spatial organization.
  • Ethics: Retrospective FCH data were ethics-approved with waived consent, while public de-identified TCGA data were exempt from ethics review.The study states that procedures adhered to the Declaration of Helsinki.

2.2 WSI Annotation and Preprocessing

Target FTUs were manually delineated on 20× or 40× H&E WSIs by trained annotators and reviewed by a senior pathologist. Tissue was standardized into non-overlapping patches, labeled by target overlap, and classified into target-containing or target-absent pools for later selection and segmentation.

  • Annotation: Three trained annotators manually delineated TLSs, glands, and blood vessels, with all annotations reviewed and finalized by a senior pathologist.The annotation criteria specified dense lymphocyte aggregates for TLSs and endothelial-lined structures for blood vessels.
  • Preprocessing: Foreground tissue was isolated using Otsu thresholding and processed at 1 µm/pixel before division into non-overlapping 256×256-pixel patches.Patches were resized to the model input size before analysis.
  • Preprocessing: The same patch coordinates were retained across classification, segmentation training, validation, and testing without later random cropping or overlap generation.Input transformations consisted of resizing, tensor conversion, and ImageNet normalization.
  • Patch labeling: Patches entirely in slide background were excluded, and any patch overlapping an annotation was labeled target-containing under an inclusive overlap rule.The rule retained partial and boundary-crossing target morphology for auxiliary classification.

2.3 Model Development

FTU-Seek uses a two-stage framework in which frozen multi-depth UNI v1 features guide patch-level hard-negative selection before static TopK segmentation training. The segmentation network then fuses multi-scale encoder features with shallow image features for dense prediction, while matched sampling strategies enable controlled comparisons.

  • Framework overview: A patch-level classifier ranks FTU-absent patches by predicted target-containing probability, and the highest-scoring negatives form compact static TopK segmentation training sets.The classifier is trained on annotated FTU-containing and FTU-absent patches, then applied within each WSI before segmentation training.
  • Classification network: The classifier extracts features from multiple relative depths of the frozen UNI v1 encoder, transforms each depth independently, aggregates the representations, and predicts binary patch labels.The implementation uses K = 10 uniformly spaced relative depths from 0.1 through 1.0, with depth-specific feed-forward heads.
  • Hard-negative selection: Hard negatives are selected independently within each WSI, using stored within-WSI ranking with predicted target-containing probability as a secondary sorting criterion.The per-WSI design limits domination by larger tissue areas and supports matched comparisons with random sampling.
  • Hard-negative selection: The nominal TopK candidate set uses K ∈ {100, 300, 500, 1000}, with fewer candidates retained when a WSI lacks enough eligible FTU-absent patches.These common values support controlled accuracy–workload comparisons rather than a prevalence-adaptive or universally optimal budget.
  • Segmentation network: The segmentation network reshapes four frozen UNI v1 Transformer-layer features into spatial maps and progressively fuses them with shallow image features to produce dense binary predictions.The four relative depths are 1/4, 1/2, 3/4, and 1; the decoder progressively restores spatial resolution.
  • Comparative strategies: Comparators include positive-only, all-tissue, random negative, and classifier-guided TopK construction, with matched random sampling using the same per-WSI negative-patch budget.These strategies isolate the contribution of classifier-guided hard-negative selection while controlling the number of sampled negatives.

2.4 Outcomes

The study assessed segmentation accuracy, classifier discrimination, and downstream FTU phenotypes using pixel-level overlap, patch-level classification, and morphology-derived measures.

  • Dice similarity coefficient measured pixel-level agreement between predicted segmentation masks and manual annotations.
  • AUC measured threshold-independent patch-level discrimination, supplemented by sensitivity, specificity, PPV, NPV, accuracy, and F1-score.
  • Sensitivity and specificity were calculated from true-positive, false-negative, true-negative, and false-positive counts.
  • PPV and NPV assessed the reliability of positive and negative classifier predictions.
  • Downstream phenotypes quantified FTU abundance, size distribution, spatial organization, vascular characteristics, and glandular shape.

2.5 Implementation Details

Implementation used frozen UNI representations with trainable task-specific modules, fixed sampling baselines, reproducible splits, and standardized inference procedures.

  • Fixed seeds, split metadata, software versions, and deterministic cuDNN settings supported reproducibility.
  • The patch classifier used frozen UNI v1 embeddings from ten relative encoder depths and a two-layer transformation head.Depth-specific features were transformed, averaged, and passed to a final classifier.
  • Segmentation used five-fold WSI-level cross-validation with a frozen UNI v1 encoder and trainable residual feature-fusion decoder.The independent test set was held out for final evaluation.
  • Matched random TopK baselines selected the same number of negative patches per WSI randomly from the FTU-absent pool.
  • Training updated only the decoder and prediction head using AdamW, mixed precision when available, and combined multiclass Dice and focal losses.
  • Inference thresholded foreground softmax probabilities at 0.5 and mapped patch predictions back to WSI coordinates.

2.6 Statistical Analysis

Statistical analysis summarized cross-validation performance, defined WSI-level Dice from accumulated pixel counts, and treated downstream clinical associations as exploratory.

  • Development and internal-test metrics were summarized as mean ± standard deviation across five cross-validation folds.
  • Random-baseline multi-seed analyses primarily assessed sampling-baseline stability rather than direct FP-find seed robustness.
  • Held-out TLS comparisons used paired WSI-level Dice differences with 95% confidence intervals, paired t-tests, and Wilcoxon signed-rank tests.The confidence intervals used the Student’s t distribution with 29 degrees of freedom.
  • Training workload was measured as retained training patches relative to all-tissue training within each cohort.
  • WSI-level Dice was computed from accumulated true-positive, false-positive, and false-negative pixels rather than averaged patch-level Dice.
  • Multiple testing used Benjamini–Hochberg FDR procedures, while outcome-derived survival cut points were treated as exploratory.
  • Downstream phenotype associations were proof-of-concept and hypothesis-generating rather than confirmatory biomarker validation.

3 Results

The cohorts differed markedly in FTU abundance and sparsity, while patch classifiers discriminated all three targets, with strongest stability for glands and greater ambiguity for blood vessels.

  • Dataset Characteristics: TLSs were substantially less frequent than blood vessels and glands, with development-set means of 16.62, 358.29, and 384.25 instances per WSI, respectively.
  • Dataset Characteristics: TLS-containing patches comprised 3.92% of development tissue patches, compared with 16.43% for BV and 36.86% for glands.
  • Dataset Characteristics: TLS occupied 0.65% of development tissue area, BV 2.46%, and glands 18.91%, establishing TLS as the most spatially sparse task.
  • Downstream Application: External TCGA cohorts supported exploratory downstream analyses of TLS survival, BV survival and MVI, and gland-related clinicopathological endpoints.
  • Patch-Level Classifier Performance: The patch classifier achieved mean AUCs of 95.92 ± 2.81% for TLS, 90.11 ± 3.11% for BV, and 98.16 ± 0.61% for glands.
  • Patch-Level Classifier Performance: TLS classification showed high NPV of 99.48 ± 0.30% but lower PPV of 38.97 ± 23.04%, consistent with extreme target rarity.
  • Patch-Level Classifier Performance: BV classification had a mean FPR of 18.40 ± 5.36%, reflecting confusion from vessel-like nonvascular tissue.
  • Patch-Level Classifier Performance: Gland probability distributions showed limited overlap and consistently high fold-wise AUCs of 97.41–98.75%.

3.3 Segmentation Performance and Training-Data Workload

FTU-Seek reduced segmentation workload by selecting classifier-ranked hard negatives, while retaining much of all-tissue performance across TLS, blood-vessel, and gland tasks. The benefit varied by FTU type and negative-patch budget.

  • Overall comparison: Positive-only training was weakest or near-weakest, whereas all-tissue training was generally strongest but required the complete tissue-patch set.FTU-absent tissue was necessary for suppressing false-positive predictions.
  • TLS: 85.14 ± 1.61% Dice was achieved by FTU-Seek Top500 using 15.8% of the all-tissue workload for TLS segmentation.Top100 achieved 85.08 ± 1.07% Dice using 6.3% of patches; Top500 improved internal-test Dice over matched random TopK by 3.48 percentage points.
  • Blood vessels: 60.38±9.76% test Dice was achieved by all-tissue blood-vessel training, while FTU-Seek Top1000 reached 57.68±6.88% using 50.0% of the workload.FTU-Seek outperformed matched random TopK sampling at all blood-vessel budgets.
  • Task dependence: Hard-negative mining was most advantageous for sparse or morphologically ambiguous FTUs, whereas broader negative coverage was needed for blood-vessel segmentation.The evaluated TopK values were exploratory operating points rather than a universally optimal budget; blood-vessel and gland cohorts lacked held-out cohorts.

3.4 Multi-Seed Stability of Random-Sampling Baselines

Additional random seeds showed limited-to-moderate variability in random-sampling baselines, while classifier-guided results remained above random-baseline averages for TLS and blood-vessel tasks.

  • Seed stability: Five total random seeds produced limited-to-moderate variability across TLS, blood-vessel, and gland segmentation, depending on FTU category and negative-patch budget.The original seed was supplemented with four additional seeds.
  • TLS: 85.08%, 84.69%, 85.14%, and 85.12% were the FTU-Seek TLS Dice values at Top100, Top300, Top500, and Top1000, respectively.These remained above the corresponding random-baseline averages of 79.86 ± 3.45%, 80.88 ± 1.55%, 81.66 ± 1.24%, and 83.92 ± 0.54%.
  • Caveat: The internal TLS test cohort contained only two WSIs, so the seed comparisons there require cautious interpretation.The independent held-out TLS cohort contained 30 WSIs and was evaluated separately.
  • Cross-task comparison: 3.98, 0.70, 2.96, and 3.47 percentage points were the blood-vessel differences between FTU-Seek and random-baseline averages from Top100 to Top1000.For glands, the corresponding differences were −1.42, +0.43, +0.43, and −0.16 percentage points.

3.6 Paired Slide-Level Analysis on the Held-Out TLS Cohort

On the independent 30-WSI TLS cohort, FTU-Seek improved slide-level segmentation over matched random TopK sampling across all tested budgets. Qualitative and exploratory downstream analyses extended the evaluation to segmentation behavior and TCGA-derived phenotypes.

  • Paired analysis: 2.58 to 8.96 percentage points was the range of mean paired improvements over matched random TopK sampling across all hard-negative budgets.All corresponding 95% confidence intervals excluded zero.
  • Paired analysis: At Top1000, FTU-Seek showed paired differences of +1.57 percentage points versus all-tissue training and +5.86 percentage points versus random-balanced training.The same 30 WSIs were used as the statistical pairing unit, with five-fold probability ensembles.
  • Qualitative comparison: FTU-Seek generally reduced scattered or fragmented false positives and refined boundaries while preserving principal TLS, gland, and blood-vessel structures.These qualitative differences were most evident for sparse targets, morphologically ambiguous background, and complex boundaries.
  • Downstream phenotypes: FTU-Seek-derived TCGA phenotypes showed inter-cohort TLS heterogeneity and exploratory associations with survival-related characteristics across TLS, blood-vessel, and gland analyses.The analyses were intended for phenotyping and hypothesis generation rather than clinical prediction validation; selected Kaplan–Meier separations were not validated or multiplicity-adjusted.

4 Discussion

FTU-Seek uses frozen pathology foundation-model representations to select morphology-aware hard negatives before segmentation training, reducing redundant background sampling while maintaining competitive segmentation performance. Its benefits vary by FTU type, and its broader validation and efficiency claims remain bounded by dataset and metric limitations.

  • Framework and mechanism: FTU-Seek ranks FTU-absent patches with a patch-level classifier and selects static per-WSI TopK hard negatives for compact segmentation training sets.The framework uses pretrained histomorphological representations before segmentation training rather than repeatedly re-mining examples during optimization.
  • Task-dependent behavior: TLSs showed the greatest efficiency gains, whereas blood-vessel segmentation benefited from larger hard-negative pools because vessel-like structures occur in diverse morphological contexts.The task-dependent pattern is linked to differences in target sparsity, class imbalance, and morphological ambiguity.
  • Task-dependent behavior: The patch-level classifier affects segmentation indirectly through negative-pool composition: misranking can omit informative confusing negatives or allocate TopK slots to less useful patches.Uncertainty-aware ranking or iterative re-mining may mitigate this error propagation but would add computational cost and variability.
  • Framework and mechanism: Frozen pathology foundation-model representations support both target-like negative identification and transferable multi-scale features for segmentation across structurally distinct FTUs.Freezing the encoder reduces trainable parameters and dependence on relatively small manually annotated datasets.
  • Downstream phenotypes: FTU-Seek-derived phenotypes captured variation in TLS abundance, spatial organization, and morphology across tumor types, while prostate-gland phenotypes reflected adverse clinicopathological characteristics.These analyses suggest the derived phenotypes retain information beyond segmentation masks for subsequent biological and clinical investigation.
  • Validation and limitations: Internal test cohorts contained only two WSIs per segmentation task, and the TCGA analyses were exploratory rather than external segmentation validation or validated biomarker studies.Larger multi-institutional studies are needed across patient populations, staining protocols, scanners, and tissue-processing procedures.
  • Validation and limitations: The independent held-out TLS analysis found higher slide-level Dice than matched random TopK across all four negative-patch budgets, but the evaluation remained within one institution.The authors describe the results as supporting robustness in the available same-institution setting, not multi-center external validation.
  • Validation and limitations: The workload metric measured retained-patch proportion rather than wall-clock time, GPU utilization, memory, or energy consumption.Comprehensive computational benchmarking is therefore still needed.

5 Conclusions

FTU-Seek extends pathology foundation models beyond feature representation by identifying target-like negative tissue before segmentation training. This reduces redundant background sampling and retained training workload while maintaining competitive segmentation performance, particularly for sparse and imbalanced FTUs.

  • 5 Conclusions: FTU-Seek uses pretrained histomorphological information to construct compact, morphology-aware segmentation training sets before training.The approach extends pathology foundation models beyond transferable feature representation.
  • 5 Conclusions: Pre-segmentation hard-negative selection reduces redundant background sampling and retained training workload while maintaining competitive segmentation performance.The clearest practical benefit was observed for sparse and imbalanced FTUs.
Loading 2609.00704v1…