PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-Supervised Virtual Staining via Morphology Preservation and Histopathological Realism Constraints

Baoshun Wang, Weiping Lin, Linwu Wang, Yihuang Hu, Baptiste Magnier, Liansheng Wang

arXiv:2609.00984v1cs.CVcs.AI

TL;DR

Virtual staining needs reliable target images but accurately registered paired data are difficult to obtain, while unpaired inputs lack direct cross-stain supervision. The paper introduces a semi-supervised framework combining Hessian-derived morphology preservation with CONCH-based histopathological realism constraints. Across H&E-to-IHC and FFPE-to-H&E tasks, it reports consistent improvements in image quality, morphology preservation, robustness, and downstream diagnostic performance.

  • Problem

    Accurately registered paired images are difficult and expensive to obtain, while unpaired source images lack corresponding targets for direct virtual-staining supervision.

  • Method

    The framework jointly uses limited paired data and abundant unpaired source images, applying Hessian-derived morphology preservation and frozen-CONCH histopathological realism constraints.

  • Results

    Across H&E-to-IHC translation for HER2 and Ki67 and FFPE-to-H&E translation, experiments show consistent improvements in image quality, morphology preservation, robustness, and downstream diagnostic performance.

  • Takeaways & Limitations

    Source-derived morphology supervision and target-stain realism guidance provide a practical approach for incorporating unpaired source images into data-efficient virtual staining.

Abstract

from arXiv · show

Virtual staining aims to computationally generate target-stained histopathological images while reducing the cost and time associated with conventional staining procedures. However, existing methods rely predominantly on strictly paired and accurately registered training data, which are difficult and expensive to obtain in routine practice. To reduce this dependence, we propose a stable semi-supervised virtual staining framework that jointly exploits both limited paired data and abundant unpaired source images. Directly incorporating unpaired images is challenging because their generated results lack corresponding targets for supervision, potentially leading to unrealistic staining, morphological degradation, or even training collapse. To obtain reliable supervision from these images, Hessian-derived morphology preservation extracts structural cues from each source image and constrains the generated output to retain tissue morphology. Histopathological realism constraints further guide the output toward plausible target-stain characteristics, preventing the source-derived structural supervision from degenerating into contour enhancement or simple color transformation. Together, the two components suppress structural and appearance drift, stabilize semi-supervised stain translation, and promote the preservation of diagnostically relevant information. Extensive experiments on H&E-to-IHC translation for Ki67 and HER2, as well as FFPE-to-H&E translation, demonstrate consistent improvements in image quality, morphology preservation, robustness, and downstream diagnostic performance. Code will be available.

I. INTRODUCTION

Virtual staining can reduce the burden of conventional tissue staining, but limited paired data and difficult registration constrain supervised learning. This framework combines paired and unpaired source images with morphology and realism supervision to improve reliable stain translation.

  • Conventional staining requires substantial labor, reagent consumption, and turnaround time, motivating computational translation between staining modalities.
  • Paired cross-stain data preserve morphology and diagnostically relevant staining patterns but are difficult to acquire because tissue correspondence is disrupted by deformation, artifacts, variation, and registration errors.
  • Unpaired source images can expand data utilization, but their generated targets lack direct supervision and may exhibit morphology distortion, unrealistic staining, or training collapse.
  • Hessian-derived morphology preservation extracts source-image structure to reduce alterations to tissue interfaces, cellular boundaries, and nuclear contours during translation.
  • Histopathological realism constraints based on a frozen CONCH model guide plausible target-stain characteristics and discourage contour enhancement or simple color transformation.
  • Across Ki67, HER2, and FFPE-to-H&E tasks, the framework reports improvements in staining quality, morphology preservation, robustness, and downstream diagnostic performance.

II. RELATED WORK

Existing virtual-staining methods either reduce registration requirements through unpaired translation or preserve correspondence through paired supervision, leaving a need for task-specific semi-supervised guidance. The paper addresses this need with complementary source-morphology and target-realism constraints.

  • A. Virtual Staining: Unpaired methods avoid spatial registration but rely mainly on domain-level matching or representation consistency, which may not ensure biomarker localization or faithful morphology.
  • A. Virtual Staining: Paired methods directly supervise stain mapping with registered source–target images, but their effectiveness depends on obtaining reliable spatial correspondence.
  • B. Semi-supervised Learning for Medical Image Translation: Semi-supervised image translation is difficult because unannotated inputs require high-dimensional synthesized targets that can reinforce inaccurate mappings during optimization.
  • B. Semi-supervised Learning for Medical Image Translation: Virtual staining additionally requires unpaired supervision that preserves source-tissue organization while producing realistic target-stain characteristics without corresponding targets.
  • B. Semi-supervised Learning for Medical Image Translation: Hessian-derived morphology preservation supplies source-side structural supervision, while histopathological realism constraints provide target-side guidance against color-only or contour-dominated shortcuts.

III. METHODOLOGY

The framework uses a shared generator for paired and unpaired source images, with supervision determined by whether a corresponding target image is available. HDMP preserves source morphology, while HRC guides realistic target-stain appearance.

  • A. Problem Formulation: Virtual staining learns a generator G: S → T that maps a source-staining image to its corresponding target-stained image.
  • A. Problem Formulation: Paired samples provide direct cross-stain supervision through registered source–target images and their corresponding generated outputs.
  • A. Problem Formulation: Unpaired samples contain source images without corresponding targets, so supervision must be derived from source morphology and expected target-stain appearance.
  • B. Framework Overview: Paired and unpaired branches share one generator, differing in their available supervision sources.
  • B. Framework Overview: HDMP extracts structural information from unpaired sources, whereas HRC uses a frozen CONCH model to encourage plausible target-stain characteristics.
  • B. Framework Overview: HDMP and HRC jointly constrain the generator: HDMP preserves spatially consistent tissue structures, while HRC discourages structure-only shortcuts and simple color transformation.

C. Hessian-Derived Morphology Preservation

Hessian-derived morphology preservation extracts structural supervision from source images so generated stains retain tissue architecture despite differing appearances. It emphasizes interfaces, cellular contours, and nuclear boundaries, while HRC is needed to prevent morphology supervision from producing unrealistic staining.

  • C. Hessian-Derived Morphology Preservation: Unpaired source images provide tissue architecture and cellular organization that can supply structural supervision when corresponding targets are unavailable.The method derives this supervision directly from each source image.
  • C. Hessian-Derived Morphology Preservation: Second-order spatial responses characterize rapid intensity transitions around tissue interfaces, glandular boundaries, cellular contours, and nuclear edges.These responses can be compared between source and generated images despite different staining appearances.
  • C. Hessian-Derived Morphology Preservation: Each image is converted to grayscale before applying the Hessian-derived response, reducing dependence on individual color channels.The resulting map emphasizes morphology-related structures around tissue interfaces, cellular contours, and nuclear boundaries.
  • C. Hessian-Derived Morphology Preservation: Morphology preservation compares each generated image's response with that of its original source image for paired and unpaired samples.This transfers morphology-related supervision without requiring identical staining appearances.
  • C. Hessian-Derived Morphology Preservation: Morphology preservation alone does not determine whether generated images exhibit realistic target staining and may encourage contour-dominated outputs or superficial color transformation.A complementary target-side constraint is therefore required.

D. Histopathological Realism Constraints

Histopathological realism constraints use a frozen CONCH vision–language model and task-specific target-stain prompts to guide generated images toward plausible modality-specific appearances. Paired samples additionally receive guidance from their corresponding target images.

  • D. Histopathological Realism Constraints: Histopathological realism constraints use a frozen CONCH vision–language model to guide generated images toward plausible target-stain appearances.They complement source-derived morphology supervision.
  • D. Histopathological Realism Constraints: A task-specific textual prompt describes the expected appearance and characteristic staining pattern of the target modality.Its normalized representation is used as the textual reference.
  • D. Histopathological Realism Constraints: The frozen CONCH text encoder produces the normalized representation of the target-stain prompt.The generated image is separately represented through the frozen CONCH image encoder.
  • D. Histopathological Realism Constraints: The image–text realism constraint encourages generated representations to remain compatible with expected target-stain characteristics.Its objective includes a temperature parameter τ.
  • D. Histopathological Realism Constraints: For paired samples, the textual reference is supplemented with sample-specific guidance from the corresponding target image.This adds target-image information to the prompt-based constraint.

1) Paired Realism Constraint:

The paired realism constraint combines general target-stain guidance with sample-specific alignment to the corresponding real target, using frozen references to stabilize appearance and morphology.

  • Paired Realism Constraint: The paired HRC objective combines image–text realism with image–image alignment to guide generated images toward plausible target staining.The image–text term provides a general target-stain reference, while image–image alignment supplies sample-specific information from the real target.
  • Paired Realism Constraint: For unpaired images, HRC relies on the target-stain prompt because no corresponding target is available for image–image alignment.
  • Paired Realism Constraint: Frozen CONCH encoders keep the target-stain reference fixed during generator optimization, reducing appearance drift.This complements morphology supervision derived from the source image.
  • Paired Realism Constraint: Together, the fixed target-stain reference and morphology supervision discourage structurally distorted outputs and unrealistic contour-preserving stain transformations.

E. Training Objective

The training objective selects paired or unpaired losses according to target availability, combining direct supervision with morphology and realism constraints while warming up on paired data first.

  • E. Training Objective: Paired batches use direct cross-stain supervision together with morphology preservation and histopathological realism constraints.
  • E. Training Objective: Unpaired batches omit direct translation and target-image alignment losses because no corresponding target image is available.
  • E. Training Objective: The mini-batch objective is selected according to whether corresponding target images are available.
  • E. Training Objective: The generator is warmed up with paired samples before activating the unpaired branch, preventing unpaired optimization from preceding a basic cross-stain mapping.
  • E. Training Objective: The framework is evaluated on Ki67, HER2, and FFPE-to-H&E virtual staining using registered paired data plus additional unpaired source images.

1) H&E-to-IHC Virtual Staining for Ki67:

The study evaluates H&E-to-IHC and FFPE-to-H&E translation using multiple baselines, complementary image and stain metrics, and matched training settings across datasets.

  • H&E-to-IHC Virtual Staining for Ki67: The HER2 task uses 3,000 registered pairs and 10,000 unpaired HER2 source patches from IHC4BC.The registered pairs include 2,500 training and 500 testing samples.
  • H&E-to-IHC Virtual Staining for Ki67: The FFPE-to-H&E task uses 2,500 registered training pairs, 1,000 testing pairs, and 7,500 unpaired FFPE patches.
  • H&E-to-IHC Virtual Staining for Ki67: Training uses matched model settings and a fixed 70-epoch paired warm-up before 30 epochs with unpaired samples.The unpaired-to-paired ratio is 2:1 unless otherwise specified.
  • H&E-to-IHC Virtual Staining for Ki67: Experiments compare representative supervised, weakly paired, diffusion-based, and pathology foundation-model virtual staining methods.The proposed framework is also combined with Pix2PixHD under the same generator backbone.
  • H&E-to-IHC Virtual Staining for Ki67: Evaluation covers image-level quality, stain-specific agreement, and downstream diagnostic utility.PSNR, SSIM, MS-SSIM, DISTS, FID, and KID assess image similarity and distributional agreement, while DAB-KL and IOD-D assess biomarker staining.

2) Stain-Specific Assessment:

Across three translation tasks, the framework improves Pix2PixHD and achieves competitive or superior evaluation results, including the strongest overall four-class HER2 scoring among compared virtual staining methods.

  • 2) Stain-Specific Assessment: Across HER2, Ki67, and FFPE-to-H&E translation, the framework consistently improves Pix2PixHD across image-level, distributional, and stain-specific evaluations.
  • 2) Stain-Specific Assessment: The method achieves competitive or superior performance across all three tasks, including biomarker-related staining and tissue-component agreement.
  • 2) Stain-Specific Assessment: Generated images show more plausible target-stain appearances while better retaining tissue organization and cellular structures.Hessian-derived morphology preservation supplies structural supervision, while HRC prevents contour enhancement or superficial color transformation.
  • 2) Stain-Specific Assessment: The proposed method achieves the strongest overall four-class HER2 scoring performance among the compared virtual staining methods.The experiment uses an external dataset with explicit HER2 annotations and compares generated IHC images with H&E and real-IHC references.
  • 2) Stain-Specific Assessment: Removing both constraints substantially degrades performance, whereas adding either HDMP or HRC improves the semi-supervised baseline and combining them performs best overall.

G. Robustness to Staining and Acquisition Variations

The framework remains robust when source images undergo staining variations or acquisition degradations, while preserving source-tissue organization. Across evaluations, it improves Pix2PixHD and achieves the highest PCC on both assessed datasets.

  • Robustness evaluation: Robustness testing introduces staining variations and acquisition degradations while preserving the underlying pathological content.Staining variations adjust hue, saturation, and brightness; degradations include noise, blur, downsampling, and radial distortion.
  • Robustness evaluation: The robustness evaluation compares original images with three modified variants, separating staining variations from acquisition degradations by row.The top row shows staining variations, whereas the bottom row shows acquisition degradations.
  • Robustness results: The proposed framework consistently improves Pix2PixHD under both input-variation types across image-level, distributional, and stain-specific evaluations.The reported results cover the Self-HER2, MIST-Ki67, and Self-FFPE evaluation settings.
  • Morphology preservation: The framework achieves the highest PCC on both MIST-Ki67 and self-FFPE, indicating stronger preservation of source-tissue organization.PCC is computed after grayscale conversion to reduce direct stain-color influence and measure agreement in spatial intensity patterns.

I. Sensitivity to the Amount of Unpaired Data

Performance generally improves as more unpaired source images are added and becomes relatively stable at an unpaired-to-paired ratio of approximately 2:1. The paper therefore uses 2:1 in its remaining experiments.

  • Data-ratio study: Performance is evaluated at unpaired-to-paired ratios of 0.5:1, 1:1, 2:1, and 3:1 while keeping the remaining settings unchanged.This experiment isolates the influence of unpaired data volume.
  • Data-ratio study: Performance generally improves as additional unpaired source images are introduced and becomes relatively stable at approximately a 2:1 ratio.Increasing the ratio beyond 2:1 provides limited additional benefit.
  • Experimental setting: The remaining experiments use an unpaired-to-paired ratio of 2:1.This setting follows the observed stabilization of performance at approximately 2:1.
  • Overall results: Experiments across HER2, Ki67, and FFPE-to-H&E translation report improvements in image quality, morphology preservation, pathology-informed metrics, robustness, and downstream diagnostic performance.The robustness and ablation results also support the stability and complementary roles of the two constraints.
  • Overall implications: The framework incorporates unpaired source images without modifying the underlying generator architecture.It jointly uses source-derived morphology supervision and target-stain realism guidance for data-efficient virtual staining.
Loading 2609.00984v1…