PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Reliable AI-Based Histological Staining: A Systematic Study of Scaling and Uncertainty in Unpaired Generative Models

Qasim Siddiqui, Adrian Friebel, Maiju Myllys, Zaynab Hobloss, Daniela Gonzalez, Ahmed Ghallab, Stefan Hoehme

arXiv:2608.24626v1cs.CV

TL;DR

The paper asks how reliably unsupervised models can virtualize Sirius Red from routine H&E for liver-fibrosis quantification when paired ground truth and additional tissue are limited. It benchmarks six architectures across 54 scaling configurations and evaluates perceptual quality, task-specific performance, and ensemble uncertainty. These axes are largely independent, so reliable model selection requires reporting and jointly considering all three.

  • Problem

    Systematic benchmarks of unsupervised H&E-to-SR models and their predictive uncertainty are limited, despite the clinical importance of SR-based collagen quantification.

  • Method

    The study benchmarks six unsupervised architectures across 54 scaling configurations, evaluates CPA MAE alongside perceptual measures, and trains deep ensembles for uncertainty maps.

  • Results

    GAN methods cluster on perceptual metrics but differ in task error and ensemble agreement, while CycleDiffusion differs across all three axes; uncertainty alone cannot replace CPA MAE evaluation.

  • Takeaways & Limitations

    Reliable virtual staining requires reporting and selecting models jointly on task-specific error, perceptual quality, and ensemble uncertainty.

  • Takeaways & Limitations

    Cycle-reconstruction calibration is only a proxy, and cross-section biological differences impose an irreducible noise floor on calibration error.

Abstract

from arXiv · show

Liver fibrosis, the principal predictor of long-term outcome in chronic liver disease, is staged from histological estimates of collagen content. Sirius Red (SR) provides the standard quantitative readout (collagen proportionate area, CPA) but is not acquired at every clinical centre and consumes tissue, time, and reagent cost beyond the routine Hematoxylin and eosin (H&E) stain. AI-based virtual staining can generate SR directly from H&E, yet systematic benchmarks of unsupervised models are scarce and their predictive uncertainty has not been quantified, even though visually plausible outputs may not faithfully reproduce the underlying tissue structure. We therefore benchmark six unsupervised image-to-image architectures (GAN-based and diffusion-based) across 54 scaling configurations on a newly released paired H&E to SR mouse liver dataset, the first open resource for this translation task. Each configuration is evaluated jointly on perceptual, distributional, and task-specific axes plus a blinded expert reader study; the best per family is then retrained as a deep ensemble, the first systematic comparison of epistemic uncertainty across unsupervised stain-to-stain architectures. Across families, perceptual quality, task-specific error, and ensemble agreement measure largely independent axes of model fitness: GAN-based methods cluster tightly on perceptual metrics yet differ substantially on task error and ensemble agreement, while the diffusion-based method (CycleDiffusion) is qualitatively different on all three. No single metric captures these differences, so reliable virtual staining requires reporting and selecting on all three jointly. The dataset, tiling pipeline, models, and evaluation code are released publicly.

Introduction

This study addresses the cost, tissue-use, and alignment burdens of multiple histological stains by evaluating unsupervised virtual SR generation from routine H&E. It benchmarks model scaling and introduces ensemble-based uncertainty analysis for this translation task.

  • Motivation: Sirius Red provides the reference collagen proportionate area measurement for liver fibrosis, but acquiring additional stains consumes tissue and laboratory resources.
  • Motivation: Unsupervised translation is better suited than supervised paired-image methods because it avoids dependence on pixel-aligned sections distorted by tissue deformation.
  • Pipeline: The virtual staining pipeline tiles H&E whole-slide images, generates virtual SR tiles, reassembles them, and quantifies CPA without a second physical SR section.
  • Study design: The benchmark spans six unsupervised architectures, three generator capacities, and three data fractions, yielding 54 configurations evaluated with CPA MAE and perceptual metrics.
  • Uncertainty: Deep ensembles of independently seeded models provide spatially resolved disagreement as an epistemic-uncertainty signal for tile-level outputs lacking ground-truth SR.
  • Dataset: The dataset contains 70 mouse-liver whole-slide images from 35 mice, with serial H&E and SR sections that are inherently unpaired.

Models

The study compares six unsupervised image-to-image architectures that extend foundational unpaired translation through contrastive alignment, shared latent spaces, or content-style decomposition.

  • The benchmark includes CycleGAN, DCLGAN, UNIT, MUNIT, UVCGAN, and CycleDiffusion as distinct unsupervised image-to-image approaches.
  • DCLGAN adds a dual patch-level contrastive loss to improve fine-grained content alignment between domains.
  • UNIT uses a shared VAE bottleneck, whereas MUNIT separates per-domain content and style codes.

5 Scaling Study: Experimental Design and Results

The study evaluates six unsupervised image-to-image architectures across 54 scaling configurations using perceptual, distributional, task-specific, and expert-reader assessments. CPA error differentiates configurations that perceptual metrics often place close together, while reader judgments track task performance.

  • Experimental Design: Six architectures were trained across three generator sizes and three data fractions, yielding 54 full-factorial configurations.Training used optimiser-step matching across data-fraction settings; CycleDiffusion used DDIM sampling over Tsample=200 steps.
  • Evaluation Metrics: Because non-adjacent H&E and SR sections cannot provide pixel-exact ground truth, test WSIs were feature-based co-registered, leaving residual misalignment that can inflate pixel-level error.Patch-based SSIM reduces sensitivity to residual misregistration, whereas FID captures population-level realism but may miss localized collagen-specific failures.
  • Evaluation Metrics: CPA MAE measures disagreement between real and virtual Sirius Red collagen proportionate area, while Patch-SSIM, LPIPS, and FID assess structural, perceptual, and distributional quality.Lower CPA MAE indicates collagen quantification more consistent with the real stain.
  • Scaling Results: CPA MAE spanned 0.008–0.171 across configurations, whereas perceptual metrics saturated within a narrow band for well-behaved GAN configurations.CycleDiffusion showed systematically lower Patch-SSIM, and some families became unstable under particular scaling conditions.
  • Scaling Results: CycleGAN-M achieved the study-wide lowest CPA MAE of 0.008 at 100% data, while DCLGAN-L achieved the lowest FID of 190.0 at 100% data.For CycleGAN at 100% data, small, medium, and large generators yielded CPA MAE values of 0.049, 0.008, and 0.085, respectively.
  • Cross-Metric and Reader Results: Metric rankings were only partly aligned: Patch-SSIM and LPIPS correlated at rs=0.82, while CPA MAE correlated with Patch-SSIM at rs=0.63 and with LPIPS at rs=0.64.The blinded reader study also separated the best and worst configurations, with generated-tile detection of 38% versus 62% and plausibility ratings favoring the best configuration.

6 Epistemic Uncertainty Analysis

Deep ensembles quantify per-tile epistemic disagreement across selected stain-translation models, revealing family-specific uncertainty patterns that differ from task error.

  • Uncertainty quantification: Ten independently seeded generators form ensembles whose output disagreement provides spatially resolved epistemic uncertainty maps for unpaired virtual staining.High disagreement flags regions where models fail to converge on a consistent translation.
  • Family comparisons: GAN families cluster at median uncertainty 37.68–41.42, whereas CycleDiffusion is lower at 23.35 but has a wider IQR of 5.89.The distributions cover 9,365 test tiles with at least 0.1% tissue coverage.
  • Calibration: GAN uncertainty moderately predicts tile-level reconstruction error with Pearson ρ = 0.50–0.66, but CycleDiffusion is uncorrelated at ρ = −0.03.No family achieves meaningful within-tile spatial calibration.

7 Discussion

The discussion shows that perceptual quality, task-specific error, and ensemble uncertainty capture distinct aspects of virtual staining performance. It also limits interpretation because the study uses a small mouse cohort and constrained reader and registration settings.

  • Scaling and metric choice: Across 54 configurations, the optimal scaling point is family-specific and cannot be reliably selected from perceptual metrics alone.Task-specific evaluation remains necessary because perceptual rankings can miss large CPA MAE differences.
  • CycleDiffusion: CycleDiffusion has lower perceptual quality but a distinct uncertainty profile, with data budget and generator size affecting CPA MAE inconsistently.Its medium generator reaches CPA MAE 0.035 at 25% data versus 0.082 at 100%, while the small generator worsens from 0.051 to 0.171.
  • Limitations: Human and ranking conclusions are bounded by five held-out paired specimens, two reader-study configurations, three readers, and loosely registered adjacent sections.The mouse bile-duct ligation model may not transfer directly to human NASH or NAFLD biopsies.

8 Conclusion

The study benchmarks unsupervised H&E-to-SR translation across model families and scaling choices while adding ensemble-based epistemic uncertainty analysis. It concludes that deployment should select models according to task error, reliability needs, and data budget, with released resources enabling further validation.

  • Conclusion: Six unsupervised architectures are benchmarked across 54 scaling configurations on a released paired mouse-liver H&E-to-SR dataset.Evaluation combines perceptual, distributional, task-specific, and per-tile ensemble uncertainty measures.
  • Practical guidance: CycleGAN-M at full data minimizes fibrosis-quantification error, DCLGAN-S best balances task accuracy and ensemble agreement, and MUNIT is most stable across data budgets.CycleDiffusion is conditionally consistent rather than uniformly more reliable than GAN alternatives.
  • Resources and next steps: The dataset, tiling pipeline, models, and evaluation code are released as I2I-Stain-Zoo, with cross-cohort validation identified as the next step.The proposed validation targets independent mouse models and human liver biopsies.

S1 Dataset Details

The study uses a mouse liver H&E–SR dataset spanning disease stages, with tissue processing, digitisation, tiling, and animal-level data splits designed for reproducible unpaired translation experiments.

  • Cohort: The cohort contains mouse liver specimens from bile-duct ligation and sham-control animals spanning four post-surgical disease stages.The source cohort is drawn from an obstructive-cholestasis experiment.
  • Histology: Each specimen contributes serial 4 µm H&E and Sirius Red sections from the same paraffin-embedded tissue block.SR selectively visualises fibrillar collagens, while non-adjacent sectioning preserves macroscopic morphology but prevents exact pixel correspondence.
  • Digitisation: Slides were digitised at 20-fold magnification at 0.221µm/pixel, with severely artefactual slides excluded before tiling.The images were exported as RGB TIFF files for downstream processing.
  • Tiling and preprocessing: Tissue masks were generated from QuPath optical-density channels before whole-slide images were divided into non-overlapping tiles.Different channels were used for H&E and SR to separate stained tissue from background.
  • Data splits and subsets: Training subsets are strict nested fractions, while animal-level splits prevent leakage and the held-out test set excludes day-63 BDL because its sections could not be co-registered.The training set contains 30 paired specimens and the test set contains 5 held-out paired specimens.
  • Public release: The release includes tissue masks, tile metadata, split assignments, image patches, pretrained checkpoints, and training scripts under CC BY 4.0.The resource is hosted as I2I-Stain-Zoo, subject to the cited source study’s data-sharing terms.

S2 Model Details

The benchmark compares six unsupervised architectures using distinct generative designs, including adversarial, shared-latent, contrastive, hybrid UNet–ViT, and diffusion-based approaches, across controlled generator sizes.

  • GAN-based models: CycleGAN uses bidirectional ResNet generators and PatchGAN discriminators trained with adversarial, reconstruction, content, and style-related objectives.Its generator objective combines adversarial training with image reconstruction, content consistency, and style consistency terms.
  • GAN-based models: UNIT assumes a shared latent space with domain-specific encoders and decoders, VAE reparameterisation, adversarial reconstruction, cycle, and KL terms.The bottleneck combines private and shared residual blocks.
  • GAN-based models: MUNIT separates domain-invariant content from domain-specific style and injects target-domain style through AdaIN-conditioned decoding.The style representation is processed by a style encoder and MLP before decoder injection.
  • GAN-based models: DCLGAN augments the CycleGAN framework with dual contrastive learning over sampled intermediate feature locations, while retaining cycle consistency and disabling identity loss.Feature patches are projected into a 256-dimensional embedding space and optimised with an InfoNCE objective.
  • Hybrid architecture: UVCGAN replaces the ResNet generator with a UNet–Vision Transformer hybrid and uses masked-image pretraining before cycle-consistent GAN finetuning.Stage 1 reconstructs randomly masked patches using an L1 loss without discriminator involvement.
  • Diffusion architecture: CycleDiffusion translates between domains through a shared Gaussian latent space learned by two independent unconditional DDPMs, without adversarial or training-time cycle-consistency losses.At inference, DDIM inversion encodes the source and DDIM sampling decodes the target; the round trip approximately reconstructs the source.
  • Model scaling: Each architecture is evaluated at small, medium, and large scales, with parameter counts defined per specified generator direction or diffusion branches.The size configurations are documented in Table S3.

S3 Implementation Details

Implementation is standardised through shared optimisation settings and step-based training, while UVCGAN and CycleDiffusion use architecture-specific schedules and inference procedures.

  • Shared training protocol: All models use Adam with learning rate 2×10^-4, β1=0.5, β2=0.999, batch size 1, linear decay, and no weight decay.Step-based training rather than epoch-based training controls comparisons across data fractions with different tile counts.
  • Training schedules: CycleGAN, UNIT, MUNIT, and DCLGAN each use 750,000 optimiser steps with alternating generator and discriminator updates.The schedule is shared across these four GAN-based architectures.
  • Training schedules: UVCGAN uses 250,000 masked-image pretraining steps followed by 500,000 cycle-consistent finetuning steps.Finetuning is initialised from the Stage 1 generator checkpoint.
  • Training schedules: CycleDiffusion trains both DDPMs simultaneously for 750,000 steps with a linear schedule over T=1000 diffusion steps and no finetuning phase.Its training uses the ε-prediction objective.
  • Inference: At inference, GAN models use one generator pass per tile, whereas CycleDiffusion uses sequential 200-step DDIM inversion and sampling passes.Generated tiles are denormalised and retain original tile identifiers for metric computation.

S4 Full Scaling Study Results

The full factorial study evaluates 54 configurations across six architectures, three generator scales, and three data fractions using multiple metrics and ensemble-based uncertainty estimation.

  • Scaling study design: 54 configurations span six models, three generator sizes, and 25%, 50%, or 100% of the training data.The reported configurations include CycleGAN, UNIT, MUNIT, DCLGAN, UVCGAN, and CycleDiffusion.
  • Evaluation: Each configuration reports Patch-SSIM, LPIPS, FID, and CPA MAE, with Patch-SSIM, LPIPS, and CPA MAE summarised over five held-out paired specimens.FID is a set-level statistic, and collapsed models are flagged when FID > 300 and Patch-SSIM < 0.20.
  • Uncertainty estimation: Deep ensembles forward each H&E tile through K=10 independently trained generators and use pixel-wise sample variance across SR outputs as the uncertainty map.This provides spatially resolved epistemic uncertainty for the selected configurations.

S5 Uncertainty Calibration Analysis

The study estimates epistemic uncertainty with deep ensembles and evaluates its calibration against cycle-reconstruction error at pixel and tile levels. Calibration is weak or model-dependent: GAN ensembles provide limited tile-level triage, while CycleDiffusion uncertainty is uninformative, so uncertainty cannot replace task-specific CPA evaluation.

  • Limitations: Calibration is defined relative to cycle-reconstruction error rather than true translation error, which is unavailable in the unpaired setting.Cycle error can underestimate failure when forward and inverse generators share systematic biases, and cross-section biology adds an irreducible ECE noise floor.
  • Within-tile spatial calibration: 0.058–0.165: all six families show weak within-tile Spearman calibration between uncertainty and cycle-reconstruction error.MUNIT is strongest at rs = 0.165 ± 0.173, whereas CycleDiffusion is near zero at rs = 0.058.
  • Across-tile calibration: ρ = 0.50–0.66: GAN families show moderate across-tile Pearson correlation, but Spearman rs = −0.18–0.21 is weaker and inconsistent.The discrepancy indicates that linear covariation is driven by a small number of high-leverage tiles rather than robust monotonic ordering.
  • Reliability and ECE: 0.055: DCLGAN achieves the lowest tile-level ECE, followed by MUNIT at 0.058, while CycleDiffusion performs worst at 0.214.Reliability curves generally fall below the identity line, and the lowest-uncertainty bin exceeds its predicted error in every family.
  • Model-specific behavior: CycleDiffusion uncertainty is uninformative because its DDIM inversion cycle error is large and varies too little across tiles to be predicted from ensemble variance.Its uncertainty is near-unrelated to error across tiles, with Pearson ρ = −0.03 and Spearman rs = −0.04.
  • Practical interpretation: GAN ensembles offer a weak but non-trivial tile-level triage signal, yet task-specific CPA MAE evaluation cannot be replaced by uncertainty screening alone.DCLGAN and MUNIT are the most defensible candidates for soft uncertainty-based pathologist review.

S6 Expert Reader Study

A blinded study compared generated SR tiles from the study-best CycleGAN configuration and study-worst CycleDiffusion configuration with real SR. Readers found the best outputs difficult to distinguish from real stains and consistently judged them more plausible than the worst outputs, although the study was small and did not rank all families.

  • Study design: Three blinded readers evaluated generated Sirius Red tiles using real-versus-generated identification and pairwise plausibility tasks.The study used only held-out data and no additional models, training runs, or data.
  • Task 1: real versus generated: 38% versus 62%: readers correctly identified best-configuration and worst-configuration tiles as generated in Task 1.The best configuration was not reliably distinguishable from real SR, whereas the worst configuration was.
  • Task 2: collagen plausibility: All three readers rated the best configuration plausible and the worst configuration implausible in Task 2, with no overlap between configurations for any reader.The comparison paired one H&E input with two generated SR tiles from the two configurations.
  • Interpretation: The reader separation followed CPA MAE ranking more closely than perceptual scores, which placed the two configurations considerably closer together.This comparison was obtained independently of the automated evaluation pipeline.
  • Limitations: The study covered two configurations and three readers, so it cannot support inter-rater agreement statistics or per-family ranking.Extension to all six families and readers at multiple centres is left for future work.
Loading 2608.24626v1…