Source-linked AI summary
Deep Learning-enabled Virtual Histological Staining of Biological Samples
Bijie Bai, Xilin Yang, Yuzhu Li, Yijie Zhang, Nir Pillar, Aydogan Ozcan
TL;DR
Deep learning-based virtual staining addresses the time, labor, and chemical burdens of conventional staining. The review describes virtual staining’s rapid, cost-effective alternatives and capabilities, while noting challenges in producing consistent ground-truth images and variation across laboratories and histotechnologists.
Problem
Conventional staining entails time and labor burdens and uses toxic staining compounds.
Method
The review examines deep learning-based virtual staining, including simultaneous generation of different stain types.
Results
Deep learning-based virtual staining enables rapid, cost-effective, and chemical-free alternatives to traditional histological staining methods.
Takeaways & Limitations
Virtual staining provides additional histological information that aids diagnostic evaluation and supports stain multiplexing.
Takeaways & Limitations
Consistent histological datasets are technically challenging to generate, and staining results vary across laboratories and histotechnologists.
Abstract
from arXiv · showhide
Histological staining is the gold standard for tissue examination in clinical pathology and life-science research, which visualizes the tissue and cellular structures using chromatic dyes or fluorescence labels to aid the microscopic assessment of tissue. However, the current histological staining workflow requires tedious sample preparation steps, specialized laboratory infrastructure, and trained histotechnologists, making it expensive, time-consuming, and not accessible in resource-limited settings. Deep learning techniques created new opportunities to revolutionize staining methods by digitally generating histological stains using trained neural networks, providing rapid, cost-effective, and accurate alternatives to standard chemical staining methods. These techniques, broadly referred to as virtual staining, were extensively explored by multiple research groups and demonstrated to be successful in generating various types of histological stains from label-free microscopic images of unstained samples; similar approaches were also used for transforming images of an already stained tissue sample into another type of stain, performing virtual stain-to-stain transformations. In this Review, we provide a comprehensive overview of the recent research advances in deep learning-enabled virtual histological staining techniques. The basic concepts and the typical workflow of virtual staining are introduced, followed by a discussion of representative works and their technical innovations. We also share our perspectives on the future of this emerging field, aiming to inspire readers from diverse scientific fields to further expand the scope of deep learning-enabled virtual histological staining techniques and their applications.
Development of a virtual staining model
Virtual staining model development requires data collection, preprocessing, and neural-network training, with supervised and unsupervised schemes differing in whether input and target images are paired. Although development can be time-intensive, validated models enable rapid, repeatable whole-slide inference without chemical staining.
- Training schemes: Supervised training requires perfectly cross-registered input and ground-truth image pairs, whereas unsupervised training does not require paired images.Registration or pretrained data-generation models may be needed to create well-matched supervised training images.
- Training schemes: Cycle-consistency frameworks such as CycleGANs learn mappings between input and ground-truth image distributions while matching color and contrast.These frameworks are commonly used for unsupervised training scenarios.
- Development requirements: Model development involves substantial data acquisition, preprocessing, network design, and training, although this is a one-time effort comparable to developing chemical staining protocols.The development stage can take substantial time, but inference follows after a satisfactory model is obtained and validated.
- Inference: A validated model can create a whole-slide virtual histological image in a few minutes on a standard computer through rapid, repeatable blind inference.Inference avoids waiting for chemical staining procedures and saves time and labor while eliminating toxic staining compounds.
Label-free virtual staining
Label-free virtual staining uses deep learning to generate histological stains from diverse microscopy modalities and biological samples. Reported applications range from standard stains to multiplexed molecular stains, in vivo skin imaging, and real-time sperm-cell evaluation.
- Scope: Deep learning virtual staining has replicated multiple stain types using different image-contrast mechanisms across diverse samples, expanding its application areas.The review summarizes these developments in Table 1 and Figure 4.
- Autofluorescence imaging: Autofluorescence imaging supports virtual generation of micro-structured, multiplexed, and molecular stains from unlabeled tissue.A single network generated multiple stains on the same tissue section, and autofluorescence was used for virtual HER2 IHC staining in unlabeled breast tissue.
- Quantitative phase imaging: Quantitative phase imaging generated virtual H&E, Jones, and MT stains matching histochemically stained counterparts in staining quality.QPI also enabled real-time virtual staining of human sperm cells for fertility evaluation.
- Other label-free modalities: Virtual staining has been demonstrated across bright-field, photoacoustic, ultraviolet, and nonlinear optical imaging modalities.Examples include H&E, picrosirius red, Giemsa, and other stains generated for tissue, blood, brain, bone, liver, and colon samples.
- Scope: A shallow pixel-wise FT-IR-to-bright-field mapping ignores 2D tissue spatial information and therefore shows limited staining performance.The limitation is attributed to the absence of deeper convolutional layers that process tissue texture.
- In vivo virtual staining: Reflectance confocal microscopy enabled in vivo virtual H&E staining of human skin without a biopsy.The approach could support rapid diagnosis of malignant skin neoplasms while avoiding unnecessary biopsies, scars, and cumbersome preparation.
Stain-to-stain transformations
Stain-to-stain transformations digitally convert one histological stain into another, commonly using accessible H&E images as input. They generate special, IHC, and immunofluorescence stains while enabling additional contrasts and comparisons from the same tissue field.
- Applications: Using stain transformations, researchers compared glomerulus segmentation accuracy under different stains within the same field of view.Such within-section comparisons are generally unavailable in standard histology because a tissue section is typically stained only once.
- Special stains: Stain-to-stain transformations generate additional contrast for applications including liver-fibrosis staging and multiplexed kidney-stain analysis.Reported outputs include trichrome, PAS, MT, and PASM stains.
- Special stains: H&E is a common source stain because it is widely accessible and cost-effective, while special stains reveal structures not shown by H&E.Deep networks have transformed H&E kidney samples into Jones silver, MT, and PAS stains, improving diagnostic accuracy in a blinded study.
- Immunofluorescence: Virtual immunofluorescence can be generated from Ki-67 IHC or H&E images to represent biomarkers in lung, bladder, and pancreatic cancer samples.IF uses fluorescent labels and can provide improved sensitivity and signal amplification through antigen recognition.
- Other transformations: Transformations can also use non-FFPE preparations and rapidly acquired stains, including Hoechst-stained fresh mouse brain images.Hoechst staining was described as very fast and relatively simple.
Training data preparation
Training-data preparation addresses registration, normalization, and domain shifts between label-free inputs and stained targets. These steps improve alignment and learnability while accommodating imaging artifacts and laboratory-dependent staining variation.
- Image registration: Supervised frameworks commonly use multi-stage registration to align input images with stained ground-truth images at pixel level.Registration may combine coarse correlation, affine feature matching, and finer elastic registration.
- Image registration: Feature-based registration methods use SIFT or SURF descriptors with RANSAC or affine transformations to align image pairs.These methods match structural features or nuclear masks and remove erroneous matches before transformation.
- Domain shifts: Domain shifts arise from imaging hardware, acquisition environments, specimen characteristics, sample preparation, and chemical staining variation.Input shifts affect label-free images, while target shifts include inconsistent color and contrast across laboratories and histotechnologists.
- Image correction: Pre-trained networks can correct non-ideal inputs such as defocus, motion blur, and readout errors before virtual staining.An autofocusing network restored randomly defocused images, enabling joint virtual staining and faster whole-slide imaging without fine autofocusing during scanning.
- Stain variation: Stain normalization uses color deconvolution, optical-density mapping, or deep learning to reduce chemically induced color and contrast variation.Deep learning-based normalization accounts for spatial tissue features, while style transfer can expose models to multiple H&E staining styles.
- Stain variation: Style-transfer augmentation can improve robustness across inter-technician, inter-laboratory, and inter-equipment H&E variations.This approach transforms H&E images into different styles for training stain-to-stain transformation networks.
Network architecture and training strategies
Virtual staining commonly uses GAN-based image-to-image architectures, with supervised methods benefiting from registered pairs and unsupervised methods addressing unpaired domains. Training combines adversarial objectives with pixel-wise, structural, cyclic, or task-specific constraints to improve fidelity, while remaining limited by hallucinations, intensity mismatch, and data availability.
- GAN-based architectures: GANs use competing Generator and Discriminator networks to produce virtually stained images that resemble histologically stained targets.The Discriminator distinguishes generated from target images, while adversarial feedback updates the Generator.
- GAN-based architectures: Standard adversarial training can mimic target colors and patterns without learning input–target correspondence, causing severe micro-scale hallucinations.Pixel-wise losses such as MAE, MSE, SSIM, Huber, reversed Huber, and color-distance terms are added to regularize training and suppress artifacts.
- Supervised learning: Supervised virtual staining is preferred when precisely registered input–target pairs exist and has succeeded across multiple tissue–stain combinations.Its pixel-wise objectives can be accurately evaluated using well-registered training datasets, but paired same-field images may be difficult or impossible to acquire for stain-to-stain tasks.
- Supervised learning: Cascaded networks improve stain-to-stain transformations by combining a virtual staining network with a jointly optimized stain-transfer network and structural losses.The C-DNN applies structural loss terms such as MAE directly to histochemically stained images from both domains.
- Unsupervised learning: CycleGAN variants enable unsupervised transformations without cross-registered pairs by learning two inverse mappings linked through cycle-consistency losses.Generators map between domains X and Y in both directions, using cycle-consistency and adversarial losses; perceptual embedding consistency can further improve performance.
- Unsupervised learning: Unsupervised performance is generally inferior to supervised learning, but it remains useful when paired image datasets are unavailable.Intensity mismatch between dark label-free inputs and bright-field stained outputs creates an additional transformation challenge addressed with intensity inversion and specialized losses.
- Recent innovations: Customized architectures and losses, including pathology consistency, parallel feature fusion, and pyramid pix2pix, have improved reported virtual staining quality.Reported examples include H&E-to-Ki-67 improvements, enhanced virtual H&E from autofluorescence, and better H&E-to-IHC transformation than some popular algorithms.
Virtual Staining Model Evaluation
Virtual staining models require both image-quality measurements and histology-aware validation. Because numerical scores may not capture diagnostically meaningful features, evaluations increasingly compare extracted pathology features and involve pathologists or downstream diagnostic models.
- Quantitative evaluation: Virtual staining models are assessed qualitatively and quantitatively using image-quality metrics when reference images are available.Common paired-reference metrics include SSIM, PSNR, MS-SSIM, MSE, and MAE.
- Quantitative evaluation: Reference-free metrics such as FID and IS evaluate generative outputs when paired input and ground-truth images are unavailable.These metrics compare statistical similarity through high-level features extracted by a trained network.
- Feature-based evaluation: Histology-aware evaluation extracts cellular or tissue features from virtual and ground-truth images and statistically compares their correspondence.Examples include nuclear, membrane, Ki-67, cytoplasm, nucleus, epithelium, lumen, tumor, and stroma features.
- Feature-based evaluation: Feature-based analyses have reported agreement between virtual outputs and chemically stained references across multiple segmented structures and stain types.Reported validations included HER2 statistical signatures, Ki-67 regions, segmented cytoplasm and nucleus, epithelium and lumen, and tumor and stroma.
- Clinical validation: Algorithmic scores cannot always reflect diagnostic value because pathologically meaningful features are complex and may not be captured by simple numerical rules.Deployment therefore requires validation that virtual images convey the same diagnostically relevant information as conventional stained slides.
- Clinical validation: Certified pathologists and downstream pathology models provide complementary validation of diagnostic information and can support scalable large-scale studies.Examples include nephropathologist confirmation of improved kidney-disease diagnosis, breast-pathologist HER2 assessment, cancer-stage grading, cell classification, and colonic-gland segmentation.
Discussion and Future Perspectives
Virtual staining offers rapid, chemical-free alternatives to conventional histological staining, while future progress depends on standardized data, higher throughput, stronger generalization, and rigorous clinical validation.
- Current impact: Deep learning-based virtual staining has enabled rapid, cost-effective, and chemical-free alternatives to traditional histological staining.These methods can preserve tissue and support additional histological information without the conventional staining workflow.
- Current impact: Virtual staining can generate multiple stains simultaneously from the same tissue section, increasing multiplexing capability and preserving more tissue.The review describes multiplexed stain generation and the ability to perform different stains on one section.
- Current impact: Virtual staining may provide additional information for downstream pathological signature detection, segmentation, malignancy classification, and diagnostic evaluation.Reported applications include pathological image analysis and downstream machine-vision tasks in digital pathology.
- Challenges: High-quality training data remain difficult to obtain because staining, tissue damage, registration, digitization, and inter-laboratory variability affect ground-truth consistency.Approximately 30% of samples were discarded in one HER2 staining study because of tissue loss or staining failures.
- Future directions: Future work should improve imaging throughput, standardized public datasets, model generalization, evaluation methods, and imaging of fresh tissue.Some label-free modalities remain slow, while fresh-tissue imaging could support intraoperative consultation and potentially reduce biopsy requirements in some locations.
- Challenges: Clinical adoption still requires broad multi-institution validation of accuracy and reliability across diverse patient tissues and pathologies.Primary diagnostic use has not yet arrived, and quantitative benchmarks should reflect diagnostic errors or uncertainties.