Source-linked AI summary
Deep learning-based virtual histology staining using auto-fluorescence of label-free tissue
Yair Rivenson, Hongda Wang, Zhensong Wei, Yibo Zhang, Harun Gunaydin, Aydogan Ozcan
TL;DR
Histological staining is time-consuming and laborious, motivating alternatives that preserve visualization of tissue features without chemical stains. The paper uses a GAN-trained convolutional network to transform one label-free auto-fluorescence image into a virtually stained image, successfully matching histological features across human tissues and stains.
Problem
Histological analysis requires lengthy, laborious preparation and staining to visualize tissue features, motivating label-free alternatives.
Method
A GAN-trained convolutional neural network transforms a single wide-field auto-fluorescence image of unlabeled tissue into a bright-field-equivalent virtually stained image.
Results
0.59 sec was sufficient to generate a virtually stained image for a ~0.33 mm × 0.33 mm field of view, with strong similarity to chemically stained images.
Takeaways & Limitations
The method could provide a digital alternative to histochemical staining and simplify tissue preparation in histopathology.
Takeaways & Limitations
Training can be challenged when histochemical staining causes tissue constituents to be lost or deformed between paired input and target images.
Abstract
from arXiv · showhide
Histological analysis of tissue samples is one of the most widely used methods for disease diagnosis. After taking a sample from a patient, it goes through a lengthy and laborious preparation, which stains the tissue to visualize different histological features under a microscope. Here, we demonstrate a label-free approach to create a virtually-stained microscopic image using a single wide-field auto-fluorescence image of an unlabeled tissue sample, bypassing the standard histochemical staining process, saving time and cost. This method is based on deep learning, and uses a convolutional neural network trained using a generative adversarial network model to transform an auto-fluorescence image of an unlabeled tissue section into an image that is equivalent to the bright-field image of the stained-version of the same sample. We validated this method by successfully creating virtually-stained microscopic images of human tissue samples, including sections of salivary gland, thyroid, kidney, liver and lung tissue, also covering three different stains. This label-free virtual-staining method eliminates cumbersome and costly histochemical staining procedures, and would significantly simplify tissue preparation in pathology and histology fields.
RESULTS · Virtual staining of tissue samples
The method was blindly evaluated on previously unseen auto-fluorescence images from label-free tissue sections and generated bright-field-equivalent virtual stains. The outputs reproduced expected H&E color schemes and histological features, with confirmation from an expert pathologist.
- Virtual staining of tissue samples: The framework was blindly tested on auto-fluorescence images from label-free tissue sections excluded from training and validation.This evaluated inference on non-overlapping samples.
- Virtual staining of tissue samples: The deep CNN transformed auto-fluorescence images into bright-field-equivalent images of the same tissue sections.
- Virtual staining of tissue samples: The results demonstrated the framework’s capability to produce virtual staining from label-free tissue auto-fluorescence images.
- Virtual staining of tissue samples: The virtual stains showed the expected color scheme for H&E-stained tissue.
- Virtual staining of tissue samples: The outputs revealed epithelioid cells, cell nuclei, nucleoli, stroma, and collagen.
- Virtual staining of tissue samples: An expert pathologist confirmed that the neural-network outputs virtually stained and revealed the same histological features as chemically stained images.
Quantification of the network output image quality
The network output was quantitatively evaluated against bright-field images of chemically stained tissue using pixel-level differences and SSIM. The comparison showed strong structural similarity, while the chemically stained images were not a true gold standard because staining and dehydration can alter tissue structure.
- Quantitative evaluation: Pixel-level differences were calculated between chemically stained bright-field images and label-free virtually stained network outputs.The virtual images were synthesized without labels or stains.
- Quantitative evaluation: SSIM quantification showed strong structural similarity between network outputs and bright-field images of chemically stained samples.The SSIM results were summarized in Table 1.
- Limitations: Chemically stained bright-field images were not a true gold standard because histochemical staining and dehydration introduce uncontrolled tissue variations and structural changes.These changes can occur during the staining process and related dehydration.
Transfer learning to other tissue-stain combinations
Transfer learning enables a prelearnt CNN from one tissue-stain combination to initialize training for a new combination, accelerating convergence and improving performance.
- Transfer learning to other tissue-stain combinations: Transfer learning uses a prelearnt CNN from a different tissue-stain combination to initialize statistical learning for virtual staining of a new combination.The approach is intended for new tissue and/or stain types.
- Transfer learning to other tissue-stain combinations: The new-combination training procedure can converge much faster while reaching an improved performance corresponding to a better local minimum in the training cost/loss function.These favorable attributes are demonstrated in Figure 5.
DISCUSSION
The study demonstrates label-free virtual staining from a single tissue auto-fluorescence image using supervised deep learning, offering a digital alternative to histochemical staining. The approach is fast after training but requires tissue/stain-specific models and careful handling of tissue changes during staining.
- Method and applicability: Supervised deep learning virtually stains label-free tissue sections from a single auto-fluorescence image captured with a standard fluorescence microscope.The approach may also support fluorescence, non-linear, holographic and optical coherence tomography modalities.
- Method and applicability: The method was demonstrated on fixed unstained tissue to enable comparison with chemically stained samples for network training and blind validation.The approach is described as broadly applicable to unsectioned, fresh tissue samples.
- Training considerations: Tissue loss or deformation during histochemical staining can mislead the training loss function when matching label-free and stained images.The issue arises because corresponding tissue features may not remain unchanged between imaging stages.
- Inference performance: After training, the feed-forward network produces virtual stains in a single non-iterative step without trial-and-error or parameter tuning.This architecture avoids iterative optimization for each new sample.
- Model specificity: The procedure trains a separate CNN for each tissue/stain combination, and a model supplied with a different combination will not perform as desired.The authors consider this acceptable because tissue and stain types are predetermined for each sample.
METHODS · Sample Preparation
The method used 2µm formalin-fixed paraffin-embedded tissue sections, acquired an initial auto-fluorescence image while unlabeled, and then processed the same slide for corresponding histochemical staining. This staining path supported training and validation but was unnecessary after network training.
- METHODS: Formalin-fixed paraffin-embedded tissue sections were prepared at 2µm thickness.
- Sample Preparation: Sections were deparaffinized using Xylene and mounted on standard glass slides with CytosealTM.
- Sample Preparation: A coverslip was placed over each mounted tissue section before imaging.
- Sample Preparation: Initial auto-fluorescence imaging used a DAPI excitation and emission filter set on the unlabeled tissue sample.
- Sample Preparation: After imaging, slides underwent Xylene treatment for approximately 48 hours before coverslip removal.
- Sample Preparation: Following coverslip removal, slides were dipped approximately 30 times in absolute alcohol, 95% alcohol, and D.I. water for ~1 min.
- Sample Preparation: The processed slides then received corresponding H&E, Masson’s Trichrome, or Jones staining procedures.
- METHODS: This tissue-processing path was used for training and validation but was not needed after the network had been trained.
Data acquisition
Label-free tissue auto-fluorescence images were acquired on a conventional fluorescence microscope with a motorized stage and automated using MetaMorph® software.
- Data acquisition: Images were captured with an IX83 Olympus fluorescence microscope equipped with a motorized stage and controlled by MetaMorph® microscope automation software.Unstained tissue samples were excited with near UV light and imaged using a DAPI filter cube.
Image pre-processing and alignment
The pre-processing pipeline registers autofluorescence input images with bright-field stained targets by matching their fields of view and addressing residual misalignment. Coarse registration can leave pixel-level discrepancies caused by sample-placement differences and slight random rotations between imaging experiments.
- Global and local registration: The registration process first matches the input and target images’ fields of view using global and local image registration.Accurate FOV matching is critical because the network learns a transformation between autofluorescence images of unstained tissue and bright-field images after staining.
- Pixel-level refinement: After FOV matching, autofluorescence and bright-field images are coarsely aligned but remain inaccurately registered at the individual-pixel level.The residual mismatch arises from slight differences in sample placement during the two microscopy experiments.
- Pixel-level refinement: Sample-placement differences can randomly produce a slight rotation of approximately 1-2 degrees between corresponding input and target images.This rotation occurs between the autofluorescence and subsequent bright-field imaging experiments.
Deep neural network architecture and training
The method uses a GAN to transform label-free autofluorescence images into corresponding bright-field images of chemically stained tissue. Its U-net generator combines multi-scale residual processing with adversarial, pixel-wise, and total-variation objectives.
- GAN framework: The GAN pairs a generator that learns the unstained-to-stained transformation with a discriminator distinguishing real stained bright-field images from generator outputs.Successful training aims to make generated images indistinguishable from stained bright-field images.
- Loss functions: The generator loss combines pixel-wise mean squared error, total variation, and discriminator prediction terms.The regularization parameters accommodate approximately 2% of the pixel-wise MSE loss and 20% of the combined generator loss, respectively.
- Generator architecture: The generator follows a U-net architecture with four down-sampling and four symmetric up-sampling steps, using multi-scale processing and residual blocks.Down-sampling connects levels through 2×2 average pooling with stride 2, reducing feature maps by a factor of 4.
- Network dimensions: The down-sampling path uses input channels 1, 64, 128, 256 and output channels 64, 128, 256, 512, while the up-sampling path maps 1024, 512, 256, 128 inputs to 256, 128, 64, 32 outputs.The final convolutional layer maps 32 channels into 3 channels, and the network uses Leaky Rectified Linear Unit activations.
- Optimization and initialization: Training uses 3×3 convolution kernels, truncated-normal initialization with standard deviation 0.05 and mean 0, zero biases, and Adam learning rates of 1×10^-4 for the generator and 1×10^-5 for the discriminator.The kernels are randomly initialized, while all network biases are initialized as 0.
Implementation details
The virtual staining network used Python 3.5.0 and TensorFlow 1.4.0, running on a Windows 10 desktop with specified Intel CPU and 64GB RAM hardware.
- Software environment: The virtual staining network was implemented in Python version 3.5.0.
- Software environment: The GAN was implemented using TensorFlow framework version 1.4.0.
- Hardware environment: The software ran on Windows 10 using an Intel Core i7-7700K CPU @ 4.2GHz and 64GB of RAM.
- Training details: Table 2 reports the number of trained patches, number of epochs, and training times.
FIGURES AND TABLES
The figures demonstrate a GAN-trained neural network that converts a single autofluorescence image of unstained tissue into a virtually stained image. Results cover H&E, Jones, and Masson’s Trichrome staining, while transfer learning improves thyroid-tissue training convergence and Table 1 evaluates image similarity and color differences.
- Virtual staining pipeline: A GAN-trained neural network rapidly transforms an autofluorescence image of an unstained tissue section into a virtually stained tissue image.This bypasses the standard chemical staining procedure used in histology.
- H&E staining results: For salivary gland tissue, virtual staining results are compared with bright-field images of the same samples after H&E histochemical staining.The autofluorescence images serve as neural-network inputs.
- Jones staining results: For kidney tissue, virtual staining results are compared with bright-field images of the same samples after Jones histochemical staining.The first two figure columns show the unstained autofluorescence inputs.
- Masson’s Trichrome results: For liver and lung tissue sections, virtual staining results are compared with bright-field images of the same samples after Masson’s Trichrome staining.Liver samples occupy rows 1–2 and lung samples occupy rows 3–4.
- Transfer learning: Transfer learning from salivary gland H&E training enables thyroid H&E staining to converge faster and reach a lower local minimum than random initialization.The new network is initialized with learned weights and biases from salivary gland tissue sections.
- Quantitative evaluation: Table 1 reports SSIM, brightness differences, and chroma differences between network outputs and bright-field images of the same chemically stained samples.Brightness and chroma differences are defined in the YCbCr color space, with average and standard deviation values reported.