Source-linked AI summary
Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology
David Tellez, Geert Litjens, Peter Bandi, Wouter Bulten, John-Melle Bokhorst, Francesco Ciompi, Jeroen van der Laak
TL;DR
CNNs trained on one laboratory’s stained images often generalize poorly to other laboratories, motivating systematic evaluation of stain augmentation and normalization. The paper compares these techniques across four multicenter classification applications and introduces an unsupervised neural-network normalization method. The experiments support using color augmentation, with normalization offering similar high performance but added computational cost.
Problem
Stain variation across laboratories can cause CNNs trained on one center’s images to underperform on images from other centers, while the techniques addressing this had not been systematically evaluated.
Method
The study compares stain color augmentation and normalization across four multicenter patch-classification tasks and trains an unsupervised neural network as an image-to-image stain-normalization model.
Results
Color augmentation substantially improved classification generalization, while neural-network normalization methods achieved similar high benchmark performance.
Takeaways & Limitations
Color augmentation should be used; normalization can be combined with it, or omitted when computational resources are limited at a negligible performance reduction and substantial speed improvement.
Takeaways & Limitations
Experiments were limited to patch-based classification, excluding segmentation, instance detection and whole-slide classification, and individual augmentation effects were not quantified.
Abstract
from arXiv · showhide
Stain variation is a phenomenon observed when distinct pathology laboratories stain tissue slides that exhibit similar but not identical color appearance. Due to this color shift between laboratories, convolutional neural networks (CNNs) trained with images from one lab often underperform on unseen images from the other lab. Several techniques have been proposed to reduce the generalization error, mainly grouped into two categories: stain color augmentation and stain color normalization. The former simulates a wide variety of realistic stain variations during training, producing stain-invariant CNNs. The latter aims to match training and test color distributions in order to reduce stain variation. For the first time, we compared some of these techniques and quantified their effect on CNN classification performance using a heterogeneous dataset of hematoxylin and eosin histopathology images from 4 organs and 9 pathology laboratories. Additionally, we propose a novel unsupervised method to perform stain color normalization using a neural network. Based on our experimental results, we provide practical guidelines on how to use stain color augmentation and stain color normalization in future computational pathology applications.
1. Introduction
The paper examines how stain variation across pathology laboratories impairs CNN generalization and systematically compares augmentation and normalization approaches across multicenter computational pathology tasks.
- Motivation: Inter-center stain variation causes CNNs trained at one pathology laboratory to underperform on images from another center.The paper links this generalization problem to differences in color distributions between laboratories.
- Approaches: Stain color augmentation simulates realistic training-data variation, while stain color normalization matches training and test color distributions.These approaches represent the two main solution categories examined in the study.
- Study aim: The study addresses limited systematic evidence by comparing these techniques across four patch-based classification applications using data from multiple external laboratories.Prior work focused on particular applications without quantifying the relationship between these techniques and CNN performance.
- Contributions: The authors systematically evaluate established augmentation and normalization algorithms and formulate stain normalization as an unsupervised image-to-image translation task.A neural network is trained to reconstruct original images from heavily augmented H&E images.
2. Materials
The materials comprise four patch-based H&E classification applications trained with Radboudumc images and tested on internal or external datasets from varied centers and staining conditions.
- Dataset design: All four classifiers were trained exclusively on Radboudumc images, while images from remaining centers were reserved for testing.The study used RGB patches of 128x128 pixels extracted from annotated regions.
- Mitosis detection: The mitosis task used 1M training patches from 14 H&E breast-cancer WSIs and 300K external patches from the TUPAC dataset.The task classified patches containing a mitotic figure at their center.
- Prostate epithelium detection: The prostate task trained on 250K patches from 25 Radboudumc WSIs and tested on differently stained Radboudumc slides plus the prostate-cedar dataset.The external Radboudumc test set had substantially different stain appearance, while prostate-cedar contributed 65K patches.
- Colorectal tissue classification: The colorectal task trained on 450K patches from 54 Radboudumc WSIs and used external datasets containing 35K and 4K patches.The training task distinguished nine colorectal tissue classes, whereas the Heidelberg dataset provided six tissue types.
- Normalization data: An unlabeled multi-organ dataset containing 500K balanced patches supported training of the stain-normalization network.Patches were aggregated from the four Radboudumc task datasets after labels were discarded.
3. Methods
The study evaluates stain color augmentation and normalization as complementary strategies for handling stain-distribution differences, including a novel unsupervised neural-network normalizer.
- Methods: The experiments compare multiple stain color augmentation and normalization methods for CNN classification.The augmentation and normalization techniques are organized and visually compared across datasets.
- Stain color augmentation: Stain color augmentation transforms homogeneous training colors into a more diverse distribution to improve generalization to varied test stains.The augmentation function maps training images from φtrain to φaugment.
- Stain color augmentation: The augmentation groups progress from basic geometric changes through morphology, brightness and contrast, HSV, and HED color transformations.Basic augmentation is included across cases; HSV and HED each have light and strong configurations.
- Stain color normalization: Stain color normalization maps training and test color distributions to a shared template distribution φnormal.The normalization function g is evaluated through several established methods and a proposed neural-network approach.
- Stain color normalization: The novel network-based normalizer learns an unsupervised many-to-one mapping from augmented color distributions to a template distribution.It is trained end-to-end to remove augmentation effects; long skip connections preserve spatial structure while the network modifies color features.
- Color-distribution analysis: Normalization clusters dataset color distributions, whereas color augmentation scatters them in HSV hue-saturation space.The analysis uses mean and standard deviation pixel intensities for hue and saturation.
4. Experimental results
Across four classification tasks, stain color augmentation was the strongest contributor to top CNN performance, while normalization improved robustness to augmentation choices without being necessary for peak performance.
- Effects of stain color augmentation: Stain color augmentation was crucial for top classification performance regardless of the normalization technique.HSV or HED augmentation outperformed brightness-and-contrast augmentation alone, which produced mediocre results.
- Effects of stain color augmentation: HED and HSV augmentation showed no substantial performance difference, while light and strong color augmentation performed similarly with a slight advantage for light.Heavy augmentation reduced internal-test performance by less than 1% on average across organs.
- Effects of stain color augmentation: Brightness-and-contrast augmentation performed best among non-color augmentation settings, followed by morphology and basic augmentation.This ordering held across almost all stain color normalization setups.
- Effects of stain color normalization: Overall top performance was achieved without color normalization, indicating that normalization was not necessary for high classification performance.Normalization nevertheless generally made classifiers more robust to different color augmentation techniques.
- Effects of stain color normalization: Network and Style normalization had no substantial performance difference, while classical LUT and Deconvolution methods did not surpass Grayscale.The authors hypothesize that classical methods may hide useful image features and add input noise.
- Computational cost: Normalizing a 50000 × 50000 RGB-pixel whole-slide image took 21.8 min with LUT, 26.0 min with Network, 111.2 min with Deconvolution, and 217.8 min with Style.These times exclude I/O delays.
5. Discussion
Stain color augmentation substantially improved generalization to unseen stain variations, while normalization alone was insufficient and sometimes unnecessary for top performance. The study also found that augmentation color choice and intensity mattered little overall, but experiments were limited to patch-based classification.
- Augmentation: Stain color augmentation drastically improved CNN classification performance across most normalization settings by improving generalization to unseen stain variations.HSV and HED transformations were the key ingredients; removing them and using BC augmentation produced lower AUC under all circumstances.
- Augmentation: HSV and HED color transformations outperformed BC augmentation under all circumstances, suggesting inter-laboratory differences were driven mainly by color rather than morphology.The comparison indicates that excluding color transformations reduced AUC consistently.
- Augmentation: CNN performance was mostly insensitive to whether HSV or HED augmentation, or light or strong intensity, was used, provided color augmentation was applied.Simpler normalization techniques required stronger augmentation to perform well.
- Normalization: The best-performing configuration used no stain color normalization, challenging the assumption that normalization is necessary for top histopathology classification performance.Normalization also added computational overhead that could substantially reduce classification speed.
- Normalization: Using stain color normalization without proper stain color augmentation produced poor performance and was insufficient to reduce generalization error caused by stain variation.The authors hypothesize that color information can promote overfitting even with excellent normalization, worsening with suboptimal normalization.
- Limitations: The study’s experiments were limited to patch-based classification and did not quantify individual augmentation techniques or broad hyperparameter ranges.Segmentation, instance detection, and whole-slide classification were excluded because of computational constraints.
6. Conclusion
Across four computational pathology applications and data from nine centers, the study recommends using stain color augmentation and selectively combining it with neural-network normalization. Omitting normalization can preserve performance while improving processing speed when resources are limited.
- Conclusion: The study quantified stain color augmentation and normalization effects across four computational pathology applications using data from nine pathology centers.The evaluation was designed to compare these techniques across multicenter classification settings.
- Conclusion: Any stain color augmentation, using HSV or HED transformation, should be used to improve robustness to stain variation.The authors recommend tuning augmentation intensity to light when normalization is enabled and strong when it is disabled.
- Conclusion: Color augmentation can be combined with neural-network stain color normalization for more robust classification performance.When computational resources are limited, normalization can be omitted with negligible performance reduction and substantially faster processing.