Source-linked AI summary
Fine-Tuning and Training of DenseNet for Histopathology Image Representation Using TCGA Diagnostic Slides
Abtin Riasatian, Morteza Babaie, Danial Maleki, Shivam Kalra, Mojtaba Valipour, Sobhan Hemati, Manit Zaveri, Amir Safarpoor, Sobhan Shafiei, Mehdi Afshari, Maral Rasoolijaberi, Milad Sikaroudi, Mohd Adnan, Sultaan Shah, Charles Choi, Savvas Damaskinos, Clinton JV Campbell, Phedias Diamandis, Liron Pantanowitz, Hany Kashani, Ali Ghodsi, H. R. Tizhoosh
TL;DR
The paper tackles the limited availability of labeled pathology images for training domain-specific representations. It develops KimiaNet from DenseNet using high-cellularity mosaics and weakly labeled TCGA patches, then evaluates it across public search and classification datasets. KimiaNet outperforms the original DenseNet and smaller CBR networks as a histopathology feature extractor.
Problem
Histopathology representation is difficult because WSIs are large and complex, while fine-tuning or training deep networks is impeded by limited labeled images and computational demands.
Method
KimiaNet fine-tunes and trains a DenseNet using high-cellularity mosaics of 1000 × 1000 patches at 20× magnification selected from weakly labeled TCGA WSIs.
Results
KimiaNet provides superior feature-extraction results to the original DenseNet and smaller CBR networks across the evaluated histopathology search and classification experiments.
Takeaways & Limitations
Fine-tuning a deep network on a sufficiently large collection of histopathology images delivered better performance than using the pre-trained network model.
Takeaways & Limitations
The high-cellularity mosaic may bias representations toward certain histologic features at the expense of visual similarities.
Abstract
from arXiv · showhide
Feature vectors provided by pre-trained deep artificial neural networks have become a dominant source for image representation in recent literature. Their contribution to the performance of image analysis can be improved through finetuning. As an ultimate solution, one might even train a deep network from scratch with the domain-relevant images, a highly desirable option which is generally impeded in pathology by lack of labeled images and the computational expense. In this study, we propose a new network, namely KimiaNet, that employs the topology of the DenseNet with four dense blocks, fine-tuned and trained with histopathology images in different configurations. We used more than 240,000 image patches with 1000x1000 pixels acquired at 20x magnification through our proposed "highcellularity mosaic" approach to enable the usage of weak labels of 7,126 whole slide images of formalin-fixed paraffin-embedded human pathology samples publicly available through the The Cancer Genome Atlas (TCGA) repository. We tested KimiaNet using three public datasets, namely TCGA, endometrial cancer images, and colorectal cancer images by evaluating the performance of search and classification when corresponding features of different networks are used for image representation. As well, we designed and trained multiple convolutional batch-normalized ReLU (CBR) networks. The results show that KimiaNet provides superior results compared to the original DenseNet and smaller CBR networks when used as feature extractor to represent histopathology images.
I. INTRODUCTION
The paper addresses the challenge of representing complex, large histopathology WSIs by investigating domain-specific fine-tuning and training of deep networks. It proposes KimiaNet and evaluates its features across pathology search and classification tasks.
- Histopathology WSIs require compact, expressive representations because they are large and contain complex patterns.
- KimiaNet features are evaluated for search and classification on three public datasets and compared with original DenseNet and smaller trained networks.
- Prior pathology studies used pre-trained, fine-tuned, or trained networks for classification, retrieval, and WSI-level diagnosis, including approaches based on VGG-19, Inception V3, ResNet, and recurrent networks.
- Deep embeddings from pre-trained networks are widely used in pathology, but the effects of fine-tuning and training from scratch across tasks remain underexplored.
- The study proposes cellMosaic, a high-cellularity patch-selection method that enables weak WSI-level labels from TCGA to train a densely connected network.
III. KIMIANET - DATA AND TRAINING
KimiaNet uses DenseNet as its architectural basis because DenseNet is established, compact, and suitable for image representation. The paper positions architecture customization as necessary for application-specific pathology tasks.
- KimiaNet is based on DenseNet, a densely connected architecture composed of dense blocks preceded by convolutional and pooling layers.
- The paper connects architecture customization with application-oriented categorization and end-user awareness in sensitive tasks.
- DenseNet was selected because it is a dependable topology with a compact footprint for image representation.
A. Public Image Datasets
The study uses three public pathology datasets spanning whole-slide and labeled-patch data. TCGA supplies the large multi-organ WSI resource used to prepare training data.
- The experiments use public TCGA, endometrial cancer, and colorectal cancer datasets.TCGA contains WSIs, while the endometrial and colorectal datasets contain labeled patches.
- TCGA provides 30,072 WSIs across 25 anatomic sites and 32 cancer subtypes, with 29,120 files readable at 20× magnification.
- Figure 1 presents the DenseNet architecture used for KimiaNet.
- The endometrial dataset contains 3,302 images across four tissue classes, with patches extracted at 20× or 10× magnification.
3) Colorectal Cancer Dataset:
The paper processes colorectal and TCGA imagery as patches and uses modified Yottixel mosaics to derive high-cellularity training representations from weakly labeled WSIs.
- The colorectal dataset contains 5,000 samples in eight classes, with 625 patches per class and labels covering tissue, cellular, debris, adipose, and background categories.
- TCGA WSIs lack regional labels, so patches must be extracted even though each slide has only a primary diagnosis label.
- The modified Yottixel pipeline clusters patches and retains the top cellularity-ranked patches to create cellMosaic representations.
- Figure 3 contrasts selected WSI patches, an 80-patch Yottixel mosaic, and a 16-patch modified cellMosaic.
- The mosaic algorithm indexes at 20×, clusters at 5×, uses 1000 × 1000 patches, and retains the top 20% by cellularity.
C. Training Data
The training dataset was constructed from TCGA slides using detailed diagnostic groupings, selected magnification and patch settings, and cellularity-based patch filtering. KimiaNet was trained in four configurations with progressively more DenseNet blocks updated.
- Dataset construction: Two TCGA primary-diagnosis classes were removed because morphology information or sufficiently represented detailed groups were unavailable.The remaining data were organized using detailed-label combinations and established tumour-type categories.
- Patch settings: 20× magnification was chosen for detailed histology patterns, while patch size and magnification were set using empirical evidence and computational convenience.Pathologists use 10× for gross features, 20× for detailed histology, and 40× for fine nuclear and cellular details.
- Patch selection: Cellularity was used to remove many benign or healthy patches because WSI-level diagnosis labels did not directly label individual training patches.The approach targeted patches with unusually high numbers of cell nuclei, which are associated with uncontrolled cell growth in carcinomas.
- Model configurations: KimiaNet-I through KimiaNet-IV trained the last one, two, three, or all DenseNet-121 blocks, respectively.Each configuration used a different extent of fine-tuning or retraining within the same DenseNet-121 architecture.
- Training outcomes: KimiaNet-IV achieved the highest accuracy, and accuracy generally increased as more DenseNet blocks were fine-tuned.Convergence was observable after about 10 epochs in most experiments.
IV. EXPERIMENTS
The experiments assessed KimiaNet features for generalization and compared them with DenseNet features using search-based evaluation. The broader setup also considered segmentation, patching, clustering, and network hyperparameters.
- Evaluation setup: KimiaNet and original DenseNet features were extracted from the last pooling layer and evaluated across three datasets for generalization.Search used k-NN with k = 3 to retrieve the most similar feature barcodes.
- Experimental considerations: Deep-network solutions for histopathology gigapixel images require tuning network parameters and additional segmentation, patching, and clustering parameters.
A. TCGA Experiments: Classification through Search
TCGA experiments evaluated feature distinctiveness through horizontal tumour-type search and vertical primary-diagnosis search. KimiaNet outperformed DenseNet across the reported search measures.
- Search tasks: Horizontal search measured tumour-type identification across the test dataset, while vertical search identified primary diagnoses within a specific tumour site.
- Evaluation metrics: 3-nearest-neighbor horizontal search used classification accuracy, whereas vertical search used F1 scores with Hamming distance on barcoded features.
- Horizontal search: 85.4% versus 44.8% was the average horizontal-search accuracy for KN-IV and DenseNet, respectively.The corresponding standard deviations were 11.6% for KN-IV and 19.9% for DenseNet.
- Horizontal search: 99% versus 72% was the brain-search accuracy for KimiaNet and DenseNet, respectively, despite brain having the lowest reported improvement at 27%.
- Vertical search: KimiaNet achieved higher F1 scores than DenseNet for all evaluated subtypes in vertical search.The greater discrimination was also visible in the t-SNE visualization.
B. Endometrium Data Experiments: Classification
The endometrium experiments compare KimiaNet features with other deep feature extractors for image classification. KimiaNet is evaluated alongside DenseNet and fine-tuned VGG-19, with confusion matrices reported for DenseNet and KimiaNet.
- B. Endometrium Data Experiments: Classification: KimiaNet was evaluated on endometrium images against fine-tuned VGG-19 and other deep feature extractors.The comparison was designed to assess pathology feature representations for classification.
- B. Endometrium Data Experiments: Classification: VGG-19 achieved 76.38% accuracy in the endometrium classification comparison.
- B. Endometrium Data Experiments: Classification: Confusion matrices were reported for DenseNet and KimiaNet features on the endometrium dataset.
- B. Endometrium Data Experiments: Classification: On colorectal images, KimiaNet achieved higher accuracy than DenseNet and fine-tuned VGG-19, while an ensemble CNN performed better.The KimiaNet confusion matrix showed a pronounced diagonal.
E. CBR Nets: Small versus large
The paper benchmarks smaller CBR networks against KimiaNet and other feature representations across search and classification visualizations. KimiaNet achieves stronger and more consistent TCGA search results than the compared smaller topologies.
- E. CBR Nets: Small versus large: Small CBR networks can produce results comparable to larger ImageNet models in some medical-image applications.Prior work compared a roughly 2M-weight Small CBR network with much larger ResNet50 and Inception-V3 models.
- E. CBR Nets: Small versus large: CBR feature performance was very low without the additional dense layer.
- E. CBR Nets: Small versus large: KimiaNet achieved a 30% accuracy difference over the second-best topology in horizontal TCGA search.The second-best topology, LargeT, had around 1.5M more parameters than KimiaNet.
- E. CBR Nets: Small versus large: KimiaNet achieved the maximum accuracy for all 12 horizontal-search classes, while no other network did so for any class.
- E. CBR Nets: Small versus large: KimiaNet’s average vertical-search F1 score was 10% higher than the second-best LargeW network.KimiaNet also showed a low standard deviation of F1 scores relative to the compared networks.
V. SUMMARY AND CONCLUSIONS
The paper addresses pathology image representation by adapting DenseNet to weakly labeled, high-resolution TCGA patches through KimiaNet and a high-cellularity mosaic. It reports improved performance over pre-trained DenseNet and identifies bias and application-domain boundaries for future work.
- V. SUMMARY AND CONCLUSIONS: Image representation is especially challenging in digital pathology because WSIs combine complex textures, polymorphism, and very large image sizes.
- V. SUMMARY AND CONCLUSIONS: KimiaNet uses clustering-based, high-cellularity mosaics to train with WSI-level labels when pixel-level annotations are unavailable.
- V. SUMMARY AND CONCLUSIONS: The study trains a densely connected network on 1000 × 1000-pixel patches at 20× magnification from TCGA using weak labels.
- V. SUMMARY AND CONCLUSIONS: Fine-tuning a deep network on sufficiently many histopathology images delivered better performance than a pre-trained network model.
- V. SUMMARY AND CONCLUSIONS: The high-cellularity mosaic facilitated training but may bias KimiaNet toward certain histologic features at the expense of visual similarities.Future work must define the histopathology applications that benefit from KimiaNet image representation.
VI. APPENDIX
The appendix lists all 32 TCGA primary-diagnosis codes and the corresponding number of evidently diagnosed patients.
- VI. APPENDIX: Table VI presents the 32 TCGA primary-diagnosis codes in alphabetical order with corresponding patient counts.TCGA denotes The Cancer Genome Atlas.