Source-linked AI summary
Iterative annotation to ease neural network training: Specialized machine learning in medical image analysis
Brendon Lutnick, Brandon Ginley, Darshana Govind, Sean D. McGarry, Peter S. LaViolette, Rabi Yacoub, Sanjay Jain, John E. Tomaszewski, Kuang-Yu Jen, Pinaki Sarder
TL;DR
Medical-image neural networks face costly, time-consuming expert annotation and limited training data. The authors pair ImageScope with iterative human correction of predictions; glomerular annotation speed rose about 4–10-fold, with nearly perfect holdout performance by iteration 4.
Problem
Medical imaging adoption of neural networks is constrained by limited curated training sets and expert annotation that is expensive, time-consuming, labor-intensive, and difficult to transfer into training data.
Source: Introduction · p. 2Method
The authors use an ImageScope-compatible human-in-the-loop pipeline in which experts correct network predictions and incorporate them into training across iterations.
Source: Conclusions · p. 8, Methods · p. 9Results
Approximately 4–10-fold increases in average glomerular annotation speed accompanied network performance reaching nearly perfect on a holdout dataset by annotation iteration 4.
Source: Results · p. 3Takeaways & Limitations
Newly added data can serve as holdout validation for qualitative review and selection of poorly predicted examples, while achievable performance remains bounded by manual annotation of all training data.
Source: Conclusions · p. 8The IFTA network was limited by the biological expertise needed for multiclass annotations, and preliminary results used only 15 annotated biopsies, a small training set.
Source: Results · p. 6
Abstract
from arXiv · showhide
Neural networks promise to bring robust, quantitative analysis to medical fields, but adoption is limited by the technicalities of training these networks. To address this translation gap between medical researchers and neural networks in the field of pathology, we have created an intuitive interface which utilizes the commonly used whole slide image (WSI) viewer, Aperio ImageScope (Leica Biosystems Imaging, Inc.), for the annotation and display of neural network predictions on WSIs. Leveraging this, we propose the use of a human-in-the-loop strategy to reduce the burden of WSI annotation. We track network performance improvements as a function of iteration and quantify the use of this pipeline for the segmentation of renal histologic findings on WSIs. More specifically, we present network performance when applied to segmentation of renal micro compartments, and demonstrate multi-class segmentation in human and mouse renal tissue slides. Finally, to show the adaptability of this technique to other medical imaging fields, we demonstrate its ability to iteratively segment human prostate glands from radiology imaging data.
Introduction
Neural networks can learn image representations for classification, but their data needs and the demands of expert annotation constrain adoption in biological and medical imaging. The paper introduces an iterative human-feedback pipeline for WSI segmentation intended to ease annotation.
Introduction
- Generalized high performance from neural networks may require tens of thousands of annotated examples, despite their ability to learn image representations for classification.Convolutional neural networks learn spatial representations using convolutional filters rather than relying on hand-engineered filtering.Source: Introduction · p. 2
- Active-learning workflows use user corrections of network predictions to identify performance gaps and refine training sets.The described applications initiate this process with low classification probabilities.Source: Introduction · p. 2
- Biological and medical adoption is constrained by scarce curated training sets and expert annotation that is expensive, time-consuming, labor-intensive, and difficult to transfer technically.Medical-image annotation requires domain expertise, making training-set creation less feasible than for traditional datasets.Source: Introduction · p. 2
- Neural-network tissue segmentation could support current diagnosis, future guideline refinement, and searchable repositories of labeled images and clinical outcomes.The passage presents these as potential benefits of digital pathology.Source: Introduction · p. 2
- H-AI-L uses repeated expert-feedback training rounds to optimize WSI segmentation and improve training efficiency with limited initial annotated slides.The pipeline is presented as a schematic for semantic segmentation of whole slide images.Source: Introduction · p. 3
Results
The results assess H-AI-L for renal WSI segmentation, including iterative glomerular annotation and multiclass renal findings, and extend the pipeline to prostate MRI. Across these tasks, the reported outcomes include improved holdout performance and reduced expert annotation.
Results
- The glomerular network was trained for five iterations on PAS- and H&E-stained mouse sections, with STZ-induced diabetic nephropathy data added at iteration 4.Testing used four holdout WSIs, including one STZ-induced WSI.Source: Results · pp. 3, 5, Table 1 · p. 5
- Glomerular annotation speed increased approximately 4 to 10-fold, with annotator time savings of 81.4%, 82%, and 72.7%; holdout performance was nearly perfect by iteration 4.These savings compare each annotator with their own baseline speed.Source: Results · pp. 3–4
- DeepZoom first identifies hotspots at 1/16th scale, then segments them at high resolution, yielding approximately 4.5-times faster predictions than a full-resolution pass.The method also obtained better F-measure than a full-resolution pass; its low-resolution pre-pass limits background seen by the high-resolution network, reducing false positives.Source: Results · pp. 4–5
- On four holdout WSIs, DeepZoom reached sensitivity 0.92 ± 0.02, specificity 0.99 ± 0.001, precision 0.93 ± 0.14, and accuracy 0.99 ± 0.001 after iteration 5.DeepZoom outperformed full-resolution analysis overall on F1 score.Source: Figure 2 · p. 4, Results · pp. 4–5
- Network predictions sometimes outperformed flawed manual annotations, including missed glomeruli or poorly drawn boundaries, especially in WSIs annotated in early iterations.The examples compare iteration-0 manual annotations with iteration-5 predictions.Source: Results · p. 5
- Preliminary holdout human WSI results show IFTA segmentation after training on 15 annotated biopsies; non-sparse IFTA regions were processed with a high-resolution pass only.The authors note that biological expertise required for multiclass annotation has limited network development.Source: Results · p. 6, Figure 5 · p. 6
- Transfer learning adapted the high-resolution network to podocyte and non-podocyte nuclei using 143 labeled mouse glomeruli, while the low-resolution glomerulus detector remained unchanged.The authors expect results to improve with more labeled training data.Source: Results · pp. 5–6
- On prostate MRI, holdout performance included sensitivity 0.88 ± 0.04 and specificity 0.99 ± 0.001; annotation fell approximately 90 percent after iteration 2 and 92% by iteration 5 at F1 > 0.88.The holdout comprised seven patients; four new patients were added to training before each iteration.Source: Results · pp. 6–7, Figure 6 · p. 7
Conclusions
The authors present an ImageScope-based human-in-the-loop pipeline for WSI segmentation, aiming to make neural-network analysis more accessible to pathologists. They report substantial annotation-time savings and describe its adaptability to different segmentation definitions and imaging modalities.
Conclusions
- 81.3%, 82.0%, and 72.7% annotation-time savings for annotators 1–3, respectively, relative to baseline; Figure 7 compares H-AI-L with baseline segmentation speed.The plotted annotation times are normalized to each annotator’s baseline speed.Source: Figure 7 · p. 8, Conclusions · p. 8
- ImageScope lets experts correct network predictions and add them to training data, while viewing predictions for qualitative assessment after each iteration.The authors argue that faster annotation and selection of poorly predicted cases may help H-AI-L reach its training-set performance upper bound more efficiently.Source: Conclusions · p. 8
- Parameter transfer supports tailoring segmentation to different clinical or research definitions and biomedical imaging modalities.Multiresolution DeepZoom analysis enables rapid prediction of sparse regions in large WSIs without sacrificing accuracy due to low-resolution analysis alone.Source: Conclusions · p. 8
- Non-sparse WSI segmentation is achievable by foregoing DeepZoom analysis.The authors also identify extensive clinical and ergonomic testing, anomaly detection, annotation-amount estimation, and native DICOM-viewer use as future work.Source: Conclusions · p. 9
Methods
H-AI-L combines expert correction of network predictions with iterative training, supported by tools for whole-slide processing, annotation conversion, and performance estimation.
Methods
- Annotators correct network predictions on new WSIs and return them to the training set, iteratively improving performance while optimizing expert annotation time.ImageScope annotations provide training input, and the resulting network predicts rough annotations for subsequent correction.Source: Methods · p. 9
- H-AI-L bridges DeepLab pixel-wise masks and ImageScope XML annotations, whose regions are represented by boundary vertices.Contour detection supports conversion to ImageScope's hierarchical XML format.Source: Methods · pp. 9–10
- Project functions automate slide chopping, augmentation, transfer learning, iterative training, and prediction, with corrected XML predictions added to training.Training blocks are 500 x 500 pixels with 50% overlap; augmentation includes flips, color shifts, and piecewise affine transformations.Source: Methods · p. 10
- For sparse renal structures, a low-resolution pass identifies hotspot regions before full-resolution segmentation.Tissue regions are first identified; blocks are screened at 1/16 resolution, stitched into a hotspot map, then extracted for pixel-wise segmentation.Source: Methods · pp. 10–11
- A validation option evaluates each iteration on holdout images using XML ground truth, while the reported animal, biopsy, and MRI data were collected under ethics approvals and consent.Validation calculates sensitivity, specificity, accuracy, and precision.Source: Methods · pp. 9, 11
- Annotation time is normalized to iteration 0 and approximated by exponential decay; the H-AI-L factor τ is lower for faster-decaying training curves.The method estimates normalized H-AI-L and baseline annotation times, then expresses time savings as a percentage.Source: Methods · pp. 11–12
Competing interests
The authors declare no competing interests; the accompanying figures depict network prediction evolution and the H-AI-L/DeepZoom workflow.
Competing interests
- The authors declare that they have no competing interests.Source: Competing interests · p. 14
- Supplemental Figure 1 is titled “Network prediction evolution.”Source: Figure 1 · p. 15
- Supplemental Figure 2 outlines DeepZoom H-AI-L semantic segmentation of whole-slide images.The accompanying overview describes low-resolution block screening, hotspot extraction, high-resolution segmentation, and predictions displayed in ImageScope.Source: Figure 2 · p. 15, Competing interests · p. 15