Source-linked AI summary
How much data is needed to train a medical image deep learning system to achieve necessary high accuracy?
Junghwan Cho, Kyewook Lee, Ellie Shin, Garry Choy, Synho Do
TL;DR
Medical image deep-learning systems need high and consistent accuracy, but privacy restrictions make large datasets difficult to assemble. This paper applies a learning-curve approach to six-class axial CT anatomy classification to estimate the training data required for a target accuracy. The curve predicted 98% accuracy at 1000 images per body class and estimated 4092 images per class for 99.5% accuracy.
Problem
The paper asks how much training data medical image deep-learning systems need to achieve high accuracy, a question not systematically explored in recent medical image publications.
Method
The study trains GoogLeNet on six anatomical CT classes at six training-set sizes and fits a weighted inverse-power learning curve to the observed accuracies.
Results
4092 training images per class were predicted to reach 99.5% accuracy, while 1000 per class yielded 98% predicted and 97.25% observed accuracy.
Takeaways & Limitations
Learning-curve extrapolation provides a methodology for estimating target training-set sizes in medical image deep-learning problems.
Takeaways & Limitations
Medical-image access and dataset scale are constrained by patient privacy and security policies, making databases with millions of cases difficult to obtain.
Abstract
from arXiv · showhide
The use of Convolutional Neural Networks (CNN) in natural image classification systems has produced very impressive results. Combined with the inherent nature of medical images that make them ideal for deep-learning, further application of such systems to medical image classification holds much promise. However, the usefulness and potential impact of such a system can be completely negated if it does not reach a target accuracy. In this paper, we present a study on determining the optimum size of the training data set necessary to achieve high classification accuracy with low variance in medical image classification systems. The CNN was applied to classify axial Computed Tomography (CT) images into six anatomical classes. We trained the CNN using six different sizes of training data set (5, 10, 20, 50, 100, and 200) and then tested the resulting system with a total of 6000 CT images. All images were acquired from the Massachusetts General Hospital (MGH) Picture Archiving and Communication System (PACS). Using this data, we employ the learning curve approach to predict classification accuracy at a given training sample size. Our research will present a general methodology for determining the training data set size necessary to achieve a certain target classification accuracy that can be easily applied to other problems within such systems.
1 INTRODUCTION
Medical images are promising for deep learning because of their standardized, high-quality structure and associated annotations, but privacy constraints make large datasets difficult to obtain. The paper therefore asks how much training data is needed to achieve accurate, consistent medical image classification and proposes a general learning-curve methodology.
- 1 INTRODUCTION: Hand-crafted features limit existing CADe and CADx systems, whereas deep learning can extract image features itself.
- Medical image uniqueness: Large medical-image datasets and parallel computing could improve automated detection and diagnosis, motivating deeper use of machine learning.
- 1 INTRODUCTION: Patient privacy and security policies make medical images difficult to access and make databases with millions of cases extremely difficult to obtain.
- 1 INTRODUCTION: The paper addresses the unstudied question of how much data is needed for high-accuracy medical image deep learning.
- 1 INTRODUCTION: It chooses learning curves to estimate necessary training-set sizes and presents a methodology intended for application to other medical image problems.
- Medical image uniqueness: Medical images offer standardized formats, quality control, and radiologist annotations that support supervised deep-learning applications.Images are stored in DICOM format, undergo quality oversight, and are accompanied by structured reports in the EHR.
2 MATERIAL AND METHOD
The study classifies six anatomical regions in axial CT images using GoogLeNet and evaluates how training-set size affects accuracy and variability. It fits a weighted inverse-power learning curve to experimental results to estimate the data required for a target accuracy.
- Data preparation: MGH PACS CT images were annotated into six anatomical classes: brain, neck, shoulder, chest, abdomen, and pelvis.Only clearly defined regions were used; transition regions were excluded from training because their regional boundaries were unclear.
- Problem setting: Anatomical variation and artifacts across body regions may require very large training datasets to achieve desired classification accuracy.
- Convolutional neural network: GoogLeNet was selected and used to learn hierarchical image features through convolutional layers and Inception modules.The network includes 22 convolutional layers, nine Inception modules, and multiple kernel sizes.
- Learning curve: Classification accuracy was modeled as an inverse power-law function of training-set size, with parameters representing learning and decay rates.The model assumes accuracy approaches 100% asymptotically.
- Learning curve: Weighted nonlinear least squares was used because larger training sets produced lower-variance accuracy observations.The selected weights were wp = {1, 1, 1, 1, 100, 150}.
3 RESULTS AND DISCUSSIONS
The CNN classified six anatomical classes from CT images, with accuracy improving and variability generally decreasing as training data increased. Learning-curve extrapolation estimated the data needed to reach a 99.5% target accuracy.
- Near-100% accuracy was achieved for all defined body parts in a representative repetition using 200 training samples.The system also classified transition-region images that were not included during training.
- Above 95% recognition rate defined correct classification, including images with varied spatial complexity and variability.The classifier identified images using features learned during training.
- Increasing training-set size improved classification accuracy and reduced misclassified images across body-part classes.The standard deviation of misclassified images generally decreased with larger training sets, except at sizes 5 and 10.
- Accuracy increased rapidly from training sizes 5 to 50, then changed little from 100 to 200 as the learning curve approached a steady state.The weighted least-squares curve fit the large-sample points better than the smaller-sample points.
- The learning curve predicted 98% accuracy at 1000 samples per body class versus 97.25% observed, and estimated 4092 samples per class for 99.5% accuracy.The authors therefore selected a baseline of 5000 samples per class for future training.
4 CONCLUSION
The conclusion frames medical image learning as promising but constrained by privacy-related access difficulties and stringent accuracy requirements. It presents learning-curve extrapolation as a transferable way to estimate training-data needs, subject to data-quality and sampling prerequisites.
- Medical images are highly suitable for deep learning and require exceptional sensitivity and specificity because of their role in diagnosis and treatment planning.
- Patient privacy laws and policies make medical images difficult to access, sharpening the question of how much data each task requires.
- Learning-curve extrapolation estimates required training-data size and produced agreeable results when applied to this system.The authors state that the method can be applied to future medical image deep-learning problems.
- Accurate learning-curve prediction requires high-quality data, systematically increased sampling points, and repeated measurements for estimating the statistical mean.
- The approach is intended for extension to disease and organ classification, abnormal-area detection, and other medical imaging modalities.