PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?

Nima Tajbakhsh, Jae Y. Shin, Suryakanth R. Gurudu, R. Todd Hurst, Christopher B. Kendall, Michael B. Gotway, Jianming Liang

arXiv:1706.00712v1cs.CVcs.LG

TL;DR

The paper asks whether fine-tuning CNNs pre-trained on natural images can replace training medical-image CNNs from scratch, which is difficult with limited labeled data. It evaluates layer-wise fine-tuning across four applications and finds that deeply fine-tuned models match or outperform fully trained CNNs, especially with limited training data.

  • Problem

    Training CNNs from scratch is difficult in medical imaging because labeled data are limited and convergence requires substantial expertise.

  • Method

    The study compares CNNs trained from scratch with pre-trained CNNs fine-tuned layer by layer across four medical imaging applications and three imaging modality systems.

  • Results

    Deeply fine-tuned CNNs performed as well as fully trained CNNs and outperformed them when limited training data were available.

  • Takeaways & Limitations

    Layer-wise fine-tuning provides a practical way to adapt tuning depth to the application and available training data.

  • Takeaways & Limitations

    Reported performance may be suboptimal because exhaustive hyper-parameter optimization was infeasible across the many time-consuming CNN experiments.

Abstract

from arXiv · show

Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: \emph{Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch?} To address this question, we considered 4 distinct medical imaging applications in 3 specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from 3 different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that (1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; (2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; (3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and (4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.

I. INTRODUCTION

The paper examines whether sufficiently fine-tuned CNNs pre-trained on natural images can replace training CNNs from scratch for medical image analysis. It evaluates this question across diverse medical imaging applications while addressing the data, computation, and optimization difficulties of full training.

  • Motivation: Training CNNs from scratch requires extensive labeled data, computational resources, and expertise to manage overfitting and convergence.These requirements are especially difficult in medical imaging, where expert annotation is expensive and diseases may be scarce.
  • Study scope: The study compares pre-trained CNNs fine-tuned for medical applications with CNNs trained from scratch across four applications and three imaging modalities.The applications include polyp detection, colonoscopy video quality assessment, pulmonary embolism detection in CT, and intima-media boundary segmentation in ultrasonography.
  • Related work: Knowledge transfer from natural images to medical images remains plausible despite substantial dissimilarity between the source and target databases.Prior research indicates that transfer success depends on database distance, while medical-imaging studies have shown transfer potential.
  • Related work: Prior medical-imaging transfer-learning studies either extracted pre-trained features for a separate classifier or adapted selected network layers to the target application.The paper positions its layer-wise fine-tuning approach within the second category while distinguishing it from feature-generation methods.

III. CONTRIBUTIONS

The paper systematically studies layer-wise fine-tuning, training-set size, and comparisons with both handcrafted methods and CNNs trained from scratch. Its evidence spans four medical imaging applications involving multiple task types and modalities.

  • III. CONTRIBUTIONS: Layer-wise fine-tuning of a pre-trained CNN produces incremental performance improvement as additional layers are updated.This distinguishes the approach from feature extraction and from fine-tuning the entire network at once.
  • III. CONTRIBUTIONS: The study analyzes how training-sample availability influences the choice between pre-trained CNNs and CNNs trained from scratch.The authors state that this issue had not been systematically addressed in the medical imaging literature to their knowledge.
  • III. CONTRIBUTIONS: The experiments compare pre-trained CNNs with handcrafted approaches and CNNs trained from scratch using medical imaging data.This broadens the comparison beyond studies that primarily evaluated pre-trained CNNs against handcrafted methods.
  • III. CONTRIBUTIONS: The paper reports consistent results across four applications involving classification, detection, and segmentation in three imaging modalities.The authors contrast this breadth with prior conclusions based on a single medical imaging application.

IV. CONVOLUTIONAL NEURAL NETWORKS (CNNS)

CNNs use convolutional layers to detect local image features hierarchically, and they are trained by minimizing a cost function with back-propagation and stochastic gradient descent.

  • Architecture: Convolutional layers connect each node to a small spatially connected subset of input neurons and use shared weights to detect local structures across images.Each shared weight set is a convolution kernel, and the resulting feature maps show the strength of detected features.
  • Architecture: A convolutional layer with n kernels learns n local features, while pooling layers reduce computational complexity and support hierarchical feature extraction.The convolution-and-pooling workflow builds increasingly abstract image representations.
  • Optimization: CNN training minimizes a cost function with respect to network weights using back-propagation.The cost is based on the probability assigned to the correct label for each training image.
  • Optimization: Stochastic gradient descent approximates the full training-set cost using mini-batches of data.The update process uses the cost calculated over each mini-batch rather than the entire dataset at every step.
  • Optimization: Layer updates incorporate a layer-specific learning rate, momentum, and a scheduling rate that decreases learning rates after each epoch.These terms determine how weights change during iterative optimization.

V. FINE-TUNING

Fine-tuning initializes a target CNN from a pre-trained network, replaces its task-specific final layer, and progressively updates more layers. The layer-wise strategy balances adaptation to the target application against the data required to avoid overfitting.

  • Initialization: Random initialization with limited labeled data can lead CNN training toward an undesirable local minimum, whereas pre-trained weights provide an alternative initialization.Training from pre-trained weights is called fine-tuning.
  • Procedure: Fine-tuning transfers pre-trained weights, replaces the final fully connected layer for the target classes, and begins by updating the new layer.The final layer has two or three neurons for the paper’s binary and three-class classification tasks.
  • Layer-wise scheme: Updating only the last layer corresponds to training a linear classifier, while updating additional fully connected layers creates progressively deeper nonlinear classifiers.Including earlier convolutional layers further adapts the network but may require more labeled data to avoid overfitting.
  • Layer-wise scheme: Early CNN layers learn broadly applicable low-level features, whereas late layers learn application-specific high-level features.This distinction motivates deciding how deeply to fine-tune according to the distance between source and target applications.
  • Layer-wise scheme: The proposed strategy starts with the last layer and incrementally includes more layers until the desired performance is reached.The paper calls updating the last few convolutional layers shallow tuning and updating all convolutional layers deep tuning.

VI. APPLICATIONS AND RESULTS

The study evaluates AlexNet across four medical-imaging applications using consistent training and fine-tuning procedures, with task-specific evaluation and convergence monitoring.

  • Study design: Four medical-imaging applications were evaluated across classification, detection, and segmentation tasks.The applications used three imaging modality systems.
  • Study design: AlexNet was used consistently across all four applications for training and fine-tuning.This supported direct comparisons between the two training scenarios.
  • Training procedure: Convergence was monitored using validation-set area under the receiver operating characteristic curve after splitting each training set into 80% training and 20% validation data.The learning parameters were selected through extensive trial-and-error experiments, with a learning rate of 0.001 ensuring convergence across applications.
  • Data preparation: Training sets used stratified image patches with equal representation of positive and negative classes.The majority negative class was randomly down-sampled while the minority positive class was retained.

A. Polyp detection

Polyp detection was tested on variable colonoscopy imagery using handcrafted, scratch-trained, and incrementally fine-tuned CNN approaches. Fine-tuning improved with deeper adaptation and remained more robust when training data were reduced.

  • A. Polyp detection: Polyps vary substantially in color, shape, and size, creating a challenging detection problem in colonoscopy videos.These appearance variations can contribute to misdetection during screening.
  • A. Polyp detection: The dataset comprised 40 colonoscopy videos, with 3,800 polyp frames and 15,100 non-polyp frames for training, plus 5,700 polyp and 13,200 non-polyp frames for testing.Each frame had a binary ground-truth image, and handcrafted candidate generation was applied to training and test frames.
  • A. Polyp detection: CNN-based scenarios significantly outperformed the handcrafted approach, while updating only the final layer produced the lowest fine-tuning performance.Fine-tuning the last two layers significantly improved sensitivity over tuning only the final layer.
  • A. Polyp detection: Deeper fine-tuning progressively improved detection, with deeply tuned CNNs achieving higher sensitivity than moderate tuning particularly at low false positive rates.Fine-tuning the last few convolutional layers was sufficient to outperform training from scratch in a low-false-positive setting.
  • A. Polyp detection: With a 25% reduction in training data, the fully trained CNN degraded dramatically while the deeply fine-tuned CNN retained relatively high performance.A significant performance gap also appeared after a 50% reduction in training data.

B. Pulmonary embolism detection

Pulmonary embolism detection compares handcrafted methods, CNNs trained from scratch, and layer-wise fine-tuned CNNs using FROC analysis. Deep fine-tuning performed on par with training from scratch and retained an advantage as training data decreased.

  • Motivation: PE detection is clinically consequential because untreated pulmonary embolism mortality may approach 30%, versus as low as 2% with early diagnosis and treatment.CT pulmonary angiography is time-consuming and its accuracy depends partly on human factors.
  • Method: The PE CNN used two-channel cross-sectional and longitudinal vessel patches converted to three-channel RGB-like inputs for AlexNet.The representation was designed to reduce variability in PE appearance and support more accurate CNN training.
  • Results: Deep fine-tuning yielded significantly higher sensitivity than shallow fine-tuning at most operating points.The FT:conv1-fc8 CNN significantly exceeded FT:fc7-fc8 at the majority of operating points, including 3 false positives per volume.
  • Results: The deeply fine-tuned CNN performed on par with the CNN trained from scratch.Its improvement over the handcrafted approach was non-significant, despite requiring minimal engineering effort.
  • Data robustness: A 50% reduction in training data produced a significant performance gap favoring deeply fine-tuned CNNs.At 25% of the original training data, both CNNs declined, while a smaller yet significant gap remained.

C. Colonoscopy frame classification

Colonoscopy frame classification evaluates whether images are informative or non-informative, comparing handcrafted methods with CNNs trained from scratch and fine-tuned at different depths. Moderate fine-tuning outperformed both shallow tuning and, at some operating points, deeper tuning while remaining robust to reduced data.

  • Task: Image quality assessment labels each colonoscopy frame as informative or non-informative.Non-informative frames commonly arise during rapid scope motion or wall contact.
  • Results: Moderate fine-tuning significantly outperformed shallow tuning and deep fine-tuning at 10% and 15% false positive rates.FT:conv4-fc8 and FT:conv5-fc8 were superior to FT:conv1-fc8 at those operating points.
  • Results: All CNN-based scenarios significantly outperformed the handcrafted approach at least one operating point.Comparisons used ROC analysis at 10%, 15%, and 20% false positive rates.
  • Results: The CNN trained from scratch outperformed shallow fine-tuning but was outperformed by moderate fine-tuning.This comparison shows that the best transfer-learning depth depended on the application.
  • Data robustness: Using 10% of the original training set caused insignificant degradation for both fully trained and deeply fine-tuned CNNs.Further reductions substantially degraded fully trained CNNs and degraded deeply fine-tuned CNNs to a lesser extent.

D. Intima-media boundary segmentation

Carotid intima-media thickness measurement requires segmenting the lumen-intima and media-adventitia interfaces in ultrasound regions of interest. The study formulates interface segmentation as three-class pixel classification and traces the interfaces from CNN-generated confidence maps.

  • Task: CIMT is the distance between the lumen-intima and media-adventitia interfaces at the carotid artery’s far wall.It supports cardiovascular risk stratification but is conventionally measured by manually tracing both interfaces.
  • Method: A 3-way CNN classified each ROI pixel as lumen-intima interface, media-adventitia interface, or non-interface.Training patches came from both interfaces and random locations far from them.
  • Method: The CNN generated separate confidence maps for the two interfaces, which were thinned and smoothed into traced boundaries.Green and red encoded the likelihoods of the lumen-intima and media-adventitia interfaces, respectively.
  • Evaluation: CIMT was computed as the average vertical distance between the two traced interfaces.Test error was measured as the average distance between expert-annotated interfaces and system-generated interfaces.
  • Results: Media-adventitia segmentation error was generally greater than lumen-intima error.The authors attribute this pattern to the more challenging image characteristics of the media-adventitia interface.

VII. DISCUSSION

The discussion finds that adequate, layer-wise fine-tuning can match or exceed full training, but the optimal tuning depth depends on the application and data characteristics. Fine-tuned models also converge faster, while the study’s scope and hyperparameter choices limit generalization.

  • Performance and tuning depth: Deeper fine-tuning generally improved performance over shallow tuning, often reaching performance comparable to or better than CNNs trained from scratch.With complete datasets, shallow tuning most often underperformed, whereas deeper fine-tuning produced comparable or superior performance.
  • Convergence: Fine-tuned CNNs reached maximum validation performance faster than CNNs trained from scratch using Xavier, MSRA, or Gaussian initialization.Convergence was assessed using validation AUC for the intima-media boundary segmentation task.
  • Application dependence: The required tuning depth varied by application, with deeper tuning especially beneficial for polyp detection and intima-media boundary segmentation.The authors associate these gains with substantial differences between those medical applications and the natural-image pretraining database.
  • Application dependence: Colonoscopy frame classification showed less benefit from deep tuning because its images shared relatively similar low-level information with ImageNet images.Fine-tuning late convolutional layers was sufficient for high-level performance in this application.
  • Study design: The experiments used AlexNet to study fine-tuning depth, rather than to maximize performance across medical imaging tasks.AlexNet was selected because a pretrained model was available in Caffe and the architecture was sufficiently deep for layer-wise comparisons.
  • Limitations and scope: Reported performance curves may be suboptimal because exhaustive hyperparameter optimization was infeasible across the many time-consuming CNN trainings.The authors state this may not change their overall conclusions because most comparisons involved pretrained models.
  • Limitations and scope: The study did not cover all medical modalities, including MR and histopathology, and focused on supervised pretrained models.The authors suggest fine-tuning may succeed in other applications but explicitly frame this as a surmise.

VIII. CONCLUSION

Across four medical imaging applications and three imaging modality systems, the study concludes that deeply fine-tuned CNNs can match fully trained CNNs and outperform them when training data are limited. The required tuning depth differs by application, supporting a layer-wise approach rather than a single universal tuning strategy.

  • Conclusion: Across 4 applications and 3 imaging modality systems, deeply fine-tuned CNNs performed as well as fully trained CNNs and outperformed them with limited training data.The results indicate that knowledge transfer from natural images to medical images is possible despite substantial database differences.
  • Conclusion: The highest-performing tuning depth differed across PE detection, colonoscopy frame classification, interface segmentation, and polyp detection.PE detection saturated after tuning late fully connected layers, while other applications required tuning late and middle or deeper layers.

SUPPLEMENTARY MATERIAL

The supplementary material reports statistical comparisons for detection, classification, and boxplot results, using task-specific operating points and α = 0.05 significance thresholds.

  • Statistical comparisons: Table S1 compares polyp-detection FROC curves at 0.01 and 0.001 false positives per frame using α = 0.05.These operating points correspond to elbows of the performance curves where sensitivity is relatively higher.
  • Statistical comparisons: Table S2 compares pulmonary-embolism FROC curves at 1, 2, 3, 4, and 5 false positives per volume using α = 0.05.Cells identify operating points where paired curves differ statistically.
  • Statistical comparisons: Table S3 compares frame-classification ROC curves at false-positive rates of 10%, 15%, and 20% using α = 0.05.The table reports pairwise comparisons at the specified operating points.
  • Statistical comparisons: Table S4 provides statistical comparisons between the boxplots shown in Fig. 9, highlighting p-values larger than 0.05.The caption identifies red highlighting as the visual encoding for those p-values.
Loading 1706.00712v1…