Source-linked AI summary
Transfusion: Understanding Transfer Learning for Medical Imaging
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, Samy Bengio
TL;DR
Medical imaging differs from ImageNet in image features, dataset scale, and task structure, while the effects of standard ImageNet transfer remain insufficiently understood. The paper benchmarks transferred and randomly initialized models, including lightweight architectures, across two medical tasks and analyzes their representations. Transfer offers little performance benefit overall, lightweight models can match standard architectures, and useful feature reuse is concentrated in early layers, with additional convergence benefits from weight scaling.
Problem
The effects and usefulness of ImageNet transfer for medical imaging are insufficiently understood despite major differences between natural-image classification and medical diagnosis.
Method
The paper benchmarks standard ImageNet architectures and lightweight CNNs under random initialization and ImageNet transfer across two medical imaging tasks, then analyzes representations and convergence.
Results
Across both medical datasets and all models, transfer learning has limited performance effect, while lightweight networks perform comparably to standard ImageNet architectures.
Takeaways & Limitations
Meaningful feature reuse is concentrated in the lowest layers, and transfer can also improve convergence through weight scaling independent of features.
Abstract
from arXiv · showhide
Transfer learning from natural image datasets, particularly ImageNet, using standard large models and corresponding pretrained weights has become a de-facto method for deep learning applications to medical imaging. However, there are fundamental differences in data sizes, features and task specifications between natural image classification and the target medical tasks, and there is little understanding of the effects of transfer. In this paper, we explore properties of transfer learning for medical imaging. A performance evaluation on two large scale medical imaging tasks shows that surprisingly, transfer offers little benefit to performance, and simple, lightweight models can perform comparably to ImageNet architectures. Investigating the learned representations and features, we find that some of the differences from transfer learning are due to the over-parametrization of standard models rather than sophisticated feature reuse. We isolate where useful feature reuse occurs, and outline the implications for more efficient model exploration. We also explore feature independent benefits of transfer arising from weight scalings.
1 Introduction
Medical imaging commonly transfers ImageNet architectures and weights, but the two domains differ in image structure, dataset scale, and class count. The paper therefore studies how useful ImageNet transfer actually is for medical tasks.
- Transfer learning in medical imaging: Transfer learning is widely used in medical imaging by fine-tuning ImageNet architectures with pretrained weights.This practice spans radiology, ophthalmology, and other medical applications.
- Open questions: Despite transfer learning’s popularity, its precise effects in medical imaging have received little study.Recent natural-image research had already challenged common assumptions about transfer and feature generality.
- Differences between domains: Medical diagnosis often relies on small local tissue variations, unlike ImageNet images that commonly have a clear global subject.Examples include retinal microaneurysms and localized chest-x-ray opacities.
- Differences between domains: Medical datasets typically contain larger images but far fewer examples than ImageNet’s roughly one million images.The cited medical datasets range from several thousand to a couple hundred thousand images.
- Differences between domains: Medical tasks use substantially fewer classes than ImageNet’s 1000-class setup, making standard architectures potentially suboptimal for diagnosis.The passage gives five classes for diabetic retinopathy and 5–14 chest pathologies.
- Study aim: The paper presents a fine-grained investigation of transfer learning for medical images.This statement introduces the study’s main contribution.
2 Datasets
The study evaluates transfer learning on retinal fundus photographs and chest x-rays, using clinically defined diagnostic tasks and AUC-based evaluation.
- Retina: The Retina dataset contains 587 × 587 retinal fundus photographs used to diagnose eye diseases including diabetic retinopathy.Diabetic retinopathy is graded on a five-class severity scale.
- Retina: Retina performance is evaluated by AUC-ROC for identifying referable diabetic retinopathy.Grades 3 and above are referable, while grades 1 and 2 are non-referable.
- CheXpert: CheXpert contains 224 × 224 chest x-ray images for diagnosing atelectasis, cardiomegaly, consolidation, edema, and pleural effusion.The study evaluates AUC for each of the five thoracic pathologies.
- Dataset comparison: The two medical datasets and ImageNet exhibit drastic differences in their visual features.Figure 1 provides example images from all three datasets.
3 Models and Performance Evaluation of Transfer Learning
The study compares standard ImageNet models and lightweight CNNs trained either from random initialization or ImageNet pretrained weights across two medical tasks. Transfer learning generally changes performance little, while smaller models remain competitive and small-data gains depend strongly on model size.
- Evaluation design: The evaluation compares random initialization with ImageNet transfer across standard ImageNet architectures and smaller convolutional networks.The study uses three repetitions for each model and initialization on both medical tasks.
- Main performance results: Across both datasets and all models, transfer learning does not significantly affect performance.The same conclusion holds when training with very limited data.
- Main performance results: Lightweight CBR networks perform comparably to standard ImageNet models despite substantially worse ImageNet accuracy.This pattern appears for diabetic-retinopathy diagnosis and chest-x-ray pathology classification.
- Model families: CBR models range from one third to one twentieth the size of standard ImageNet models.The family varies convolutional filter size, channel count, and depth.
- Chest x-ray results: In chest x-rays, transfer learning produces mixed gains: it is slightly worse for three pathologies but helps for edema and pleural effusion.The affected pathologies are atelectasis, cardiomegaly, and consolidation.
- Very small data regime: With 5000 Retina datapoints, transfer helps large models more than smaller CBR models.The authors identify model size as a confounding effect in the very small data regime.
- Representational analysis: For larger models, pretrained and randomly initialized training produces more different hidden representations than for smaller models.CCA similarity is lower relative to the random-initialization baseline for larger models.
4 Representational Analysis of the Effects of Transfer
CCA analyses show that transfer and random initialization produce different representations in larger models, but meaningful feature reuse is concentrated in the lowest layers. Larger models change less during training, consistent with overparametrization, while smaller models move more and do not retain the same initialization patterns.
- Representational differences: CCA comparisons show that larger models learn representations that differ between pretrained and randomly initialized training, whereas smaller CBR models learn more similar functions.Representations from random initialization and pretrained weights are less similar than representations from two random initializations for ResNet and Inception.
- Training dynamics: For ResNet, initialization similarity remains strongly correlated with converged similarity, with R^2 = 0.75, 0.84, unlike CBR-Small.The correlation appears for both pretrained-versus-random and two-random-initialization comparisons.
- Training dynamics: Larger models change less during training, especially in lower layers, even from random initialization, indicating that feature reuse alone does not explain their stability.This pattern is interpreted as evidence of overparametrization for the medical task.
- Layerwise reuse: Feature reuse is mostly restricted to the lowest two layers or stages, where pretrained representations remain more similar to their initialization than randomly initialized representations.Per-layer similarity identifies the bottom two layers as the only location with significantly greater initialization similarity for pretrained weights.
- Filter visualizations: Conv1 visualizations show limited ResNet filter movement, while CBR-Small changes more and does not learn Gabor filters from scratch or retain all pretrained Gabors.These visual results provide a concrete counterpart to the layerwise CCA findings.
5 Convergence: Feature Independent Benefits and Weight Transfusion
The paper separates transfer’s convergence benefits from feature reuse and tests how much pretrained information is needed. Better weight scaling speeds convergence, while the lowest layers provide the largest reuse-related gains and support slimmer hybrid designs.
- Feature-independent benefits: Pretrained weights speed convergence through feature-independent benefits attributed to better weight scaling, separate from sophisticated feature reuse.The study explicitly investigates faster convergence as a benefit that can persist even when feature reuse is limited.
- Weight transfusion: Using pretrained weights from the lowest two layers or stages has the biggest effect on convergence, supporting their disproportionate role in feature reuse.Weight transfusion evaluates contiguous pretrained subsets while randomly initializing the remaining layers.
- Hybrid approaches: Synthetic Gabor filters provide an alternative to pretrained conv1 initialization in the explored hybrid transfer approaches.The paper presents synthetic features as a way to retain flexibility while using fewer pretrained weights.
- Hybrid approaches: A hybrid ResNet that reuses pretrained weights through Block2 and slims the remaining network matches full transfer learning in performance and convergence.The remainder is randomly initialized and trained end to end with half the channels.
- Weight transfusion: The largest convergence gain occurs with the earliest transfused layers, despite a ResNet block containing multiple reused layers.This result further supports concentrating pretrained reuse in the lowest part of the network.
6 Conclusion
Across two large medical tasks, transfer learning provides limited performance gains, while lightweight models can match standard ImageNet architectures. Representation analyses attribute some transfer differences to overparametrization and localize meaningful reuse to the lowest layers, while weight scaling improves convergence.
- Conclusion: Across two large medical tasks, transfer learning offers limited performance gains and much smaller architectures perform comparably to standard ImageNet models.The conclusion combines benchmark results across standard and lightweight architectures.
- Conclusion: Representational similarity analyses provide evidence that standard ImageNet models are overparametrized for the medical tasks.The paper also finds meaningful feature reuse concentrated at the lowest layers.
- Conclusion: Hybrid transfer approaches retain transfer benefits while enabling more flexible and efficient model designs.The paper additionally identifies feature-independent convergence speedups from better weight scaling.
A Details on Datasets, Models and Hyperparameters
The experiments use two medical imaging tasks, standard ImageNet architectures, and small CBR convolutional networks under fixed preprocessing and training settings. Models are evaluated with task-specific input sizes, optimizers, and output formulations.
- Datasets: The Retina dataset contains around 250k training images and 70k test images, split by patient ID to prevent accidental train-test similarity.The patient-level split follows standard medical-dataset practice.
- Datasets: CheXpert contains 223k training chest X-rays with binary disease indicators extracted automatically from metadata.The dataset supports diagnosis of multiple thoracic pathologies.
- Preprocessing: Preprocessing uses mild random cropping, ImageNet mean-variance normalization, hue and contrast augmentation, and dataset-specific flip policies.Random flips are used for Retina but not for chest X-rays.
- Model outputs: Retina experiments replace the 1000-class ImageNet head with a five-class diabetic-retinopathy head, using sigmoid outputs in a multi-label binary-classification framework.The same five-class head formulation is described for the chest X-ray experiments.
- Models: CBR models use repeated convolution, batch normalization, and ReLU layers followed by max pooling, with fixed convolutional filter sizes within each architecture.The family includes CBR-LargeT, CBR-LargeW, CBR-Small, and CBR-Tiny variants.
- Training settings: Retina models use 587 × 587 inputs, Adam, learning rate 0.001, and batch size 8, while chest X-ray models use 224 × 224 inputs, SGD with momentum, and batch size 32.Learning-rate scheduling for chest X-rays is inherited from the ImageNet training pipeline.
B Additional Dataset Size Results
Additional experiments varying training-set size find that transfer learning provides only a fractional improvement around 50k datapoints.
- Around 50k datapoints, transfer learning returns to only a fractional improvement.
C CCA Details
The paper adapts CCA to compare large convolutional activations by controlling sampled patches and neuron activation vectors. This avoids computationally expensive flattening while addressing CCA's sensitivity to activation dimensions.
- CCA is implemented for convolutional activations whose dimensions vary substantially across network layers.Activation shapes range from (n, 294, 294, 64) at conv1 to (n, 19, 19, 1024) at the end of block 3.
- The procedure samples p image patches and d neuron activation vectors before applying CCA to d x n h_L w_L activation matrices.
- The sampling is repeated across samples of d and n.
- The adapted method works better than averaging spatial dimensions or flattening all neurons, which is too computationally expensive.
D Additional Results from Representation Analysis
Additional representation analyses support the view that model size and network depth shape how transfer learning changes learned representations. Larger models change less during training, while pretraining effects are concentrated in lower layers.
- Table 5 compares representations across random seeds for Resnet-50 and CBR-Small using pretrained weights and random initializations.
- Larger models show more similar representations after pretraining than smaller models when compared against random initialization.
- For higher layers, pretrained and randomly initialized representations become more similar, with fine-tuning changing representations primarily at the top.
- Figures 9 and 10 compare randomly initialized, ImageNet-initialized, and converged first-layer filters for CBR-Small and Resnet-50 on CheXpert.
E The Fixed Feature Extraction Setting
Fixed-feature experiments compare pretrained and random features across retinal and chest-x-ray tasks. Pretrained features help substantially on the Retina task, but the advantage is smaller on chest x-rays and higher-layer initialization adds little.
- The fixed-feature setting freezes pretrained layers through L and trains only layers L + 1 onward.
- Figures 10 and 11 illustrate first-layer filter comparisons and the intuition that larger models move less through training.
- Pretrained ImageNet features perform significantly better than random features on Retina, while the gap is significantly closer on chest x-rays.
- There is little performance difference between initializing all layers with pretrained weights and initializing only through layer L.
F Additional Results on Feature Independent Benefits and Weight Transfusions
Additional experiments show that transfer’s feature-independent benefits arise mainly from parameter scaling, while useful reuse is concentrated in lower layers and diminishes at higher layers.
- Feature-independent benefits: Mean Var initialization can converge much faster than random initialization through better parameter scaling transferred from pretrained weights.On Retina, its effectiveness is less pronounced in smaller, shallower models; on CheXpert, it benefits convergence for all diseases except Cardiomegaly.
- Feature reuse: ImageNet features outperform random fixed features on Retina, while the advantage is much closer on CheXpert.The experiments freeze layers up to L and train only layers L+1 onwards.
- Batch normalization: BN Identity Init performs slightly better than alternative batch-normalization schemes, while directly transferring ImageNet batch-normalization parameters performs worst.The non-batchnorm layers use Mean Var initialization in these comparisons.
- Weight distributions: Mean Var initialization converges at a similar speed to sampling from or shuffling the full empirical ImageNet weight distribution.This suggests further convergence improvements may require modeling correlations between weights.
- Weight transfusion: Reusing pretrained weights in the lowest layers produces the largest speedups, with diminishing returns when transferring higher layers.The same pattern appears for ResNet-50 and CBR-LargeW, although higher-layer reuse changes slightly more for the smaller model.
- Model architecture: A slim ResNet-50 without pretrained weights converges significantly more slowly than the hybrid approach.The slim model halves the channels in Block3 and Block4.