A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis

1 Introduction

• Supervised ImageNet fine-tuning is standard amid scarce medical annotations, but subtle image details make fine-grained representations relevant to classification and segmentation.The paper contrasts ImageNet's coarse-grained object classification with iNat2021's fine-grained data.Source: §1 · pp. 1–2
• The study asks whether iNat2021 pre-training, self-supervised ImageNet models, and medical-image continual pre-training improve transfer to medical tasks.The motivating questions concern fine-grained features, self-supervised transferability, and bridging the natural-medical domain gap.Source: §1 · p. 2
• Table 1 frames the benchmark across seven tasks varying in label structure, modality, organ, disease, and data size.The task suite includes classification and segmentation.Source: §1 · p. 2, Table 1 · p. 3
• Fine-grained pre-training benefits segmentation, coarse-grained features prevail in classification, self-supervised ImageNet models outperform supervised ImageNet in mean performance per task, and continual pre-training can bridge the domain gap.The benchmark covers seven medical tasks across diseases, organs, and modalities.Source: §1 · pp. 2–3

2 Transfer Learning Setup

• Seven challenging tasks span diseases, organs, and modalities, with official splits used when available and otherwise an 80%/20% train/test split.The tasks include imbalanced classes, limited data, and small scanning areas for pathology of interest.Source: §2 · p. 3
• Evaluations control preprocessing, architecture, and transfer hyperparameters, using ResNet-50 for classification and U-Net with a ResNet-50 encoder for segmentation.All target-model parameters are fine-tuned; AUC evaluates classification and Dice evaluates segmentation.Source: §2 · p. 3
• The benchmark compares supervised and self-supervised models pre-trained on ImageNet, iNat2021, CheXpert, and ChestX-Ray14.In-domain supervised models are initialized randomly or fine-tuned from ImageNet; results report mean and standard deviation over ten runs with independent two-sample t-tests.Source: §2 · p. 4

• The study evaluates transferability of 14 popular SSL methods using officially released models, all pretrained on ImageNet with ResNet-50, against a supervised ImageNet ResNet-50 baseline across seven diverse medical tasks.Source: §3 · pp. 6–7
• Across transfer settings, iNat2021 favors segmentation, at least three self-supervised models beat supervised ImageNet per task, and domain-adapted models outperform corresponding ImageNet and in-domain models.Source: Figure 1 · p. 5, Figure 2 · p. 6, Table 2 · p. 8
• A 500K-image iNat2021 mini model still outperforms ImageNet in segmentation, supporting data granularity rather than pretraining-set size as the explanation.Source: §3 · p. 5
• Self-supervised methods more often improve transfer on segmentation than classification; their broader image attention is proposed to support more precise localization under larger domain shifts.Source: §3 · p. 7
• No single self-supervised method dominates every task, although at least three outperform supervised ImageNet on each task and leading methods accelerate target-model training.Source: §3 · p. 7
• The authors attribute self-supervised transferability to holistic, task-agnostic features, while fine-grained local representations suit segmentation and coarser high-level features suit classification.Source: §3 · pp. 4, 5, 7
• Domain adaptation sequentially pre-trains ImageNet models on medical data, with ImageNet→CheXpert and ImageNet→ChestX-ray14 evaluated across five chest X-ray tasks.Source: §3 · pp. 7–8
• ImageNet→CheXpert surpasses ImageNet and CheXpert on DXC14, and adapted models also dominate at LXS, TXC, and PXS; performance falls on DXC5.Source: §3 · p. 8

• The study’s supported conclusions are that fine-grained representations suit segmentation, top self-supervised ImageNet models outperform supervised ImageNet, and continual in-domain pretraining strengthens ImageNet models.Source: §4 · p. 9
• The evaluation comprises 1,420 experiments, but self-supervised models use ImageNet pretraining and a ResNet50 backbone.Source: §4 · p. 9
• Future work could test modern self-supervised models pretrained on iNat2021 or medical images across architectures for deeper transfer-learning insights.Source: §4 · p. 9

• iNat2021 contains 2.7M training images across 10K species, while iNat2021 mini contains 500K images across the same species and has narrower visual diversity than ImageNet.Source: A Datasets · p. 13
• ChestX-ray14 has 112K frontal X-rays and reports mean AUC over 14 diseases; CheXpert has 224K images and reports mean AUC over five manually annotated test diseases.Source: A Datasets · p. 13
• The medical benchmarks include pneumothorax segmentation on 10K chest X-rays, pulmonary embolism detection from more than 7K CT scans, and tuberculosis detection using 662 chest X-rays.Source: A Datasets · p. 13
• Montgomery provides 138 chest X-rays with lung masks, while DRIVE provides 40 retinal images with vessel masks; both support segmentation evaluated by mean Dice.Source: A Datasets · p. 14

B Implementation

• All experiments use a ResNet-50 backbone, fine-tune every downstream-network layer, and run each method ten times with averages, standard deviations, and independent two-sample t-tests reported.Classification models add a task-specific head; segmentation models use a U-Net with a ResNet-50 encoder.Source: B Implementation · p. 14
• Images are resized by modality, with standard augmentation for classification and modality- and task-specific augmentation for segmentation.X-ray, fundoscopic, and CT images are resized to 224×224, 512×512, and 576×576, respectively.Source: B Implementation · p. 14
• Adam is used throughout, with ReduceLROnPlateau for classification, cosine decay for segmentation, early stopping, and task-specific learning rates.Early stopping uses 10% of training data as validation.Source: B Implementation · pp. 14–15
• Table 3 evaluates iNat2021 mini on medical segmentation tasks; its caption reports equal or better performance than ImageNet with fewer than half as many pre-training samples.Best and second-best results are bolded and underlined, respectively.Source: Table 3 · p. 15
• Table 4 benchmarks supervised iNat2021 and ImageNet pre-training across seven medical tasks, with iNat2021 favored for segmentation and ImageNet for classification.The best model in each application is bolded.Source: Table 4 · p. 15

C Ablation study on iNat2021 mini dataset

• With 500K images—less than half ImageNet's sample count—iNat2021 mini achieves equal or better segmentation performance, supporting fine-grained data rather than pre-training size as the explanation for its advantage.The passage attributes the advantage to the fine-grained nature of iNat2021 data.Source: C Ablation study on iNat2021 mini… · p. 15
• Across seven medical tasks, the table caption reports that self-supervised ImageNet models outperform the supervised ImageNet model.The best model is bolded, and models outperforming the supervised baseline are underlined.Source: Table 5 · p. 16

D Tabular results

• Tables 4 and 5 present the tabulated results for Figures 1 and 2 in the main paper, respectively.Source: D Tabular results · p. 16

E Convergence Time Analysis

• Fine-tuning from iNat2021 improves performance on all segmentation tasks and considerably accelerates training in two of three compared with ImageNet.The table reports average performance and training epochs over ten runs for each model and target task.Source: Table 6 · p. 17, E Convergence Time Analysis · p. 17
• The best self-supervised models provide better or equivalent performance and accelerate training versus supervised ImageNet; results vary by task and model.SwAV and Barlow Twins outperform with fewer epochs on DXC14; SeLa-v2, DeepCluster-v2, and SwAV do so on PXS; all evaluated self-supervised models do so on VFS.Source: Table 7 · p. 17, E Convergence Time Analysis · pp. 17–18
• Domain-adapted models converge faster than corresponding ImageNet models in most cases, and the table caption reports higher performance across tasks with faster training in most cases.The comparison includes ImageNet→CheXpert and ImageNet→ChestX-ray14 models; transfer is not applicable when pre-training and target tasks are the same.Source: Table 8 · p. 18, E Convergence Time Analysis · p. 18
• With equal training epochs, leading self-supervised models outperform supervised ImageNet on both DXC5 and ECC.The comparison uses ten epochs for DXC5 and one for ECC.Source: E Convergence Time Analysis · p. 18

Contrastive methods

• MoCo-v1 and SimCLR-v1 bring augmented views of an image closer while distinguishing them from negatives; MoCo-v1 uses a memory bank and momentum encoder, whereas SimCLR-v1 uses negatives within large batches.MoCo-v2 strengthens the MoCo-v1 baseline with a projection head, extra augmentations, cosine decay, and longer training; SimCLR-v2 adds projection-head capacity and a MoCo memory mechanism.Source: F Self-supervised Learning Methods · pp. 18–19

Non-contrastive methods

• BYOL avoids negative pairs by matching online-encoder predictions to target-encoder features, with a momentum-updated target encoder to prevent collapse.Source: F Self-supervised Learning Methods · p. 19

Proxy-task methods

• PIRL uses Jigsaw shuffling or image rotations as proxy tasks with an NCE loss and memory bank; this benchmark uses Jigsaw, which outperforms rotation.Source: F Self-supervised Learning Methods · p. 19

Clustering methods

• DeepCluster-v2 alternates clustering-based pseudo-labeling and feature learning, while SeLa-v2 frames self-labeling as optimal transport solved with Sinkhorn-Knopp [7].Both updated methods use stronger augmentation, an MLP projection head, cosine decay, and multi-cropping [7].Source: F Self-supervised Learning Methods · pp. 19–20

Hybrid methods

• PCL combines MoCo-style contrastive learning with clustering prototypes and ProtoNCE, while SwAV uses online Sinkhorn-Knopp assignments and predicts one view’s codes from another [7].SwAV’s multi-cropping strategy can also improve other methods [7].Source: F Self-supervised Learning Methods · p. 20

View design

• InfoMin proposes task-dependent views that retain downstream-task label information while discarding irrelevant factors, reducing mutual information between views.Source: F Self-supervised Learning Methods · p. 20

Redundancy reduction

• Barlow Twins trains two encoders to make their output cross-correlation matrix approach identity, maximizing view similarity while minimizing representation-component redundancy [41].Source: F Self-supervised Learning Methods · p. 20