Source-linked AI summary
CheXtransfer: Performance and Parameter Efficiency of ImageNet Models for Chest X-Ray Interpretation
Alexander Ke, William Ellsworth, Oishi Banerjee, Andrew Y. Ng, Pranav Rajpurkar
TL;DR
ImageNet-pretrained models are commonly used for chest X-ray interpretation, but the transfer assumptions about architecture rankings and pretrained weights had not been systematically investigated. The paper compares architectures and weights on CheXpert and finds no ImageNet–CheXpert performance relationship, alongside benefits from pretraining and truncation. Truncation makes pretrained models more parameter-efficient without a statistically significant performance drop.
Problem
ImageNet transfer for chest X-ray interpretation assumes better ImageNet architectures perform better and pretrained weights help, but the relationship had not been systematically investigated.
Source: §1 · p. 2Method
The study compares 16 ImageNet architectures with and without pretraining on CheXpert and evaluates truncations of pretrained models.
Source: §1 · p. 2, §2.3 · p. 3, §3.2 · p. 3Results
ImageNet and CheXpert performance were uncorrelated, while pretraining boosted performance and truncation achieved 3.25x greater parameter efficiency on average without a statistically significant drop.
Source: §5 · pp. 6–8Takeaways & Limitations
The findings support evaluating architectures on medical datasets rather than assuming ImageNet performance predicts CheXpert performance.
Source: §5 · p. 7The study does not evaluate truncated models without pretrained weights.
Source: §4.4 · p. 6
Abstract
from arXiv · showhide
Deep learning methods for chest X-ray interpretation typically rely on pretrained models developed for ImageNet. This paradigm assumes that better ImageNet architectures perform better on chest X-ray tasks and that ImageNet-pretrained weights provide a performance boost over random initialization. In this work, we compare the transfer performance and parameter efficiency of 16 popular convolutional architectures on a large chest X-ray dataset (CheXpert) to investigate these assumptions. First, we find no relationship between ImageNet performance and CheXpert performance for both models without pretraining and models with pretraining. Second, we find that, for models without pretraining, the choice of model family influences performance more than size within a family for medical imaging tasks. Third, we observe that ImageNet pretraining yields a statistically significant boost in performance across architectures, with a higher boost for smaller architectures. Fourth, we examine whether ImageNet architectures are unnecessarily large for CheXpert by truncating final blocks from pretrained models, and find that we can make models 3.25x more parameter-efficient on average without a statistically significant drop in performance. Our work contributes new experimental evidence about the relation of ImageNet to chest x-ray interpretation performance.
1 INTRODUCTION
The paper investigates whether ImageNet architecture rankings and pretrained weights transfer to chest X-ray interpretation. It finds no relationship between ImageNet and CheXpert performance, but reports benefits from pretraining and truncating pretrained models.
1 INTRODUCTION
- ImageNet and CheXpert performance showed no relationship for models without pretraining (Spearman ρ=0.08) or with pretraining (ρ=0.06).The authors suggest ImageNet architecture improvements may not translate to medical imaging tasks.Source: §1 · p. 2
- Without pretraining, largest-versus-smallest model differences were 0.005 AUC for ResNet, 0.003 for DenseNet, and 0.004 for EfficientNet, while differences between families exceeded 0.006.The authors conclude that family choice influences performance more than within-family size.Source: §1 · p. 2
- ImageNet pretraining yielded an average boost of 0.016 AUC across architectures, with greater boosts for smaller models (Spearman ρ= −0.72 with number of parameters).The boost was statistically significant across architectures.Source: §1 · p. 2
- 3.25x greater parameter efficiency was achieved on average by truncating pretrained models, without a statistically significant performance drop.The authors suggest truncation can yield lighter models while preserving architecture design features.Source: §1 · p. 2
2 RELATED WORK
Prior work examined ImageNet transfer, medical-task architecture performance, and methods for reducing model complexity. This paper extends those discussions to a broader set of contemporary architectures and describes truncation as a form of layer-wise pruning.
2.1 ImageNet Transfer
- Kornblith et al. [15] found ImageNet accuracy highly correlated with transfer accuracy on many image-classification datasets, but the relationship was weaker without pretraining (Spearman ρ=0.59).The paper extends this transfer analysis to CheXpert and the medical setting.Source: §2.1 · p. 2
2.2 Medical Task Architectures
- Earlier CheXpert studies reported strong performance from DenseNet and ResNet models, while this paper examines 16 contemporary architectures and extends prior work to pretraining, ImageNet–CheXpert relationships, and architecture design.Raghu et al. [19] found little pretraining improvement for two models; this study reports a boost for 12 of 16 architectures. Bressem et al. [3] found smaller models could perform similarly to deeper ones on AUPRC.Source: §2.2 · p. 2, §2.1 · p. 2
2.3 Truncated Architectures
- Model truncation is presented as layer-wise pruning: removing final pretrained layers, appending a classification layer, and fine-tuning the resulting model on the target task.The related work also describes efficiency approaches including factorization, compact filters, distillation, and parameter pruning.Source: §2.3 · p. 3
3 METHODS
The methods compare pretrained and randomly initialized architectures on CheXpert, then evaluate truncated pretrained variants and visualize class activation maps. The study reports test-set results with bootstrap confidence intervals and a specified significance threshold.
3.1 Training and Evaluation Procedure
- Models were fine-tuned and evaluated on CheXpert, using average AUC across six tasks; the final ensemble used ten checkpoints selected by validation-set average AUC.CheXpert contains 224,316 chest X-rays from 65,240 patients labeled for 14 observations; results are reported on the test set.Source: §3.1 · p. 3
- The study estimated 95% confidence intervals with 1,000 bootstrap replicates and assessed significance at p=0.05.The reported interval endpoints were the 2.5th and 97.5th percentiles of the replicate distribution.Source: §3.1 · p. 3
3.2 Truncated Architectures
- Four architectures were truncated by removing final blocks, then given a global-average-pooling and fully connected classification block before fine-tuning.The truncated variants were DenseNet121, MNASNet, ResNet18, and EfficientNetB0; ImageNet weights initialized the retained model, while the appended classification block was randomly initialized.Source: §3.2 · p. 3
3.3 Class Activation Maps
- Grad-CAM visualizations combine final convolutional feature maps using weights based on positive partial derivatives with respect to class score.The resulting map is scaled by predicted probability, upsampled to the input resolution, and overlaid on the image.Source: §3.3 · p. 3
4 EXPERIMENTS
Across the experiments, ImageNet accuracy did not track CheXpert AUC, while pretraining boosted performance and final-block truncation reduced model size without a significant AUC decrease.
4.1 ImageNet Transfer Performance
- ImageNet top-1 accuracy had no monotonic relationship with CheXpert AUC, either without pretraining (ρ=0.08) or with pretraining (ρ=0.06).Models that succeed on ImageNet therefore do not necessarily succeed on CheXpert.Source: Figure 2 · p. 4, §4.1 · p. 4
4.2 CheXpert Performance and Efficiency
- ρ=0.791 without pretraining and ρ=0.565 with pretraining describe positive size–AUC relationships, yet architecture family mattered more than size within a family.DenseNet, ResNet, and Inception typically outperformed EfficientNet and MobileNet regardless of model size.Source: §4.2 · pp. 4–6
4.3 ImageNet Pretraining Boost
- ImageNet pretraining provided an average 0.015 AUC boost, with larger boosts tending to occur in smaller architectures (Spearman ρ=−0.718).Within a family, increasing model size did not meaningfully increase the pretraining boost.Source: §4.3 · p. 6
4.4 Truncated Architectures
- Final-block truncation saved 1.4x–4.2x the parameters without a significant CheXpert AUC decrease across four model families.Deeper truncations significantly reduced performance in MNASNet, DenseNet121, and ResNet18; the truncated models were not evaluated without pretrained weights.Source: Table 2 · p. 6, §4.4 · p. 6
5 DISCUSSION
The discussion finds that ImageNet performance does not predict CheXpert performance, while architecture family, pretraining, and model truncation each matter in different ways. Truncating pretrained models can improve parameter efficiency without a statistically significant performance drop.
5 DISCUSSION
- ImageNet performance showed no statistically significant relationship with CheXpert performance, unlike transfer to typical image-classification datasets [15].The discussion notes possible differences in task characteristics and data, including grayscale images and similar spatial structures across chest X-rays.Source: §5 · pp. 6–7
- Most models benefited significantly from ImageNet pretraining, with smaller models tending to benefit more (Spearman ρ=−0.72).The paper defines pretraining boost as the increase in average CheXpert AUC with pretraining versus without it.Source: Figure 4 · p. 7, §5 · p. 7
- For models without pretraining, architecture family may influence CheXpert performance more than model size; newer ImageNet-search architectures underperformed older DenseNet and ResNet models.The authors suggest CheXpert or other large medical datasets as potential architecture-search benchmarks.Source: §5 · p. 7
- Further truncation more effectively localized Atelectasis and traced the hila and vessel branching for Edema in the displayed class activation maps.Figure 5 compares DenseNet121 with DenseNet121Minus1 and DenseNet121Minus2.Source: Figure 5 · p. 8
- The authors identify the relationship between chest X-ray architectures and self-supervised pretraining as a direction for future work.They point to recent self-supervised pretraining work as a basis for investigation.Source: §5 · p. 8
- 3.25x greater parameter efficiency was achieved on average by truncating final blocks of pretrained models, without a statistically significant performance drop.The authors suggest this may yield lighter models and higher-resolution class activation maps for clinical settings.Source: §5 · p. 8