Source-linked AI summary
Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification
Ivo M. Baltruschat, Hannes Nickisch, Michael Grass, Tobias Knopp, Axel Saalbach
TL;DR
The paper asks how deep-learning design choices affect multi-label chest X-ray classification amid limited and inconsistent comparative evidence. It systematically evaluates ResNet variants, transfer learning, X-ray-specific training, resolution, and patient metadata using repeated resampling. Overall, the X-ray-specific model incorporating non-image data achieves the best reported results, although performance varies substantially across splits and label noise complicates clinical assessment.
Problem
Limited medical-imaging datasets and differing experimental setups make comparative evidence for chest X-ray deep learning difficult to interpret.
Method
The study compares ResNet-50 initialization and transfer-learning strategies, architecture variants, and patient metadata using 5-fold re-sampling for multi-label ChestX-ray14 classification.
Results
The CXR-trained model incorporating view position, patient age, and gender achieves the best overall results; fine-tuning improves average AUC from 0.730 to 0.819, while the large variant adds 0.002.
Takeaways & Limitations
The optimized ResNet-38-large-meta reaches state-of-the-art results in five of fourteen classes, supporting systematic evaluation of architecture and metadata choices.
Takeaways & Limitations
Substantial split-dependent variability and approximately 10% label noise make true network performance and clinical impact difficult to assess.
Abstract
from arXiv · showhide
The increased availability of X-ray image archives (e.g. the ChestX-ray14 dataset from the NIH Clinical Center) has triggered a growing interest in deep learning techniques. To provide better insight into the different approaches, and their applications to chest X-ray classification, we investigate a powerful network architecture in detail: the ResNet-50. Building on prior work in this domain, we consider transfer learning with and without fine-tuning as well as the training of a dedicated X-ray network from scratch. To leverage the high spatial resolution of X-ray data, we also include an extended ResNet-50 architecture, and a network integrating non-image data (patient age, gender and acquisition type) in the classification process. In a concluding experiment, we also investigate multiple ResNet depths (i.e. ResNet-38 and ResNet-101). In a systematic evaluation, using 5-fold re-sampling and a multi-label loss function, we compare the performance of the different approaches for pathology classification by ROC statistics and analyze differences between the classifiers using rank correlation. Overall, we observe a considerable spread in the achieved performance and conclude that the X-ray-specific ResNet-38, integrating non-image data yields the best overall results. Furthermore, class activation maps are used to understand the classification process, and a detailed analysis of the impact of non-image features is provided.
1 Introduction
The paper addresses limited and inconsistent evidence for deep learning-based chest X-ray classification by systematically comparing design choices and incorporating clinically relevant patient information. It focuses on transfer learning, architecture changes, and non-image features using repeated evaluation.
- Motivation: 23,000 chest X-rays went formally unreviewed at one UK hospital over 12 months, illustrating the demand for scalable CXR assessment.Three patients with lung cancer suffered significant harm after their CXRs were not properly assessed.
- Motivation: Reliable ground-truth annotation and limited medical-imaging datasets complicate deep-learning development and motivate transfer learning.Transfer learning uses knowledge from another domain or task to support learning in the target domain.
- Research gap: Prior ChestX-ray14 studies used differing architectures, losses, augmentations, dataset splits, and evaluation setups, hindering faithful comparison.The paper notes that performance depends significantly on the selected split.
- Study scope: The study systematically evaluates weight initialization and transfer learning, large-input ResNet architectures, and non-image features.The evaluation uses a 5-fold re-sampling scheme.
- Study scope: Because prior ChestX-ray14 work primarily analyzed images, the proposed architecture adds age, gender, and view position to the learned image representation.These features are intended to leverage information available in the dataset and used by radiologists during diagnosis.
2 Methods
The methods cast pathology detection as 15-label classification and compare ResNet-50 initialization, transfer-learning, resolution, depth, and metadata configurations. The models use ChestX-ray14 images together with selected patient features in one architecture.
- Classification setup: Pathology detection is formulated as multi-label classification with binary label vectors and 15 labels, including an explicit “No Finding” label.The model learns a classification function by minimizing a loss over training sample-label pairs.
- Initialization and transfer learning: The study compares ResNet-50 training from scratch with ImageNet-pretrained off-the-shelf and fine-tuned initialization.Off-the-shelf training adapts only the classifier layer, whereas fine-tuning retrains one or more layers; here all convolutional layers were retrained.
- Architectures: Architectural variants use one input channel, 448 × 448 inputs with added pooling, and altered depths through ResNet-38 and ResNet-101.The large-input design targets small structures such as nodules and masses, while the one-channel variant is X-ray-specific.
- Non-image features: The metadata model concatenates image features with encoded view position, gender, and scaled age; metadata-only classification achieved an average AUC of 0.61.This experiment indicates that the three non-image features contain information relevant to disease classification.
3 Experiments and Results
The experiments evaluate multi-label chest X-ray classifiers across resampling splits, initialization schemes, architectures, non-image features, and model depths. Results show substantial split-dependent variability, with the from-scratch ResNet-50-large-meta performing best in resampling and the shallower ResNet-38-large-meta performing best on the official split.
- Evaluation setup: Five resampled splits used 70% training, 10% validation, and 20% testing, assigning each patient’s acquisitions to one subset.This design assessed generalization while avoiding patient-level overlap between subsets.
- Evaluation setup: Dataset splits substantially affected outcomes, with Hernia showing standard deviations of up to 0.05 because it had the fewest positive samples.The result makes comparisons based on a single random split difficult.
- Model comparisons: Fine-tuning improved average AUC from 0.730 to 0.819, while ResNet-50-large exceeded fine-tuning by 0.002 average AUC.The high-resolution variant particularly improved results for nodules by 0.018 AUC and masses by 0.006 AUC.
- Non-image features: Adding non-image features slightly increased average AUC, and the from-scratch ResNet-50-large-meta achieved the best overall resampling performance with 0.822 average AUC.The evaluated features were patient age, gender, and view position.
- Model comparisons: Prediction correlations were 0.93 among from-scratch models without non-image features, whereas models with non-image features had correlations between 0.32 and 0.47.The authors report that exclusively X-ray-trained models were both most accurate and most consistent.
- Interpretability and limitations: Grad-CAM showed correct pneumothorax localization in one example but highlighted a drain in a negative example, indicating reliance on drains as a feature.The authors therefore describe the clinical utility of ChestX-ray14 as an open issue because of label noise and medical interpretation concerns.
- Limitations: Performance spread was attributed not only to data splits but also to random model initialization and stochastic optimization.These factors further complicate direct comparisons between reported results.
- Comparison to other approaches: On the official split, reducing depth increased average AUC from 0.785 for ResNet-101-large-meta to 0.795 for ResNet-50-large-meta and 0.806 for ResNet-38-large-meta.Compared with the reported 0.807 average AUC from Guendel et al., ResNet-38-large-meta achieved 0.806 while using 185,000 fewer training images.
4 Discussion and Conclusion
The study finds that X-ray-specific models incorporating non-image data achieve the strongest overall results, while comparisons remain sensitive to dataset partitioning and label quality. Clinical deployment therefore remains constrained by noisy labels and treated findings, motivating cleaner evaluation data and further architectural work.
- X-ray-specific models incorporating view position, patient age, and gender achieve the best overall results.
- ResNet-38-large-meta achieves state-of-the-art results in five of fourteen classes against Guendel et al. on the official split.
- Performance comparisons are difficult because results vary substantially across dataset partitions, especially for the sparsely sampled Hernia class.
- ChestX-ray14’s reported 10% label noise and treated pneumothorax findings make true performance and clinical interpretation difficult to assess.
- Future work will examine other architectures, label-dependency modeling, and segmentation information.