Source-linked AI summary
Learning to diagnose from scratch by exploiting dependencies among labels
Li Yao, Eric Poblenz, Dmitry Dagunts, Ben Covington, Devon Bernard, Kevin Lyman
TL;DR
Medical diagnosis requires simultaneous prediction of many potentially subtle and inconsistently labeled abnormalities despite scarce data and concerns about bias from unrelated pre-training. The paper combines a densely connected image encoder with an LSTM-based recurrent decoder that exploits dependencies among 14 chest x-ray labels, training from scratch. Experiments on the largest publicly available chest x-ray dataset report state-of-the-art performance, while alternative metrics support clinically meaningful comparisons.
Problem
Medical diagnosis involves large-scale multi-label prediction under scarce data, subtle or inconsistently labeled abnormalities, and concerns that unrelated pre-training may introduce clinical bias.
Method
The paper combines a densely connected image encoder with a recurrent neural network decoder that exploits conditional dependencies among abnormality labels, training models from scratch.
Results
The baseline model significantly outperformed the current state-of-the-art, while considering label dependencies produced significant benefits across all four reported metrics.
Takeaways & Limitations
The proposed metric set provides meaningful quantification of chest x-ray diagnostic performance and facilitates comparison with future work.
Takeaways & Limitations
Learning label interdependencies remains limited in exploration and may learn biased dependencies when training data poorly represent realistic pathology distributions.
Abstract
from arXiv · showhide
The field of medical diagnostics contains a wealth of challenges which closely resemble classical machine learning problems; practical constraints, however, complicate the translation of these endpoints naively into classical architectures. Many tasks in radiology, for example, are largely problems of multi-label classification wherein medical images are interpreted to indicate multiple present or suspected pathologies. Clinical settings drive the necessity for high accuracy simultaneously across a multitude of pathological outcomes and greatly limit the utility of tools which consider only a subset. This issue is exacerbated by a general scarcity of training data and maximizes the need to extract clinically relevant features from available samples -- ideally without the use of pre-trained models which may carry forward undesirable biases from tangentially related tasks. We present and evaluate a partial solution to these constraints in using LSTMs to leverage interdependencies among target labels in predicting 14 pathologic patterns from chest x-rays and establish state of the art results on the largest publicly available chest x-ray dataset from the NIH without pre-training. Furthermore, we propose and discuss alternative evaluation metrics and their relevance in clinical practice.
1 INTRODUCTION
Medical diagnosis poses a large-scale multi-label prediction problem complicated by scarce data, subtle and inconsistently labeled abnormalities, and possible bias from pre-training. This work uses label dependencies, trains without external pre-training, and evaluates clinically relevant metrics on chest x-ray diagnosis.
- 1 INTRODUCTION: Medical diagnosis requires accurate prediction across many pathological outcomes, turning radiology tasks into large multi-label classification problems.Chest x-rays may contain dozens of patterns corresponding to hundreds of potential pathologies.
- 1 INTRODUCTION: Conditional dependencies among labels may improve prediction across categories while increasing data utilization and statistical efficiency.
- 1 INTRODUCTION: The work trains models from scratch without extra-domain data and directly compares them with ImageNet-pre-trained results.This tests whether pre-training can be removed when sufficient medical data are available.
- 1 INTRODUCTION: The study investigates alternative metrics intended to make automatic chest x-ray diagnosis more clinically interpretable.
- 1 INTRODUCTION: Without pre-training, a carefully designed baseline ignoring label dependencies outperformed the pre-trained state-of-the-art by a large margin.
- 1 INTRODUCTION: Explicitly modeling conditional dependencies produced superior diagnostic results over models that do not consider interdependencies across the proposed metrics.
2 RELATED WORK
Prior work commonly uses convolutional or recurrent architectures for medical image classification, but binary relevance treats diagnostic labels as independent. This paper instead uses recurrent decoding to model label dependencies, with a sigmoid sequence design suited to presence and absence predictions.
- 2 RELATED WORK: Earlier medical imaging applications primarily relied on 2D or 3D ConvNets for classification, detection, and segmentation.
- 2 RELATED WORK: Binary relevance decomposes multi-label classification into independent binary problems but ignores dependencies that can be important in medical diagnosis.
- 2 RELATED WORK: Researchers have modeled label dependencies through label-power-set prediction, dependency-aware losses, and sequences of classifiers factoring the joint distribution.
- 2 RELATED WORK: The proposed architecture uses 2D ConvNets as image encoders and recurrent neural network decoders to model dependencies among abnormality outputs.
- 2.3 KEY DIFFERENCES: Unlike a prior chest x-ray model using softmax and a fixed stopping limit, this model predicts each abnormality’s presence or absence with sigmoid across all abnormality steps.The design permits absence predictions to be passed forward and is intended to reduce per-class overcalls and false alarms.
- 2.3 KEY DIFFERENCES: Relative to Wang et al. (2017), the study models label dependencies instead of using binary relevance and trains from scratch on the ChestX-ray8 dataset.ChestX-ray8 is described as the largest public x-ray dataset at that time.
3 MODELS
The model notation represents each chest x-ray as an input image and each diagnostic target as a binary abnormality vector, with parameters and Bernoulli means defined for the model.
- 3 MODELS: The input image x has dimensions w × h × c, representing width, height, and channel.
- 3 MODELS: The target y is a binary vector of dimensionality T, the total number of abnormalities.For abnormality t, y_t = 0 denotes absence and y_t = 1 denotes presence.
- 3 MODELS: Subscripts identify examples, θ denotes the union of model parameters, and m contains Bernoulli-distribution means for each abnormality.
3.1 DENSELY CONNECTED IMAGE ENCODER
The paper adapts DenseNet design for high-resolution medical images and limited training data, modifying the architecture to preserve localized image information while remaining smaller.
- 3.1 DENSELY CONNECTED IMAGE ENCODER: DenseNet establishes shortcut connections between all pairs of layers at different depths in a deep neural network.
- 3.1 DENSELY CONNECTED IMAGE ENCODER: The proposed encoder adapts DenseNet to medical images whose small localized regions may require resolutions higher than the 256 × 256 typical for natural images.
- 3.1 DENSELY CONNECTED IMAGE ENCODER: The model is made much smaller than standard designs to address the limited training-set sizes described for the medical problem.
3.2 INDEPENDENT PREDICTION OF LABELS
This section formulates independent multi-label prediction by factorizing the label distribution and optimizing each label’s Bernoulli likelihood separately. Parameter sharing can still encourage feature reuse and reduce overfitting.
- Independent factorization: The independent model factorizes the conditional label distribution as P(y|x) = product over t of P(y_t|x).This assumes that knowing one label provides no additional information about any other label.
- Image encoder: The image encoder uses a compact DenseNet variant with four ConvBlocks per DenseBlock to keep the total parameter count small.The encoder produces a vector representation for the decoder.
- Parameter sharing: Shared parameters allow classifiers to reuse learned features and increase the example-to-parameter ratio, which alleviates overfitting.
- Training: During training, the model maximizes the sum of log probabilities for the observed labels under Bernoulli distributions.Each Bernoulli mean is parameterized as m_t = sigmoid(f(x, θ)).
- Inference: At inference, each binary label is generated independently by choosing the most probable value, equivalent to a 0.5 classification threshold.The decision is y_t* = arg max P(y_t|x, θ).
3.3 EXPLOITING HIGHER-ORDER DEPENDENCIES AMONG LABELS
This section replaces independent label prediction with an LSTM sequence model that explicitly factorizes the joint label distribution through conditional dependencies. The decoder predicts binary labels sequentially, using a fixed ordering and greedy inference.
- Higher-order dependencies: The joint label distribution is factorized into conditional terms, allowing each label to depend on the preceding labels and the input image.This formulation does not assume independence among abnormalities.
- LSTM decoder: The model treats multi-label classification as fixed-length sequence prediction with an LSTM decoder conditioned on an encoded image representation.The formulation omits attention and does not learn when to stop.
- Conditional prediction: The decoder computes each conditional factor as a Bernoulli distribution whose mean is produced sequentially from the image representation and prior label information.
- Output design: Sigmoid outputs are used instead of softmax because sparse labels could bias a softmax decoder toward an end-of-sequence class and miss rare abnormalities.The paper also contrasts this design with softmax-based sequential prediction.
- Label ordering: There are T! mathematically equivalent label orderings, and the experiments examine how two distinct orderings affect training.Some orderings may produce models that are easier to train.
- Inference: Although joint inference is intractable, the model uses greedy search, equivalent to beam search with size 1, because it performs similarly in practice.Each factor is discretized using a 0.5 threshold.
4 EXPERIMENTS
Experiments use the largest publicly available chest-x-ray dataset and evaluate models with probabilistic, similarity, and sensitivity/specificity metrics. Results show that the independent baseline outperforms prior state of the art, while modeling label dependencies improves all four proposed metrics.
- Dataset: The dataset contains 112,120 frontal-view chest x-rays annotated for the presence or absence of 14 abnormalities.Images were rescaled to 1024 × 1024, and the data were split into training, validation, and test sets following Wang et al. (2017).
- Performance metrics: The benchmark reports NLL, AUC, DICE, PESS, and PCSS to assess probabilistic performance and clinical sensitivity and specificity.PESS averages sensitivity and specificity over examples, whereas PCSS averages them over abnormalities; both use a 0.5 threshold.
- Training procedures: Data augmentation is used to combat overfitting through translations, rotations, and scaling of 512 × 512 input images.Training also uses ADAM, learning-rate reduction when validation performance stalls, and early stopping.
- Training procedures: The three models are constrained to have roughly equal parameter counts, with narrower encoders for the LSTM-decoder models.The experiments also compare frequency-based and alphabetical label orderings for dependency factorization.
- Quantitative results: The independently modeled baseline significantly outperforms the previous state of the art, while label dependencies provide significant benefits across all four proposed metrics.When sufficiently trained, the choice between frequency-based and alphabetical label ordering has only a marginal effect.
5 CONCLUSION
The paper presents a two-stage end-to-end model that combines a densely connected image encoder with a recurrent decoder and trains it from scratch for chest-x-ray diagnosis. The baseline significantly outperforms the current state of the art, while the authors caution that learning label interdependencies requires further study because limited or mixed-label data can produce biased relationships.
- Conclusion: The proposed model combines a densely connected image encoder with a recurrent neural network decoder to exploit dependencies between diagnostic labels.The model is trained from scratch to capture application-specific features.
- Conclusion: The baseline model significantly outperforms the current state of the art, and the proposed metrics quantify this performance for future comparisons.The conclusion describes the approach as feasible and effective for computer-assisted chest-x-ray diagnosis.
- Limitations and future work: Further experimentation is required because limited or mixed-label training data may produce biased interdependencies among pathologies and symptomatic patterns.The authors identify ontology-based labeling with consistent relational structure as a direction for future study.