Source-linked AI summary
Thoracic Disease Identification and Localization with Limited Supervision
Zhe Li, Chong Wang, Mei Han, Yuan Xue, Wei Wei, Li-Jia Li, Li Fei-Fei
TL;DR
Accurate disease identification and localization require annotated radiology images, but location annotations are limited and expensive. The paper presents a unified model that uses class information and scarce location annotations for both tasks, significantly outperforming the reference baseline in classification and localization.
Problem
Chest X-ray analysis needs accurate disease identification and localization, but detailed location annotations are expensive and available for only a small subset of images.
Method
A unified model jointly performs disease identification and localization using whole-image CNN features, patch-grid processing, disease labels, and limited bounding-box annotations.
Results
The model significantly outperforms the state-of-the-art or reference baseline in both disease identification and localization despite limited bounding-box annotations.
Takeaways & Limitations
The same underlying prediction model can leverage image-level labels and limited localization annotations for joint chest X-ray diagnosis and localization.
Takeaways & Limitations
Localization performance is not as good as the reference when the evaluation threshold is small.
Abstract
from arXiv · showhide
Accurate identification and localization of abnormalities from radiology images play an integral part in clinical diagnosis and treatment planning. Building a highly accurate prediction model for these tasks usually requires a large number of images manually annotated with labels and finding sites of abnormalities. In reality, however, such annotated data are expensive to acquire, especially the ones with location annotations. We need methods that can work well with only a small amount of location annotations. To address this challenge, we present a unified approach that simultaneously performs disease identification and localization through the same underlying model for all images. We demonstrate that our approach can effectively leverage both class information as well as limited location annotation, and significantly outperforms the comparative reference baseline in both classification and localization tasks.
1. Introduction
The paper targets chest X-ray analysis that identifies diseases and localizes their abnormal regions despite scarce location annotations. It proposes a unified approach using image-level disease labels and limited bounding-box information for both tasks.
- Clinical chest X-ray analysis should provide both disease classifications and spatial evidence supporting those classifications.
- Disease localization is difficult because abnormalities vary in size and location, while multiple sites may share homogeneous visual features.Atelectasis may occur locally anywhere in either lung, whereas Cardiomegaly typically covers much of the chest around the heart.
- Detailed localization annotations are expensive, motivating models that perform well with only a small amount of location annotation.The dataset combines disease labels with a small subset containing region-level bounding-box annotations.
- The proposed unified approach simultaneously improves disease identification and localization using only a small amount of X-ray images with disease location information.It produces disease-indicating regions rather than strictly predicting object-detection bounding boxes.
- The approach applies a CNN to whole-image information and slices the image into a patch grid to capture local disease information.Bounding-box images provide patch labels through overlap, while disease-label-only images are handled as a multiple-instance learning task.
- The model significantly improves accuracy over the published state of the art for both disease identification and localization despite limited bounding-box annotations.Qualitative results also show strong correspondence between radiologist annotations and detected disease regions.
2. Related Work
Related work spans object detection, medical disease diagnosis, and multiple instance learning. The paper adapts grid-based patch processing and develops an end-to-end unified model for images with mixed annotation types.
- Object detection: Object-detection methods use shared CNNs, region proposals, or direct bounding-box prediction, but generally require sufficiently annotated bounding boxes.The paper instead treats image regions as grid cells and each patch as a classification target.
- Medical disease diagnosis: Medical disease-diagnosis research has used image-text models and large chest X-ray datasets for disease classification.
- Multiple instance learning: Multiple instance learning assigns labels to image-level bags containing many instances such as patches, supporting image-level prediction from patch evidence.Prior medical-imaging methods include patch-level CNNs, decision fusion, ranking, and end-to-end multiple-instance losses.
- Multiple instance learning: This paper builds an end-to-end unified model that uses both image-level labels and bounding-box annotations.
3. Model
The model unifies disease classification and localization in one end-to-end architecture, using CNN features, patch-level predictions, and supervision from both annotated and unannotated images.
- Joint formulation: The unified model jointly performs disease identification and localization, allowing both tasks to be trained end-to-end within the same prediction framework.It is designed to use images with disease labels and limited bounding-box information.
- Image and patch representation: A CNN encodes the input image into abstract feature maps, which are resized into a P × P patch grid for local disease prediction.The patch slicing layer uses bilinear interpolation for upsampling and max-pooling for downsampling.
- Recognition network: A fully convolutional recognition network transforms resized feature maps through 3 × 3 and 1 × 1 convolutions into K class probabilities for every patch.Batch normalization and ReLU follow the initial convolution, and the intermediate patch representation uses c∗ = 512 channels.
- Supervision: For images with bounding boxes, patches overlapping the box are positive and other patches are negative, making localization supervision fully supervised at patch level.Partially overlapped patches are treated as positive for the corresponding disease class.
- Supervision: For images without bounding boxes, multi-instance learning assumes at least one positive patch for each labeled disease, providing image-level supervision.At test time, the model computes image-level class probabilities from patch predictions.
- Localization generation: At localization generation, patches whose class probability exceeds Ts = 0.5 are activated, producing non-rectangular class-specific regions rather than strict bounding boxes.The full model predicts a probability score for each patch, and activated patches collectively represent the localization.
4. Experiments
Experiments on the NIH Chest X-ray dataset test classification and localization under limited bounding-box supervision, comparing annotated and unannotated data combinations against a reference model. The unified model improves classification for most diseases and localization with increasing annotation, while unannotated data helps selectively.
- Experimental setup: The NIH Chest X-ray dataset contains 112,120 images with 14 disease labels and 984 bounding boxes covering 880 images.Labels are derived from radiology reports, and the images are resized from 1024 × 1024 to 512 × 512 pixels without augmentation.
- Experimental setup: The evaluation uses 5-fold cross-validation to compare the model with a reference baseline and to vary annotated and unannotated training data.Experiments examine bounding-box supervision, the contribution of unannotated images, and disease-identification performance under different data ratios.
- Disease identification: The model achieves better AUC scores for most diseases than the reference model, with especially significant improvements for small objects such as Mass and Nodule.Shared features and patch-level supervision help small abnormalities stand out, while Hernia has larger standard error because its 227 samples lack bounding-box annotations.
- Disease identification: Bounding-box supervision improves classification and can reduce the amount of unannotated data needed for similar AUC scores.For Edema, 80% annotated images plus 40% unannotated images (45,200 total) outperform 80% unannotated images alone (88,892 images).
- Disease identification: 0.07 versus 0.12 AUC degradation occurs for Cardiomegaly when unannotated data decreases from 80% to 20%, with versus without annotated images, respectively.With no unannotated images, performance is significantly degraded for some diseases because positive samples are absent for six disease types; Cardiomegaly is an exception with AUC = 0.8685.
- Disease localization: Localization accuracy increases as annotated images rise from 0% to 80%, whereas additional unannotated images produce disease-dependent gains, peaks, or declines.Bounding-box supervision is necessary for accurate localization; with fixed annotation, too many negative or unannotated samples can degrade performance for some diseases.
- Disease localization: At T(IoU)=0.6, Cardiomegaly reaches 73.42% accuracy versus 16.03% for the reference model, while Mass reaches 14.92% versus 0%.Across varying IoU thresholds, the model outperforms the reference baseline in most cases and retains localization capability at larger thresholds.
- Qualitative results: Qualitative results show localization that corresponds strongly with radiologist annotations and radiological signs, including enlarged hearts, fibrosis, edema, and consolidation.The model can also highlight multiple suspicious regions for one disease when the dataset provides only one ground-truth box.
5. Conclusion
The paper proposes a unified model that jointly performs disease identification and localization with limited localization annotations, outperforming the state-of-the-art algorithm quantitatively and qualitatively.
- The unified model jointly models disease identification and localization using limited localization annotation data.
- Quantitative and qualitative results show that the method significantly outperforms the state-of-the-art algorithm.
Supplementary material
The supplementary experiments examine numerical stability, annotation requirements, unannotated-data effects, localization comparisons, and performance under an updated data split.
- Numerical normalization is used to keep patch probabilities away from values that make product terms numerically unstable.The method normalizes probabilities from [0, 1] to [0.98, 1] because products over many patches can underflow in single-precision arithmetic.
- Without annotated images, image-level supervision and probabilistic approximation from patch predictions do not produce accurate localization for most disease types.
- Localization accuracy increases for each disease type as the amount of annotated training data rises from 0% to 80%.
- With 80% annotated images, adding unannotated images can produce an accuracy increase, a peak, and then an accuracy fall for most diseases.Some diseases achieve their best accuracy without unannotated images; the authors suggest lowering the weight of unannotated images as a possible remedy.
- Using 80% annotated and 50% unannotated images for training, the model achieves higher localization accuracy than the reference model across various T(IoR) values.The comparison evaluates the remaining 20% annotated images in each fold, and the table reports that the model outperforms the reference baseline in most threshold cases.
- The updated data split places all bounding-box-annotated images in the test list, preventing annotation information from improving training.The updated split also separates images at the patient level and evaluates variants of ResNet and DenseNet alongside the baseline.