Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation

Hoo-Chang Shin; Kirk Roberts; Le Lu; Dina Demner-Fushman; Jianhua Yao; Ronald M Summers

Source-linked AI summary

Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation

Hoo-Chang Shin, Kirk Roberts, Le Lu, Dina Demner-Fushman, Jianhua Yao, Ronald M Summers

arXiv:1603.08486v1Mar 28, 2016cs.CV

TL;DR

The paper addresses the limited use of image captioning for medical images and the challenge of biased, context-poor disease labels. It trains CNNs and RNNs on chest X-rays, reports, and MeSH annotations, then recurrently mines joint image/text contexts for image relabeling. The recurrent cascade achieves improved image annotation results when those joint contexts are used.

Problem
Medical image captioning and report-mining methods had limited ability to describe disease context, while chest-X-ray data contain substantial normal-versus-diseased imbalance.
Method
The model trains CNNs on disease labels mined from annotations, trains RNNs to describe context, and recurrently uses CNN/RNN outputs to create joint image/text labels.
Results
Improved image annotation results are achieved when joint image/text contexts are used to relabel images and subsequently train the CNN/RNN framework.
Takeaways & Limitations
The framework detects diseases and describes contextual attributes including location, severity, and affected organs from chest X-rays and reports.
Takeaways & Limitations
Aggressive anonymization achieved 100% de-identification precision but rendered a few findings uninterpretable, including a disease-related word replaced by “XXXX”.

Abstract

from arXiv · show

Despite the recent advances in automatically describing image contents, their applications have been mostly limited to image caption datasets containing natural images (e.g., Flickr 30k, MSCOCO). In this paper, we present a deep learning model to efficiently detect a disease from an image and annotate its contexts (e.g., location, severity and the affected organs). We employ a publicly available radiology dataset of chest x-rays and their reports, and use its image annotations to mine disease names to train convolutional neural networks (CNNs). In doing so, we adopt various regularization techniques to circumvent the large normal-vs-diseased cases bias. Recurrent neural networks (RNNs) are then trained to describe the contexts of a detected disease, based on the deep CNN features. Moreover, we introduce a novel approach to use the weights of the already trained pair of CNN/RNN on the domain-specific image/text dataset, to infer the joint image/text contexts for composite image labeling. Significantly improved image annotation results are demonstrated using the recurrent neural cascade model by taking the joint image/text contexts into account.

1. Introduction

The paper extends image understanding beyond disease classification by annotating chest X-rays with disease contexts such as location, severity, and affected organs. It addresses class imbalance and incomplete single-disease labels with a recurrent CNN/RNN cascade.

Motivation: Prior CNN/RNN captioning work largely targeted natural-image datasets such as Flickr8k, Flickr30k, and MSCOCO rather than medical image reports.
Motivation: Medical image descriptions can improve understanding by specifying where a disease is, how severe it is, and which organ is affected.Such contextual annotations also support retrieval queries over large collections of medical images.
Contribution: The study uses OpenI chest X-rays and reports to detect diseases and describe their location, severity, and affected organs.The dataset includes image annotations associated with the radiology reports.
Data bias: Normal cases comprise 37% of the dataset, versus 12% for opacity and 9% for cardiomegaly, motivating regularization for normal-versus-diseased bias.The dataset contains 7,284 images, including 2,696 normal cases, 840 opacity cases, and 655 cardiomegaly cases.
Cascade model: Single disease labels collapse distinct contexts, whereas the recurrent cascade re-labels images with joint image/text contexts before retraining CNNs and generating annotations.For example, granulomas in different locations or with different descriptors receive different contextual labels.

2. Related Work

Earlier work addressed image captioning, radiology label mining, and disease detection separately, but did not establish contextual description of medical images from reports. This paper positions its approach as combining disease recognition with human-like context description.

Image captioning: CNN/RNN image captioning advanced natural-image description, while related work also explored mid-level RNN representations for inferring image labels.
Radiology report mining: Radiology image-and-report methods generally mined disease names but provided limited contextual information and often used datasets dominated by diseased cases.The paper contrasts this with the prevalence of normal cases in reality.
Image/text mining: Web-based image-label mining enabled natural-language search, but generating descriptions from image/text pairs had not yet been demonstrated in that line of work.
Medical image analysis: Prior chest-X-ray studies detected diseases, classified views, or segmented anatomy, whereas learning image contexts from text and regenerating human-like descriptions remained unstudied.The authors describe their study as the first to combine disease detection with contextual description from radiology image/report data.

3. Dataset

The dataset combines anonymized OpenI radiology reports, chest X-rays, and MeSH annotations. Aggressive de-identification improves privacy but can make some findings uninterpretable and can remove disease-related words.

Dataset: The OpenI subset contains 3,955 radiology reports and 7,470 associated chest X-rays from the Indiana Network for Patient Care.The dataset was fully anonymized, with reported de-identification precision of 100%.
Dataset: Reports follow the common diagnostic format of comparison, indication, findings, and impression sections.
Limitations: Aggressive automated de-identification falsely replaced at least one word possibly indicating disease with “XXXX,” rendering some findings uninterpretable.Radiology reports may also contain patient information that cannot be inferred from image content.
Annotations: MeSH terms are manually annotated for each OpenI radiology report and used to train the model.Controlled-vocabulary annotation is associated with improved image retrieval quality in prior work.
Annotations: The twenty most frequent MeSH terms co-occur with other terms in more than 60% of cases, while normal cases have no overlap and comprise 37% of the dataset.This overlap prevents assigning a single image label directly from MeSH terms.

4. Disease Label Mining

Because chest X-rays differ substantially from ImageNet images and MeSH terms frequently overlap, the paper mines recurring annotation patterns to create image labels. These labels support CNN training and an 80%/10%/10% data split.

Label construction: Chest X-rays differ significantly from ImageNet images, so the paper trains CNNs using labels mined from radiology annotation patterns.
Label construction: Frequent annotation patterns with limited overlap are sampled to assign labels to chest X-rays and train CNNs with cross-entropy.This follows earlier approaches that mine disease labels from radiology reports.
Dataset splitting: Seventeen unique MeSH-term patterns appear in at least 30 cases, enabling training, validation, and testing splits of 80%/10%/10%.At least 10 cases are placed in each validation and testing set.
Label construction: The mined labels include terms such as scoliosis, osteophyte, spondylosis, and fractures/bone, while some frequent terms lack unique appearance patterns because they co-occur with other diseases.

5. Image Classification with CNN

The CNN classification pipeline addresses severe normal-versus-diseased imbalance using regularization and augmentation, then evaluates NIN and GoogLeNet models. Combining batch normalization with data dropout improves NIN validation accuracy by about 2%, while GoogLeNet achieves about 4% higher accuracy and images duplicated rather than cropped add about 3%.

CNN training: The 17 disease annotation patterns are used to label images and train CNNs while regularization addresses normal-versus-diseased bias.The default classifier is Network-In-Network because it is small, fast to train, and comparable to AlexNet.
CNN training: Normal cases comprise 71% of images in the selected patterns, so smaller disease classes are augmented by randomly cropping 224 × 224 regions from 256 × 256 images.
Regularization: Mini-batch normalization and data dropout are combined to regularize CNN training against the normal-versus-diseased imbalance.Data dropout randomly removes an excessive proportion of normal cases during minibatch sampling, while normalization uses balanced minibatch statistics.
Regularization: About 2% higher validation accuracy is obtained when batch normalization and data dropout are combined, whereas either technique alone does not significantly improve performance.
Model complexity: GoogLeNet achieves approximately 4% higher accuracy, with a further approximately 3% increase when images are duplicated instead of cropped for balancing.The GoogLeNet experiment applies both batch normalization and data dropout.

6. Annotation Generation with RNN

The paper trains LSTM and GRU recurrent networks to generate annotation word sequences from CNN image embeddings. BLEU evaluation compares their sequence-generation performance, while training choices and model behavior differ between the two architectures.

RNN implementations: RNNs learn annotation sequences from CNN embeddings, with LSTM and GRU tested as recurrent implementations.The CNN embedding initializes the RNN state, and the network predicts the following annotation words.
LSTM: LSTM uses input, forget, and output gates to update its memory and output vectors.The notation defines h_t as the state vector, e_h_t as new memory, and m_t as the output vector.
GRU: GRU uses update and reset gates to compute a new state and final state vector.The paper identifies z_t as the update gate and r_t as the reset gate.
Training: Annotations are constrained to five time steps, with shorter sequences zero-padded using the end-of-sentence token.Descriptions longer than six words are excluded because only nine cases exceed six words.
Evaluation: BLEU scores evaluate generated annotations against reference sequences across training, validation, and test images.BLEU-N is additionally evaluated only for annotations containing at least N words.
Evaluation: GRU is easier to compare empirically than conclusively: LSTM is easier to train, while carefully tuned GRU models tend to achieve higher average scores.The reported hyperparameter settings differ between the final LSTM and GRU models, including learning rate, decay, and dropout.

7. Recurrent Cascade Model for Image Labeling with Joint Image/Text Context

The recurrent cascade model uses joint image/text context vectors to create finer-grained disease labels, retrain CNN/RNN models, and improve annotation generation. Context-aware relabeling separates annotations that share a disease name but differ in location, size, multiplicity, or other attributes.

Motivation: Single disease labels merge annotations with distinct locations, sizes, or multiplicities, motivating context-aware image labeling.Examples include calcified granuloma in the right upper lobe, a small lesion in the left lung base, and multiple lesions.
Joint context construction: RNN state vectors are mean-pooled across annotation sequences to form joint image/text context vectors initialized from CNN image embeddings.The RNN processes each annotation word while sharing parameters across the trained RNNs.
Joint context construction: Joint image/text vectors assign different labels to annotations describing the same disease when their contexts differ.The vectors summarize both image context and word sequence, distinguishing location- and multiplicity-specific calcified granuloma descriptions.
Relabeling: Calcified granuloma cases increase from 139 to 414, while opacity cases increase from 65 to 207 after context-aware relabeling.The relabeling retains cases previously filtered because of multiple diseases or long description sequences.
Retraining: For disease labels with at least 170 cases, k-means clustering divides joint context vectors into subgroups, increasing labels from 17 to 57 before retraining.The CNN is fine-tuned with the additional labels, followed by new RNN training and annotation generation.
Evaluation: The final model achieves better overall BLEU scores than before joint image/text context was used, especially for BLEU-N with N > 1.GRU performs slightly better on average, whereas LSTM obtains higher BLEU-1 scores; the comparison is empirical.

8. Conclusion

The paper presents a framework that learns disease detection and contextual description from chest X-rays and radiology reports. It reports higher text-generation performance when joint image/text contexts are used to relabel images before subsequent CNN/RNN training.

Conclusion: The framework learns to detect diseases and describe their contexts from chest X-rays and accompanying radiology reports with MeSH annotations.The reported contexts include disease location, severity, and affected organs.
Conclusion: Higher text-generation performance is achieved when joint image/text contexts are used to relabel images and subsequently train the CNN/RNN framework.The conclusion attributes the improvement to exploiting joint contexts during image relabeling and later training.
Conclusion: The study mines a publicly available radiology image-report dataset to classify and detect disease while describing disease context in a human-observer-like manner.The authors state that this combined use of radiology images and reports had not previously been studied to their knowledge.

A. More Examples of the Chest X-ray Dataset

This appendix provides additional examples of the OpenI chest X-ray dataset and visualizes joint image/text context representations. The visualization illustrates that annotations for one disease can separate into context-dependent categories.

Dataset examples: OpenI provides paired chest X-ray images, reports, and annotations used in the dataset examples.Additional examples of these paired materials are shown in Figure 7.
Context visualization: t-SNE projections of about fifty calcified granuloma and opacity annotations visualize their joint image/text vectors in two dimensions.The projected vectors separate annotations describing the same disease into different categories based on context.

C. More Annotation Generation Examples

Additional generation examples show promising disease and context predictions, while exposing boundaries from rare diseases and unmodeled X-ray views. In lateral views, the system cannot identify whether a disease is on the left or right.

More annotation generation examples: The system produces promising predictions for disease labels and contextual attributes in additional chest X-ray annotation examples.The examples include cases shown in Figures 10 and 11.
More annotation generation examples: Rare diseases are more difficult to detect than frequent diseases in the additional examples.The cited rare cases include pulmonary atelectasis, spondylosis, density, foreign bodies, atherosclerosis, costophrenic angle, and deformity.
Limitations: Lateral X-ray views do not permit identification of whether a disease is located on the left or right.Because the dataset contains limited disease cases, the method treats each image-report pair as a sample and does not account for different views.
Future work: Future work proposes accounting for different views and collecting more data to better represent rare diseases.These are stated as two directions for improving prediction accuracy.