Clinically Accurate Chest X-Ray Report Generation

Guanxiong Liu; Tzu-Ming Harry Hsu; Matthew McDermott; Willie Boag; Wei-Hung Weng; Peter Szolovits; Marzyeh Ghassemi

Source-linked AI summary

Clinically Accurate Chest X-Ray Report Generation

Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, Marzyeh Ghassemi

arXiv:1904.02633v2Apr 4, 2019cs.CVcs.CL

TL;DR

Radiology report generation requires clinical accuracy beyond readable text, but prior methods often miss this domain-specific priority. The paper proposes hierarchical topic-to-sentence generation with reinforcement learning for readability and clinical coherence, and reports improvements across language and clinical-efficacy metrics on MIMIC-CXR and Open-I. The authors also identify poor recall and limitations of standard NLG metrics as important scope boundaries.

Problem
Existing radiology report-generation methods often prioritize readability while failing to adequately account for clinical accuracy and radiology-specific report structure.
Method
The system hierarchically generates report topics and topic-conditioned sentences, then uses reinforcement learning with CIDEr and the Clinically Coherent Reward to optimize readability and clinical accuracy.
Results
The system outperforms competitive baselines across readability and clinical-efficacy metrics on both MIMIC-CXR and Open-I.
Takeaways & Limitations
Domain-aware optimization can align generated reports more closely with clinical disease labels while retaining readable report text.
Takeaways & Limitations
The model has much worse CheXpert recall than precision, reflecting a CCR and reinforcement-learning setup that favors precision.

Abstract

from arXiv · show

The automatic generation of radiology reports given medical radiographs has significant potential to operationally and improve clinical patient care. A number of prior works have focused on this problem, employing advanced methods from computer vision and natural language generation to produce readable reports. However, these works often fail to account for the particular nuances of the radiology domain, and, in particular, the critical importance of clinical accuracy in the resulting generated reports. In this work, we present a domain-aware automatic chest X-ray radiology report generation system which first predicts what topics will be discussed in the report, then conditionally generates sentences corresponding to these topics. The resulting system is fine-tuned using reinforcement learning, considering both readability and clinical accuracy, as assessed by the proposed Clinically Coherent Reward. We verify this system on two datasets, Open-I and MIMIC-CXR, and demonstrate that our model offers marked improvements on both language generation metrics and CheXpert assessed accuracy over a variety of competitive baselines.

1. Introduction

The paper targets clinically accurate chest X-ray report generation, addressing gaps in systems that prioritize readability over accurate clinical descriptions. Its domain-aware system hierarchically generates topic-conditioned sentences and uses reinforcement learning to optimize readability and clinical accuracy.

Radiology report generation must prioritize accurate clinical descriptions, while prior natural-language-generation methods often treat descriptive accuracy as secondary to readability.
The proposed system hierarchically predicts unconstrained report topics and generates one sentence for each topic.This structure exploits templated radiology reports while preserving freedom for diverse free-form text.
Reinforcement learning jointly optimizes readability through CIDEr and clinical accuracy through concordance of CheXpert disease-state labels.The clinical objective aligns labels extracted from generated reports with labels from ground-truth reports.
The work evaluates the system on MIMIC-CXR and Open-I, reporting improvements in natural-language-generation metrics and CheXpert concordance over competitive baselines.

2. Background & Related Work

Radiology reports summarize image findings and impressions for clinical communication, creating a challenging image-to-text task. Related work spans multimodal learning, neural language generation, image captioning, and report-generation systems using hierarchical, retrieval, template, or reinforcement-learning approaches.

Radiology: Radiologists summarize positive, negative, or uncertain findings and their locations and severity, while impressions integrate diagnoses across report sections and prior studies.
Radiology: Identifying abnormalities is difficult because of variation, atypical cases, and imaging information overload, motivating machine-learning assistance for report drafting or direct clinical communication.
AI on Radiology Data: Image-report datasets support computational models for image annotation and automatic report generation, with Table 1 summarizing available chest X-ray datasets.
Radiology Report Generation: Prior radiology systems combine convolutional and recurrent networks, attention, hierarchical decoders, templates, retrieval, and reinforcement learning for report generation.
Language Generation: Natural-language generation seeks realistic, meaningful, linguistically correct discrete sequences, while recurrent systems can remain fluent without reaching human-level quality.
Image Captioning: Image captioning generates readable and accurate text conditioned on images, and reinforcement learning has been used to fine-tune generated text.
Proposed Reward Modeling: The proposed model encodes images, decodes sentence topics, generates words with visual attention, and can apply language, clinical, or combined rewards during policy learning.

3. Methods

The method generates chest X-ray findings hierarchically by predicting sentence topics before decoding words, then uses reinforcement learning to balance language quality with clinical coherence. Its clinical reward aligns disease-state distributions between generated and ground-truth reports using CheXpert-derived labels.

The model focuses on generating the findings section because it is the most direct annotation from radiological images.
Hierarchical generation: The CNN-RNN-RNN architecture encodes the image, generates sentence topics sequentially, and decodes each topic into words with visual attention.A stop signal ends sentence-topic generation when u > 0.5.
Hierarchical generation: The word decoder conditions vocabulary probabilities on the mean visual feature, sentence topic, and visual-sentinel-modulated feature map.The sentinel lets the model attend to image regions or effectively look away from them.
Clinically coherent reinforcement learning: CIDEr optimization alone can favor disease names without correct positive or negative disease states, motivating a clinically oriented reward.For example, it may prefer “mild pleural effusion” over the correct report “no pleural effusion.”
Clinically coherent reinforcement learning: The Clinically Coherent Reward uses CheXpert labels to align disease-state distributions between generated and ground-truth reports.CheXpert labels mentions as positive, negative, uncertain, or absent, while the reward estimates conditional disease-state probabilities from these labels.
Clinically coherent reinforcement learning: The reward assumptions treat diseases as rare, approximate negative or absent labels as negative states, assign high precision to positive labels, and set βu = 0.5 for uncertainty.These assumptions are adjustable and are reported to perform well in the experiments.
Clinically coherent reinforcement learning: Reinforcement learning combines the natural-language and clinical-coherence losses as L(θ) = LNLG(θ) + λLCCR(θ), with λ controlling their relative importance.The clinical reward baseline uses a separate exponential moving average for each reward term.

4. Experiments

Experiments evaluate the report-generation system on Open-I and MIMIC-CXR using language and clinical metrics against prior models and baselines. The study also includes dataset preprocessing, ablations, and post-hoc duplicate-sentence removal.

Datasets: The experiments use MIMIC-CXR and Open-I chest X-ray/report datasets.
Datasets: After removing duplicated radiographs, MIMIC-CXR contains 327,281 images and 141,783 reports.The original dataset contains 473,057 images and 206,563 reports from 63,478 patients.
Datasets: After excluding Open-I entries without findings sections, the dataset contains 6,471 images and 3,336 reports, with a 948-token vocabulary.Open-I initially contains 7,471 images and 3,955 reports.
Evaluation: Models are compared using metrics spanning natural-language similarity and clinical adequacy, including CIDEr-D, ROUGE-L, BLEU, and disease-category accuracy, precision, and recall.The clinical evaluation covers 14 thoracic-disease and support-device categories.
Baselines: Comparisons include 1-NN, Show and Tell, Show, Attend, and Tell, TieNet, prior state-of-the-art systems, and medical report-generation baselines.
Post-processing: Removing exact duplicate sentences improves readability but slightly degrades natural-language-generation metrics.
Ablations: Ablations separately evaluate natural-language reward, clinical-coherence reward, and the full objective.The full model combines both rewards.
Evaluation: An unsupervised Noise-RNN language model provides context for metric scores, while recurrent models use beam search with beam size 4.

5. Results & Discussion

The results show that combining language fluency with clinical coherence improves report generation beyond optimizing readability alone, while qualitative review reveals persistent errors, especially for positive findings and medical devices.

Quantitative Results: The full model achieves the highest clinical disease annotation accuracy while retaining decently high natural-language generation metrics.Optimizing CIDEr alone produces strong language metrics but clinical meaningfulness is not significantly above the major negative class.
Quantitative Results: MIMIC-CXR provides 16.7% positive disease annotations versus 5.4% for Open-I, producing a 156 times difference in diseased-patient images.The authors identify both larger data volume and a higher proportion of positive annotations as advantages for training encoder-decoder models.
Quantitative Results: Ours (CCR) achieves the highest overall precision, while Ours (full) is the runner-up and both models show reduced recall after optimizing for accuracy.The results indicate a trade-off: pursuing accuracy can improve precision but harm recall.
Qualitative Results: Qualitatively, the full model follows radiologists’ report flow and identifies major findings such as cardiomegaly, pleural effusion, and atelectasis.Both the full model and TieNet generate reasonable descriptions for negative cases, but neither identifies every finding in positive cases.
Qualitative Results: Manual review found recurring errors in supportive lines and tubes, lung lesions, chest tubes or PICC lines, and positive lung-parenchymal findings.TieNet tends to mention nasogastric tubes, whereas the full model tends to mention tracheostomy or endotracheal tubes.
Qualitative Results: The small qualitative sample cannot establish superiority over TieNet, and neither system describes most chest-radiograph findings well enough for clinical deployment.The authors conclude that significant progress remains necessary despite reasonable quantitative metrics.

6. Conclusion

The system achieves strong CheXpert precision but has substantially worse recall, and several limitations remain, including repeated-sentence post-processing and isolated-image operation. The authors emphasize that automatic language metrics may not reflect clinical efficacy and that further progress is needed before clinical deployment.

Limitations and future work: Removing repeated sentences is an ugly but necessary post-processing step that improves readability while motivating future diversity-focused work.The paper presents eliminating repetition as an unresolved limitation of the current system.
Limitations and future work: The model processes images individually rather than summarizing ordered radiograph series for a patient.The authors identify use of all available information as a direction for improving report quality.
Limitations and future work: Strong CheXpert precision is accompanied by much worse recall, reflecting the reward's emphasis on accuracy and precision.The authors suggest optimizing Recall at Fixed Precision or Fβ depending on the clinical application.
Reflections on trends in the field: CheXpert's rule-based, English-specialized labeling makes the approach harder to generalize across datasets and languages.The authors propose learning-based labeling as a more universal direction, including for Spanish corpora such as PadChest.
Reflections on trends in the field: NLG metrics are limited substitutes for readability and may not indicate real clinical efficacy, even when reinforcement learning improves automatic scores.The paper warns that metric optimization can be gamed and that clinical translation remains unclear.
Conclusion: The authors conclude that their system outperforms baselines across readability and clinical efficacy metrics on MIMIC-CXR and Open-I.This conclusion is paired with acknowledged limitations in evaluation and deployment readiness.

Appendix A. Implementation Details

The implementation uses a CNN-LSTM architecture with image and view-position inputs, followed by cross-entropy pretraining and REINFORCE optimization. The reward combines CIDEr-D-based readability with the Clinically Coherent Reward.

Encoder: The DenseNet-121 encoder maps 256 × 256 × 3 images from 8×8×1024 features into 256-dimensional embeddings with dropout p = 0.5.The encoder provides K = 64 spatial features and dφ = 1024 before projection.
Encoder: View position is represented as a one-hot embedding concatenated with the image embedding before decoding.This incorporates patient posture relative to the X-ray machine into later decoder inputs.
Decoder: The LSTM uses 256-dimensional image, word-embedding, and hidden representations, with pretrained Gensim word embeddings.The decoder receives the projected image embedding as its visual input.
Training Details: Models are first trained with Adam and cross-entropy for 64 epochs, then optimized with REINFORCE for another 64 epochs.The second stage uses a lower initial learning rate of 10^-5 and the same annealing schedule.
Training Details: CIDEr-D supplies the readability reward, while CCR uses EMA momentum γ = 0.95 and weighting factor λ = 10.These settings define the reward components and their relative scale in the full model.

Appendix B. TieNet Re-implementation

The TieNet re-implementation combines image encoding, attention-based sentence decoding, and joint report-image disease classification. The authors compare only its report-generation component because the original implementation was unavailable.

Overview: TieNet stands for Text-Image Embedding Network and contains an image encoder, attention-based sentence decoder, and Joint Learning Network.The joint representation combines information from generated text and image attention maps.
Encoder: The re-implemented encoder uses 256 × 256 × 3 images and produces 14 × 14 × 2048 features from ResNet-101.Convolutional blocks conv2 through conv4 are fine-tuned during training.
Decoder: View position is concatenated with image features, while attention over image pixels and prior words drives LSTM word generation.The decoder computes attention weights at each decoding step.
Joint Learning Network: The Joint Learning Network combines decoder hidden states and attention maps to predict thoracic disease labels.It forms global representations for both reports and images before classification.
Joint Learning Network: The implementation combines classification and report-generation losses as L_overall = αL_C + (1 − α)L_R.The model uses M = 14 CheXpert labels for joint classification.
Training: The decoder is trained with Adam and cross-entropy using batch size 32 for 64 epochs, with BLEU-based learning-rate decay.The joint learning network is trained separately according to the re-implementation details.
Training: Because TieNet's original implementation was unavailable and view information was added, re-implementation results may differ from the original paper.Only report generation is compared with the proposed model.