PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichment

Jianbo Yuan, Haofu Liao, Rui Luo, Jiebo Luo

arXiv:1907.09085v2eess.IVcs.CVcs.MM

TL;DR

Generating radiology reports requires expertise and can be error-prone. This paper combines multi-view image processing with medical-concept-enriched decoding and reports state-of-the-art performance against baseline approaches.

  • Problem

    Automatic chest X-ray report generation matters because image interpretation requires extensive expertise and is prone to human error.

    Source: §1 · p. 1

  • Method

    The model combines same-domain chest X-ray encoder pretraining, cross-view-consistent multi-view attention, and medical concepts incorporated through word-level decoder attention.

    Source: §1 · pp. 2–3

  • Results

    The integrated framework outperforms state-of-the-art baselines in experiments.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that the approach can simplify and expedite report generation and assist human experts in decision-making.

    Source: §4 · p. 8

    The authors identify the limited corpus scale of IU-RR as a limitation.

    Source: §3 · p. 8

Abstract

from arXiv · show

Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limited. To cope with these practical challenges, we propose a generative encoder-decoder model and focus on chest x-ray images and reports with the following improvements. First, we pretrain the encoder with a large number of chest x-ray images to accurately recognize 14 common radiographic observations, while taking advantage of the multi-view images by enforcing the cross-view consistency. Second, we synthesize multi-view visual features based on a sentence-level attention mechanism in a late fusion fashion. In addition, in order to enrich the decoder with descriptive semantics and enforce the correctness of the deterministic medical-related contents such as mentions of organs or diagnoses, we extract medical concepts based on the radiology reports in the training data and fine-tune the encoder to extract the most frequent medical concepts from the x-ray images. Such concepts are fused with each decoding step by a word-level attention model. The experimental results conducted on the Indiana University Chest X-Ray dataset demonstrate that the proposed model achieves the state-of-the-art performance compared with other baseline approaches.

1 Introduction

The paper proposes a chest X-ray report-generation model that combines multi-view image information with report-derived medical concepts. It targets limited clinical training data and the need for accurate paragraph-length reports, particularly for medical content.

1 Introduction

  • The model uses frontal and lateral chest X-rays to generate radiology reports, addressing challenges of limited clinical data, paragraph-level generation, and high precision for medical content.The study uses the Indiana University Chest X-ray radiology report dataset; reports include descriptive findings and concise impressions.Source: §1 · p. 2
  • The encoder is pretrained on CheXpert chest X-rays for domain-specific features, while a cross-view consistency loss encourages consistent representations across views.The multi-task scheme combines image classification and report generation to support radiology-related feature learning.Source: §1 · pp. 2–3
  • Sentence-level attention selectively synthesizes complementary frontal and lateral image features rather than treating the views independently or combining them equally.The paper argues that lateral images can provide complementary information and that separate views may otherwise yield inconsistent reports.Source: §1 · pp. 2–3
  • A hierarchical LSTM decoder uses word-level attention over report-derived medical concepts to improve the accuracy of diagnoses, locations, and observations.The encoder is fine-tuned to recognize frequent medical concepts extracted from the training reports.Source: §1 · p. 3
  • The integrated framework is reported to outperform state-of-the-art baselines, and the encoder’s uncertain radiographic observations are visualized to guide expert attention.The paper presents uncertainty visualization as an added benefit for further analysis in practice.Source: §1 · p. 3

2 Methodology

The framework combines a multi-view CNN encoder with a concept-enriched hierarchical LSTM decoder for radiology report generation. It uses large-scale chest X-rays for observation pretraining and a smaller paired image-report dataset for report generation.

2 Methodology

  • CheXpert [5] supplies large-scale images for encoder pretraining, while Indiana University Chest X-ray supports report-generation training and evaluation.The framework has a multi-view CNN encoder and a concept-enriched hierarchical LSTM decoder.Source: §2 · pp. 3–4

2.1 Image Encoder

  • The ResNet-152 encoder predicts 14 radiographic observations with binary cross-entropy and uses a mean-squared-error loss to encourage consistent predictions across frontal and lateral views.It also produces global features and local features from the last CNN block for report generation.Source: §2.1 · p. 4
  • Semrep3 extracts normalized report concepts occurring at least 80 times, yielding 69 concepts; the pretrained encoder is fixed and augmented with a concept classifier.This addresses noise and lack of normalization in MTI concepts.Source: §2.1 · p. 4

2.2 Hierarchical Decoder

  • The hierarchical decoder generates reports sentence by sentence, comparing direct view concatenation, early fusion, and late fusion of visually attended view features.A sentence LSTM produces sentence states, and a word LSTM generates language; at each word step, attention over predicted medical concepts combines an attended concept feature with the previous word embedding.Source: §2.2 · pp. 4–5

3 Experiment

Experiments on IU-RR show that the proposed report-generation models generally outperform the compared baselines, while report quality depends on accurate medical-concept predictions. CheXpert pretraining and cross-view consistency improve radiographic-observation classification.

Data Collection

  • CheXpert supplies 224,316 multi-view images from 65,240 patients, while IU-RR provides 3,074 multi-view report samples split into 615 test and 2,459 training/validation samples.IU-RR reports without multi-view images or with fewer than three sentences were filtered out; findings and impression sections were concatenated. CheXpert was chosen over ChestX-ray14 because its labeler was reported as more reliable [5].Source: §3 · p. 6

Chest Radiographic Observations

  • CheXpert pretraining performs best for observation classification, cross-view consistency improves results, and max-fused multi-view predictions achieve the best result.ImageNet pretraining performs marginally better than no pretraining; the paper attributes CheXpert’s advantage to large-scale in-domain data.Source: Table 1 · p. 6, §3 · p. 6

Radiology Report Generation

  • The proposed models generally outperform the report-generation baselines; attention-based multi-view fusion improves on MvH, and medical concepts especially boost METEOR, with errors limiting gains.MvH+AttL+MC* uses ground-truth concepts and serves as an upper bound on improvement from medical concepts.Source: Table 2 · p. 7, §3 · p. 7

Discussion

  • Without explicit abnormality information in word-level decoding, AttL struggles to generate abnormalities and locations; concept-prediction errors can also propagate into reports.Not all predicted concepts appear in generated reports, and a prediction such as “right” instead of “right lung” can affect the output.Source: §3 · p. 7, Figure 3 · p. 8
  • The limited IU-RR corpus makes unseen sentences and syntax errors difficult to handle; the authors suggest unpaired text pretraining as a possible remedy.The discussion also notes that similar decoder hidden states may arise without sentence-decoder constraints, and suggests stacked attention to focus on different image sub-regions.Source: §3 · pp. 7–8

4 Conclusions

The paper presents an encoder-decoder model that combines multi-view visual attention with medical concepts in a hierarchical LSTM decoder, achieving state-of-the-art performance. It also visualizes uncertain radiographic observations to focus expert attention for further analysis.

4 Conclusions

  • The proposed model achieves state-of-the-art performance by combining late-fusion visual attention over multi-view information with medical concepts in a hierarchical LSTM decoder.The authors say this approach better exploits both visual and textual contents.Source: §4 · p. 8
  • The automatic interpretation approach is intended to simplify and expedite radiology report generation and assist human experts in decision making.Source: §4 · p. 8
  • The model extracts and visualizes uncertain radiographic observations so experts can direct further attention to them for analysis.Source: §4 · p. 8

Related papers

Loading 1907.09085v2…