Source-linked AI summary
Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichment
Jianbo Yuan, Haofu Liao, Rui Luo, Jiebo Luo
TL;DR
Reliable radiology report generation is difficult because it must link complex visual content to accurate language, while paired image-report datasets remain limited. The paper combines domain-specific multi-view encoding, selective late fusion, and medical-concept-enriched hierarchical decoding, achieving state-of-the-art performance against baseline approaches on the Indiana University Chest X-Ray dataset.
Problem
Radiology report generation requires accurate visual-language linking and extensive expertise, while open-access paired medical image-report datasets remain limited.
Method
The model pretrains a multi-view encoder on CheXpert, enforces cross-view consistency, uses sentence-level late fusion, and enriches hierarchical decoding with medically extracted concepts.
Results
The integrated framework achieves state-of-the-art performance compared with baseline approaches on the Indiana University Chest X-Ray dataset.
Takeaways & Limitations
The approach is presented as a way to simplify and expedite radiology report generation while assisting human experts in decision making.
Takeaways & Limitations
The IU-RR corpus has limited scale, motivating future exploration of unpaired textual data for decoder pretraining.
Abstract
from arXiv · showhide
Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limited. To cope with these practical challenges, we propose a generative encoder-decoder model and focus on chest x-ray images and reports with the following improvements. First, we pretrain the encoder with a large number of chest x-ray images to accurately recognize 14 common radiographic observations, while taking advantage of the multi-view images by enforcing the cross-view consistency. Second, we synthesize multi-view visual features based on a sentence-level attention mechanism in a late fusion fashion. In addition, in order to enrich the decoder with descriptive semantics and enforce the correctness of the deterministic medical-related contents such as mentions of organs or diagnoses, we extract medical concepts based on the radiology reports in the training data and fine-tune the encoder to extract the most frequent medical concepts from the x-ray images. Such concepts are fused with each decoding step by a word-level attention model. The experimental results conducted on the Indiana University Chest X-Ray dataset demonstrate that the proposed model achieves the state-of-the-art performance compared with other baseline approaches.
1 Introduction
The paper targets reliable chest X-ray report generation, using multi-view images and medical concepts to address limited data, complex visual-language links, and medical-content correctness.
- The framework generates radiology reports from frontal and lateral chest X-ray images, whose reports support diagnosis and follow-up recommendations.
- Limited IU-RR scale motivates domain-specific encoder pretraining on large-scale CheXpert chest X-ray images rather than general ImageNet data.
- Multi-view features are selectively synthesized with sentence-level attention, while cross-view consistency encourages consistent predictions across frontal and lateral views.
- A hierarchical LSTM decoder uses medical concepts and word-level attention to improve deterministic medical contents such as diagnoses, locations, and observations.
- The integrated framework outperforms state-of-the-art baselines and visualizes uncertain radiographic observations for further expert analysis.
2 Methodology
The methodology combines a multi-view CNN encoder with a concept-enriched hierarchical LSTM decoder, integrating radiographic observations, selective view fusion, and medical concepts.
- Image Encoder: The multi-view CNN encoder uses ResNet-152 to classify radiographic observations and provide visual features for report generation.
- Image Encoder: CheXpert pretraining and cross-view consistency support 14-observation classification from paired frontal and lateral images.
- Image Encoder: The encoder produces global average-pooled features and local features from the last CNN block for downstream attention.
- Medical Concepts: Semrep3 extracts 69 normalized medical concepts occurring frequently in training reports for decoder enrichment.
- Hierarchical Decoder: A hierarchical decoder generates reports sentence by sentence using sentence-level and word-level LSTM decoders.
- Hierarchical Decoder: Late fusion applies separate visual attentions to multi-view features before fusing decoder hidden states, unlike direct concatenation or early fusion.
- Hierarchical Decoder: At each word step, predicted word embeddings attend over medical concepts, and the attended concept feature is fused into the next decoding state.
3 Experiment
Experiments evaluate encoder pretraining, multi-view fusion, and radiology report generation on CheXpert and IU-RR. CheXpert pretraining with cross-view consistency improves observation classification, while attention-based fusion and medical concepts generally improve report generation over baselines; the IU-RR corpus remains limited.
- Datasets: CheXpert contains 224,316 multi-view chest x-ray images from 65,240 patients, while the processed IU-RR evaluation set contains 3,074 multi-view samples.CheXpert covers 14 common radiographic observations; IU-RR provides paired images and reports for generation evaluation.
- Chest Radiographic Observations: CheXpert pretraining outperforms ImageNet pretraining and no pretraining for radiographic observation classification, and cross-view consistency further improves results.The best classification result uses multi-view prediction fusion with a max operation: CX+CVC+F.
- Radiology Report Generation: The report-generation evaluation uses BLEU, METEOR, and ROUGE to compare captioning, radiology-generation, retrieval-based, and proposed model variants.The proposed variants include multi-view hierarchical decoding with early or late attention fusion and medical concepts; an oracle medical-concept variant provides an upper bound.
- Radiology Report Generation: The proposed models generally outperform state-of-the-art baselines, while multi-view attention fusion improves over the base MvH model and medical concepts provide additional improvement.Late fusion with medical concepts is evaluated as MvH+AttL+MC, with MvH+AttL+MC* serving as an oracle condition.
- Limitations: The study identifies the limited IU-RR corpus scale as a limitation and suggests unpaired textual data for decoder pretraining as a possible response.This limitation concerns the available report-generation training data.
4 Conclusions
The paper presents an encoder-decoder model that uses late-fusion visual attention for multi-view radiology images and medical concepts in a hierarchical LSTM decoder. It reports state-of-the-art performance and highlights uncertain-observation visualization as a practical benefit for expert analysis.
- Conclusions: The model combines multi-view visual attention in a late-fusion architecture with medical concepts enriched into a hierarchical LSTM decoder.The conclusion states that this combination better exploits visual and textual contents.
- Conclusions: The authors report state-of-the-art performance and propose that automatic interpretation can simplify and expedite radiology report generation.The stated application is assistance for human experts in decision making.
- Conclusions: Uncertain radiographic observations are extracted and visualized to direct more expert attention toward uncertainties for further practical analysis.The conclusion presents this as an added benefit of the model.