Source-linked AI summary
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports
Baoyu Jing, Zeya Wang, Eric Xing
TL;DR
Automatic CXR report generation must handle relationships between Findings and Impression and imbalanced normality versus abnormality. The paper introduces a two-stage framework with CMAS agents specialized for planning, normality, and abnormality. Experiments on two datasets report state-of-the-art performance and meaningful, fluent reports that accurately describe detected abnormalities.
Problem
Existing CXR report-generation methods rarely modeled the Findings–Impression relationship or the differing distributions of normal and abnormal content.
Method
The paper combines a two-stage Findings-to-Impression framework with CMAS, comprising Planner, Abnormality Writer, and Normality Writer agents.
Results
The proposed CMAS generated meaningful and fluent reports and accurately described detected abnormalities in experiments on two CXR report datasets.
Takeaways & Limitations
Integrating report structure supports high-quality CXR report generation within the evaluated datasets and tasks.
Abstract
from arXiv · showhide
Chest X-Ray (CXR) images are commonly used for clinical screening and diagnosis. Automatically writing reports for these images can considerably lighten the workload of radiologists for summarizing descriptive findings and conclusive impressions. The complex structures between and within sections of the reports pose a great challenge to the automatic report generation. Specifically, the section Impression is a diagnostic summarization over the section Findings; and the appearance of normality dominates each section over that of abnormality. Existing studies rarely explore and consider this fundamental structure information. In this work, we propose a novel framework that exploits the structure information between and within report sections for generating CXR imaging reports. First, we propose a two-stage strategy that explicitly models the relationship between Findings and Impression. Second, we design a novel cooperative multi-agent system that implicitly captures the imbalanced distribution between abnormality and normality. Experiments on two CXR report datasets show that our method achieves state-of-the-art performance in terms of various evaluation metrics. Our results expose that the proposed approach is able to generate high-quality medical reports through integrating the structure information.
1 Introduction
CXR report generation is difficult because reports contain structured Findings and Impression sections, while prior methods largely ignored their relationships and differing normality-abnormality distributions. The paper addresses this with a two-stage framework and a cooperative multi-agent system.
- CXR report generation seeks to produce detailed findings and diagnoses from images while reducing radiologists’ reporting workload.
- Findings describe normal and abnormal observations in detail, whereas Impression summarizes Findings into diagnostic conclusions.
- Existing methods ignored both the Findings–Impression relationship and the different distributions of normal and abnormal findings or conclusions.
- The framework uses two stages, borrowing from image captioning and text summarization to generate Impression from Findings.
- CMAS decomposes generation into abnormality examination followed by detailed description, using Planner, Abnormality Writer, and Normality Writer agents.
- The paper evaluates report quality and abnormality-term detection quantitatively, and examines generated-report quality and properties qualitatively.
2 Related Work
Related work spans visual captioning, CXR report generation, and multi-agent reinforcement learning. Prior CXR systems include single-sentence, hierarchical, and disease-classification approaches, while multi-agent research distinguishes independent and cooperative systems.
- Visual Captioning: Visual captioning research developed CNN-RNN models, attention mechanisms, hierarchical LSTMs, and deep reinforcement-learning approaches.
- Chest X-ray Image Report Generation: CXR report-generation research includes tag prediction, joint report generation and disease classification, and hierarchical co-attention language models.
- Chest X-ray Image Report Generation: Earlier CXR methods had limitations including low BLEU scores for a single-sentence model and a tendency to produce normal findings in a hierarchical model.
- Multi-Agent Reinforcement Learning: Multi-agent reinforcement learning integrates agents focused on different subtasks and distinguishes independent from cooperative systems.
- The paper’s framework is presented through figures depicting CMAS and the Show, Describe and Conclude strategy.
3 Overall Framework
The framework separates report generation into Findings and Impression modules. Findings describe image regions, while Impression concludes from the generated Findings and the input image, explicitly modeling their summarization relationship.
- The framework contains separate Findings and Impression modules for CXR report generation.
- Findings: The Findings module examines image areas and generates descriptions for them.
- Impression: After Findings are generated, the Impression module produces a conclusion using the Findings and the input CXR image.
- CMAS models the text-generation process within each module.
4.1 Overview
Within each module, CMAS coordinates a Planner with Normality and Abnormality Writers through repeated state-sharing loops. The Planner selects continuation or a writer, and the complete system is trained using rewards from report comparisons.
- CMAS uses Planner, Normality Writer, and Abnormality Writer agents to generate findings or impressions from CXR images.
- Generation proceeds through several loops in which agents take sequential actions.
- Writers share local states containing prior actions to form a global state with the image and previously generated sentences.
- Using the global state, the Planner decides whether to stop or select the Normality Writer or Abnormality Writer for the next sentence.
- A selected writer refreshes its memory from the global state and generates words sequentially from its local state.
- After generation, a reward module compares the report with ground truth, and REINFORCE trains the whole system from that reward.
4.2 Policy Network
The policy network uses shared visual-language state encoding and a planner that selects whether to stop or invoke the normality or abnormality writer. Writer outputs feed back into the global state for sequential report generation.
- Global State Encoder: A shared single-layer LSTM Global State Encoder dynamically updates each loop from a multimodal context vector.The context jointly represents visual features and the previous sentence, while shared weights reduce computation across agents.
- Global State Encoder: The context vector combines attended visual features with the previous writer’s local state through concatenation and a nonlinear transformation.Visual features come from an intermediate CNN layer, and the writer’s final hidden state supplies the local linguistic state.
- Planner: The planner decides whether to terminate generation or select the Normality Writer or Abnormality Writer for the next sentence.The indicator uses 0 for STOP, 1 for NW, and 2 for AW.
- Writers: Separate normality and abnormality writers address the report’s imbalance, where normal sentences usually outnumber abnormal sentences by 4-12 times.Both writers use single-layer LSTMs in the reported implementation.
- Writers: The selected writer is initialized with the global state and BOS, generates words until EOS or T, and returns its final hidden state to update the next global state.The final hidden state becomes the local state vector used by the Global State Encoder in the next loop.
4.3 Reward Module
The reward module evaluates normal and abnormal report sentences separately with BLEU-4, then assigns sentence-level rewards from incremental score changes and discounts them over generation.
- Sentence-level rewards: BLEU-4 scores are computed separately for abnormal and normal sentences in each generated paragraph.The paragraph is represented as paired collections of abnormal and normal sentences with corresponding ground-truth sentences.
- Sentence-level rewards: The immediate reward for a generated sentence is the change in its BLEU-4 score relative to the preceding sentence.This applies to the first n generated abnormal and normal sentences.
- Discounted rewards: The training objective uses a discounted reward for each generated sentence.The reward module defines the discounted quantity after introducing the immediate score difference.
4.4 Learning
CMAS trains its cooperating agents with reinforcement learning using expected discounted reward, while imitation learning pre-trains the policy network to provide a useful initialization.
- Reinforcement learning: The three CMAS agents jointly generate a paragraph while maximizing the discounted reward for each sentence.The agents are Planner, Normality Writer, and Abnormality Writer.
- Reinforcement learning: The paragraph loss is defined as the negative expected reward under the complete CMAS policy network.The policy network is denoted by πθ.
- Reinforcement learning: The reinforcement-learning gradient uses REINFORCE and weights joint sentence-action log likelihood by the sentence reward.The joint negative log likelihood is decomposed into the agents’ negative log likelihoods and action probabilities.
- Imitation learning: Imitation learning with cross-entropy loss pre-trains the policy network because training the agents from scratch with reinforcement learning is difficult.Balancing coefficients weight the Planner, Normality Writer, and Abnormality Writer losses.
4.5 CMAS for Impression
The Impression module extends CMAS by conditioning on both the CXR image and the generated Findings, thereby modeling Impression as a conclusion based on Findings and the image.
- Impression module: Unlike the Findings module, the Impression module receives the image and the generated Findings sentences as inputs.The generated Findings contain Nf sentences and are incorporated into the module’s global state.
- Impression module: The remaining CMAS architecture for Impression is unchanged from the Findings module, with the generated Findings encoded through an extended multimodal context.The Findings representation uses a soft-attention vector analogous to the visual-attention representation.
5 Experiments
Experiments on IU-Xray and CX-CHR evaluate report generation, impression generation, abnormality detection, qualitative outputs, and template learning. CMAS variants generally outperform baselines, while qualitative analyses show strong normal-finding generation alongside some missed or mislocalized abnormalities.
- Datasets: IU-Xray contains 3,955 reports and 7,470 images, while CX-CHR contains reports for 35,500 unique patients.Both datasets include chest X-ray images and structured report sections such as Findings and Impression.
- Evaluation: The evaluation combines BLEU, ROUGE, CIDEr, abnormality precision, and average False Positive Rate.Abnormality detection is evaluated separately for generated textual reports on both datasets.
- Findings generation: CMAS outperforms baseline methods on almost all Findings metrics across both datasets, while hierarchical models consistently outperform single-sentence models.CMAS-IL and CMAS-RL lead the other hierarchical models, supporting the use of within-section structure.
- Ablation study: CMAS-IL exceeds the independently trained-agent combination, and CMAS-RL consistently outperforms CMAS-IL on all Findings metrics.The results are attributed to communication through the Planner and the effectiveness of reinforcement learning.
- Impression generation: CMAS-based Impression models outperform corresponding SoftAttvision and SoftAtttext baselines, with visual input generally stronger than text-only input.Comparisons among CMAS variants also indicate that text information can improve performance to some degree.
- Abnormality detection: CMAS-RL achieves the highest precision and lowest average FPR on both datasets for Findings abnormality detection, but not the lowest FPR for Impression detection.The authors note that lowering FPR can be achieved by generating normal sentences, as illustrated by CMAStext.
6 Conclusion
The conclusion presents a framework that exploits report structure through explicit between-section modeling and a cooperative multi-agent system. Its agents are trained with REINFORCE, and experiments support meaningful, fluent reports that describe detected abnormalities.
- The framework explicitly models between-section structure with a two-stage approach and within-section structure with CMAS.
- CMAS comprises Planner, Abnormality Writer, and Normality Writer agents, and the entire system is trained with REINFORCE.
- Extensive quantitative and qualitative experiments show that CMAS generates meaningful, fluent reports and accurately describes detected abnormalities.