Source-linked AI summary
Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
Christy Y. Li, Xiaodan Liang, Zhiting Hu, Eric P. Xing
TL;DR
Medical image report generation must produce long, coherent reports that cover heterogeneous findings and use medical terminology accurately. HRGR-Agent combines template retrieval with neural generation through hierarchical decision-making and reinforcement learning, achieving state-of-the-art results on two datasets, high terminology-detection precision, and improved human preferences.
Problem
Medical image reports require long, coherent narratives with broad content coverage, specific terminology, and appropriate ordering, while retrieval and generation each have complementary weaknesses.
Method
HRGR-Agent uses a high-level retrieval policy to select template sentences or invoke low-level generation, jointly trained with sentence-level and word-level reinforcement-learning rewards.
Results
HRGR-Agent achieves state-of-the-art performance on two medical report datasets across automatic metrics, human evaluation, and medical terminology detection.
Takeaways & Limitations
The approach generates robust reports that balance concise template sentences with complicated, diverse sentences and accurately detects medical abnormalities.
Abstract
from arXiv · showhide
Generating long and coherent reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse report generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an off-the-shelf template database, or invoke a low-level generation module to generate a new sentence. HRGR-Agent is updated via reinforcement learning, guided by sentence-level and word-level rewards. Experiments show that our approach achieves the state-of-the-art results on two medical report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical report contents. In addition, our model achieves the highest detection accuracy of medical terminologies, and improved human evaluation performance.
1 Introduction
Medical image reports require coherent multi-sentence narratives with broad content coverage, precise terminology, and appropriate ordering. HRGR-Agent combines retrieval and generation through hierarchical reinforcement learning to address these requirements.
- Medical reports must maintain plausible logic and consistent topics across multiple sentences or paragraphs.
- Medical report generation requires coverage of normal and abnormal findings, specific medical terminology, and prescribed content ordering.Findings may include heart size, lung opacity, bone structure, and abnormalities such as effusion, pneumothorax, and consolidation.
- Retrieval performs well for common template-like normal findings, whereas generation can miss rare abnormalities such as effusion.
- HRGR-Agent lets a retrieval policy choose between template retrieval and sentence generation, then jointly trains both modules with sentence-level and word-level rewards.Its template database incorporates human prior knowledge from available medical reports.
- HRGR-Agent bridges rule-based retrieval and learning-based generation so each module learns complementary sentence types through policy learning.The generation module learns diverse, complicated sentences while retrieval learns template-like sentences.
- On two medical report datasets, HRGR-Agent achieves state-of-the-art performance across automatic metrics, human evaluation, and medical terminology detection.Generated reports balance concise template sentences with complicated and diverse sentences.
2 Related Work
Prior work includes visual captioning and hybrid template-generation methods, while reinforcement learning has been applied to optimize sequence-generation objectives.
- Visual Captioning and Report Generation: Visual captioning models usually generate short sentences, whereas report generation requires informative multi-sentence outputs with content selection, relation generation, and ordering.
- Template Based Sequence Generation: Existing template-generation approaches either use retrieval as limited latent guidance or encourage generators to imitate template-like sequences.
- Reinforcement Learning for Sequence Generation: Reinforcement learning has been used for sequence generation because cross-entropy training can suffer from exposure bias and may not directly optimize metrics such as CIDEr, ROUGE, or BLEU.
3 Approach
HRGR-Agent generates medical reports hierarchically by producing sentence topics, then selecting retrieval or generation for each topic. Its modules are trained with sentence- and word-level CIDEr-based reinforcement rewards.
- Hierarchical report generation: The model encodes medical images into a context vector, then hierarchically generates topic states before predicting each sentence's words.A sentence decoder uses attention and recurrent layers to produce topic states and a stop-control probability.
- Retrieval-generation policy: The template database contains frequent training-corpus sentences that typically describe general observations and can be reused in reports.This design reflects observed reporting practice in which doctors follow patterns while modifying statements for individual cases.
- Retrieval-generation policy: A retrieval policy chooses between generating a new sentence and retrieving one of |T| candidate templates for each topic state.The action distribution includes automatic generation and template retrieval; the highest-probability action determines which module is activated.
- Sentence generation: The generation module produces words conditioned on the current topic state, image context, and previous word through recurrent attentional decoding.Its word probabilities are computed over the vocabulary after combining topic, visual-context, and word-embedding information.
- Hierarchical reinforcement learning: Sentence-level delta CIDEr rewards assess each sentence's contribution to the report, while word-level delta CIDEr rewards assess generated words.Discounted sentence-level rewards update the retrieval policy, and discounted word-level rewards update the generation module.
- Hierarchical reinforcement learning: The overall objective maximizes the generated report's reward against the ground-truth report, with separate losses for retrieval-policy and generation modules.The retrieval action may be automatic generation or any template in the database, and both modules receive policy updates.
4 Experiments and Analysis
Experiments evaluate HRGR-Agent on two medical report datasets using automatic, terminology-detection, and human metrics. Results show strong performance, with retrieval supporting structure and generation supporting rare abnormalities.
- Datasets: IU X-Ray contains 7,470 paired chest X-ray images and reports, while CX-CHR provides 33,236 selected patient samples with Chinese reports.Both datasets are split by patient into training, validation, and testing sets at a 7:1:2 ratio.
- Experimental setup: Template databases contain 28 frequent templates for IU X-Ray and 97 for CX-CHR, grouping sentences with equivalent meanings and minor linguistic variation.Candidates are selected by document frequency thresholds in the training sets.
- Automatic evaluation: HRGR-Agent outperforms non-retrieval or non-hierarchical baselines on both datasets, while reinforcement fine-tuning raises CX-CHR CIDEr by 0.73 over HRG.Retrieval also surpasses Generation on automatic scores, supporting structured report generation when combined with neural generation.
- Medical terminology evaluation: HRGR-Agent achieves the highest medical abnormality terminology precision and is only slightly below CoAtt in average false positive.The authors associate this result with robustness in detecting rare abnormal findings.
- Human evaluation: HRGR-Agent receives much higher human preference than Generation and CoAtt on CX-CHR and IU X-Ray, respectively.Human evaluation covers content coverage, specific terminology accuracy, and language fluency.
- Qualitative analysis: Qualitative examples show HRGR-Agent reports are generally longer, balance template and generated sentences, and detect abnormal findings at a higher rate.Figure 3 marks medical abnormality terms and italicizes text retrieved from the template database.
5 Conclusion
HRGR-Agent bridges human prior knowledge and generative neural networks via reinforcement learning for robust medical image report generation. On two datasets, it achieves state-of-the-art performance, high precision for abnormal-finding detection, and strongest human preference.
- HRGR-Agent bridges human prior knowledge and generative neural networks via reinforcement learning for medical image report generation.
- The model achieves state-of-the-art performance on two medical image report datasets.
- HRGR-Agent generates robust reports with high precision in medical abnormal-findings detection and best human preference.
A Policy Update Algorithm
The policy update algorithm alternates sentence-level decisions between retrieval and generation while applying the corresponding reinforcement-learning rewards. Retrieved templates update only the retrieval policy, whereas generated sentences update both modules through separate rewards.
- When a template is retrieved, only the retrieval policy is updated using sentence-level reward.
- When automatic generation is selected, the generation module receives word-level reward while the retrieval policy receives sentence-level reward.
- CNN and an image encoder extract visual features and a context vector before sentence-level policy decisions begin.
- At each sentence time step, a sentence decoder produces a topic state and the retrieval policy selects a template index.
- The selected route either retrieves a template sentence or invokes the generation module to produce the sentence.
B DenseNet Pretraining
The experiments pretrain DenseNet on ChestX-ray8 for thorax-disease classification and adapt visual features for medical report generation. The implementation adds feature-processing layers and uses template groups that consolidate sentences with equivalent meanings.
- DenseNet is pretrained on ChestX-ray8 using multi-label classification and fine-tuned on CX-CHR for 20 common thorax disease labels.ChestX-ray8 contains 108,948 frontal-view X-ray images from 32,717 patients, labeled for 14 common thorax diseases.
- The visual backbone adds lateral and convolutional layers, producing 16 × 16 × 256 feature maps from the second dense block.These features provide higher-resolution details and more location information without expanding total feature size.
- Each template groups sentences with the same meaning but slightly different expressions, and the retrieval step returns only the first sentence.
C Template Database
The template database is built from frequent training-corpus sentences, grouping variants with the same meaning to support retrieval. Dataset-specific document-frequency thresholds determine which sentences enter the database.
- Templates are selected from the most frequent training-corpus sentences above a document-frequency threshold.The thresholds are 100 for IU X-Ray and 500 for CX-CHR.
- Sentences with the same meaning but slightly different language variation are grouped into one template.