PaperlayerView exact revision on arXiv

Source-linked AI summary

MAC-RRG: Iterative Multi-Agent Collaboration for X-ray Radiology Report Generation

Futian Wang, Yuhan Qiao, Xiao Wang, Dan Xu, Yuehang Li, Zhixiang Guo, Yaowei Wang, Jin Tang

arXiv:2609.26124v1cs.AI

TL;DR

X-ray report generation requires clinically consistent descriptions, while existing approaches are described as limited by static knowledge fusion, single-round generation, and insufficient multi-source knowledge. MAC-RRG uses specialized agents to retrieve complementary evidence from an initial report and iteratively refine it; the authors report improved interpretability and disease diagnosis and analysis across multiple mainstream datasets.

  • Problem

    Existing RRG approaches are described as limited by static knowledge fusion, single-round generation, and insufficient multi-source knowledge despite the need for clinically consistent diagnostic reports.

    Source: §I · pp. 1–2

  • Method

    MAC-RRG generates an initial report, uses specialized agents to retrieve structured graph relations and complementary textual evidence, and fuses these sources to guide iterative refinement.

    Source: §I · pp. 2–3, A. Overview · p. 4

  • Results

    Experiments on multiple mainstream X-ray report datasets demonstrate efficacy, with improved interpretability and disease diagnosis and analysis reported versus conventional large-model and knowledge-graph-enhanced methods.

    Source: §I · p. 3

  • Takeaways & Limitations

    Automated X-ray reporting may ease radiologists' workload and serve as an auxiliary diagnostic tool, especially where radiology practitioners are scarce.

    Source: §I · p. 1

    Refinement depends on the initial draft: missed or incorrectly described abnormalities may lead agents to retrieve incomplete or less relevant evidence.

    Source: F. Limitation Analysis · p. 12

Abstract

from arXiv · show

Despite the remarkable progress of LLM-based and knowledge graph-augmented Radiology Report Generation (RRG) methods, existing techniques still suffer from inherent defects. Conventional LLM-only models lack structured medical prior knowledge, resulting in frequent medical hallucinations and low diagnostic interpretability. Current knowledge graph-enhanced schemes adopt static one-round knowledge fusion with single-source knowledge, incapable of dynamic knowledge updating according to generation feedback. This paper proposes a novel Multi-Agent Collaborative iterative framework for X-ray Radiology Report Generation, termed MAC-RRG. Inspired by multi-agent technology, our framework constructs a closed-loop optimization paradigm based on task decoupling and collaborative reasoning. Specifically, the framework first generates a preliminary radiology report from input X-ray images via a vision encoder and a basic LLM. Subsequently, a multimodal knowledge graph (MM-KG) agent mines structured disease correlation and anatomical knowledge from medical knowledge graphs, while an auxiliary knowledge agent extracts unstructured domain knowledge from public medical databases. The multi-source knowledge acquired by dual agents is fused and embedded to guide the LLM in iteratively refining the initial report. Extensive quantitative and qualitative experiments on mainstream X-ray RRG datasets, including IU X-ray, MIMIC, and CheXpert Plus, fully verify the superiority of our proposed method. The source code and pre-trained models have been released on https://github.com/Event-AHU/Medical_Image_Analysis

I. INTRODUCTION

MAC-RRG addresses limits in X-ray report generation by iteratively refining an initial report with complementary medical knowledge mined by specialized agents. The authors report improved interpretability and disease diagnosis and analysis compared with conventional large-model and knowledge-graph-enhanced methods.

I. INTRODUCTION

  • Existing LLM-based methods lack structured medical priors, while knowledge-graph methods fuse static, single-source knowledge without feedback-driven refinement.The introduction also identifies entity omissions and biased descriptions in initial reports as errors that one-shot pipelines cannot continuously correct.Source: §I · p. 2
  • The introduction presents task-decoupled agents and iterative interaction as a response to static knowledge fusion, single-round generation, and limited use of multiple knowledge sources.The proposed workflow uses initial-report feedback for dynamic retrieval and correction.Source: §I · p. 2
  • The framework generates an initial report from image tokens, extracts its entities, retrieves graph relations and textual medical evidence, then fuses both knowledge streams to guide LLM refinement.The MM-KG agent mines structured knowledge, while a second agent retrieves domain knowledge from public medical sources.Source: §I · pp. 2–3
  • MAC-RRG uses collaborative agents to mine and fuse supplementary knowledge over initial X-ray reports for iterative report enhancement.Its components include an MM-KG agent, a knowledge agent, and a knowledge fusion module.Source: §I · p. 3
  • Experiments on multiple mainstream X-ray report datasets are reported to show improved interpretability and disease diagnosis and analysis versus conventional large-model and knowledge-graph-enhanced methods.The introduction states that the experiments demonstrate the method's efficacy.Source: §I · p. 3

II. RELATED WORKS

Prior X-ray report generation progressed from visual encoder–decoder models to LLMs and medical-knowledge methods. Related work also emphasizes structured clinical entities and relations as important to report consistency.

A. Radiology Report Generation

  • Earlier RRG models used CNN encoders with recurrent or hierarchical decoders, followed by Transformer and memory mechanisms; LLM-based models improve generation but visual alignment alone may not ground each entity in external evidence.The section notes continuing risks of abnormality omission, vision–language mismatch, and template-like reports.Source: A. Radiology Report Generation · p. 3

B. Medical Knowledge Graphs

  • Medical-knowledge methods incorporate anatomy, disease concepts, and clinical relations through graphs, retrieval, and knowledge distillation.RadGraph and Chest ImaGenome represent report entities and relations or chest X-rays as anatomy-centered scene graphs.Source: B. Medical Knowledge Graphs · p. 3
  • The reviewed work frames report quality as depending on clinically consistent links among diseases, anatomical locations, attributes, temporal changes, and adjacent findings.Source: B. Medical Knowledge Graphs · p. 3

C. Retrieval-Augmented Generation and Agentic Workflows

The section contrasts similarity-based retrieval with entity-targeted agent workflows for X-ray reporting. MAC-RRG uses entities from an initial draft to retrieve graph and textual evidence, fuse it with image tokens, and refine the report.

C. Retrieval-Augmented Generation and Agentic Workflows

  • Existing chest X-ray retrieval methods use image- or report-level similarity, which may return evidence that does not match the current sample's specific entities.The section describes retrieval as a source of external evidence that can compensate for missing or unstable medical knowledge in model parameters.Source: C. Retrieval-Augmented Generation and… · pp. 3–4

A. Overview

  • MAC-RRG assigns graph-relation retrieval and textual-evidence retrieval to separate agents, then fuses both knowledge streams to refine the initial report.The overview encodes graph neighborhoods and retrieved text, integrates their tokens with image tokens, and projects them into the LLM hidden space.Source: A. Overview · p. 4, C. Retrieval-Augmented Generation and… · p. 4

B. Initial Report Generation and Entity Extraction

  • The initial draft supplies a shared set of anatomy and disorder entities for both retrieval branches, restricting queries to image-conditioned clinical concepts and keeping their evidence comparable.The extraction process segments the draft, normalizes text, matches a radiology entity dictionary, and merges duplicate mentions.Source: B. Initial Report Generation and… · p. 4

C. MM-KG Agent

The MM-KG Agent expands entities in the preliminary report into local knowledge-graph neighborhoods. It encodes central entities, neighbors, and relation labels for fusion with retrieval-based knowledge.

C. MM-KG Agent

  • For each report entity, the agent finds matching graph triples, ranks their edges by frequency, and retains the top-Kg neighboring entities.In the current setting, each entity is linked to at most ten neighbors (Kg = 10).Source: C. MM-KG Agent · p. 4
  • The central entity, neighboring entities, and relation labels are encoded with frozen Bio_ClinicalBERT and fused with retrieval-based knowledge representations.Source: C. MM-KG Agent · p. 4
  • The graph encoding aims to preserve relations among anatomical regions, imaging findings, and diseases by providing local graph context rather than processing concepts independently.Source: C. MM-KG Agent · p. 4

D. Knowledge Agent

The Knowledge Agent retrieves unstructured medical text as a complement to graph evidence. It builds entity-level knowledge items from chest X-ray question-answer samples and retrieves compact evidence relevant to entities in the draft report.

D. Knowledge Agent

  • The retrieval corpus is built from PubMedVision chest X-ray visual question-answering samples, which an LLM summarizes into knowledge items collected as snippets.Source: D. Knowledge Agent · p. 5
  • Queries are grounded in the preliminary report, target entity, and entity type, and retrieved knowledge items are designed to be focused and less redundant than raw web text or complete reports.Source: D. Knowledge Agent · p. 5
  • The retriever ranks candidate chunks by similarity and reranks them, retaining one chunk per entity in the current experiments after retrieving ten candidates.Source: D. Knowledge Agent · p. 5
  • Overlapping retrieved chunks are merged and deduplicated, then encoded with frozen Bio_ClinicalBERT using mean pooling.Source: D. Knowledge Agent · p. 5
  • The retrieval branch supplies descriptive evidence that may be absent from graph triples, while encoding chunks as continuous vectors avoids concatenating long text prompts and keeps the LLM input compact.Source: D. Knowledge Agent · p. 5

E. knowledge Fusion Module

The Knowledge Fusion Module combines graph and retrieval knowledge with image tokens to condition report generation. It encodes graph neighborhoods and retrieved chunks before concatenating their projected representations with visual features.

E. knowledge Fusion Module

  • Graph and retrieval knowledge are complementary, but directly feeding both into the LLM could create an excessively long, redundant prompt, motivating a knowledge fusion module.Source: E. knowledge Fusion Module · p. 6
  • A relation-aware graph attention encoder uses each central entity as a query and adds relation embeddings to neighbor keys to encode local graph context.Source: E. knowledge Fusion Module · p. 6
  • The graph encoder concatenates entity outputs and dynamically pads them into graph knowledge tokens, while the retrieval branch encodes deduplicated chunks with Bio-ClinicalBERT.Source: E. knowledge Fusion Module · p. 6
  • Graph and retrieval representations are projected into the visual-feature space and concatenated with image tokens to form the multimodal representation.The fused representation is projected into the LLM hidden space and used with the generation prompt as input for report generation.Source: E. knowledge Fusion Module · p. 6

F. Loss Function

MAC-RRG trains report generation with an autoregressive language-modeling objective conditioned on a multimodal prefix. Only target report tokens contribute to the loss.

F. Loss Function

  • The multimodal prefix combines visual, graph, and retrieval tokens with the textual prompt embedding to condition generation of the target report.Source: F. Loss Function · p. 7
  • The target report probability is factorized autoregressively, conditioned on the multimodal prefix and preceding report tokens.The target report is denoted Y = (y1, y2, ..., yT), where T is the number of report tokens.Source: F. Loss Function · p. 7
  • Training minimizes average negative log-likelihood over target report tokens, masking prompt and multimodal-token labels so only report tokens contribute to the loss.Source: F. Loss Function · p. 7

IV. EXPERIMENTS

The experiments evaluate MAC-RRG on three chest X-ray report datasets using both language-generation metrics and clinical abnormality-recognition metrics.

IV. EXPERIMENTS

  • Evaluation covers IU-Xray, MIMIC-CXR, and CheXpert Plus, with dataset-specific splits following prior protocols for IU-Xray and MIMIC-CXR.IU-Xray contains 7,470 images and 3,955 reports; MIMIC-CXR contains 377,110 images and 227,835 reports.Source: A. Datasets and Evaluation Metric · p. 7
  • BLEU, ROUGE-L, METEOR, and CIDEr assess linguistic generation quality, including n-gram similarity, sequence overlap, lexical variation, and informative expressions.The metrics use n-gram precision, longest common subsequence, morphological and synonym matching, and TF–IDF-weighted n-grams, respectively.Source: A. Datasets and Evaluation Metric · p. 7
  • Precision, Recall, and F1 assess recognition of clinically relevant abnormalities, with F1 combining precision and recall.Precision reflects false-positive predictions, while recall reflects the proportion of actual positive labels identified.Source: A. Datasets and Evaluation Metric · p. 7

B. Implementation Details

The experiments compare MAC-RRG across public X-ray report-generation benchmarks and describe its retrieval-and-fusion implementation. Reported results are competitive across datasets, with strong clinical efficacy on CheXpert Plus.

B. Implementation Details

  • Swin Transformer and Llama2-7B anchor the pipeline; entities retrieve up to 10 KG neighbors and reranked chunks from a 393-entry text memory, then fused features enter LLM space.Frozen Bio ClinicalBERT encodes KG nodes, neighbors, and relation edges into 768-dimensional features.Source: B. Implementation Details · pp. 7–8

C. Comparison on Public Benchmark Datasets •

D. Ablation Study •

The component analysis finds that combining structured graph knowledge, retrieved textual evidence, and knowledge fusion improves report-generation metrics. Further ablations select a balanced checkpoint and settings for neighborhood size, entity types, corpus size, and fusion.

Component Analysis

  • The full model improves BLEU-1 from 0.361 to 0.375 and clinical F1 from 0.260 to 0.295 over baseline.It also reaches BLEU-4 0.105 and METEOR 0.151, compared with baseline scores of 0.101 and 0.145.Source: D. Ablation Study • · p. 10

Analysis of Checkpoint Selection Strategy

  • MAC-RRG-ckpt2 is selected for balancing language quality and clinical factual accuracy, with BLEU-4 0.105 and Precision 0.350.The alternative ckpt1 has BLEU-4 0.106, while ckpt3 has the highest F1 and Recall but lower Precision.Source: D. Ablation Study • · p. 10

Analysis on different numbers of neighbors

  • topK=10 yields the best BLEU-4 (0.105) and METEOR (0.151) among tested neighborhood sizes; larger neighborhoods may add redundant or weakly relevant information.Across topK 3–20, BLEU-4 ranges 0.102–0.105 and METEOR 0.146–0.151.Source: D. Ablation Study • · p. 10, Table V · p. 10

Analysis of contextual entity types

  • Anatomy + Disorder is adopted: it achieves BLEU-4 0.105 and METEOR 0.151, while adding device entities reduces CIDEr from 0.121 to 0.115.Anatomy-only has the highest CIDEr score, 0.133; all entity types lower ROUGE-L and METEOR to 0.265 and 0.147.Source: D. Ablation Study • · p. 11, Table VI · p. 11

Analysis of different numbers of chunks

  • The nominal 400-chunk setting is selected to balance linguistic quality and clinical efficacy; increasing the corpus to 1,500 chunks degrades all NLG metrics.The selected checkpoint scores F1 0.295, Precision 0.350, and Recall 0.274, versus 0.278, 0.334, and 0.263 for the 1,000-chunk setting.Source: D. Ablation Study • · p. 11, Table VII · p. 11

Analysis of different fusion strategies

  • Direct concatenation achieves the best BLEU-4 and METEOR while maintaining competitive ROUGE-L; self-attention and cross-attention do not improve performance consistently.The authors suggest additional attention may amplify retrieval noise or weakly relevant graph neighbors.Source: D. Ablation Study • · p. 11, Table VIII · p. 11

E. Visualization •

The visualizations show how the two agents organize complementary evidence and compare generated reports with ground truth. MAC-RRG is reported to align more closely with ground-truth findings than R2GenGPT in the qualitative examples.

Visualization of Structured and Textual Knowledge Evidence

  • The MM-KG Agent organizes draft-report concepts into entity-centered subgraphs, with relations encoding anatomical associations, modifier relationships, and diagnostic associations.The illustrated subgraphs cover cardiomediastinal findings, atelectasis, pleural effusion, thoracic structures, and pulmonary edema.Source: E. Visualization • · p. 11
  • The Knowledge Agent retrieves text chunks that provide richer descriptive context and broader co-occurrence patterns than the MM-KG Agent’s concise relational evidence.The illustrated chunks concern cardiomegaly, pulmonary vascular congestion or edema, pleural effusion, costophrenic angle blunting, and lower-lung abnormalities.Source: E. Visualization • · pp. 11–12
  • The two agents’ structured and textual evidence jointly supports knowledge fusion and refinement of the initial report.The example begins with a draft describing cardiomediastinal enlargement, possible edema, a small left effusion, and other findings.Source: E. Visualization • · pp. 11–12

Report Generation

  • MAC-RRG reports show stronger consistency with ground truth than R2GenGPT for both positive findings and negative observations in the CheXpert Plus examples.Figure 5 marks matching sentences for MAC-RRG in yellow, R2GenGPT in cyan, and matches shared by both in pink.Source: E. Visualization • · p. 12, Figure 5 · p. 12

F. Limitation Analysis

MAC-RRG’s iterative refinement depends on the quality of its initial draft, and the tested attention-based fusion strategies do not consistently improve performance under the current setting. The conclusion reports competitive or superior results on multiple NLG metrics and identifies directions for further work.

F. Limitation Analysis

  • Errors or omissions in the initial draft can lead both agents to retrieve incomplete or less relevant external evidence.The paper identifies more robust entity extraction and draft correction as important for improving iterative refinement.Source: F. Limitation Analysis · p. 12
  • Self-attention and cross-attention do not consistently improve overall performance under the current setting, so direct concatenation is used as the fusion strategy.The authors suggest added attention interactions may amplify weakly relevant retrieved chunks or noisy graph neighbors.Source: F. Limitation Analysis · pp. 12–13

V. CONCLUSION

  • Experiments on IU X-Ray, CheXpert Plus, and MIMIC-CXR show competitive or superior performance on multiple NLG metrics, especially high-order BLEU scores.The conclusion associates the high-order BLEU results with more coherent medical phrases and disease-related descriptions.Source: §V · p. 13
  • The paper identifies robust entity extraction, larger medical knowledge bases, uncertainty-aware retrieval, and adaptive fusion as future research directions for improving generalization and clinical trustworthiness.It concludes that role-specialized agents provide complementary structured and textual evidence for LLM-based report generation.Source: §V · p. 13

Related papers

Loading 2609.26124v1…