PaperlayerView exact revision on arXiv

Source-linked AI summary

MedPAO-VL: Interpretable and Protocol-Compliant Agent for Structured Radiology Report Generation

Shrish Shrinath Vaidya, Gowthamaan Palani, Minmini Selvam, Gokulraja Srinivasaraja, Ganapathy Krishnamurthi

arXiv:2610.21794v1cs.CV

TL;DR

Automated radiology reporting still faces a challenge in producing accurate, structured reports with interpretable reasoning. MedPAO-VL combines uncertainty-aware visual concepts, concept-grounded generation, and iterative correction before protocol structuring. Its reported results include improved concept coverage with the PAO loop and strong clinical report-generation metrics.

  • Problem

    End-to-end radiology report models can be black boxes, while post-hoc structuring does not reason from primary visual evidence, leaving explainability and generative capability uncombined.

    Source: §1 · p. 2

  • Method

    MedPAO-VL predicts visual concepts, generates a concept-grounded narrative, checks it for semantic discrepancies, iteratively refines it, and structures the report to a clinical protocol.

    Source: §1 · p. 3, §3.3 · p. 8

  • Results

    The PAO full, cold-cache condition raises concept coverage ratio from 0.602 to 0.803 and improves ChexBERT-F1, RadGraph-F1, and GREEN scores over the deterministic baseline.

    Source: §5.3 · p. 14

  • Takeaways & Limitations

    The approach combines visual explainability and auditable intermediate reasoning with protocol-compliant structured radiology report generation.

    Source: §6 · p. 15

Abstract

from arXiv · show

Despite the remarkable generalization capabilities of Large Language Models, generating clinically accurate and structured radiology reports from medical images remains a significant challenge. Current end-to-end approaches function as black boxes, prone to factual hallucinations and generating ungrounded findings. This lack of reliability and failure to adhere to domain-specific structuring protocols, fundamentally limits their clinical utility. We present MedPAO-VL, a multimodal agent that shifts the paradigm from monolithic generation to explainable, iterative, protocol-compliant structured report generation. Unlike deterministic models, our plan-act-observe loop functions as a goal-oriented agent: it first perceives the Chest X-Ray through an explainable vision module designed to capture inherent diagnostic uncertainty, then iteratively refines the generated report by observing semantic discrepancies and self-correcting its generation for better alignment with visually detected clinical findings. Evaluation of our framework demonstrates that this agentic reasoning improves clinical correctness and completeness, achieving an expert-rated accuracy score of 3.5/5 and a structural score of 4.3/5. By prioritizing pathology specific explainability and iterative refinement over opaque end-to-end mapping, MedPAO-VL offers a transparent and clinically viable alternative for automated structured radiology report generation.

1 Introduction

Automated radiology reporting must address both accurate, structured output and trustworthy, auditable reasoning. MedPAO-VL combines multimodal visual reasoning with an agentic, protocol-driven framework to generate structured reports from CXRs.

1 Introduction

  • Automated reporting is needed amid rising imaging volumes, but high-stakes deployment also requires interpretable, auditable outputs and clinically safe failure modes.Direct image-to-report models are described as black boxes, while post-hoc structuring operates on existing text rather than primary visual evidence.Source: §1 · pp. 1–2
  • The research gap is that existing approaches do not combine multimodal visual reasoning with transparent, protocol-driven auditability.The paper identifies this gap as compromising trustworthiness in automated radiology pipelines.Source: §1 · p. 2
  • MedPAO-VL extends MedPAO with multimodal capabilities and embeds explainability into report generation rather than adding it afterward.The stated contributions include an uncertainty-aware Vision Concept Module and evaluation of factual consistency and protocol compliance.Source: §1 · p. 3
  • MedPAO-VL unifies visual perception with a Plan-Act-Observe loop for protocol-compliant report generation and step-wise auditable reasoning.Its uncertainty-aware vision module is designed to provide explainable visual recognition.Source: §1 · p. 3

2 Related Works

Radiology report generation has progressed from encoder-decoder systems to transformer, vision-language, and agent-based approaches. Explainability work includes spatial and concept-level methods, but the paper identifies unresolved issues in uncertainty handling and concept verification.

2.1 Radiology Report Generation

  • Radiology report generation evolved from CNN–LSTM encoder-decoder models to CNN–LLM, ViT–LLM, and native vision-language and generalist medical AI systems.Other approaches add memory, retrieval, contrastive learning, lesion features, or medical knowledge graphs; agent systems decompose reporting tasks.Source: §2.1 · p. 3
  • Although foundational models and language models produce fluent narratives, the cited work says they struggle with clinical-protocol structure and complex reasoning or factual hallucinations.The subsection also describes agent approaches using specialized tasks and concept bottlenecks to ground report generation.Source: §2.1 · pp. 3–4

2.2 Explainable AI in Report Generation

  • Explainability methods use spatial cues such as region-guided detections, eye-gaze supervision, and interpretable attention maps, or concept-level intermediate representations to guide report generation.Some methods extract structured clinical entities from reports to guide generation.Source: §2.2 · p. 4
  • Most cited methods assume undocumented pathologies are absent, while general-purpose LLMs extract clinical concepts without radiologist verification; PromptMRG is the stated exception to the first limitation.Source: §2.2 · p. 4

3 Methodology

MedPAO-VL combines visual concept extraction, concept-grounded narrative generation, and an iterative Plan-Act-Observe loop, followed by protocol-based report structuring. Its vision module preserves uncertain findings and produces pathology-specific visual explanations.

3 Methodology

  • MedPAO-VL extends MedPAO with a Vision Concept Module, Concept Grounded Report Generator, and agentic Plan-Act-Observe loop for iterative refinement.The system is designed to generate protocol-compliant structured radiology reports.Source: §3 · p. 4, Figure 2 · p. 5

3.1 Vision Concept Module

  • The Vision Concept Module predicts present, absent, or uncertain for each target pathology, distinguishing true absence from a finding omitted in reporting.This three-class formulation is intended to mitigate the implicit negative bias created when unmentioned or uncertain findings are treated as negatives.Source: Mitigating Implicit Negative Bias · pp. 4–5
  • The Pathology Slot Module turns backbone features into pathology-specific context vectors through iterative slot refinement.Its pathology-specific slots are designed to extract features relevant to their corresponding pathologies.Source: Explainable Pathology Slot Module · p. 5
  • Softmax over image patches lets each pathology slot attend to evidence independently, supporting co-occurring findings and yielding inherent pathology-specific spatial maps.This differs from slot competition over patches and supplies visual grounding from the forward pass.Source: Explainable Pathology Slot Module · pp. 5–6
  • A shared classifier predicts three classes from final pathology context vectors, while the similarity regularizer discourages redundant attention maps across pathologies.The regularizer uses cubed pairwise attention similarity and penalizes positive off-diagonal redundancy while allowing anti-correlation.Source: Shared Classification Head · p. 6, Interpretability-Aware Training… · pp. 6–7

3.2 Concept Grounded Report Generator

  • The report generator conditions on both the CXR and present/absent clinical concepts, making narrative statements traceable to discrete findings while retaining image context for fluent descriptions.Source: §3.2 · p. 7

3.3 The Agentic PAO Loop

  • The PAO loop checks generated clinical entities against the vision concept list, replans and refines discrepant sections, then structures the report under a clinical protocol.It continues for up to N iterations or until the concept list is covered by current findings; dynamic caching reuses frequent ABCDEF mappings to bypass ontology-mapping and categorizer calls on cache hits.Source: §3.3 · p. 8

4 Experiments

The experiments evaluate MedPAO-VL’s report generation on MIMIC-CXR and IU X-RAY, and examine whether its pathology slots encode clinically meaningful relationships and produce anatomically faithful attention maps.

4.1 Dataset & Pre-processing

  • The curated MIMIC-CXR training set contains approximately 50,000 image-report-concept triplets and 236 radiologist-validated standardized concept groups.Reports were selected for HISTORY, FINDINGS, and IMPRESSION sections; FINDINGS supplied concepts normalized to SNOMED CT.Source: §4.1 · p. 8

4.2 Implementation Details

  • The Vision Concept Module uses a fixed MedSigLIP encoder and 236 pathology-specific slots, while MedGemma 4B is fine-tuned for report generation.The generator was first tuned on 10,000 samples for ablations and then trained on 50,000 samples.Source: §4.2 · p. 9

4.3 Report Generation Evaluation

  • On MIMIC-CXR, the 50K generator achieves CheXbert-F1 0.617 and RadGraph-F1 0.427, outperforming MAIRA-2 and Flamingo-CXR on these clinical metrics.Its CheXbert Micro-F1 is 0.706; on IU X-RAY, the GREEN mean score is 0.733.Source: Unstructured Report Evaluation · p. 9, Table 2 · p. 10
  • Structured reports were assessed by two blinded senior radiologists on 50 stratified MIMIC-CXR cases, comparing the agent with MedGemma-27B multimodal reports.Accuracy measures clinical correctness of findings, while Structural Score measures conformity to the predefined reporting protocol.Source: Structured Report Evaluation · pp. 9, 11, Table 3 · p. 10

4.4 Interpretability and Representational Analysis

  • Pathology-slot similarities align related pneumonia and CHF findings while placing opposing conditions in negative or opposing directions.For pneumonia, hazy bilateral opacities and likely loculated fissural fluid each have cosine similarity +0.985; for mild CHF, vascular congestion is +0.893.Source: Latent Orthogonality · pp. 11–12, Figure 3 · p. 11
  • Without ABCDEF-category training, slot activations cluster with anatomical categories, including spatially separated Airways, Breathing, and Circulation clusters.The analysis uses t-SNE on the 236-dimensional slot activation space and a stratified sample of 4,000 slot instances per category.Source: Latent Orthogonality · p. 12, Figure 4 · p. 12, Emergent Anatomical Organization in… · p. 13
  • PSM attention maps localize co-occurring findings to distinct, anatomically faithful regions of a representative chest X-ray.Examples include costophrenic-angle attention at the lung bases and cardiomegaly attention across the cardiac silhouette; the maps arise during the forward pass without post-hoc approximation.Source: Figure 5 · p. 12, Spatial Explainability · p. 13

5 Ablations

The ablations examine successive vision-module changes, concept grounding, and the PAO loop, with iterative correction improving concept coverage and caching reducing inference time.

5.1 Vision model ablations

  • The shared-classifier PSM variant improves performance over PSM alone while reducing parameter count; similarity loss minimally changes metrics but diversifies pathology-specific attention maps.PSM also modestly improves on the baseline and enables interpretable attention maps.Source: Table 4 · p. 13, §5.1 · p. 14

5.2 Impact of Concept Grounding

  • Adding medical concepts to the finetuned MedGemma-4B model improves performance, placing the complete model ahead of the other tested variants.The finetuned model also outperforms both the 4B and 27B zero-shot baselines.Source: Table 5 · p. 13, §5.2 · p. 14

5.3 PAO Loop Ablations

  • 0.803 concept coverage with PAO Full, Cold Cache, versus 0.602 for the deterministic baseline; CheXbert-F1, Radgraph-F1, and GREEN also improve.The PAO loop averages 2.8 iterations, while mean LLM calls rise from 6.0 to 11.2 before warm-cache savings.Source: §5.3 · p. 14, Table 6 · p. 14
  • 45% lower inference time with caching, from 295s to 163s, while quality metrics remain unchanged.The cache retrieves previously computed concept-to-ABCDEF mappings.Source: Table 6 · p. 14, §5.3 · p. 15

6 Conclusion

MedPAO-VL combines a Vision Concept Module, a Concept Grounded Report Generator, and an iterative PAO loop to produce structured, protocol-compliant reports with auditable reasoning. Its diagnostic reach is bounded by a 236-group vocabulary.

6 Conclusion

  • The system combines visual localization, uncertainty preservation, and ontology adherence through three components: the Vision Concept Module, Concept Grounded Report Generator, and iterative PAO loop.The conclusion describes the resulting reports as reliable, factually consistent, and protocol-compliant.Source: §6 · p. 15
  • A predefined vocabulary of 236 concept groups limits documentation of ultra-rare anomalies.Future work proposes dynamically expanding the vocabulary and extending the framework to multimodal imaging and broader diagnostic workflows.Source: §6 · p. 15

Supplementary Material

The supplementary material outlines the ABCDEF CXR review protocol, MedPAO components, and technical details of the PAO loop and concept cache.

A The ABCDEF Protocol for Systematic CXR Interpretation

  • ABCDEF is a sequential CXR review framework intended to evaluate anatomical and radiological domains before forming an impression, reducing omission risk.Its categories cover Airways, Breathing, Circulation, Diaphragm and Below, External structures, and Foreign Material.Source: A The ABCDEF Protocol for Systematic… · p. 22, Table 7 · p. 23
  • The protocol checks Airways, Breathing, and Circulation, including tracheal position, lung and pleural findings, cardiac silhouette, and mediastinal contours.Examples include assessing the cardiothoracic ratio on a standard PA film and inspecting pulmonary vascularity.Source: A The ABCDEF Protocol for Systematic… · p. 22
  • The remaining checks cover the diaphragm and visible abdomen, external chest-wall structures, and positioning of devices, lines, and tubes.Examples include free subdiaphragmatic air, fractures, and verifying tube and catheter locations.Source: A The ABCDEF Protocol for Systematic… · pp. 22–23

B Overview of MedPAO Components

  • MedPAO combines concept extraction, ontology mapping and filtering, protocol categorization, and structured report generation under an LLM orchestrator.The inherited tools ground concepts in SNOMED CT and use source sentences as explainable reasoning traces.Source: B Overview of MedPAO Components · p. 24

C.1 Plan-Act-Observe: Technical Implementation

C.2 Concept Grounded Report Generator Implementation

The report generator uses a quantized, LoRA-finetuned MedGemma-4B model. Evaluation compares blinded radiologist ratings of generated reports using clinical-accuracy and protocol-structure scores.

C.2 Concept Grounded Report Generator Implementation

  • MedGemma-4B is loaded in 4-bit NF4 and fine-tuned with LoRA on 50,000 reports for four epochs, using a single NVIDIA A6000 GPU.The setup uses rank 32, alpha 64, dropout 0.05, and AdamW with a 2 × 10^-4 learning rate.Source: §C.2 · p. 26

D.1 Structured Report Evaluation Metrics

  • Radiologists blindly compare system-generated reports with MedGemma-27B reports alongside CXR images and MIMIC-CXR references, scoring clinical accuracy and protocol adherence.Accuracy measures clinical correctness; structural score measures adherence to the defined reporting protocol.Source: §D.1 · p. 27

D.2 Extended Radiologist score evaluation

Radiologist ratings favored MedPAO-VL on high scores for accuracy and structure, although inter-rater agreement was fair to moderate. The data pipeline extracted, categorized, standardized, and frequency-filtered concepts, with radiologist validation supporting the retained groups.

D.2 Extended Radiologist score evaluation

  • Inter-rater agreement ranged from fair to moderate (quadratic weighted κ = 0.22–0.51), reflecting subjectivity in report-quality judgments.MedPAO-VL accuracy agreement (κ = 0.222) was lower than MedGemma-27B (κ = 0.510).Source: §D.2 · p. 27
  • 60% of MedPAO-VL accuracy assessments scored 4 or 5, versus 44% for MedGemma-27B; the corresponding structure figures were 90% and 70%.The evaluation cohort was N=50 per rater, limiting formal significance testing; Fig. 7 shows the radiologist score distribution.Source: §D.2 · p. 29, Figure 7 · p. 29

E.1 Data Curation

  • The curation pipeline extracted concepts with source sentences, classified them as present or absent, mapped them to SNOMED CT, and retained groups occurring at least 100 times, yielding 236 labels.Absent concepts were converted to present forms for standardization and restored afterward; low-frequency groups were filtered from the final vision-model labels.Source: §E.1 · pp. 29, 31

E.2 Radiologist validation on frequency filtering

  • Radiologists judged the 236 accepted groups clinically meaningful; rejected groups were mainly technical descriptors or entities too rare for reliable learning.The authors state that removal reduces class imbalance and noise without materially affecting clinical usability; rare entities may be added as more data become available.Source: §E.2 · p. 32

E.3 Data Analysis

The analysis describes how concept counts vary across reports and standardized groups, and how a nine-concept threshold separates normal from complex reports for data splitting.

E.3 Data Analysis

  • Standardization shifts the distribution toward fewer concepts per report, primarily because low-frequency concepts are removed; Fig. 9 compares raw and standardized distributions.Fig. 10 shows the most frequent raw concepts.Source: §E.3 · p. 32, Figure 9 · p. 33, Figure 10 · p. 33
  • Reports with fewer than nine extracted concepts are categorized as normal and those with more as complex; complex reports have higher mean and median word counts, and are fewer in number.This separation supports balanced train-validation-test splits for the grounded report generator; Fig. 11 presents category word-count distributions.Source: §E.3 · pp. 32–33, Figure 11 · p. 34, Table 13 · p. 34
  • SNOMED CT groups contain varying numbers of present and absent concepts, and their occurrence frequencies vary across the dataset.Figs. 12 and 13 show the present/absent counts and frequency patterns across standardized groups.Source: §E.3 · p. 33, Figure 12 · p. 34, Figure 13 · p. 35

F.1 Emergent Clinical Co-occurrence in Latent Space

The pathology-slot analysis reports mostly orthogonal latent representations alongside localized correlations between co-occurring findings. Further analyses describe slot specialization and the effect of similarity loss on attention maps.

F.1 Emergent Clinical Co-occurrence in Latent Space

  • 0.91 cosine similarity links biapical scarring and bibasilar airspace opacities, while the broader pathology embedding space is predominantly orthogonal.The authors interpret localized positive correlations as encoding clinical co-occurrence rather than noise; Fig. 14 shows the global heatmap and a Breathing-category cross-section.Source: §F.1 · pp. 35–36, Figure 14 · p. 35

F.2 Clinical Manifold Analysis and Concept Specialization

  • Per-pathology t-SNE projections show localized positive-instance clusters for rare devices and chest tubes, and class-conditional separation for pulmonary edema.The authors report that slots specialize in distinct visual signatures, including in complex multipathological images; Fig. 15 presents representative per-slot projections.Source: §F.2 · p. 36, Figure 15 · p. 37

F.3 Impact of Similarity loss on Attention Maps

  • Without similarity loss, attention concentrates on limited image regions; adding it produces more diverse, spatially meaningful maps aligned with pathologies.The authors state that similarity loss also improves explanation-map interpretability and quality; Fig. 16 compares maps with and without the loss.Source: §F.3 · pp. 36–37

G End-to-End System Walkthrough

This walkthrough traces MedPAO-VL from visual perception and report generation through iterative correction of omitted concepts to ontology mapping and structured findings. It shows the plan-act-observe workflow alongside concept extraction, ABCDEF categorization, and report refinement.

Phase 1: Visual Perception

  • The vision module outputs clear lungs, a normal cardiomediastinal silhouette, prior sternotomy and bypass surgery, and absent tiny pleural effusion and pneumothorax.It estimates present, absent, or uncertain probabilities for each pathology and produces spatial attention maps.Source: G End-to-End System Walkthrough · pp. 37, 39, Figure 17 · p. 38

Phase 2: Initial Report Generation

  • The concept-grounded generator uses the visual findings to produce an initial narrative describing clear lungs, stable contours, and sternotomy wires and mediastinal clips.The process is described as generating findings strictly from grounded visual concepts and CXR analysis.Source: G End-to-End System Walkthrough · p. 39

Phase 3: Self Correction Loop

  • The first extraction and SapBERT-based overlap check identify “no tiny pleural effusion” and “no pneumothorax” as concepts missing from the initial report.The extractor pairs clinical findings with their source sentences, while the checker matches extracted report concepts against vision-module concepts and returns missing concepts.Source: G End-to-End System Walkthrough · pp. 39–40
  • After re-prompting, the report adds that there is no pleural effusion or pneumothorax, and the next extraction identifies both findings as report concepts.The walkthrough then compares the expanded extracted concept set with the vision-module concepts again.Source: G End-to-End System Walkthrough · pp. 40–41

Phase 4: Report Structuring

  • The structuring phase maps extracted concepts to SNOMED CT, but the shown matches include “stable cardiomediastinal silhouette” mapped to “Stable angina” and “no pneumothorax” to “Perinatal pneumothorax.”The stated purpose is to obtain clinical information for structural grounding in text.Source: G End-to-End System Walkthrough · p. 42

G End-to-End System Walkthrough

  • The structured-report generator turns categorized concepts and their source sentences into ABCDEF findings, as illustrated by a sample report.The example places clear lungs and absent pleural findings in B, stable cardiomediastinal and hilar contours in C, and sternotomy wires and mediastinal clips in F.Source: G End-to-End System Walkthrough · p. 47

H LLM prompts

  • In the PAO loop, the LLM selects a tool and parameters using available tools and prior steps, then evaluates tool output against the original query to continue or return a final response.The observation prompt includes retry and replanning mechanisms.Source: H LLM prompts · p. 47, Figure 18 · p. 48, Figure 19 · p. 49
  • The prompt set extracts clinical concepts with source sentences, classifies them into ABCDEF, and conditions structured report generation on the protocol while prohibiting unsupported findings.Concept extraction uses five diverse examples spanning normal findings to complex pathologies.Source: H LLM prompts · p. 47, Figure 20 · p. 50, Figure 21 · p. 51
  • A finetuned multimodal concept-grounded generator uses the chest X-ray and clinical concepts to produce a fluent narrative, and is reused to generate findings flagged as missing.The reprompting step is constrained to generate only missing findings rather than repeat or regenerate the full report.Source: H LLM prompts · p. 47, Figure 23 · p. 53, Figure 24 · p. 54

Related papers

Loading 2610.21794v1…