PaperlayerView exact revision on arXiv

Source-linked AI summary

UR$^{2}$-MLLM: Uncertainty-aware Revisit Reasoning in Multimodal Large Language Models for Radiology Report Generation

Yucheng Chen, Yang Yu, Jiazhou Zhou, Yufei Shi, Yongying Lan, Yichi Zhang, Liyi Li, Si Yong Yeo

arXiv:2608.22217v1cs.CV

TL;DR

Radiology report-generation reasoning lacks a dynamic way to revisit uncertain image regions, despite the iterative visual re-examination used in diagnosis. UR2-MLLM adds uncertainty perception, supervised detect-and-copy revisits, and visual-grounding reinforcement learning; across in-distribution and out-of-distribution benchmarks, it reports consistent linguistic and clinical improvements.

  • Problem

    Existing radiology report-generation methods provide static visual evidence without dynamically revisiting uncertain regions during reasoning.

  • Method

    UR2-MLLM trains uncertainty perception, constructs multimodal revisit trajectories with detect-and-copy, and refines visual grounding through reinforcement learning.

  • Results

    UR2-MLLM achieves state-of-the-art performance with consistent improvements across linguistic and clinical efficacy metrics on in-distribution and out-of-distribution benchmarks.

  • Takeaways & Limitations

    Uncertainty-aware visual revisit reasoning is presented as a basis for more reliable and clinically aligned radiology report generation.

  • Takeaways & Limitations

    The evaluation is confined to chest X-ray benchmarks with automatic metrics and lacks external clinical validation and radiologist human-in-the-loop assessment.

Abstract

from arXiv · show

Radiologists generate diagnostic reports through iterative and selective revisiting of suspicious regions to refine their interpretations. Recent multimodal large language models (MLLMs) for radiology report generation (RRG) have shifted from text-only reasoning toward a ``Thinking-with-Images'' paradigm, incorporating visual evidence into the reasoning process. However, existing methods provide static visual evidence without a dynamic revisit mechanism during reasoning, neglecting how radiologists re-examine uncertain observations. To this end, we propose an Uncertainty-aware Revisit Reasoning MLLM (UR$^{2}$-MLLM) framework that dynamically revisits uncertain regions during reasoning for RRG. UR$^{2}$-MLLM is first equipped with uncertainty perception by training on an uncertainty-aware dataset. We then construct a multimodal reasoning trajectory dataset together with a detect-and-copy mechanism, which guides when and where to revisit. Finally, a visual grounding reward refines this behavior through reinforcement learning, aligning the revisited regions with corresponding anatomical structures. Experiments on MIMIC-CXR and IU-Xray show that UR$^{2}$-MLLM achieves state-of-the-art performance, highlighting the value of uncertainty-aware visual revisit reasoning for reliable and clinically aligned report generation.

1 Introduction

Radiology report generation involves uncertain, iterative interpretation, but existing reasoning paradigms either disconnect reasoning from images or provide only static visual evidence. UR2-MLLM addresses this gap by dynamically revisiting uncertain anatomical regions during reasoning and achieves state-of-the-art results.

  • Motivation: Radiologists repeatedly revisit suspicious regions and cross-reference visual findings with prior knowledge while refining diagnostic interpretations.This motivates modeling report generation as decision-making under uncertain observations.
  • Existing paradigms: Text-only reasoning encodes the image once, decoupling subsequent reasoning from visual evidence and potentially accumulating errors or hallucinated findings.The reasoning trajectory proceeds purely in textual space after image encoding.
  • Existing paradigms: Think-with-Images methods provide static visual evidence but lack an explicit mechanism to identify and refine uncertain regions as reasoning progresses.This single-pass approximation treats perception as deterministic despite ambiguity from overlapping anatomy or subtle pathologies.
  • Proposed framework: UR2-MLLM dynamically determines when and where to revisit visual evidence based on uncertainty and corresponding anatomical regions.The framework explicitly models both revisit timing and spatial focus.
  • Proposed framework: The three-stage pipeline trains uncertainty perception, supervises multimodal revisit trajectories with detect-and-copy, and uses visual-grounding reinforcement learning.Hedging expressions are aligned with anatomical regions; a visual grounding reward aligns revisited regions with subsequent textual claims.
  • Results: Extensive experiments on IU-Xray and MIMIC-CXR report state-of-the-art performance with consistent improvements in efficiency, faithfulness, and interpretability.The supplied passage states the overall experimental conclusion without reporting individual metric values.

2 Related Work

Related work on radiology report-generation reasoning includes uncertainty estimation and multimodal reasoning, but visual evidence is either absent after encoding or incorporated without dynamic re-examination. UR2-MLLM is framed against these existing categories.

  • Uncertainty modeling: Radiology uncertainty methods estimate uncertainty probabilistically, through consistency across perturbed inputs or generations, or through supervision using hedging cues.The cited passage identifies Monte Carlo dropout, variational latent models, output variation, and supervised extraction of hedged language as representative approaches.
  • Reasoning in MLLMs: MLLM radiology report-generation methods increasingly include explicit reasoning traces.The passage introduces reasoning traces as a recent development in this area.
  • UR2-MLLM framework: Figure 2 presents UR2-MLLM through multimodal reasoning trajectory construction, a training-pipeline overview, and stage-wise training details.The framework’s Stage 1 constructs an uncertainty-aware corpus associating anatomical regions with hedged report phrases.
  • Reasoning in MLLMs: Text-based reasoning generates traces purely in textual space after image encoding, preventing image re-examination to verify or refine claims.Static visual reasoning instead incorporates global image features or region-level cues.

3 Methodology

UR2-MLLM models radiology report generation as uncertainty-aware, visually grounded reasoning with iterative region revisits. Its three-stage methodology combines uncertainty perception, multimodal revisit trajectories with detect-and-copy, and reinforcement learning using visual grounding rewards.

  • UR2-MLLM uses three training stages: uncertainty-aware perception, grounded multimodal reasoning, and clinically faithful generation via reinforcement learning.
  • The overall objective addresses conventional maximum-likelihood training’s failure to explicitly model iterative visual revisits under uncertainty by optimizing a multidimensional reward.
  • Stage 1: Uncertainty-aware Perception: Stage 1 aligns hedging expressions with anatomical regions in chest X-rays, giving the model a perceptual prior that diagnostic uncertainty is region-specific.
  • Stage 2: Grounded Multimodal Reasoning: Stage 2 augments sentence-level reasoning traces with hedged region references, detect calls, bounding boxes, patch indices, and post-revisit summaries.
  • Stage 2: Grounded Multimodal Reasoning: The detect-and-copy mechanism extends the vocabulary with image-patch pointers, allowing each decoding step to emit either a text token or a patch pointer.
  • Stage 3: Clinically Faithful Generation via RL: Stage 3 uses a visual grounding reward that compares each revisited region with its follow-up summary against random-region baselines, then combines it with format, clinical, and coherence rewards.

4 Experiments

Experiments on MIMIC-CXR and IU X-ray show strong report-generation and clinical-efficacy performance, while ablations and revisit analysis examine the value and behavior of dynamic visual revisiting.

  • Overall results: UR2-MLLM achieves state-of-the-art performance across experiments on MIMIC-CXR and IU X-ray, with improvements in efficiency, faithfulness, and interpretability.
  • MIMIC-CXR comparison: UR2-MLLM surpasses CURV on five of six MIMIC-CXR NLG metrics and outperforms CURV across all five reported clinical-efficacy metrics.METEOR is the exception among NLG metrics, scoring 28.66 versus CURV’s 30.43.
  • Out-of-distribution evaluation: UR2-MLLM leads all six IU X-ray NLG metrics and nearly all clinical-efficacy metrics, including 31.81 BLEU-1, 7.11 BLEU-4, 40.75 METEOR, and 41.20 ROUGE-L.CheXbert Mi-F1 is the only metric without a lead: 74.15 versus CURV’s 74.36.
  • Ablation study: Stage 1 improves clinical F1 from 4.75 to 12.35, while adding Stages 2 and 3 raises performance to 27.79 BLEU-1, 28.66 METEOR, and 13.96 clinical F1.Stage 1 slightly degrades NLG metrics because it focuses on structured uncertainty enumeration rather than full report generation.
  • Ablation study: Dynamic visual revisit outperforms both no revisit and static vision, reaching 27.79 BLEU-1 compared with 10.72 and 16.13, respectively.Clinical F1 likewise reaches 13.96, versus 3.87 without visual revisit and 6.13 with static vision.
  • Visual reasoning evaluation: On 3,858 MIMIC-CXR test cases, Stage 3 raises trigger rate from 25.5% to 51.6% and recall from 25.6% to 64.3%, although precision and spatial overlap remain limited.The ground-truth revisit base rate is 35.0%, so Stage 3 shifts behavior from undertriggering to actively invoking revisits.

5 Conclusion

UR2-MLLM models radiologists’ iterative revisiting of suspicious regions by triggering visual revisits from uncertainty and grounding them anatomically. It reports consistent gains across in-distribution and out-of-distribution benchmarks on linguistic and clinical efficacy metrics.

  • UR2-MLLM explicitly models when and where to revisit visual evidence during radiology report generation.
  • The framework treats diagnostic uncertainty as an explicit trigger for revisiting visual evidence as reasoning unfolds.
  • UR2-MLLM progressively learns uncertainty perception, localized visual grounding, and reinforcement-learning-based revisit refinement.
  • Across in-distribution and out-of-distribution benchmarks, UR2-MLLM consistently improves linguistic and clinical efficacy metrics.
  • Further analysis attributes these gains to grounded visual revisits rather than imitation of surface reasoning patterns.

Limitations

The paper identifies uncertainty modeling, reasoning data, spatial supervision, and evaluation scope as limitations. These constraints leave the clinical reliability and deployment readiness of UR2-MLLM unresolved.

  • Textual hedging cues provide only a coarse, weakly constrained signal for diagnostic uncertainty.The authors suggest radiologist confidence ratings or calibrated probability estimates as stronger alternatives.
  • The Stage 2 multimodal reasoning trajectory corpus is relatively small and partly based on LLM-rewritten traces, which may introduce upstream biases.The corpus contains approximately 2K trajectories.
  • Spatial grounding relies on silver-standard Chest ImaGenome anatomical annotations, constraining revisit-supervision precision and applicability where region-level annotations are unavailable.
  • Evaluation is confined to chest X-ray benchmarks with automatic metrics and lacks external clinical validation or radiologist human-in-the-loop assessment.The authors therefore leave real-world clinical reliability an open question.

Ethics Considerations

The study states that it follows established ethical standards and uses public datasets solely for their intended research purposes. It reports no personally identifiable information, sensitive content, human subjects, or annotators.

  • All datasets are publicly available and used solely for their intended research purposes.
  • The datasets contain no personally identifiable information or sensitive content, posing no reported privacy or confidentiality risks.
  • The study involved neither human subjects nor annotators.

A.1 Dataset

The study builds its training and evaluation resources from MIMIC-CXR and Chest ImaGenome, then constructs stage-specific uncertainty and reasoning corpora. Reports are evaluated using complementary language-generation and clinical-efficacy measures.

  • MIMIC-CXR contains over 370,000 chest X-ray images paired with free-text radiology reports.
  • Chest ImaGenome supplies anatomical bounding boxes and structured scene graphs for MIMIC-CXR studies.
  • The authors construct an uncertainty-aware perception corpus D1 containing approximately 101k samples for Stage 1.
  • Natural language generation quality is measured with BLEU-1/2/3/4, METEOR, and ROUGE-L against ground-truth reports.
  • Clinical efficacy evaluates factual agreement on 14 CheXpert pathologies using Accuracy, Macro-F1, and Micro-F1 from CheXbert labels.Entity F1 and complete F1 from RadGraph are also reported.

A.3 Implementation Details

UR2-MLLM uses a three-stage training pipeline that builds uncertainty perception, constructs multimodal revisit trajectories, and applies reinforcement learning. The implementation combines curated LLM-generated supervision with specified optimization and inference settings.

  • Optimization and inference: Training initializes from Qwen2.5-VL-3B and uses 4 NVIDIA A6000 GPUs, BF16 precision, AdamW, cosine decay, and 5% warm-up.Stages 1 and 2 run for 3 supervised epochs each; Stage 3 runs GRPO for 1 epoch with G = 4 trajectories per prompt, β = 0.04, and clip range ϵ = 0.2.
  • Optimization and inference: Inference uses nucleus sampling with temperature T = 0.7 and top-p = 0.9.These settings define the decoding procedure used for generated reports.
  • Training pipeline: Stage 1 converts reports into an uncertainty-aware perception dataset using section parsing, uncertainty extraction, and per-phrase classification calls.The dataset supports Stage 1 supervised fine-tuning and links uncertain findings to anatomical regions.
  • Training pipeline: Stage 2 rewrites uncertain report sentences with hedge cues and region references, then appends revisit summaries to populate reasoning trajectories.Each uncertain sentence is paired with an ImaGenome phrase, anatomy label, and bounding box before generating the reasoning trace.
  • Training pipeline: Stage 3 applies GRPO reinforcement learning to the Stage 2 objective without modifying the prompt.The prompt structure therefore remains fixed while reward shaping refines model behavior.

C Uncertainty Analysis

The uncertainty analysis shows that model training narrows and reshapes hedging language relative to radiologist reports. SFT establishes generic-hedge overuse, while the revisit reward changes the mid-frequency vocabulary rather than eliminating top-marker concentration.

  • Corpus and marker distribution: 276,778 reports yielded 155,542 hedge instances from a closed vocabulary of 35 uncertainty terms.Matching was case-insensitive, whole-word, and longest-first to avoid double-counting multi-word expressions.
  • Corpus and marker distribution: The five most common ground-truth markers were likely (2.69 per thousand words), appears (2.03), may (1.63), consistent with (1.22), and could (0.81).The ground-truth distribution was relatively balanced, with no single cue dominating radiologist phrasing.
  • Training-induced shifts: All model variants make appears the most frequent marker, while SFT reaches 4.34 versus 2.03 per thousand words in ground truth.SFT also raises suggestive of to 2.17 versus 0.49 and possible to 1.89 versus 0.41.
  • Training-induced shifts: The over-reliance on a small group of generic hedges is established during Stage 2 SFT and further accentuated by GRPO.The analysis attributes the shift to training dynamics rather than to GRPO alone.
  • Revisit-reward effect: Removing the revisit reward increases mid-frequency cues such as could (1.51 versus 0.81 in ground truth), possibly (1.26 versus 0.33), and might (0.31 versus 0.08).The no-revisit model broadens the hedging vocabulary, whereas full-reward GRPO concentrates hedging on frequent markers.

D.2 LLM-based Reasoning Evaluation

The reasoning evaluation compares model variants using GPT-4o judgments of generated thinking traces across multiple dimensions. SFT leads in clinical plausibility and faithfulness, while GRPO changes behavior through reward shaping and increases revisit triggering.

  • Model comparison: Stage 2 SFT achieves the highest clinical plausibility at 2.26 and faithfulness at 2.38 among the compared variants.The corresponding GRPO and no-revisit scores are 2.02 and 1.92 for clinical plausibility, and 2.26 and 2.20 for faithfulness.
  • Model comparison: The three variants differ mainly in which reasoning dimension they favor rather than in overall quality.The table is intended as a per-dimension diagnostic rather than a global ranking.
  • Reasoning-trace construction: The Stage 2 prompt uses hedge rewrites, object tokens, and detect calls to anchor uncertain regions in reasoning traces.The prompt includes a self-correction fallback and a closed vocabulary of hedge terms.
  • Reward-shaped behavior: Stages 2 and 3 use an identical prompt, while reward shaping raises the detect() trigger rate from 25.5% to 74.5%.The comparison attributes this behavioral drift to reward shaping rather than prompt engineering.

E Case Study

The case studies show both visually grounded reasoning on a subtle finding and a failure on severe, multifinding pathology. The successful trace aligns regional grounding with the report, while the failure exposes over-commitment during global diagnosis.

  • Overview: Figure 8 contains two MIMIC-CXR case studies illustrating one successful prediction and one challenging failure scenario.The cases are presented to examine model behavior qualitatively.
  • Successful case: In the successful case, the model matches the report’s conclusion about low lung volumes exaggerating contours while heart size remains top normal.It separately detects the right and left lungs, grounds them with bounding boxes, and evaluates the cardiac silhouette in context.
  • Failure case: In the failure case, the model correctly grounds some regions but reports heart failure with pulmonary edema, cardiomegaly, and a moderate right pleural effusion that the radiologist does not confirm.The underlying report describes severe multifinding pathology including fibrosis, interstitial opacities, and pulmonary edema.
  • Failure case: The error originates when regional findings transition to the global differential diagnosis, where the model over-commits to a familiar disease pattern.The trace makes this failure location explicit despite the incorrect final output.
Loading 2608.22217v1…