X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning

• CXR interpretation has advanced from classification to open-ended explanations, but models still struggle to ground diagnostic criteria in relevant anatomy and often hallucinate measurements.Prior work includes vision-language models, reinforcement learning, and specialized segmentation and lesion-localization tools (Bai et al., 2025; Shao et al., 2024; Fan et al., 2025; Seibold et al., 2023).Source: §1 · pp. 1–2
• Current approaches struggle to connect broad visual cues with localized anatomy and quantitative metrics, faithfully integrate tool evidence, and screen measurements for acquisition applicability.These gaps can leave diagnoses fragmented or unsupported and allow inapplicable quantitative evidence into reasoning.Source: §1 · p. 2
• X-Ragent selectively routes task-specific evidence, integrates tool observations with visual findings, and exposes quantitative measurements only when acquisition conditions are satisfied.The policy learns which optional evidence to seek, while a separate validity gate controls measurement admission.Source: §1 · pp. 2–3
• X-Ragent raises ChestAgentBench accuracy by 8.6 points over the strongest baseline and improves across all seven CXR interpretation capabilities.The fully held-out benchmark is excluded from post-training.Source: §1 · p. 3
• Rad-CoT supplies 7.7K structured diagnostic trajectories for agentic SFT, followed by GRPO with an accuracy-gated reward for selective tool use and faithful measurement interpretation.The reward applies process-level terms only to correct trajectories.Source: §1 · p. 3

2.1 PROBLEM FORMULATION & WORKFLOW DESIGN

• The policy selects optional visual-refinement or knowledge-lookup calls, while a separate validity gate controls quantitative measurements during two-turn diagnosis.The routed observations are integrated with the image and query to produce reasoning and a final answer.Source: §2.1 · p. 4, Figure 3 · p. 4

2.2 TRAINING DATA CONSTRUCTION

• Rad-CoT contains 7,715 samples: 3,715 routed two-turn examples and 4,000 report-style diagnostic reasoning examples.Routed trajectories reveal observations only after the corresponding action and filter malformed, leaky, or unverifiable generations.Source: §2.2 · pp. 4–5

2.3 AGENTIC SUPERVISED FINE-TUNING

• Agentic SFT provides a cold start before GRPO, training structured reasoning and tool-use behaviors while masking injected observations from the loss.Prompt-elicited reasoning and direct GRPO from the base policy underperform the supervised initialization.Source: §2.3 · p. 5

2.4 AGENTIC REINFORCEMENT LEARNING

• GRPO combines answer accuracy, format compliance, radiographic evidence consistency, and auxiliary-tool cost, with process terms restricted to correct trajectories.Measurement consistency applies only to applicable, query-relevant measurements; visual-refinement and knowledge-lookup tools receive no explicit process supervision.Source: §2.4 · pp. 5–6

3.1 MAIN RESULTS

• X-Ragent achieves state-of-the-art results: 71.7% on ChestAgentBench, 77.3% average across CheXbench subsets, and 93.0% on SLAKE.Source: §3.1 · p. 6, Table 1 · p. 7
• ChestAgentBench is fully held out, and X-Ragent beats the strongest baseline across all seven capabilities, with 14.6–18.3-point gains over its base model.Source: §3 · p. 6, §3.1 · p. 7

3.2 ABLATION STUDY

• SFT supplies the largest single gain (+10.8), while RL adds more on held-out ChestAgentBench (+3.0) than CheXbench (+1.2).Source: §3.2 · p. 7, Table 2 · p. 7
• Tool access adds 4.3 points on ChestAgentBench and 4.1 on CheXbench, but 0.9 on near-saturated SLAKE; gains concentrate on fine-grained reasoning tasks.Source: §3.2 · pp. 7–8
• Gating evidence-consistency and tool-cost terms by answer correctness gives the best results on all three benchmarks; the ungated combination underperforms either term alone.Source: Table 3 · p. 7, §3.2 · p. 8
• Tool invocation rates are higher on base-model errors: 64.9% versus 43.6% on ChestAgentBench and 61.5% versus 45.9% on CheXbench.Source: §3.2 · p. 8, Table 4 · p. 8
• Removing task components has distinct effects: tool trajectories matter across benchmarks, report-style CoT most on held-out ChestAgentBench, and VQA mainly on SLAKE.Removing tool-trajectory supervision lowers accuracy by 4.5, 6.1, and 7.0 points on ChestAgentBench, CheXbench, and SLAKE; removing report-style CoT costs 5.5 points on held-out ChestAgentBench, while removing VQA costs 10.5 points on SLAKE.Source: §3.2 · p. 8, Table 5 · p. 8
• Validity gating raises non-frontal or atypical accuracy from 69.17% to 70.15% (+0.98 points), leaves frontal accuracy unchanged at 72.18%, and yields all six net corrections on validity-sensitive inputs.In the case study, exposing an inapplicable asymmetry measurement flips the correct answer to an incorrect one; withholding it preserves the visually supported answer.Source: §3.2 · p. 8, Table 6 · p. 8, Figure 4 · p. 9

4 RELATED WORK

• Multimodal-CoT [Zhang et al., 2024], LLaVA-CoT [Xu et al., 2025], and VLM-R1 [Shen et al., 2025] structure multimodal reasoning, but use fixed inputs rather than acquiring task-specific evidence.Source: §4 · p. 9
• General agents such as ReAct [Yao et al., 2023], Toolformer [Schick et al., 2023], and HuggingGPT [Shen et al., 2023] coordinate tools, but do not address clinical validity or medical-imaging evidence integration.Source: §4 · p. 9
• Unlike MMedAgent [Li et al., 2024a] and MedRAX [Fallahpour et al., 2025], X-Ragent learns a two-turn CXR interaction with a reward considering correctness, tool cost, and measurement consistency.Source: §4 · p. 9

5 CONCLUSION

• The framework combines learned evidence seeking with validity-gated quantitative augmentation for evidence-grounded CXR reasoning.Agentic SFT and GRPO train the policy to acquire auxiliary evidence selectively and admit quantitative measurements only when clinically valid.Source: §5 · pp. 9–10
• X-Ragent achieves state-of-the-art performance across three CXR benchmarks, including an 8.6-point gain on the fully held-out ChestAgentBench.The paper also reports consistent improvements across all seven interpretation capabilities.Source: §5 · p. 10

AI USE STATEMENT

• During manuscript preparation, LLMs were used for translation, language polishing, and pre-submission feedback.Source: AI USE STATEMENT · p. 10
• Locally deployed Qwen3.5-122B-A10B-FP8 generated synthetic Rad-CoT trajectories, which the authors filtered and verified.Source: AI USE STATEMENT · p. 10
• A locally deployed Qwen3.5-27B served as the evidence_lookup knowledge source, while LLMs did not independently determine scientific hypotheses, methods, or conclusions.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• The study uses only de-identified retrospective data and collects no new data.Datasets are used under their applicable agreements and licenses.Source: ETHICS STATEMENT · p. 10
• X-Ragent is a research prototype, not a medical device, and should not be used for clinical decision-making.Source: ETHICS STATEMENT · p. 10

A APPENDIX

• The appendix provides additional experimental and tool-suite details, extended related work, and prompts used in the analysis.Source: A APPENDIX · p. 17

B.1 MORE EXPERIMENTAL DETAILS

• For ChestAgentBench, overall performance is question-level accuracy across 2,500 questions, while its seven capability subsets overlap.CheXbench overall accuracy is an unweighted macro average across OpenI, Rad-ReStruct, and SLAKE; standalone SLAKE uses exact-match accuracy on 114 questions.Source: §B.1 · p. 17
• SFT initializes from Qwen3-VL-8B-Instruct for one epoch at a 1 × 10−5 learning rate and effective batch size 16, with the vision encoder frozen.The language policy learns evidence routing and evidence-integrated reasoning.Source: §B.1 · p. 17
• The figure distinguishes report-style single-turn examples from two-turn routed SFT examples, and marks GRPO examples reused from routed SFT versus multiple-choice questions unseen during SFT.Source: Figure 5 · p. 18
• Any well-formed correct trajectory receives higher reward than any incorrect one under Eq. 7, so tool cost cannot override diagnostic correctness.Source: §B.1 · p. 18

B.2 RAD-COT CONSTRUCTION AND QUALITY CONTROL

• Rad-CoT contains 7,715 SFT instances—3,715 routed and 4,000 report-style—and 4,715 GRPO examples, including 1,000 questions unseen during SFT.Report-style examples are used only for SFT because they lack rule-verifiable answers.Source: §B.2 · p. 18, Table 8 · p. 19
• Report-style rationales are generated from PA MIMIC-CXR-JPG studies, with reference findings retained as targets, then filtered from 6,000 candidates to 4,000 trajectories.Filtering removes leakage and malformed or excessively finding-similar reasoning; valid rationales must contain 40–900 words, with reasoning–finding similarity above 0.82 rejected.Source: §B.2 · p. 19
• MCQs are generated from MIMIC-CXR-JPG without report-derived cues, while ChestAgentBench uses disjoint Eurorad cases and evaluation-image and sibling-report overlap are checked.The SLAKE-overlapping images are also removed from the closed-ended VQA subset.Source: §B.2 · pp. 19–20
• The 3,715 routed examples reveal tool observations only after the first-turn action, and SFT masks those external observations from the language-modeling loss.This trains action selection and evidence use rather than imitation of tool outputs.Source: §B.2 · p. 19

B.3 EFFECT OF PROMPT-ELICITED REASONING

• Direct answering scored 55.1%, 65.5%, and 72.8% on ChestAgentBench, CheXbench, and SLAKE; the strongest reasoning prompts trailed by 4.0, 5.4, and 10.5 points.The comparison changed only the system prompt; all settings used the same checkpoint, inputs, decoding, and evaluation protocol.Source: Table 9 · p. 20, §B.3 · p. 20
• The authors use the prompt results, together with weaker RL-only performance than supervised initialization, to motivate SFT before GRPO refinement.Source: §B.3 · p. 20

B.4 REWARD DETAILS

• Measurement-consistency reward eligibility is the product of query relevance and backend validity, and each measurement contributes at most once.Query relevance affects reward eligibility, not whether the measurement executes.Source: §B.4 · p. 20
• Auxiliary-tool cost counts policy-generated calls, including repeated calls, but excludes backend validity checks and deterministic measurement routines.Source: §B.4 · p. 21

B.5 MORE ABLATION STUDIES

• Removing all modules drops accuracy by 4.3 points to 67.4%; auxiliary interpretation contributes 2.2 points, visual enhancement 1.6, and objective measurement 0.9.Forcing all policy-callable tools on every question remains 0.7 points below learned routing, which uses tools on 53.2% of questions.Source: §B.5 · p. 21
• Default validity-gated measurements raise accuracy from 71.0% to 71.7% on ChestAgentBench and from 76.1% to 77.3% on CheXbench versus policy-routed measurement.The policy-routed variant is close to removing measurement entirely; the default backend supplies measurements whenever acquisition conditions are satisfied.Source: §B.5 · p. 21

B.6 MORE CASE STUDIES

• The case study reaches the correct diagnosis by combining a focused ROI with retrieved criteria, while withholding an implausible CTR of 1.79.The absent measurement is treated as unavailable rather than normal.Source: §B.6 · pp. 21–22, Figure 6 · p. 22

C TOOL SUITE DETAILS

• Policy-routed auxiliary calls acquire visual or interpretive evidence, whereas an independent backend computes measurements subject to an acquisition-validity check.The policy-callable tools are roi_crop, region_parse, and evidence_lookup.Source: C TOOL SUITE DETAILS · p. 22, Table 12 · p. 23

C.1 VISUAL REFINEMENT TOOLS

• region_parse localizes a target structure or finding, and roi_crop magnifies its bounding-box region while retaining the original radiograph for joint reasoning.Parsing returns a segmentation mask and tight axis-aligned bounding box.Source: §C.1 · pp. 22–23

C.2 AUXILIARY INTERPRETATION VIA EVIDENCE LOOKUP

• evidence_lookup returns condition-level radiographic manifestations and discriminating features for the policy to reconcile with patient-specific visual findings.It supplies general knowledge rather than a patient-specific diagnosis.Source: §C.2 · p. 23

C.3 VALIDITY-GATED OBJECTIVE MEASUREMENTS

• Quantitative measurements are withheld when applicability checks fail, and absence denotes unavailable or inapplicable evidence rather than a normal finding.The backend computes CTR, lung-field asymmetry, and thoracic shift for applicable inputs.Source: §C.3 · pp. 23–24

C.4 TOOL EXECUTION AND OBSERVATION INJECTION

• During SFT, tool observations are inserted after their corresponding actions and masked from language-modeling loss, training tool selection and evidence integration rather than tool-output imitation.Source: §C.4 · p. 24