PaperlayerView exact revision on arXiv

Source-linked AI summary

ChestX-Reasoner: Advancing Radiology Foundation Models with Reasoning through Step-by-Step Verification

Ziqing Fan, Cheng Liang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, Weidi Xie

arXiv:2504.20930v2cs.AIcs.CLcs.CV

TL;DR

Medical AI often lacks the structured reasoning processes used in clinical diagnosis. ChestX-Reasoner mines reasoning chains from clinical reports, trains with supervised fine-tuning and process-reward reinforcement learning, and outperforms medical and general-domain MLLMs in reasoning and outcome accuracy.

  • Problem

    Medical AI models often overlook the structured, step-by-step reasoning processes that radiologists document in clinical reports.

    Source: §1 · p. 1

  • Method

    ChestX-Reasoner mines reasoning chains from clinical reports and uses supervised fine-tuning followed by reinforcement learning with process rewards.

    Source: §1 · p. 3, §4 · p. 10

  • Results

    ChestX-Reasoner achieves state-of-the-art reasoning and outcome accuracy across all five Chest X-ray benchmark tasks against general and specialized medical models.

    Source: §4 · p. 11

  • Takeaways & Limitations

    Clinical reports can provide scalable, factual, and verifiable reasoning supervision for medical reasoning MLLMs.

    Source: §4 · pp. 10–11

    The study focuses on Chest X-rays and currently relies on reasoning-augmented samples and rule-based factuality scores for process supervision.

    Source: §4 · pp. 11–12

Abstract

from arXiv · show

Recent advances in reasoning-enhanced large language models (LLMs) and multimodal LLMs (MLLMs) have significantly improved performance in complex tasks, yet medical AI models often overlook the structured reasoning processes inherent in clinical practice. In this work, we present ChestX-Reasoner, a radiology diagnosis MLLM designed to leverage process supervision mined directly from clinical reports, reflecting the step-by-step reasoning followed by radiologists. We construct a large dataset by extracting and refining reasoning chains from routine radiology reports. Our two-stage training framework combines supervised fine-tuning and reinforcement learning guided by process rewards to better align model reasoning with clinical standards. We introduce RadRBench-CXR, a comprehensive benchmark featuring 59K visual question answering samples with 301K clinically validated reasoning steps, and propose RadRScore, a metric evaluating reasoning factuality, completeness, and effectiveness. ChestX-Reasoner outperforms existing medical and general-domain MLLMs in both diagnostic accuracy and reasoning ability, achieving 16%, 5.9%, and 18% improvements in reasoning ability compared to the best medical MLLM, the best general MLLM, and its base model, respectively, as well as 3.3%, 24%, and 27% improvements in outcome accuracy. All resources are open-sourced to facilitate further research in medical reasoning MLLMs.

1 Introduction

ChestX-Reasoner addresses the limited use of structured clinical reasoning in medical AI by mining reasoning chains from radiology reports and training with process supervision. It is evaluated with RadRBench-CXR and improves both reasoning ability and outcome accuracy over medical, general-domain, and base-model baselines.

Motivation

  • ChestX-Reasoner targets radiology diagnosis by leveraging step-by-step reasoning processes documented in clinical reports.The approach follows radiologists from low-level observations through patient-specific information to diagnostic impressions.Source: §1 · p. 1

Dataset construction

  • Clinical reports provide scalable reasoning supervision because findings and impressions can be converted into question-answer pairs and diagnostic reasoning chains.GPT-4o assists with generating questions, extracting observations, and refining reasoning narratives.Source: §1 · pp. 1, 3

Model development

  • The two-stage training framework combines supervised fine-tuning with reinforcement learning using process rewards for clinical-observation correctness.SFT uses answer-only and reasoning-augmented data, while RL evaluates generated observations against report sentences.Source: §1 · p. 3

Evaluation resources

  • RadRBench-CXR contains 59K visual question answering samples and 301K clinically validated reasoning steps, while RadRScore evaluates factuality, completeness, and effectiveness.The benchmark is built from MIMIC-CXR, CheXpert, and MS-CXR-T and includes radiologist review of mined reasoning.Source: §1 · p. 3

Results

  • ChestX-Reasoner improves reasoning ability by 16%, 8.5%, and 18% over the best medical baseline, best general baseline, and Qwen2VL-7B, respectively, while outcome accuracy improves by 3.3%, 24%, and 27%.These comparisons use RadRBench-CXR and CheXbench against medical and general-domain foundation models.Source: Figure 1 · p. 2, §1 · p. 3

2 Benchmarks

The benchmark combines report-derived reasoning chains with diverse Chest X-ray VQA tasks and evaluates reasoning through RadRScore. Its construction uses automated mining, explicit quality controls, and factuality-focused validation.

Task coverage

  • The benchmark spans binary, single-disease, multiple-disease, anomaly-detection, and temporal-comparison tasks across reports from MIMIC-CXR, CheXpert, and MS-CXR-T.RSNA and SIIM are additionally used for evaluation without associated clinical reports.Source: §2 · p. 3

Reasoning mining

  • Reasoning chains are mined by generating diagnostic plans, extracting report evidence, and refining the resulting steps into coherent logical sequences.GPT-4o automates plan construction, answer extraction, and narrative refinement.Source: §2 · p. 4

RadRScore

  • RadRScore evaluates factuality, completeness, and effectiveness using model, ground-truth, and report-derived observation sets.Factuality checks report-supported model observations, completeness measures coverage of ground-truth observations, and effectiveness measures meaningful ground-truth reasoning steps.Source: §2 · p. 4

Benchmark construction

  • RadRBench-CXR is constructed as a visual-language reasoning benchmark from generated VQA samples, mined reasoning chains, and reasoning-quality metrics.The benchmark integrates the data and evaluation components into one resource.Source: §2 · pp. 3–4

Quality control

  • The benchmark includes 1.2M answer-only training samples, 7.3K answer-only test samples, 59K reasoning-augmented training samples, and 1K reasoning-augmented test samples.Reasoning samples with factuality Rf below 1 are filtered out; before filtering, the method reached Rf=0.82 versus 0.60 for GPT-4o and 0.59 for Qwen2VL-72B.Source: §2 · pp. 5–6

3 Results: Reasoning Enhanced Medical VLM (ChestX-Reasoner)

ChestX-Reasoner outperforms medical and general MLLMs in reasoning ability and outcome accuracy across five chest X-ray tasks, while ablations show that combining SFT, RL, and process supervision is important.

Reasoning Ability

  • 18%: ChestX-Reasoner improves average RadRScore over its Qwen2VL-7B base model, reaching 0.531 versus 0.349 across five tasks.GPT-4o and Qwen2VL-72B reach 0.472 and 0.407, while MedDr-40B reaches 0.367.Source: §3.1 · p. 7

Outcome Accuracy

  • 3%: ChestX-Reasoner improves overall outcome accuracy over CheXagent-3B across five tasks, despite using 1.2M rather than 8.5M training samples.It also improves 27% over Qwen2VL-7B and 24% over GPT-4o.Source: §3.2 · p. 9
  • 17%: ChestX-Reasoner raises anomaly-detection RaTEscore to 0.621 from CheXagent-3B’s 0.529 and exceeds all listed general-domain baselines.For disease diagnosis and temporal comparison, its reported accuracy is comparable to CheXagent-3B and higher than the cited general and other medical baselines.Source: §3.2 · p. 9

Ablation on Training Strategy

  • SFT and RL are both necessary: outcome-only RL reaches average RadRScore 0.343 and accuracy 0.471, while reasoning-only SFT reaches 0.423 and 0.566.The paper attributes these limitations to missing reasoning knowledge under RL alone and exact-ground-truth reliance under SFT alone.Source: §3.3 · p. 10
  • 5%: Adding answer supervision during SFT improves reasoning, while process-reward removal still improves reasoning 14.5% and outcome accuracy 9.5% over the SFT(RO)+RL(O) configuration.These comparisons support answer supervision for domain alignment before later reasoning optimization.Source: §3.3 · p. 10
  • 2.5%: Process supervision increases average RadRScore from 0.512 without process reward to 0.525 in the final model, with a corresponding improvement in outcome accuracy.The result links process supervision to stronger reasoning performance and diagnostic accuracy within this ablation.Source: §3.3 · p. 10

4 Discussion

ChestX-Reasoner develops process-supervised radiology reasoning by mining clinical reports, introducing a benchmark and metric for evaluating reasoning quality. The study finds that report-derived supervision and process rewards improve reasoning and outcome performance while highlighting limitations in modality coverage and supervision design.

Contributions

  • ChestX-Reasoner combines clinical-report reasoning extraction with stepwise process supervision to mirror structured radiological diagnosis.The approach uses clinical reports as a scalable source of clinically grounded reasoning chains and introduces a process-supervised radiology foundation model.Source: §4 · pp. 10–11
  • RadRBench-CXR evaluates diagnostic outcomes and reasoning quality, while RadRScore measures reasoning factuality, completeness, and effectiveness.The benchmark is derived from real clinical reports and extends evaluation beyond outcome accuracy.Source: §4 · p. 11

Key findings

  • RadRScore factuality reached 0.82 for report-mined reasoning, versus 0.60 for GPT-4o and 0.59 for QwenVL-72B.Radiologist evaluations also reported average factuality, validity, and completeness scores of 7.83, 7.25, and 7.97, respectively, on a -10 to 10 scale.Source: §4 · p. 11
  • Direct reinforcement learning on general-domain base models performed poorly and unstably, so supervised fine-tuning is needed as a medical cold start.Without medical initialization, models were prone to convergence failures and could not reliably develop clinical reasoning skills.Source: §4 · p. 11
  • Process rewards improved reasoning quality and outcome accuracy over outcome-only reinforcement learning, supporting their importance for medical reasoning MLLMs.The study attributes the improvement to explicit supervision of intermediate clinical observations rather than relying only on final-answer rewards.Source: §4 · p. 11

Limitations

  • The study is limited to Chest X-rays and relies on reasoning-augmented samples plus rule-based factuality scores, leaving broader modalities and stronger supervision methods for future work.The authors identify CT, MRI, ultrasound, larger models, medical pretraining, and medical-specific reward models as directions for extension.Source: §4 · p. 12

5 Methods

ChestX-Reasoner combines supervised fine-tuning on answer-only and reasoning-augmented data with reinforcement learning that adds process rewards. This design supports both direct answers and step-by-step reasoning while addressing weaknesses of training on either data type alone.

Training pipeline

  • ChestX-Reasoner uses supervised fine-tuning followed by GRPO-based reinforcement learning with additional process rewards.The pipeline begins with answer-only and reasoning-augmented data, then evaluates output format, answer accuracy, and reasoning factuality.Source: §5 · p. 12, §5.2 · p. 14

Data formulation

  • Reasoning-augmented samples contain clinical reports and mined reasoning sentences, while answer-only samples lack ground-truth reasoning and reports.The dataset is divided into DR and DA according to whether process supervision is available.Source: §5.1 · p. 13

Inference modes

  • Two instruction formats enable reasoning mode with CoT prompts or direct-response mode without intermediate reasoning, reducing inference time when reasoning is unnecessary.Both formats are paired with expected outputs during training.Source: §5.1 · p. 13

Training pipeline

  • Training exclusively on answer-only data overfits answer patterns, whereas reasoning-only tuning underuses labels and can prevent direct answer generation.The authors therefore combine both data types during supervised fine-tuning.Source: §5.2 · p. 13

Reinforcement learning

  • The reinforcement-learning reward combines output-format and outcome-accuracy signals for all samples, adding process-factuality reward for reasoning-augmented samples.Process factuality measures the correctness of generated clinical observations against report sentences.Source: §5.2 · p. 14

Code availability

The authors release the experimental code, including tools for creating reasoning processes from reports and training ChestX-Reasoner.

Code availability

  • The released code covers reasoning-process creation from reports and ChestX-Reasoner training, while model weights and variants are distributed separately.The model weights are released in ChestX-Reasoner-Model, with additional variants planned for HuggingFace.Source: Code availability · p. 14

6 Supplementary

The supplementary materials provide training configurations, benchmark examples, instruction formats, and qualitative comparisons across diagnostic reasoning tasks.

Training details

  • Supplementary Tables 1 and 2 document training hyperparameters for the first and second training stages.The first-stage setup uses AdamW, cosine scheduling, a 2e-6 learning rate, batch size 512, and 8 Tesla A100 GPUs.Source: Table 1 · p. 18, Table 2 · p. 18, §6.1 · p. 18
  • The experiments use Qwen2VL-7B as the base model because of its reasoning performance in mathematics, programming, and general vision-language domains.The model is used as the initialization for ChestX-Reasoner.Source: §6.1 · p. 18

Qualitative cases

  • Supplementary Figures 1–4 present cases covering binary diagnosis, multiple-disease diagnosis, anomaly detection, and temporal comparison analysis.The cases compare outputs from ChestX-Reasoner, GPT-4o, and Qwen2VL-72B alongside questions, answers, and extracted reasoning processes.Source: Figure 1 · p. 19, Figure 2 · p. 20, Figure 3 · p. 21

Benchmark materials

  • Supplementary Table 3 presents mined reasoning processes and their corresponding instructions for RadRBench-CXR cases.The benchmark examples are organized by task type, input instruction, and extracted reasoning process.Source: Table 3 · p. 23, §6.3 · p. 24

Instruction formats

  • The supplementary materials show two instruction formats: one requesting only an answer and another requesting reasoning inside <think> tags plus an answer inside <answer> tags.These formats support direct-response and reasoning modes in RadRBench-CXR.Source: §6.2 · p. 24

Related papers

Loading 2504.20930v2…