PaperlayerView exact revision on arXiv

Source-linked AI summary

MedAgent-R1: Faithfulness-Aware Reinforcement Learning for Evidence-Grounded Medical Reasoning

Jiangwang Chen, Chenghao Zhang, Hengxing Cai

arXiv:2608.30676v1cs.AI

TL;DR

Medical reasoning agents need justifications that clinicians can verify, yet outcome-only RL can improve accuracy while degrading evidence faithfulness. The paper introduces faithfulness-gated training with retrieval-aware auxiliary signals and reports lower fabrication, higher evidence coverage, preserved accuracy, and improved HealthBench Safety. The authors note that evaluation and deployment scope remain limited.

  • Problem

    Outcome-only RL can produce correct medical answers with fabricated, unsupported justifications, making clinical reasoning difficult to verify and potentially misleading.

  • Method

    MedAgent-R1 combines observation-masked SFT with a faithfulness-gated RL reward, supplemented by retrieval validity and conciseness signals.

  • Results

    MedAgent-R1 reduces fabrication to 4.7%, achieves 82.6 EC-F1 without sacrificing accuracy, and improves HealthBench Safety by 13.2 points.

  • Takeaways & Limitations

    Explicitly rewarding evidence grounding separates faithfulness from answer accuracy and improves trustworthy medical agent behavior under the evaluated retrieval protocol.

  • Takeaways & Limitations

    The experiments primarily use a 7B backbone, leave 70B+ behavior untested, and measure evidence grounding rather than causal faithfulness.

Abstract

from arXiv · show

When medical AI systems hallucinate clinical reasoning, the consequences extend beyond incorrect answers: fabricated justifications that superficially reference retrieved evidence can mislead clinicians into unsafe treatment decisions. Medical reasoning agents must therefore produce not only correct answers but also faithful justifications that clinicians can verify against cited evidence. We identify a systematic failure mode in RL-trained retrieval agents: outcome-only rewards improve accuracy while degrading faithfulness, a phenomenon we term confident hallucination. The agent learns to answer from parametric memory and backfill plausible but unsupported justifications; citation fabrication rates rise from 16.5% to 31.8% even as accuracy improves by 5 points over the supervised baseline. We address this with a faithfulness-gated reward design: accuracy credit is conditioned on evidence grounding via a hard gate, complemented by retrieval validity and conciseness signals that close exploitation paths unique to agentic retrieval. The resulting system, MedAgent-R1, reduces citation fabrication from 31.8% to 4.7% and raises evidence completeness from 58.7 to 82.6 while maintaining 75.1% accuracy, with 13.2-point gains on HealthBench Safety. Under the same agentic retrieval setup, MedAgent-R1 outscores GPT-4o on faithfulness-specific dimensions (Factual Support 4.55 vs. 4.25; Overclaiming 4.40 vs. 4.15) while remaining below GPT-4o in overall accuracy, suggesting that explicit faithfulness training yields evidence-grounding gains not achieved by scaling alone.

1 Introduction

Medical retrieval agents can improve answer accuracy while producing unfaithful, potentially misleading justifications. MedAgent-R1 addresses this failure with faithfulness-gated rewards and reports substantially better grounding without sacrificing accuracy.

  • 1 Introduction: Clinicians need reasoning that can be audited against cited evidence, because correct answers with fabricated justifications can mislead treatment decisions.A fabricated benefit estimate may encourage unsafe confidence despite a correct recommendation.
  • 1 Introduction: Outcome-only rewards give full credit to correct answers regardless of whether their reasoning faithfully reflects retrieved evidence.This creates an incentive to answer from parametric memory and backfill plausible citations.
  • 1 Introduction: Citation fabrication rose from 16.5% to 31.8% under outcome-only RL even as accuracy improved by 5 points over SFT, a failure termed confident hallucination.Nearly one-third of claims lacked citations or cited unsupported evidence.
  • 1 Introduction: MedAgent-R1 conditions accuracy credit on evidence grounding and adds validity and conciseness signals to block retrieval-specific exploitation paths.The framework also uses observation-masked SFT for faithful cold-start initialization.
  • 1 Introduction: 4.7% fabrication and 82.6 EC-F1 were achieved without sacrificing accuracy, while faithfulness-specific scores exceeded GPT-4o under the same retrieval setup.The benchmark evaluates whether claims are cited and entailed, as well as overlap with physician reference claims.

2 Related Work

Prior retrieval and medical RL systems improve evidence access, search behavior, or answer quality, but generally do not separate faithfulness from accuracy as an explicit training signal. This paper positions its contribution as addressing that unresolved grounding problem in agentic retrieval.

  • 2 Related Work: RL-trained retrieval systems extend from cold-start reasoning to open-domain search and iterative multi-step research, but reward final-answer quality.The cited systems include DeepSeek-R1, R1-Searcher, Search-R1, DeepResearcher, WebGPT, and GopherCite.
  • 2 Related Work: Medical RAG systems improve evidence supply through corpus benchmarking, query refinement, reflection tokens, or biomedical knowledge graphs without constraining evidence use.These approaches focus on what and when to retrieve rather than whether reasoning is grounded in cited sources.
  • 2 Related Work: Faithfulness evaluation work provides tools for factual precision and citation quality, while the paper identifies agentic retrieval as introducing an exploitation challenge absent from standard generation.The challenge arises because retrieval and generation can be jointly manipulated.
  • 2 Related Work: Medical preference, process-reward, and verifiable-reasoning methods improve answer quality or coherence but leave the accuracy–faithfulness trade-off unaddressed.The cited methods include UltraMedical, Med-PRM, and HuatuoGPT-o1.

3 Method

MedAgent-R1 trains a medical retrieval agent to produce correct answers and evidence-grounded explanations, addressing shortcuts that arise when outcome-only rewards ignore faithfulness. Its staged training and gated reward combine grounded initialization with signals for faithfulness, retrieval validity, and conciseness.

  • Task Formulation: Faithful reasoning requires every explanation claim to be entailed by evidence the agent itself retrieved, unlike standard question answering with externally fixed sources.The setting jointly targets answer correctness and verifiable grounding of the evidence summary.
  • Agent Architecture: The agent iteratively reasons, retrieves observations from clinical sources, and synthesizes cited evidence before producing an answer.Its trajectory contains reasoning steps, retrieval actions, observations, an evidence summary, and a final answer.
  • Cold Start via Faithful SFT: Observation-masked SFT filters teacher trajectories for correctness and NLI-verified faithfulness, while masking tool observations to prevent memorization of retrieved content.Training therefore begins with grounded reasoning and learned retrieval behavior rather than unfaithful patterns.
  • Faithfulness-Aware Reinforcement Learning: The total reward gates binary accuracy on positive faithfulness, then adds faithfulness, retrieval-validity, and conciseness signals to block unfaithful and manipulable trajectories.Correct-but-unfaithful trajectories receive no accuracy credit; validity penalizes irrelevant retrieval, while conciseness addresses over-retrieval gaming.
  • Optimization: Group Relative Policy Optimization uses the gated reward to increase within-group separation between faithful and unfaithful trajectories, strengthening the learning signal for grounded reasoning.The objective also regularizes against the SFT policy through a KL term.

4 Experiments

The experiments evaluate accuracy, reasoning faithfulness, and clinical safety using shared retrieval access, MedFaith-Eval, and HealthBench. Results show outcome-only RL can improve accuracy while degrading evidence grounding, whereas faithfulness-aware RL improves fabrication, completeness, safety, and selected faithfulness dimensions.

  • 4.1 Setup: The evaluation measures closed-form accuracy, reasoning faithfulness, and clinical safety across medical benchmarks, MedFaith-Eval, and HealthBench.MedFaith-Eval uses physician-annotated evidence claims; HealthBench evaluates a stratified 1,000-consultation subset.
  • 4.2 Main Results: 75.12% accuracy and 4.38 HRS were highest among same-scale models with shared retrieval access.GPT-4o achieved 84.32% accuracy and 4.52 HRS under the same agentic loop, while MedAgent-R1 scored higher on Factual Support and Overclaiming.
  • 4.2 Main Results: Accuracy rose from 69.6 to 74.6, while fabrication increased from 16.5% to 31.8% and evidence completeness fell from 65.1 to 58.7 under outcome-only RL.This pattern identifies confident hallucination: correct answers are produced while evidence-based reasoning is abandoned.
  • 4.2 Main Results: 4.7% fabrication and 82.6 EC-F1 show that faithfulness-aware RL reverses the outcome-only degradation, with the lowest fabrication across models.The full model surpassed GPT-4o in the same agentic loop on EC-F1 and fabrication despite GPT-4o’s scale advantage.
  • 4.2 Main Results: +13.2 HealthBench Safety points were obtained over outcome-only RL, whereas outcome-only RL improved Accuracy by nearly 10 points but barely changed Safety.MedAgent-R1 scored 64.5 Safety versus 61.5 for GPT-4o under this internal protocol.
  • 4.2 Main Results: Composite HRS rose from 3.42 to 3.65 despite outcome-only RL degrading Factual Support and Overclaiming.Faithfulness-aware RL repaired Factual Support from 2.82 to 4.55 and Overclaiming from 3.00 to 4.40.
  • 4.3 Analysis: How Confident Hallucination Manifests: Faithfulness declined monotonically while accuracy rose under outcome-only RL, but both rewards rose together and plateaued by approximately step 800 under faithfulness-aware RL.The results indicate progressive exploitation of parametric shortcuts during outcome-only training.
  • 4.4 Ablation Study: Removing the faithfulness gate raised fabrication to 12.8%, removing all faithfulness signals raised it to 28.5%, and removing observation masking reduced EC-F1 to 76.5.Validity prevented unfocused retrieval, while conciseness had minimal standalone effect.

5 Conclusion

The paper identifies confident hallucination as a failure of outcome-only RL and proposes faithfulness-gated rewards to constrain evidence grounding. MedAgent-R1 reduces fabrication while preserving accuracy and improving safety.

  • Outcome-only RL teaches retrieval agents to answer correctly while fabricating ungrounded justifications, with fabrication increasing from 16.5% to 31.8%.
  • A faithfulness-gated reward reduces fabrication to 4.7% without sacrificing accuracy and yields a 13.2-point gain on HealthBench Safety.
  • The conclusion argues that trustworthy agentic reasoning requires reward designs constraining reasoning quality, not only outcome quality.

Limitations

The study's evidence is strongest at smaller model scales and for measured evidence grounding, with broader clinical, causal, benchmark, and latency boundaries remaining.

  • The primary experiments use a 7B backbone, while a 14B replication confirms transfer; behavior at 70B+ remains untested.
  • NLI-based faithfulness may miss subtle unfaithfulness, and the method enforces evidence grounding rather than causal faithfulness.
  • MedFaith-Eval and the benchmarks do not establish cross-benchmark generalization or capture longitudinal reasoning and decisions under incomplete information.
  • The approximately 8-second query overhead may be prohibitive in latency-sensitive settings.

Ethics Statement

This research system is intended for medical question answering rather than direct clinical deployment, and it uses public benchmarks and de-identified vignettes instead of patient data.

  • The system is not intended for direct clinical deployment without regulatory validation.
  • No patient data were used; training and evaluation relied on publicly available medical QA benchmarks and de-identified clinical vignettes.
  • Physician annotators and human evaluators were compensated at standard consulting rates and consented to publication of their contributions.

Reproducibility

The paper documents shared retrieval infrastructure, datasets, training configurations, evaluation procedures, and released reproducibility artifacts. Experiments use controlled benchmark sampling and physician-annotated evidence references.

  • Retrieval Infrastructure: All models share dense MedCPT, entity-centric UMLS/DrugBank, and hybrid BM25-semantic PubMed retrieval modalities.
  • Training Data: Training uses MedQA, MedMCQA, and PubMedQA data with strict test separation, while MMLU-Medical is reserved for out-of-distribution evaluation.
  • Model and Training: The policy backbone is Qwen2.5-7B-Instruct; SFT uses 38K trajectories and GRPO runs for 1000 steps with group size G = 8.
  • Dual-Teacher Data Details: GPT-4o and Qwen2.5-72B-Instruct form a dual-teacher relay combining clinical planning with reliable tool-call formatting.
  • NLI Model: Faithfulness scoring uses a frozen DeBERTa-v3-large NLI model with threshold τ = 0.7 selected through annotated-pair grid search.
  • Evaluation: Evaluation samples 800 stratified questions, generates full responses with shared retrieval, and compares model claims against physician-annotated reference evidence.

B.1 Citation Faithfulness Evaluation

The evaluation separates citation faithfulness from claim completeness and composite reliability, showing that outcome-only RL can improve aggregate scores while worsening evidence grounding. MedAgent-R1 substantially reduces fabrication and improves faithfulness-specific evaluation under comparable retrieval conditions.

  • Metrics: EC-F1 measures overlap with physician reference claims, while Citation Faithfulness tests whether cited observations entail the claims they support.Together, the metrics address incomplete evidence coverage and irrelevant or insufficient citations.
  • Automated results: 31.8% fabrication under outcome-only RL is the highest reported rate, reflecting citation-format mimicry without reliable evidence grounding.Fabrication includes uncited claims or claims unsupported by their cited observations.
  • Automated results: 4.7% fabrication for MedAgent-R1 is consistent with its entailment-based faithfulness objective.The full model requires evidence entailment during training, unlike outcome-only RL.
  • Human verification: 91.3% human–GPT-5.5 agreement supports the automated entailment judge, although disagreements cluster around implicit medical inferences.Physicians accepted such inferences more often than GPT-5.5 in the validation sample.
  • Human verification: Human fabrication rates were 29.9% for outcome-only and 5.2% for MedAgent-R1, confirming a 24.7-point inter-model gap.The validation sample was stratified, so the authors interpret its direction and magnitude rather than as unbiased corpus-level estimates.
  • Reference-model comparison: MedAgent-R1 reaches higher Factual Support and Overclaiming scores than GPT-4o despite lower Medical Correctness, indicating faithfulness gains beyond scale alone.The comparison uses the same agentic loop for GPT-4o and MedAgent-R1.

D Full HealthBench Results

The HealthBench evaluation uses a controlled 1,000-consultation subset with shared retrieval and scoring conditions, while explicitly limiting comparison with the official leaderboard. Additional analyses examine reward-model robustness, retrieval behavior, specificity, paraphrasing, and efficiency.

  • Evaluation setup: 1,000 HealthBench consultations are evaluated in a stratified subset balanced by medical specialty and complexity tier.The underlying benchmark contains 5,000 multi-turn consultations scored by physician rubrics.
  • Comparability: Official HealthBench scores are not directly comparable because this evaluation changes retrieval access, scoring methodology, and benchmark coverage.The authors state that within-study comparisons remain valid under identical conditions.
  • HealthBench results: MedAgent-R1’s safety is comparable to or slightly above GPT-4o under the shared agentic retrieval protocol.The table caption identifies safety as the dimension where MedAgent-R1 approaches or exceeds GPT-4o.
  • Efficiency: Iterative reasoning-retrieval methods incur higher latency than batch retrieval because generation occurs sequentially between retrieval calls.The efficiency comparison reports tokens, retrieval calls, and latency under identical hardware conditions.
  • Robustness: Three NLI reward models are tested with fixed hyperparameters, and all variants are evaluated by the same GPT-5.5 judge.The robustness table reports that general-domain NLI provides substantial gains, with medical fine-tuning adding modest improvement.
  • Potential reward gaming: Query-answer cosine similarity is 0.41 for MedAgent-R1 versus 0.38 for SFT-only, indicating no systematic query manipulation.The small difference is interpreted as negligible tautological retrieval.
  • Potential reward gaming: MedAgent-R1 produces 2.3 specific medical entities per sentence versus 2.1 for outcome-only, while evidence-sentence ROUGE-L rises to 0.34 from 0.31 for SFT-only and 0.26 for outcome-only.Manual inspection found that 78% of sampled trajectories synthesized across multiple evidence passages.

H Evidence Ablation at Inference

Inference-time evidence ablation distinguishes models that genuinely condition on retrieved observations from models that preserve accuracy through parametric recall. The full model is strongly evidence-dependent, whereas outcome-only RL largely ignores retrieval content.

  • Ablation design: The ablation compares normal inference with irrelevant observations and with observations removed entirely.Replacement preserves observation count and format markers while changing the content; removal uses a no-information placeholder.
  • Results: The outcome-only model loses only 5.18 points when evidence is replaced with irrelevant content, indicating limited dependence on retrieved evidence.The table caption interprets this as evidence that the model largely ignores retrieval content.
  • Results: 16.62 points is the full model’s accuracy loss when relevant observations are replaced, from 75.12 to 58.50.Removing observations entirely lowers accuracy further to 54.63.
  • Interpretation: The ablation provides causal evidence that outcome-only RL uses a parametric shortcut, while faithfulness-aware training makes answers condition on retrieved evidence.The authors characterize the outcome-only model as retrieving evidence it does not use.
  • Reward sensitivity: The selected faithfulness reward weight is λ3=0.5, balancing accuracy and faithfulness; larger values produce diminishing EC-F1 gains and lower accuracy through hedging.At λ3=0, the system is equivalent to outcome-only RL.
  • Agent protocol: The agent’s prescribed pipeline iteratively retrieves observations, reasons from them, cites each factual claim, and answers only after evidence-based reasoning.The instructions require direct support for every claim and prohibit claims beyond retrieved observations.

L Human Evaluation Protocol

Human evaluation uses blinded, independently scored physician judgments across clinical accuracy, evidence usage, and safety, with substantial inter-annotator agreement. The results corroborate automated evidence-grounding metrics and show strong alignment between physician ratings and Factual Support.

  • Protocol: Three board-certified physicians independently evaluate blinded model outputs after calibration, using randomized presentation order.The evaluators include two internal medicine physicians and one emergency medicine physician.
  • Sampling: 200 MedFaith-Eval examples are stratified by difficulty, medical topic, and answer correctness for human assessment.Topic strata include pharmacology, pathophysiology, clinical management, and diagnostics.
  • Scoring: Physicians score clinical accuracy, evidence usage quality, and recommendation safety on five-point scales, while flagging factual, logical, citation, and safety errors.This protocol evaluates both output quality and whether cited evidence supports the claims.
  • Agreement: Inter-annotator agreement is substantial: α=0.78 for clinical accuracy, 0.72 for evidence usage, and 0.81 for safety.All reported agreement values exceed the stated α>0.67 threshold.
  • Human results: 4.32 versus 2.87 is the mean evidence-usage score for MedAgent-R1 and outcome-only RL, respectively, with p<0.01.The paired comparison applies Bonferroni correction.
  • Human results: Spearman ρ=0.84 links physician evidence-usage scores with automated Factual Support, supporting correspondence between the two evaluations.The correlation is significant at p<0.001.
  • Contamination checks: MedAgent-R1 scores 73.0% on 100 newly authored questions, compared with 75.12% benchmark performance, reducing concern that benchmark results reflect memorization.The new questions cover 2025 guideline updates and were authored after data collection cutoff.

N Scaling Analysis

Scaling analysis shows that outcome-only RL preserves the confident-hallucination failure at 14B, whereas faithfulness-aware RL remains effective and improves evidence grounding at larger scale.

  • N Scaling Analysis: 14B outcome-only RL degrades EC-F1 by 7.3 points, comparable to the 7B model’s 6.4-point drop despite higher accuracy.The 14B model reaches 77.82% accuracy, but its HRS remains comparable to the 7B outcome-only model.
  • N Scaling Analysis: Faithfulness-aware RL eliminates the outcome-only degradation at both 7B and 14B scales.The scaling result supports explicit evidence-grounding incentives rather than relying on increased model capacity alone.
  • N Scaling Analysis: 85.1 EC-F1 and 4.52 HRS are reached by the 14B full model, improving over the 7B full model’s 82.6 EC-F1 and 4.38 HRS.The full model also lowers fabrication from 4.7% at 7B to 4.3% at 14B.
Loading 2608.30676v1…