Source-linked AI summary
CARE: Causally-Aligned Reasoning Exploration for Medical Large Language Models
Yucheng Zhou, Peng Luo, Qianning Wang, Chengzhong Xu, Jianbing Shen
TL;DR
Medical reinforcement learning can reward correct answers reached through spurious shortcuts and suffer unstable credit assignment, while expert annotations remain costly. CARE curates training experiences using causal sufficiency and proximal learnability, and consistently outperforms strong baselines across diverse medical multimodal and text-only benchmarks while improving reasoning consistency.
Problem
Outcome-based medical reinforcement learning lacks reliable reasoning supervision, enabling spurious shortcuts and unstable optimization while expert-annotated data remain costly to scale.
Method
CARE curates experiences through agreement-based causal verification and dynamic entropy-bounded proximal learnability, then optimizes them with dual-stream exploration and replay.
Results
CARE consistently outperforms strong comparable baselines across diverse medical multimodal and text-only benchmarks while reducing correct-but-inconsistent reasoning.
Takeaways & Limitations
Intrinsic curation governed by causal sufficiency and proximal learnability improves reasoning consistency and training stability within the evaluated medical benchmarks.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have shown strong potential for medical reasoning, yet the scarcity and cost of expert-annotated data constrain their progress. While reinforcement learning offers a scalable alternative, standard outcome-based methods in medicine often suffer from autoregressive credit assignment failure and gradient variance explosion. This leads to the "Right Answer, Wrong Reason" trap, where models inadvertently reinforce spurious correlations and dataset shortcuts rather than valid clinical deduction. In this work, we propose Causally-Aligned Reasoning Exploration (CARE), a theoretically grounded framework for intrinsic experience curation. CARE is built upon two rigorous conditions for high-quality training trajectories: Causal Sufficiency, which utilizes an agreement-based self-verification mechanism to mimic $do$-calculus interventions and effectively debias gradients; and Proximal Learnability, which employs dynamic entropy bounds to select experiences within the model's zone of proximal development for variance-bounded optimization. These rigorously filtered experiences are optimized via a dual-stream objective that combines on-policy group-relative exploration with difficulty-weighted experience replay. Extensive experiments on diverse medical multimodal and text-only benchmarks demonstrate that CARE consistently outperforms other strong competitors, substantially reducing correct-but-inconsistent reasoning and improving training stability.
1 Introduction
CARE addresses failures of outcome-based reinforcement learning in medical reasoning by curating trajectories that satisfy Causal Sufficiency and Proximal Learnability. Across multimodal and text-only benchmarks, CARE outperforms similarly scaled models while reducing correct-but-inconsistent reasoning.
- Evaluation: CARE consistently outperforms other similarly scaled models across diverse multimodal and text-only medical benchmarks while reducing correct-but-inconsistent reasoning.The evaluation covers clinical VQA and complex diagnostic reasoning, and the analysis links reduced inconsistency to mitigating shortcut learning.
- Motivation: Outcome-based RL in medicine suffers from autoregressive credit-assignment failure and exploding gradient variance, reinforcing spurious correlations in the “Right Answer, Wrong Reason” trap.Medical reasoning lacks inexpensive, reliable automated verifiers, so outcome supervision can misalign rewards with the reasoning process; heterogeneous data further destabilize learning signals.
- Method: CARE reframes self-improvement as intrinsic experience curation governed by Causal Sufficiency, using Agreement-Based Self-Verification as a gradient-debiasing mechanism against shortcut learning.An experience is admitted only when the model can reproduce the final decision using solely its generated rationale.
- Method: Proximal Learnability filters trajectories with dynamic entropy bounds, providing variance-bounded optimization over effective experiences within the model’s zone of proximal development.This mechanism addresses the instability caused by indiscriminately reinforcing correct rollouts across medical data with widely varying information density.
2 Related Work
Related work traces LLM foundations from policy-gradient and actor-critic reinforcement learning through PPO and Decision Transformers, while also noting early medical LLMs including HealthGPT, MMedLM, Team et al. (2025), and Hulu-Med.
- Reinforcement-learning foundations: Policy Gradient and Actor-Critic methods established foundational reinforcement-learning approaches that shaped later LLM development.
- Reinforcement-learning foundations: PPO provided a more stable alternative to TRPO, while Decision Transformers extended sequence modeling for LLM-related reinforcement-learning research.
- Medical LLMs: Early medical LLM work included HealthGPT, MMedLM, Team et al. (2025), and Hulu-Med (Jiang et al., 2025b).
3 Theoretical Analysis: Causal Misalignment and Optimization Stability
The analysis shows that outcome-based reinforcement learning can amplify spurious rationales through autoregressive credit-assignment failure. CARE addresses this by filtering causally insufficient trajectories and restricting learning to a variance-bounded, informative difficulty window.
- Theoretical Guarantees: Together, Causal Sufficiency debiases updates toward valid reasoning, while Proximal Learnability constrains optimization to a stable, low-variance region for efficient self-improvement.The framework addresses both spurious-correlation reinforcement and gradient instability in self-improving medical LLMs.
- Credit Assignment Failure: Under standard outcome supervision, a correct answer strictly increases the likelihood of any spurious shortcut rationale, reinforcing “Right Answer, Wrong Reason” behavior.Because the scalar correctness reward is broadcast across rationale and answer tokens, the optimizer cannot distinguish valid deduction from shortcut-driven correctness.
- Causal Sufficiency: Causal Sufficiency uses do(R = r) verification to eliminate gradient contributions from rationales whose answers are not recoverable without the input.For spurious rationales, masking x causes verification to fail and zeros their trajectory gradients, directing updates toward valid causal reasoning.
- Proximal Learnability: Proximal Learnability restricts training to a dynamic NLL window, ensuring bounded gradient variance while preserving a non-vanishing learning signal.The guarantees are Var(ĝ|y ∈W) ≤ C · τ^2_high and ||E[ĝ|y ∈W]|| ≥ ϵ > 0.
4 Methodology
CARE aligns policy optimization with valid clinical reasoning by replacing naive outcome supervision with intrinsic curation of reasoning trajectories. It admits only clinically valid, learnable, and causally sufficient experiences, then optimizes them through combined on-policy exploration and difficulty-weighted replay.
- Framework Setup: CARE shifts training from binary outcome supervision to intrinsic experience curation for multimodal clinical queries with structured rationale–answer outputs.The framework targets settings where training pairs provide diagnostic answers but lack expert-annotated rationales, and constructs curated rewards and replay data from multiple rollouts.
- Unified Admission: The unified admission function hierarchically filters trajectories by outcome validity, proximal learnability, and causal sufficiency.Outcome validity requires agreement with the ground-truth decision; learnability uses length-normalized NLL and a quantile-based dynamic window to avoid trivial shortcuts and high-variance hallucinations.
- Causal Sufficiency: Agreement-based self-verification tests whether the rationale independently reproduces the answer after masking the original query, enforcing causal sufficiency.Trajectories with disagreement between the verification answer and the generated answer are penalized, making the rationale a sufficient statistic for the decision and reducing dependence on dataset artifacts.
- Training Procedure: Training alternates rollout generation, dynamic learnability-window updates, causal and learnability filtering, and policy updates using the dual-stream objective LCARE = Lon + λLrep.The admission function supplies both the augmented reinforcement-learning signal and the replay-buffer gatekeeper.
- Optimization Objective: CARE combines on-policy GRPO, which standardizes admission scores within rollout groups, with replay of curated trajectories weighted toward the model’s capability frontier.The replay stream samples trajectories that pass curation and applies importance weights proportional to sequence likelihood, while a KL penalty limits deviation from the reference policy.
5 Experiments
CARE consistently outperforms strong medical baselines across multimodal and text-only benchmarks, while its causally aligned curation improves reasoning consistency, optimization stability, and robustness to spurious clinical distractors. Ablations and training analyses attribute these gains to causal self-verification and proximal learnability filtering.
- Medical Multimodal Benchmarks: CARE achieves the strongest overall performance among medical VLMs, outperforming prior domain-specific models across all evaluated multimodal benchmarks and surpassing its Hulu-Med-7B backbone.Applying CARE to HuatuoGPT-V also produces consistent performance boosts, supporting gains beyond backbone architecture.
- Medical Text Benchmarks: CARE consistently outperforms HuatuoGPT-V and Hulu-Med across all evaluated medical text benchmarks, including complex reasoning tasks such as MMLU-Med and MedQA.The reported gains indicate improvements in both general medical knowledge deduction and multimodal perception.
- Methodological Baselines and Ablation Study: +0.4% average gain from standard outcome-based GRPO over the Hulu-Med-7B SFT baseline is marginal, motivating CARE’s component-level comparison.Table 3 evaluates Standard GRPO and CARE components on PMC-VQA, MedQA, and MMMU-Med.
- Mitigating Shortcut Learning (Right Answer, Wrong Reason): Among cases where both methods answer MedQA correctly, CARE’s clinical consistency evaluation tests whether rationales provide valid causal reasoning rather than spurious shortcuts.The evaluation samples 500 queries and uses majority voting by three human experts.
- Optimization Stability via Proximal Learnability: CARE stabilizes training by rejecting variance-exploding, low-probability trajectories, producing monotonic improvement and a higher convergence ceiling than Standard GRPO.Standard GRPO shows sharp spikes and reward degradation, whereas CARE’s proximal learnability filter stabilizes gradient behavior and constrains updates to a stable region.
- Dynamics of Causal Self-Verification: 48.5% of correct trajectories are initially rejected by causal self-verification, declining to 15% as CARE reinforces causally aligned trajectories.This training dynamic evaluates whether the agreement-based mechanism is excessively restrictive.
6 Conclusion
CARE is introduced as a self-improvement framework that mitigates causal misalignment in outcome-based medical reinforcement learning through intrinsic experience curation.
- 6 Conclusion: CARE addresses causal misalignment inherent in outcome-based medical reinforcement learning through a self-improvement framework.The framework is designed to improve how medical reinforcement-learning trajectories guide optimization.
- 6 Conclusion: CARE replaces naive outcome monitoring with rigorous intrinsic experience curation governed by causal sufficiency and proximal learnability.These two principles define the framework’s criteria for selecting training experiences.
- 6 Conclusion: This curation directs policy updates toward valid clinical deduction rather than spurious dataset shortcuts.The framework’s intended benefit is causal alignment between the reasoning used for updates and clinically valid deduction.