PaperlayerView exact revision on arXiv

Source-linked AI summary

Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approach

Haolin Li, Shuyang Jiang, Ruipeng Zhang, Jiangchao Yao, Ya Zhang, Yanfeng Wang

arXiv:2604.11547v1cs.LGcs.CL

TL;DR

Medical reasoning development is limited by scarce high-quality data, particularly for rare diseases, while existing trace-distillation approaches are costly and yield limited gains. MedSSR synthesizes knowledge-enhanced, distribution-controllable questions, uses policy-generated pseudo-labels, and trains through self-supervised then supervised RL. Across Qwen and Llama experiments, it outperforms baselines, with gains up to 5.93% on rare-disease tasks.

  • Problem

    Medical reasoning lacks high-quality reasoning data, especially for rare diseases, while synthetic medical data risks low fidelity and factual errors.

  • Method

    MedSSR combines medical knowledge-enhanced, distribution-controllable question synthesis with policy-generated pseudo-labels and intrinsic-to-extrinsic semi-supervised RL.

  • Results

    MedSSR outperforms baselines and existing medical LLMs across Qwen and Llama evaluations, with gains up to 5.93% on rare-disease tasks and 3.91% in general.

  • Takeaways & Limitations

    MedSSR provides an efficient pathway for scaling medical reasoning without relying on costly long-trace distillation.

  • Takeaways & Limitations

    Experiments did not include larger LLMs because of computational constraints, and full-scale expert validation of synthetic data remains desirable.

Abstract

from arXiv · show

While large language models hold promise for complex medical applications, their development is hindered by the scarcity of high-quality reasoning data. To address this issue, existing approaches typically distill chain-of-thought reasoning traces from large proprietary models via supervised fine-tuning, then conduct reinforcement learning (RL). These methods exhibit limited improvement on underrepresented domains like rare diseases while incurring substantial costs from generating complex reasoning chains. To efficiently enhance medical reasoning, we propose MedSSR, a Medical Knowledge-enhanced data Synthesis and Semi-supervised Reinforcement learning framework. Our framework first employs rare disease knowledge to synthesize distribution-controllable reasoning questions. We then utilize the policy model itself to generate high-quality pseudo-labels. This enables a two-stage, intrinsic-to-extrinsic training paradigm: self-supervised RL on the pseudo-labeled synthetic data, followed by supervised RL on the human-annotated real data. MedSSR scales model training efficiently without relying on costly trace distillation. Extensive experiments on Qwen and Llama demonstrate that our method outperforms existing methods across ten medical benchmarks, achieving up to +5.93% gain on rare-disease tasks. Our code is available at https://github.com/tdlhl/MedSSR.

1 Introduction

Medical reasoning research is constrained by scarce, especially rare-disease, reasoning data, while existing trace-distillation methods are costly and yield limited rare-disease gains. MedSSR addresses these gaps with knowledge-controlled synthesis and semi-supervised RL.

  • Only 22% of existing medical-dataset questions are reasoning-intensive, and just 3% of those concern rare diseases.
  • Existing methods distill long chain-of-thought traces from proprietary models, then use supervised fine-tuning followed by reinforcement learning.
  • Rare-disease gains remain below a 3% improvement ceiling even with fully supervised GRPO, while API-based trace generation is expensive.
  • Synthetic medical data requires structured knowledge because low fidelity and factual errors are especially unacceptable in healthcare.
  • MedSSR synthesizes complex reasoning questions from real-world seeds, optionally retrieves rare-disease knowledge, and controls the rare-disease proportion.
  • MedSSR combines self-supervised RL on pseudo-labeled synthetic data with supervised RL on human-annotated real data.
  • 5.93% average improvement on Qwen rare-disease tasks and 3.91% in general demonstrate gains over baselines and existing medical LLMs.

2 Related Work

Related medical-reasoning methods use distilled traces, knowledge graphs, process reward models, or synthetic data, but reported approaches provide limited rare-disease gains. MedSSR instead uses proprietary models for question generation rather than long reasoning traces.

  • HuatuoGPT-O1 distills reasoning traces from GPT-4o through supervised fine-tuning before applying a second-stage reinforcement-learning procedure.
  • MedReason incorporates knowledge graphs during chain-of-thought generation to improve factual accuracy.
  • MedS3 and MedPRM generate reasoning paths for process-reward-model training, then use those models to supervise the policy model.
  • Existing methods fail to provide considerable gains for rare diseases, whereas MedSSR uses proprietary models to generate questions instead of long reasoning traces.
  • Synthetic-data research has become a strategy for mitigating data scarcity in healthcare and other settings where real-world data collection is challenging.

3 Method

MedSSR combines distribution-controllable medical question synthesis with semi-supervised reinforcement learning. It uses pseudo-labeled synthetic data for self-supervised training before applying supervised RL to verified real data.

  • Preliminary and Training Objective: The optimization maximizes answer-verification rewards and regulates policy updates through a KL-divergence coefficient β.GRPO provides the advantage computation used in the two sequential training phases.
  • Data Synthesis: MedSSR first synthesizes questions with controllable rare-disease content using seed questions, medical knowledge, and a proprietary generation model.Rare-disease knowledge is retrieved and concatenated with seed questions to form prompts for synthetic-question generation.
  • Data Synthesis: The threshold α controls rare-disease knowledge injection, while ρ ≥ α leaves out additional disease-specific knowledge.The resulting synthetic dataset has a tunable focus on rare diseases.
  • Pseudo-label Generation: MedSSR uses the policy model to generate multiple responses, extracts their answers, and assigns pseudo-labels by majority voting.The resulting labeled synthetic dataset supplies rewards aligned with the policy model’s learning trajectory.
  • Pseudo-label Generation: Offline pseudo-labeling mitigates reward hacking by preventing the policy from collapsing to identical answers during online self-supervised training.The authors report that offline voting enables stable training for more than 1000 steps.
  • Semi-supervised Training: Training proceeds from self-supervised RL on pseudo-labeled synthetic data to supervised RL on human-annotated real data.The first phase exploits intrinsic policy capability, while the second uses ground-truth rewards to extend reasoning beyond that boundary.

4 Experiments

Experiments evaluate MedSSR across rare-disease and general medical benchmarks, varying rare-disease ratios, training strategies, labeling methods, and synthetic-data quality. MedSSR improves rare-disease and general performance, with controlled knowledge injection and offline pseudo-labeling supporting its training design.

  • Main Results: 5.93% average improvement over the supervised baseline is achieved on rare-disease tasks with Qwen, compared with 3.70% for Llama.MedSSR also outperforms existing medical and general LLM baselines.
  • Main Results: 3.91% further average improvement is obtained on general medical benchmarks for Qwen, while MedSSR surpasses the GRPO baseline across all nine datasets for both models.The rare-disease ratio is controlled at 25% for the reported main results.
  • Ablation Study: 25% rare-disease content gives the best overall trade-off, although general performance peaks at 13% and rare-disease performance peaks at 33%.Increasing the ratio to 50% reduces performance on both task groups.
  • Ablation Study: +2.29 average gain on rare-disease tasks outweighs a -0.21 average change on general tasks when comparing 25% with 13% rare-disease content.Knowledge injection improves three general datasets while slightly reducing performance on others.
  • Ablation Study: The intrinsic-to-extrinsic two-stage strategy outperforms one-stage and reversed two-stage training across all benchmarks.Stage-I self-supervised training matches or slightly surpasses supervised training using the same amount of real data.
  • Further Discussion: 0.50% of 200 physician-evaluated synthetic samples were marked incorrect and harmful by senior doctors, while average plausibility reached 4.80/5.0.Eight physicians, divided equally between senior and junior groups, independently evaluated the samples.
  • Further Discussion: Offline voting produces steadily increasing reward and performance curves resembling ground-truth supervision, unlike online voting, whose reward reaches 1.0 while performance drops.The study attributes this stability to decoupling label generation from the training loop.

5 Conclusion

MedSSR addresses medical reasoning data scarcity and limited rare-disease improvement through knowledge-enhanced synthesis and semi-supervised reinforcement learning. Evaluations across diverse benchmarks and architectures demonstrate its effectiveness.

  • MedSSR generates distribution-controllable medical questions using knowledge-enhanced data synthesis.
  • MedSSR first elicits intrinsic abilities on synthetic data, then enhances them with ground-truth supervision.
  • Evaluations across diverse benchmarks and architectures demonstrate MedSSR's effectiveness.

Limitations

The study reports two limitations: experiments did not extend beyond 14B-parameter models, and synthetic data lacked full human-expert validation. The authors expect the methodology may generalize to larger models, while noting that full-scale verification remains desirable for deployment.

  • Experiments did not include larger LLMs because of computational constraints, with testing reaching 14B parameters.The authors state that the framework enables efficient scaling and may generalize to larger models.
  • MedSSR synthetic data has not been fully validated by human experts.Eight physicians assessed a subset, providing preliminary evidence of clinical validity, but full-scale expert verification remains desirable for real-world deployment.

Ethical Consideration

The paper describes research-use safeguards and methodological choices relevant to responsible medical reasoning development. It uses public medical knowledge sources and avoids long proprietary reasoning traces while addressing reward hacking through offline voting.

  • MedSSR combines synthetic and real data in a semi-supervised RL framework and uses offline voting on generated data to avoid reward hacking.
  • The medical knowledge corpus draws on PubMed, Wikipedia, StatPearls, and textbooks.
  • Rare-disease selection uses a deduplicated list of 12,445 diseases from four certified databases.
  • GPT-4.1 is used in practice as the question generator.

B.4 Expert verification of synthetic data

The authors evaluated MedSSR synthetic question–answer data with physicians and expanded the assessment to 500 samples. Reported error and harmful rates remained low, while average plausibility reached 4.85/5.0.

  • The synthetic dataset contains generated questions paired with pseudo-labels assigned by the policy model itself.Human evaluation compares this data against gold standards.
  • Eight physicians assessed the clinical validity of 200 synthetic question–answer pairs labeled by Qwen3-8B-Base.
  • 0.80% error rate and 0.60% harmful rate were reported across the expanded physician evaluation.The expanded assessment covered 500 synthetic samples.
  • 4.85/5.0 was the average plausibility score in the expanded evaluation.Three physicians assessed 300 additional samples under the same criteria.

C Experimental Details

The experiments evaluate MedSSR across diverse medical benchmarks, model configurations, and implementation settings. Dataset construction includes rare-disease coverage and deduplication, while training and evaluation use specified frameworks and controls.

  • Dataset Details: The evaluation spans medical, biomedical, clinical, rare-disease, and expert-level benchmarks with separately specified test or training splits.The datasets include MedBullets, ReDis-QA, BioASQ, MedQA, MedMCQA, MedXpertqa, MMLU-Med, PubMedQA, HEAD-QA, and NEJM/Lancet.
  • Dataset Details: 2,122 samples form RareDis-Sub after removing 189 overlapping ReDis-QA samples and adding 951 rare-disease samples from nine general benchmarks.The construction aims to prevent potential data leakage while expanding rare-disease evaluation coverage.
  • Training Configuration: The experiments use detailed RL configurations, with Llama-specific learning-rate and KL-loss tuning because its RL training is prone to collapse.Offline label voting uses the same configurations as training, and knowledge retrieval uses top-k=4.
  • Variability Analysis: Table 9 reports standard deviations across all models and benchmarks, supplementing the main results with variability estimates.These deviations were omitted from the main results because of page limitations.
  • Evaluation Configuration: Evaluation uses vLLM with recommended decoding settings and a second decoding pass applying logit bias to avoid invalid outputs.Token counts are computed with the official tokenizers where applicable, tiktoken for MedSSR questions, and the public tokenizer for the SFT baseline.

D.1 Standard Deviation of Main Results

The standard-deviation analysis supplements the main results with variability estimates and reports lower variation for MedSSR than key Qwen reference systems. Figure 8 provides reward and performance curves with error bars.

  • Standard-Deviation Results: MedSSR exhibits consistently lower standard deviations than fully supervised baselines and the base model on Qwen.The comparison covers all models and benchmarks reported in the detailed breakdown.
  • Figure 8: Figure 8 reports rewards and performance curves for offline labeling beyond 1K steps, using average test-set performance with error bars.The figure is intended to show both training signals and performance variability over longer training.
  • Interpretation: The method is summarized as achieving higher average performance with superior training stability, supporting its suitability for safety-critical deployment.This conclusion follows the reported Qwen variability analysis.

D.2 Offline Voting Enables Long-term Stable Training

The analysis examines whether offline voting supports stable long-term self-supervised RL and how synthetic-data composition, training cost, and model scale affect performance. Results favor offline voting, a 25% rare-disease ratio, and MedSSR’s scalability across Qwen sizes.

  • Offline Voting Enables Long-term Stable Training: Offline voting avoids the reward hacking and early collapse associated with online self-supervised reward strategies.The resulting pseudo-labeled dataset supplies model-aligned intrinsic rewards for self-supervised RL.
  • Offline Voting Enables Long-term Stable Training: Beyond 1,000 steps, raw reward and running average increase steadily while remaining correlated with actual model performance, without sharp reward growth or collapse.The authors interpret this as evidence that offline voting can scale RL training, especially for larger models.
  • Rare-Disease Ratio: +2.29% improvement over the 13% natural-distribution ratio is achieved at the 25% rare-disease ratio, which best balances rare-disease and general-task performance.Performance initially rises with the rare-disease ratio, peaks, and then declines.
  • Comparison under Similar Cost: +3.91% gain over the 43K fully supervised baseline is delivered by MedSSR with 43K synthetic plus 43K real samples, versus +1.12% from doubling real data.The comparison uses an equivalent total data budget and is presented as evidence that the synthetic data is reasoning-targeted rather than merely additional volume.
  • Model Scaling: More than 10% average improvement on rare-disease tasks occurs at every tested Qwen scale, while general-task gains rise from +8.5% at 1.7B to +10.6% at 8B.The experiments cover Qwen3 models from 1.7B through 14B and report strong performance across scales.
  • Knowledge Injection: Rare-disease knowledge injection produces targeted, clinically grounded questions about Worster-Drought Syndrome with improved relevance and factual grounding.The comparison is made between synthesis with and without disease-specific knowledge injection.

E.2 Failure Mode Analysis

The failure analysis shows that majority voting can reproduce a coherent but incorrect base-model belief. The authors argue that such failures are rare and can be corrected by subsequent supervised training on real data.

  • Failure Mode Analysis: An 8/8 majority vote selected cervical radiculopathy because the base model confidently misinterpreted thenar atrophy and neck pain.The model incorrectly treated thenar atrophy as inconsistent with severe carpal tunnel syndrome.
  • Failure Mode Analysis: Systematic errors can propagate through majority voting, so pseudo-labels remain vulnerable when the base model shares the same incorrect knowledge.The example demonstrates that voting consistency does not guarantee factual correctness.
  • Failure Mode Analysis: The analyzed error was internally consistent rather than random guessing or logically contradictory reasoning.The failure therefore reflects a systematic knowledge error in the model’s medical content.
  • Mitigation: Physician evaluation reportedly finds these failures extremely rare, while supervised training on human-annotated real data can help correct errors introduced during self-supervised training.The two-stage curriculum supplies external supervision after pseudo-label-based RL.
  • Data Synthesis Constraints: The synthesis prompts require novel, clinically accurate, self-contained multi-step reasoning questions with plausible but uniquely correct options.The knowledge-enhanced variant additionally incorporates rare-disease knowledge when useful.
Loading 2604.11547v1…