PaperlayerView exact revision on arXiv

Source-linked AI summary

HuatuoGPT-3: RL-Only Domain Adaptation from Base Models

Junying Chen, Xinyuan Xie, Ziniu Li, Wenyuan Gu, Jianquan Li, Xiang Wan, Guangjun Yu, Ruoyu Sun, Haizhou Li, Benyou Wang

arXiv:2610.05966v1cs.CLcs.AI

TL;DR

Domain adaptation seeks to make general-purpose language models expert in a target domain, but SFT+RL may restrict exploration and complicate optimization, while pure on-policy RL has a cold-start problem. OnePO uses teacher outputs as transient guidance, strengthening learning on informative tokens and retiring outputs as the policy improves. With 20K samples, it reaches 67.2 on HealthBench and outperforms SFT+RL under the same setting.

  • Problem

    SFT+RL may reduce exploration diversity and add multi-stage optimization complexity, while pure on-policy RL can be slow to acquire target-domain capability without external guidance.

    Source: §1 · p. 1

  • Method

    OnePO uses teacher outputs as transient guidance, strengthening learning on informative low-probability tokens and retiring outputs once their rewards no longer exceed the current on-policy frontier.

    Source: §1 · p. 2

  • Results

    With 20K samples, OnePO adapts Qwen3-8B-Base to 67.2 on HealthBench, outperforming SFT+RL under the same setting.

    Source: §6 · p. 12

  • Takeaways & Limitations

    The results support RL-only training as a practical and effective route for domain adaptation without a preceding SFT stage.

    Source: §6 · p. 12

    The authors state that the models are not intended for clinical deployment without rigorous validation by medical professionals.

    Source: Impact Statement · p. 13

Abstract

from arXiv · show

Domain adaptation aims to turn a general-purpose large language model (LLM) into an expert for a target domain. While the dominant SFT+RL pipeline offers a convenient cold start, it may reduce exploration diversity and introduces additional complexity through multi-stage optimization. These limitations motivate RL-only adaptation. However, pure on-policy RL suffers from a cold-start problem, while mixed-policy RL still falls short: informative tokens in teacher outputs are learned too slowly in early training, and stale teacher outputs can hinder later improvement. We identify these two failure modes as Gradient Starvation and Teacher-Distribution Anchoring. To address them, we propose One-stage Policy Optimization (OnePO), which treats teacher outputs as transient guidance for policy improvement. OnePO combines Adaptive Objective Evolution to strengthen learning on informative low-probability teacher tokens and Teacher Retirement to discard teacher outputs once the current policy can surpass them. On medical adaptation, OnePO achieves 67.2 on HealthBench (Total) with only 20K training samples, outperforming SFT+RL and pure RL by 2.7 and 7.4 points, respectively. We further scale OnePO to produce HuatuoGPT-3, an open-source medical LLM series whose 27B variant reaches 70.1 on HealthBench (Total) and 71.4 on HealthBench Professional, surpassing frontier models such as GPT-6 Astra. Models and code are available at https://github.com/FreedomIntelligence/HuatuoGPT-3.

1 Introduction

SFT+RL offers a convenient cold start but may restrict later exploration and adds multi-stage optimization complexity, while pure on-policy RL faces a cold-start problem. OnePO uses transient teacher guidance to address mixed-policy RL’s early and late failure modes, and the paper validates it on medical benchmarks.

Diagnosis of Existing RL Paradigms

  • Pure on-policy RL has a cold-start problem, while mixed-policy RL learns useful teacher tokens too slowly early and can pull the policy toward weaker behavior later.The paper names these mixed-policy failure modes Gradient Starvation and Teacher-Distribution Anchoring.Source: §1 · pp. 1–2

A New RL-Only Paradigm: OnePO

  • OnePO treats teacher outputs as transient guidance, strengthening learning on informative low-probability tokens and retiring outputs once their rewards no longer exceed the on-policy frontier.Its mechanisms are Adaptive Objective Evolution and Teacher Retirement.Source: §1 · p. 2

Validation in the Medical Domain

  • With 20K training samples, OnePO outperforms pure RL and SFT+RL on HealthBench, while HuatuoGPT-3’s 27B variant reaches 71.4 on HealthBench Professional.The paper reports scaling OnePO to an open-source medical LLM series.Source: §1 · p. 2

Diagnosis of Existing RL Paradigms

  • The paper identifies these two mixed-policy failure modes and reports validating the diagnosis through controlled pilot studies.The introduction frames RL-only adaptation as an alternative to SFT+RL, which may reduce exploration diversity and adds multi-stage optimization complexity.Source: §1 · pp. 1–2

2 Motivation

The section motivates RL-only adaptation as an alternative to SFT-based multi-stage training and diagnoses why pure on-policy and mixed-policy RL each have shortcomings. It then presents OnePO as transient teacher guidance that changes as the policy improves.

2.1 Why RL-only Adaptation?

  • SFT-based adaptation offers an easy imitation-based cold start but can reduce later diversity, while RL-only methods better preserve exploration; OnePO aims to combine an easy cold start with later autonomy.The section also notes that multi-stage objectives and data shifts can increase optimization complexity and exacerbate forgetting.Source: Table 1 · p. 3, §2.1 · p. 3

2.2 A Diagnosis of RL Through the Lens of Mixed-Policy

  • Pure on-policy RL can struggle to generate positively rewarded outputs early, while mixed-policy RL adds teacher trajectories but still has two weaknesses.The section frames these as a cold-start problem and the limitations diagnosed in the following subsections.Source: §2.2 · p. 3

2.2.1 Weakness I: Gradient Starvation

  • Gradient Starvation occurs when informative teacher tokens receive weak gradient signals, especially because high-quality teacher outputs can have low probability under the current policy.Under the described ratio-based surrogate, the gradient for a sampled token scales with its current-policy probability.Source: §2.2.1 · p. 3
  • In the AHA-Medicine pilot, OnePO reaches the 1.0 fictitious-fact recognition ceiling, while SFT+RL plateaus at the 0.5 imitation reward and mixed-policy RL remains near zero.Pure RL fails because it never observes the synthetic fact; the study tests learning and reasoning beyond imitation under zero data leakage.Source: §2.2.1 · p. 4, Figure 1 · p. 4

2.2.2 Weakness II: Teacher-Distribution Anchoring

  • In the anchoring pilot, mixed-policy RL improves early but later saturates, whereas Teacher Retirement achieves substantially better performance.The variants differ in whether teacher outputs are progressively removed.Source: Figure 1 · p. 4, §2.2.2 · p. 5

2.3 Solution: Teacher Outputs as Transient Guidance

  • OnePO treats teacher outputs as transient guidance, strengthening learning on informative low-probability tokens early and retiring outputs once the policy can match or surpass them.Adaptive Objective Evolution counters Gradient Starvation; Teacher Retirement counters Teacher-Distribution Anchoring.Source: §2.3 · p. 5

3 Methodology

OnePO uses teacher outputs as temporary guidance in RL-only domain adaptation, strengthening their learning signal when the policy is weak and retiring them as on-policy performance improves. Adaptive Objective Evolution addresses weak learning on low-probability teacher tokens, while Teacher Retirement limits persistent teacher influence.

3 Methodology

  • OnePO retains teacher guidance early, then shifts toward on-policy exploration as training progresses.The method combines Adaptive Objective Evolution with Teacher Retirement to address gradient starvation and teacher-distribution anchoring.Source: Figure 2 · p. 5, §3 · pp. 5–6

3.1 Adaptive Objective Evolution

  • OnePO corrects the teacher-output objective with a probability floor and gradient rescaling to address vanishing learning signals for low-probability tokens.The floor gives such tokens room to grow before clipping, while rescaling restores SFT-level gradient magnitude.Source: §3.1 · p. 6
  • Low-probability teacher tokens with positive advantage receive a strong, selective learning signal rather than applying likelihood learning to every teacher token.The probability-aware, advantage-weighted token gradient strengthens learning on informative tokens.Source: §3.1 · p. 6
  • As tokens are absorbed and their probabilities rise, the teacher-output objective smoothly reduces to standard GRPO without a manual schedule.Clipping creates a soft ceiling that prevents overfitting to the teacher distribution.Source: §3.1 · p. 7

3.2 Teacher Retirement

  • Teacher Retirement keeps a teacher output only when its reward exceeds the best on-policy reward in the group, replacing a randomly selected on-policy output while preserving group size.As on-policy performance improves, fewer teacher outputs pass this threshold and training returns to standard on-policy RL without a manual schedule.Source: §3.2 · p. 7

4 Main Experiments

The experiments test OnePO’s medical-domain training setup, benchmark performance, and key design choices. OnePO outperforms SFT+RL in the reported controlled comparison, while ablations examine reward composition, teacher retirement, robustness, and transfer beyond medicine.

4.1 Training Setup

  • The 20K training set jointly combines 10K closed-ended and 10K open-ended medical examples, using exact-match and rubric rewards across medical benchmark evaluations.Closed-ended signals are described as strengthening reasoning, while rubric-based signals improve open-ended generation quality.Source: §4.1 · pp. 7–8, §4.2 · p. 8

4.3 Main Results

  • 67.2/44.5 on HealthBench Total/Hard versus 64.5/40.7 for SFT+RL (+2.7/+3.8); HuatuoGPT-3-9B/27B reach 68.3/71.4 on HealthBench Professional.OnePO also consistently surpasses SFT+RL across open-ended and closed-ended benchmarks; HuatuoGPT-3 models sit above baseline scaling trends with fewer parameters.Source: Table 2 · p. 9, Figure 4 · p. 9, §4.3 · pp. 9–10

4.4 Ablations and Analysis

  • Mixed rewards balance open-ended and closed-ended performance, and surpass rubric-only training on open-ended evaluation.Verifiable-only training is stronger on closed-ended tasks but only slightly improves open-ended quality; rubric-only shows the opposite pattern.Source: Figure 5 · p. 10, §4.4 · p. 10
  • Removing the probability floor causes the largest HealthBench drop (65.4→49.1); removing rescaling or Teacher Retirement also degrades results (65.4→57.3; 65.4→52.7).Without Teacher Retirement, performance falls below Pure RL (59.8) and approaches the GPT-5 Chat teacher (51.9).Source: §4.4 · p. 10, Table 3 · p. 10
  • Max > Mean > Min > None retirement thresholds; looser thresholds produce earlier, more severe training plateaus.The strict Max threshold retires teacher outputs once on-policy rollouts surpass them, allowing autonomous exploration beyond imitation.Source: Figure 6 · p. 10, §4.4 · pp. 10–11
  • Switching from DeepSeek-V3.2 (thinking) to Qwen3-8B (thinking) as teacher causes only a modest drop, and both teachers are mostly retired by training’s end.The stronger teacher phases out more slowly because its outputs remain above the on-policy frontier longer.Source: Figure 7 · p. 10, §4.4 · p. 11
  • Preliminary writing and law experiments with Qwen3-4B-Base show OnePO consistently improving over SFT+RL, offering initial evidence beyond medical adaptation.Writing uses rubric-based rewards and law uses exact-match verifiable rewards.Source: §4.4 · p. 11
  • OnePO scores 54.2 on IFEval and 92.3 on GSM8K, exceeding SFT+RL (48.8/89.7) on both metrics.The paper reports the highest GSM8K score among training paradigms and says OnePO can preserve or improve general reasoning during domain adaptation.Source: §4.4 · p. 11

5 Related Work

Related work covers on-policy and off-policy RL, reward design for verifiable and open-ended tasks, and domain adaptation for specialized LLMs. The paper positions OnePO as temporary teacher guidance followed by on-policy exploration.

5.1 On-Policy and Off-Policy RL

  • On-policy RL is limited by current-policy sampling in unfamiliar domains; imitation and mixed-policy alternatives can respectively retain SFT limitations or keep models tied to external outputs.OnePO instead absorbs teacher outputs early and retires them to return to on-policy exploration.Source: §5.1 · p. 11

5.2 RLVR and Rubric-Based RL

  • Exact-answer or execution-based rewards suit verifiable tasks, while rubric-based rewards provide denser criteria for open-ended quality dimensions.The medical setting combines objective signals for closed-ended questions with rubrics for safety- and communication-sensitive clinical responses.Source: §5.2 · p. 12

5.3 Domain Adaptation for Specialized LLMs

  • Specialized LLMs have been developed across medicine, law, and finance, but most still rely on multi-stage training; this paper studies medical adaptation through RL alone.Source: §5.3 · p. 12

6 Conclusion

The conclusion presents OnePO as an RL-only framework for domain adaptation, using teacher outputs selectively before returning to on-policy improvement. Its medical experiments include a Qwen3-8B-Base adaptation and scaled HuatuoGPT-3 variants.

6 Conclusion

  • OnePO bypasses SFT through Adaptive Objective Evolution and Teacher Retirement, selectively absorbing teacher knowledge while avoiding teacher-distribution anchoring.Source: §6 · p. 12
  • With 20K samples, OnePO adapts Qwen3-8B-Base to achieve 67.2 on HealthBench and outperform SFT+RL under the same setting.Source: §6 · p. 12
  • Applying OnePO across backbones and model sizes produces HuatuoGPT-3 9B and 27B variants; the paper concludes RL-only training can be a practical domain-adaptation route without preceding SFT.Source: §6 · p. 12

Impact Statement

The impact statement frames the work as medical-domain research and notes both potential healthcare benefits and risks. It sets a boundary against clinical deployment without rigorous professional validation.

Impact Statement

  • The work aims to advance LLM domain adaptation, with experiments conducted in the medical domain.Source: Impact Statement · p. 12
  • Medical LLMs could improve healthcare accessibility but may produce inaccurate or harmful advice.Source: Impact Statement · p. 12
  • The model is a research prototype and is not intended for clinical deployment without rigorous validation by medical professionals.The statement also encourages responsible use and human oversight in safety-critical domains.Source: Impact Statement · p. 13

A Limitations

The limitations concern reliance on external signals, the breadth of domain validation, and the scale and teacher setup tested. Technical notes explain why standard mixed-policy surrogates can weakly update informative low-probability teacher tokens and why persistent teacher outputs can hinder later improvement.

A Limitations

  • Weak or biased teachers may slow early learning or limit domain coverage, while model-based grading cannot fully prevent reward misspecification or hacking.Grader validation mitigates this concern but does not replace expert evaluation.Source: A Limitations · p. 19
  • Broader validation is still needed in specialized domains where reward design is less mature or expert criteria are hard to formalize.The main experiments are medical; writing and legal results are also reported.Source: A Limitations · p. 19
  • OnePO has not been validated on 100B+ frontier models, and controlled experiments used only one teacher output per prompt.Multiple teachers or outputs could improve coverage but would alter retirement dynamics; this is left for future work.Source: A Limitations · p. 19

B Technical Notes Supporting the Motivation

  • The technical notes address weak effective gradients on informative low-probability teacher tokens and later hindrance from teacher outputs that no longer extend the on-policy frontier.Pilot-study details are presented separately in Appendix C.Source: B Technical Notes Supporting the… · p. 19

B.1 Gradient Starvation on Low-Probability Teacher-Output Tokens

This section examines gradient starvation in mixed-policy RL and OnePO’s responses to low-probability teacher tokens and stale outputs. It also characterizes the unordered distribution of retained on-policy samples, conditioned on the retained subset.

B.1 Gradient Starvation on Low-Probability Teacher-Output Tokens

  • Standard mixed-policy gradients retain a multiplicative dependence on the current policy’s token probability, starving informative low-probability tokens; OnePO removes that dependence.SFT instead gives lower-probability targets stronger correction signals.Source: §B.1 · pp. 20–21

B.2 Teacher-Distribution Anchoring Beyond the On-Policy Frontier

  • When a teacher reward is at or below the on-policy frontier, fixed-size replacement cannot raise the group’s maximum reward and may lower it, motivating retirement.OnePO retains teacher outputs only while they exceed the current on-policy frontier.Source: §B.2 · p. 21

C.1 Pilot I: AHA-Medicine

  • The AHA-Medicine pilot tests learning fictional knowledge absent from pretraining, with rewards for restating the fact and recognizing that it is fictitious.GPT-4.1-mini serves as the judge; the setup concentrates supervision on teacher tokens initially assigned near-zero probability.Source: §C.1 · p. 22

C.2 Pilot II: Anchoring

  • The anchoring pilot compares retaining teacher outputs throughout RL with Teacher Retirement, using the same setup and a shared cold-start fine-tuned initialization to reduce early gradient-starvation effects.The model is fine-tuned on 2K examples sampled from the 20K training set, then both variants train on that same 20K set.Source: §C.2 · pp. 22–23

D.1 Low-Probability Regime

  • In the low-probability regime, OnePO’s corrected update removes the token-probability factor and provides strong learning only up to a soft saturation scale around c(1 + ϵ).With c ≈ 0.1, the correction turns off as a token becomes learnable, limiting drift toward the teacher distribution.Source: §D.1 · p. 23

D.2 In-Support Regime

  • Once a teacher token has sufficient support under πθold, OnePO turns off the low-probability amplification and returns to the standard clipped policy-optimization form.The corrected teacher-output term then reduces to the same ratio form as standard GRPO.Source: §D.2 · p. 24

E Details of Teacher Retirement

  • Teacher Retirement keeps the group size fixed by replacing on-policy outputs with retained teacher outputs; under i.i.d. on-policy rewards, random replacement preserves the distribution of the unreplaced draws.The algorithm retains teacher outputs only when their rewards exceed the on-policy maximum and retires the rest.Source: E Details of Teacher Retirement · p. 24, §E.1 · p. 24, §E.2 · p. 25

E.2 Replacement View

  • Exchangeability makes the retained on-policy index set uniform over subsets of size G − n, so its unordered multiset matches that of any fixed G − n coordinates.This holds after conditioning on the realized retained subset S and n = |S|.Source: §E.2 · p. 25

F Dataset Construction

The appendix describes generating open-ended medical prompts and rubrics from case reports, then illustrates the resulting open- and closed-ended task formats. Its examples include a case-report task scored with positive and negative rubric criteria and a multiple-choice task with a tagged answer.

F.1 Prompt for Open-Ended Question and Rubric Generation

  • Reference prompts are questions posed to the LLM, while reference rubrics are scoring rules in which higher scores indicate better performance..Source: §F.1 · p. 27
  • Prompts should be challenging and diverse, with corresponding rubrics tailored to the case and limited to no more than 8 items.GPT-5 Chat converts filtered PMC-OA case reports into prompt and rubric sets, using strict JSON output with user turns.Source: §F.1 · pp. 25, 27

F.2 Data Examples

  • The closed-ended example requires a multiple-choice response containing only the selected option inside an answer tag.The example gives Eduardo Bassini as the reference answer.Source: §F.2 · pp. 27–28
  • The open-ended example converts a PMC-OA case report into a challenging task accompanied by up to 8 rubric items that positively or negatively score clinically relevant criteria.The illustrated prompt concerns a 28-year-old patient with a brachial artery aneurysm.Source: §F.2 · p. 28

G Details of Dataset and Benchmarks

The controlled medical dataset combines closed-ended questions with open-ended case-report tasks, and the paper evaluates medical and general capabilities across distinct benchmark types. The appendix describes the task sources and evaluation protocols.

G Details of Dataset and Benchmarks

  • 20K training tasks comprise 10K closed-ended and 10K open-ended medical tasks.Source: G Details of Dataset and Benchmarks · p. 29
  • 10K closed-ended tasks are high-difficulty medical multiple-choice questions sampled from MedQA and MedMCQA training sets, with standard answers used for verifiable rewards.Source: G Details of Dataset and Benchmarks · p. 29
  • 10K open-ended tasks are built from PMC-OA case reports using GPT-5 Chat-generated prompts and rubrics, then filtered against the source case for hallucinations or clinically unreasonable criteria.The authors state this construction can scale to larger corpora with limited manual annotation, followed by targeted quality checks.Source: G Details of Dataset and Benchmarks · p. 29
  • Five medical benchmarks assess domain-specific capabilities, while IFEval and GSM8K assess general instruction-following and mathematical reasoning capabilities.IFEval and GSM8K are evaluated zero-shot on official sets and are not used for training.Source: G Details of Dataset and Benchmarks · p. 29
  • HealthBench uses rubric-based grading for open-ended health responses, while closed-ended benchmarks report exact-match accuracy (%).HealthBench Professional covers physician-authored care consult, writing and documentation, and medical research tasks.Source: G Details of Dataset and Benchmarks · p. 29

H Ablation of Reward Data Composition

The reward-composition ablation finds that combining rubric-based and verifiable rewards balances open- and closed-ended performance. Additional comparisons examine SFT+RL and the probability floor used by OnePO.

H Ablation of Reward Data Composition

  • Verifiable-only reward scores 64.5% on Medbullets and 80.5% on MMLU-Pro but 24.2% on HealthBench Total; rubric-only scores 63.7% on HealthBench Total but 44.8% on Medbullets.Verifiable reward performs strongly on closed-ended tasks but barely improves open-ended quality over the 22.1% base score; rubric reward shows the opposite pattern and may over-constrain reasoning for concise, definite answers.Source: H Ablation of Reward Data Composition · p. 30
  • 65.4% HealthBench Total, 64.0% Medbullets, and 81.2% MMLU-Pro with mixed rewards give the best overall trade-off across open- and closed-ended tasks.The mixed setting uses the full 20K samples and both reward types.Source: Table 7 · p. 29, H Ablation of Reward Data Composition · p. 30

I SFT Baseline on HealthBench

  • With the same GPT-5 Chat teacher source, SFT+RL scores 63.6 on HealthBench versus 59.8 for Pure RL, while OnePO uses the same teacher outputs in reward-driven optimization.HealthBench evaluates held-out open-ended clinical response quality rather than imitation fidelity; the passage attributes poor generalization to unseen rubric criteria as a possible issue for naive SFT.Source: I SFT Baseline on HealthBench · p. 30

J Sensitivity to Probability Floor

  • Setting the probability floor c to 0 causes a large performance drop; c = 0.1 gives the best overall trade-off.Very small floors inadequately correct low-probability tokens, while very large floors increase anchoring risk.Source: J Sensitivity to Probability Floor · p. 30

K Effect of KL Regularization in the AHA-Medicine Pilot

The pilot tests OnePO with KL constraints, followed by cross-domain evaluation and checks of rubric quality, grader reliability, and reward stability. Together, these tests examine the method’s behavior across settings and evaluation checks.

K Effect of KL Regularization in the AHA-Medicine Pilot

  • Under KL constraints, OnePO absorbs the synthetic AHA-Medicine knowledge, while removing either the probability floor or gradient rescaling fails to reach reward ≥0.5 within 100 steps.KL constraints make updates more conservative but do not remove OnePO’s effect; the rollout-based KL estimate confirms reduced policy movement.Source: K Effect of KL Regularization in the… · p. 31, Table 10 · p. 31

L Cross-Domain Validation

  • OnePO improves over SFT+RL in both writing and law under the same teacher source, using rubric-based rewards for writing and exact-match rewards for law.The evaluation uses Qwen3-4B-Base and 10K samples per task.Source: Table 13 · p. 32, L Cross-Domain Validation · p. 32

M Rubric Filtering and Grader Validation

N Reward Stability

  • Open-ended rubric reward improves smoothly, while closed-ended exact-match reward also rises steadily during training.The paper interprets these curves as evidence of stable optimization signals in this setting when dense rubric rewards are combined with verifiable rewards.Source: N Reward Stability · p. 33, Table 16 · p. 33

Related papers

Loading 2610.05966v1…