PaperlayerView exact revision on arXiv

Source-linked AI summary

On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting

Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, Jingren Zhou

arXiv:2508.11408v3cs.LGcs.AI

TL;DR

Existing SFT–RL integration can disrupt established response patterns and overfit expert data, motivating a unified off-policy/on-policy treatment. CHORD makes SFT a dynamically weighted auxiliary objective within on-policy RL, combining global and token-wise controls. Experiments report stable learning and improvements over SFT-then-RL and existing approaches.

  • Problem

    Existing SFT–RL integration can disrupt established response patterns and induce overfitting to expert data, while sequential SFT-then-RL can remain suboptimal and fragile.

  • Method

    CHORD reframes SFT as a dynamically weighted auxiliary objective within on-policy RL, using global coefficient µ and token-wise function ϕ(·) to control expert-data influence.

  • Results

    Extensive experiments demonstrate that CHORD outperforms the SFT-then-RL paradigm and existing approaches.

  • Takeaways & Limitations

    CHORD provides fine-grained control over off-policy expert data while integrating expert demonstrations with on-policy exploration and maintaining training stability.

  • Takeaways & Limitations

    The framework remains sensitive to configurations for µ and ϕ(·), making reduced hyperparameter dependence an open practical objective.

Abstract

from arXiv · show

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are two prominent post-training paradigms for refining the capabilities and aligning the behavior of Large Language Models (LLMs). Existing approaches that integrate SFT and RL often face the risk of disrupting established response patterns and inducing overfitting to expert data. To address this, we present a novel investigation into the unified view of SFT and RL through an off-policy versus on-policy lens. We propose CHORD, a framework for Controllable Harmonization of On- and Off-Policy Reinforcement Learning via Dynamic Weighting, which reframes SFT not as a separate stage but as a dynamically weighted auxiliary objective within the on-policy RL process. Based on an analysis of off-policy expert data's influence at both holistic and granular levels, we incorporate a dual-control mechanism in CHORD. Specifically, the framework first employs a global coefficient to holistically guide the transition from off-policy imitation to on-policy exploration, and then applies a token-wise weighting function that enables granular learning from the expert, which promotes on-policy exploration and mitigates disruption from off-policy data. We conduct extensive experiments across various practical tasks, providing empirical evidence that CHORD achieves a stable and efficient learning process. By effectively harmonizing off-policy expert data with on-policy exploration, CHORD demonstrates significant improvements over baselines. We release the implementation at https://github.com/modelscope/Trinity-RFT/tree/main/examples/mix_chord to inspire further research.

1 INTRODUCTION

SFT and RL offer complementary benefits, but sequentially combining them can disrupt established response patterns and overfit expert data. CHORD instead harmonizes off-policy imitation with on-policy exploration through dynamic weighting.

  • SFT learns expert response patterns but depends on expert-data quality and quantity, may generalize poorly beyond memorization, and is vulnerable to exposure bias.
  • RL supports exploration and generalization through feedback on on-policy generations, but can be inefficient and cause policy degradation or over-exploitation.
  • The SFT-then-RL paradigm does not consistently outperform pure RL, despite intending to combine expert guidance with on-policy learning.
  • SFT on Deepseek-R1 expert data exhibits a “shift-readapt-overfit” progression, disrupting established patterns before recovery and eventual overfitting.
  • CHORD reframes SFT as a dynamically weighted auxiliary objective within on-policy RL, using global coefficient µ and token-wise weighting function ϕ(·).The token-wise function down-weights highly divergent expert tokens to maintain stability.
  • Extensive experiments report that CHORD outperforms SFT-then-RL and existing approaches while selectively absorbing expert knowledge without stifling reasoning capabilities.

2 PRELIMINARIES

SFT optimizes a language-model policy against static expert demonstrations, whereas RL optimizes it from rewards on generated trajectories. GRPO implements the RL side with grouped responses, normalized advantages, and a PPO-style clipped objective.

  • SFT is an off-policy paradigm that adjusts policy πθ to mimic a static dataset of high-quality expert demonstrations.
  • RL is an on-policy paradigm that maximizes expected reward R(τ) from generated trajectories.
  • The SFT objective minimizes the negative log-likelihood of expert responses using an empirical mini-batch estimate.
  • GRPO samples K responses per prompt, evaluates each with a reward function, and updates πθ using a PPO-style clipped surrogate objective.
  • GRPO computes each response’s advantage from its reward relative to the group mean and standard deviation, with a small stability constant.
  • The token-wise importance-sampling ratio reweights actions sampled under πsample; under strict on-policy sampling, the ratio equals 1.

3 CHORD: HARMONIZING OFF-POLICY AND ON-POLICY LEARNING

CHORD unifies SFT and RL by embedding expert-data learning as a dynamically weighted auxiliary objective within on-policy training. Its global and token-wise controls aim to transition from imitation to exploration while reducing disruption from off-policy data.

  • 3.1 THE SHIFT-READAPT-OVERFIT PROGRESSION WHEN UTILIZING OFF-POLICY DATA: The SFT-then-RL paradigm can be fragile when expert response patterns diverge from the model’s established patterns, producing a shift-readapt progression and possible overfitting.The observed progression makes the influence of off-policy expert data difficult to control and can yield suboptimal solutions.
  • 3 CHORD: HARMONIZING OFF-POLICY AND ON-POLICY LEARNING: CHORD reframes SFT as a dynamically weighted auxiliary objective within on-policy RL instead of a separate tuning stage.Its hybrid loss combines the RL and SFT losses, with µ ∈ [0, 1] governing their trade-off.
  • 3.2 CONTROLLING THE INFLUENCE OF OFF-POLICY EXPERT DATA VIA µ: A decaying µ provides a flexible transition from off-policy imitation to on-policy optimization by reducing expert-data influence during training.Training begins with a large µ and gradually shifts focus toward on-policy exploration before overfitting on expert data.
  • 3.2 CONTROLLING THE INFLUENCE OF OFF-POLICY EXPERT DATA VIA µ: Applying a decay schedule to µ yields notable performance gains over the SFT-then-RL paradigm, but CHORD-µ can still disrupt exploration and impose the expert’s verbose response pattern.These observations motivate adding a finer-grained control beyond the global coefficient.
  • 3.3 ENHANCING THE STABILITY OF OFF-POLICY LEARNING VIA ϕ(·): Importance Sampling down-weights low-probability expert tokens but can sharply collapse policy entropy, whereas unweighted mixed training sharply raises entropy relative to pure RL.These opposing effects motivate a weighting function that avoids both disruption and excessive suppression of exploration.
  • 3.3 ENHANCING THE STABILITY OF OFF-POLICY LEARNING VIA ϕ(·): CHORD adds a token-wise weighting function ϕ(·) to modulate each expert token’s gradient contribution and stabilize learning from off-policy data.The dual-control mechanism combines global control through µ with fine-grained token-level weighting.

4 EXPERIMENTS

Experiments evaluate CHORD on mathematical reasoning and tool-use tasks, finding consistent gains, improved training stability, and task-appropriate response patterns. Analyses also examine weighting schedules, expert sources, and weaker policy models.

  • Datasets, Models, and Evaluations: CHORD is evaluated on mathematical reasoning and practical tool-use tasks, with Qwen2.5-7B-Instruct serving as the policy model for math experiments.The math setup uses OpenR1-Math-220k, with AIME24, AIME25, AMC, and MMLU-Pro evaluation.
  • Model Performance: CHORD-µ outperforms SFT-best+RL across all math benchmarks, improving AMC by +2.4, AIME24 by +1.0, and AIME25 by +1.6.It also achieves better overall results than SFT-then-RL baselines on tool-use tasks.
  • Model Performance: CHORD-ϕ consistently outperforms baselines and achieves the best performance on both reasoning problems and tool-use tasks.Its token-wise weighting selectively applies SFT to non-disruptive tokens while retaining on-policy exploration.
  • Response Patterns: 2,444 tokens for math and 120 tokens for tool use show that CHORD-ϕ learns comprehensive reasoning while retaining concise task-specific responses.Pure RL similarly lengthens math responses from 659 to 1,423 tokens while shortening tool-use responses from 147 to 118 tokens.
  • Analysis on the Effects of µ and ϕ(·): Dynamic µ outperforms fixed schedules, while token-wise ϕ stabilizes entropy and supports continuous reward increases.Fixed µ can force accommodation of divergent reasoning patterns; ϕ reduces premature entropy collapse and large entropy spikes.
  • Analysis on the Effects of µ and ϕ(·): No single token-weighting design is expected to be optimal across all tasks, datasets, and models, although the proposed design is reported as robust and effective.The study tests entropy-based, clipping, and focal-loss variants.
  • Further Analysis: CHORD-µ and CHORD-ϕ outperform Pure RL and SFT+RL across expert sources, with imitation-heavy methods benefiting more from stylistically similar experts.CHORD-ϕ also achieves faster convergence and higher final rewards in diverse non-verifiable domains.
  • Further Analysis: Weaker Qwen2.5-3B-Instruct models are more susceptible to performance collapse when trained on the same off-policy expert data.The experiments characterize weaker models as limited both in on-policy exploration and expert-data absorption.

5 RELATED WORKS

Related work combines expert data and on-policy generations through dataset mixing, generation guidance, interleaved SFT and RL, and unified sample-level objectives. This study focuses on tuning an instruct model that already has an established response pattern.

  • Existing Integration Strategies: Prior approaches mix expert trajectories with on-policy generations, guide generation with experts, or interleave RL updates and SFT steps.SRFT more recently combines data mixing with a sample-level SFT loss.

6 CONCLUSIONS AND FUTURE DIRECTIONS

The paper concludes that sequential SFT-then-RL can disrupt established response patterns, motivating CHORD’s unified weighting approach. Future work targets stabilization, adaptive tuning, deeper pattern-shift analysis, and heterogeneous expert mixtures.

  • Conclusions: CHORD unifies SFT and RL by using global µ for transition control and token-wise ϕ(·) for selective expert-data absorption.The framework is intended to improve over sequential pipelines and other baselines.
  • Future Directions: Future work includes stabilizing high-variance RL dynamics and developing adaptive µ and ϕ(·) tuning to reduce hyperparameter dependence.The authors also identify deeper analysis of diverse chain-of-thought pattern shifts as an open direction.
  • Experimental Configuration: The experiments use Adam with β1 = 0.9 and β2 = 0.999, learning rates from {1×10−6, 5×10−6, 1×10−5}, and K = 8 rollouts per update.The maximum response length is 16k tokens, with separate training-step and batch-size settings for math and tool-use tasks.

A.2 IMPLEMENTATION DETAILS

Implementation details cover task-specific rewards, evaluation metrics, prompting, algorithms, and hardware for the mathematical reasoning and tool-use experiments.

  • Rewards: Mathematical reasoning uses a hierarchical reward scheme that rewards correct answers and format adherence, with integer-answer problems sampled for precise evaluation.A correct final answer receives +1.0 reward.
  • Implementation: SFT is implemented with LLaMA-Factory and RL with Trinity-RFT across 8 NVIDIA A100 and 8 NVIDIA H20 GPUs.
  • Evaluation: Accuracy is reported as avg@32 on AIME24 and AIME25 and avg@8 on AMC, using the best validation-selected checkpoint.These reporting choices are intended to reduce variance and support fair comparisons.
  • Prompts: Math prompts require hidden reasoning followed by <think> formatting and a boxed final answer, while MMLU-Pro requires an integer option index.Tool-use experiments follow the setup and prompt from Zhang et al. (2025a).

B.1 ADAPTIVE TUNING µ

Adaptive weighting of the SFT loss can reduce its contribution as reward improves, but its effectiveness remains sensitive to the chosen reward threshold.

  • Adaptive schedule: The adaptive strategy sets µ′ = max(0, τ − reward_mean), gradually removing the SFT component once average reward exceeds threshold τ.Experiments tested thresholds τ = 0.5 and τ = 0.7.
  • Results: A reward threshold of 0.5 achieves an overall ToolACE score of 78.1, competitive with the fixed-decay approach.
  • Limitations: The adaptive schedule remains dependent on task-specific threshold tuning, and threshold 0.7 causes significant performance degradation.Excessive SFT may continue even when rewards are moderately high, disrupting optimization.
  • Limitations: The adaptive approach is presented as a proof of concept rather than a simpler tuning procedure, because it introduces the reward threshold as another hyperparameter.Robust and generalizable adaptive schemes remain future work.

B.2 VARYING THE ϕ FUNCTION

Token-wise weighting substantially affects the stability and performance of SFT/RL integration across tool-use and mathematical reasoning tasks. CHORD-ϕ remains consistently strong, whereas aggressive or coarse weighting strategies can collapse or underperform.

  • Weighting variants: CHORD-ϕ achieves consistently strong performance across tool-use and mathematical reasoning benchmarks.Its weighting function is ϕ(p) = p × (1 − p).
  • Weighting variants: Entropy Top reaches 17.2 on AIME2024, but entropy-based gains are less consistent across benchmarks than CHORD-ϕ.
  • Weighting variants: IS Clip has limited effectiveness and underperforms the pure RL baseline on the tool-use task.
  • Weighting variants: Focal Loss severely destabilizes training and causes significant performance collapse on both task types.It aggressively up-weights low-probability, high-surprise tokens.
  • Analysis: The choice of token-wise weighting function significantly affects training stability and final performance across domains.The proposed ϕ instantiation down-weights tokens at both probability extremes.

B.3 VARYING EXPERT DATA SOURCE

The expert data source matters partly because its reasoning patterns may shift the base model by different amounts. CHORD-µ benefits from closer pattern alignment, while CHORD-ϕ remains strong across experts of different quality and style.

  • Quantitative comparison: Using Qwen2.5-72B-Instruct data, CHORD-µ scores 78.1 versus 77.6 with DeepSeek-R1 data.Qwen2.5-72B-Instruct has reasoning patterns closer to LLaMA3.2-3B-Instruct.
  • Quantitative comparison: CHORD-ϕ scores 78.3 versus 78.5 across the two expert data sources despite Qwen2.5-72B-Instruct being the weaker expert.This supports robustness to differences in expert strength and stylistic mismatch.
  • Baseline comparison: The baseline SFT-best + RL method improves from 76.1 to 77.7 when using Qwen2.5-72B-Instruct data.The comparison links effectiveness to pattern shift, not expert quality alone.
  • Pattern analysis: DeepSeek-R1 produces more verbose, elaborate reasoning, whereas Qwen2.5-72B-Instruct is more concise and closer to LLaMA3.2-3B-Instruct’s direct style.The qualitative pattern difference aligns with the quantitative comparison in Table 6.

B.4 NON-VERIFIABLE TASKS

On RaR-Medicine, CHORD generalizes beyond tasks with deterministic verification, with both global and token-wise variants outperforming pure RL. The weaker-policy experiment further shows that naive SFT can be unstable, while CHORD-ϕ preserves learning stability.

  • RaR-Medicine: CHORD-µ and CHORD-ϕ achieve testset scores of 80.6 and 81.3, compared with 76.8 for pure RL.These results are reported on the RaR-Medicine medical question-answering task.
  • RaR-Medicine: CHORD-ϕ converges faster and reaches higher final rewards than pure RL on RaR-Medicine.CHORD-µ exhibits a similar shift-readapt pattern to the main experiment.
  • Weaker policy models: Naively fine-tuning the weaker Qwen2.5-3B-Instruct model with SFT causes a performance collapse.
  • Weaker policy models: With the same 5k SFT samples, Qwen2.5-7B-Instruct improves AIME2024 accuracy from 11.7% to 15.8%.
  • Weaker policy models: For weaker models, the trade-off is difficult because both on-policy exploration and absorption of off-policy expert data can be constrained.CHORD-ϕ maintains stability, while more expert data could potentially make SFT-leaning methods more favorable.

B.6 DIVERSE MODEL ARCHITECTURES

CHORD extends evaluation beyond dense Qwen and LLaMA models to Phi-mini-MoE-instruct, while its dual weighting controls expert-data influence and training stability. Across schedule comparisons and related settings, the framework is presented as broadly applicable and less sensitive to global-coefficient tuning.

  • Diverse Model Architectures: The Phi-mini-MoE-instruct model shows significant performance collapse with naive SFT, whereas CHORD-ϕ achieves the highest performance.This evaluates CHORD beyond dense Qwen and LLaMA models.
  • Dynamic Weighting: CHORD’s dual-control mechanism combines a global coefficient µ with a token-wise weighting function ϕ(·) to regulate expert-data influence at different granularities.The framework uses global control for overall influence and token-wise control for fine-grained weighting.
  • Dynamic Weighting: ϕ(·) makes CHORD more robust to the specific µ schedule, reducing the need for meticulous global-coefficient tuning.This robustness simplifies practical application of the framework.
  • Dynamic Weighting: CHORD-tune-both yields slightly better reward progression with warmup-then-decay µ, but the final performance gap is not significant.The comparison appears in Figure 11.
  • Evaluation Scope: The evaluation includes tool-use curves, BFCL comparisons, and MMLU-Pro results, with BFCL averages weighted by instance counts.All methods use the same system prompt format for BFCL evaluation.

D.1 THE INFLUENCE OF OFF-POLICY DATA ON BASE VS. INSTRUCTION MODELS

Off-policy expert data affects base and instruction models differently: established instruction policies are vulnerable to disruptive distribution shifts, whereas base-model rewards improve monotonically. CHORD addresses this setting with token-wise adaptive weighting and shows stronger performance than directly mixed Zero-RL methods, while combining expert imitation with on-policy learning across distinct task types.

  • D.1 THE INFLUENCE OF OFF-POLICY DATA ON BASE VS. INSTRUCTION MODELS: Instruction models face amplified off-policy-control and stability challenges because they already contain an established policy.This contrasts with the less constrained policy state of base models.
  • D.1 THE INFLUENCE OF OFF-POLICY DATA ON BASE VS. INSTRUCTION MODELS: Distributional mismatch between instruction-model policies and expert reasoning patterns can trigger disruptive updates and performance collapse.The mismatch is more consequential because instruction models have sharply peaked policies.
  • D.1 THE INFLUENCE OF OFF-POLICY DATA ON BASE VS. INSTRUCTION MODELS: Base-model reward improves monotonically when training mixes on-policy and off-policy data, unlike the instability observed for instruction models.Figure 13 provides the cited empirical observation using fixed µ = 0.1.
  • D.1 THE INFLUENCE OF OFF-POLICY DATA ON BASE VS. INSTRUCTION MODELS: The paper targets established instruction-following models rather than the Zero-RL setting, which is more practical when starting from a base model is infeasible.The authors specifically note that base-model methods are ineffective for tool-use tasks.
  • D.1 THE INFLUENCE OF OFF-POLICY DATA ON BASE VS. INSTRUCTION MODELS: CHORD significantly outperforms LUFFY and SRFT on the tool-use comparison while using 500 expert demonstrations without prompt-level alignment.LUFFY and SRFT require aligned expert demonstrations for all 5,000 training prompts.
  • D.1 THE INFLUENCE OF OFF-POLICY DATA ON BASE VS. INSTRUCTION MODELS: CHORD’s token-wise ϕ(·) selectively absorbs expert patterns, avoiding the indiscriminate updates caused by trajectory-level or sample-level weighting.The paper links this mechanism to better expert-data efficiency and training stability on instruction-tuned models.
  • D.2 ON DIFFERENT TASK-RELATED PERFORMANCE: MATH benefits from expert step-by-step reasoning, whereas tool-use tasks favor concise responses and can suffer from naive imitation of verbose expert trajectories.The two domains therefore create different challenges for combining offline SFT and online RL.
  • D.2 ON DIFFERENT TASK-RELATED PERFORMANCE: The combined approach produces comprehensive MATH reasoning and concise, efficient tool calls, demonstrating selective task-specific absorption of expert knowledge.The authors present this as evidence of robustness and versatility across diverse applications.

E CASE STUDIES

The case studies compare RL-only, SFT-only, and CHORD generation patterns on a time-interval problem. CHORD-ϕ combines concise structure with expert-style verification, while CHORD-µ resembles the more verbose SFT-only pattern.

  • Generation Pattern Comparisons: The case study compares generation patterns from RL-only, SFT-only, and CHORD models.The examples use a bus-trip duration problem and include a Qwen2.5-7B-Instruct response.
  • Generation Pattern Comparisons: RL-only produces concise, structured, and rigid reasoning that follows fixed connectors without self-correction or alternative paths.This is efficient but may yield less robust solutions.
  • Generation Pattern Comparisons: SFT-only and CHORD-µ generate verbose, exploratory think-aloud responses with meta-commentary, backtracking, and intermediate-step verification.The passage describes CHORD-µ as having a similar reasoning pattern to SFT-only.
  • Generation Pattern Comparisons: CHORD-ϕ combines clean logical structure with selected expert verification strategies, including alternative methods and cross-checking.The examples use checkpoints at 800 training steps.
  • Bus-Trip Case Study: The bus-trip example uses departure at 7:43 a.m. and arrival at 8:22 a.m., decomposing the interval into 17 and 22 minutes.The correct total is 39 minutes.
  • Bus-Trip Case Study: The corrected calculation confirms 39 minutes by both interval addition and conversion to minutes since midnight.The conversion computes 502 - 463 = 39 minutes.
Loading 2508.11408v3…