PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Self-Weighted Fine-Tuning

Haiquan Wen, Yiwei He, Bei Peng, Guangliang Cheng

arXiv:2609.00734v1cs.LG

TL;DR

SFT weights expert demonstrations uniformly, while RL-based methods adapt updates at higher sampling cost and with instability on hard reasoning tasks. OSW-FT estimates query success online and rescales expert-anchored SFT updates, yielding consistent improvements over SFT on small-to-medium Qwen3 models with a favorable limited-rollout trade-off.

  • Problem

    SFT treats mastered and unresolved demonstrations identically, while RL adaptivity can require substantial sampling and become unstable on hard reasoning tasks.

  • Method

    OSW-FT uses a small number of inference-only rollouts to estimate success rate and reweight each expert-trajectory SFT loss without changing its optimization direction.

  • Results

    OSW-FT consistently improves over standard SFT on small-to-medium Qwen3 models across multiple challenging benchmarks.

  • Takeaways & Limitations

    OSW-FT offers a favorable performance–rollout trade-off for binary-verifiable reasoning when the online rollout budget is limited, using 2 online rollouts in the experiments.

  • Takeaways & Limitations

    OSW-FT assumes a reliable binary verifier and high-quality expert trajectories, and has not been validated for continuous rewards or open-ended generation.

Abstract

from arXiv · show

Standard supervised fine-tuning (SFT) assigns the same explicit loss weight to every expert demonstration, regardless of the model's changing competence over training queries. Reinforcement learning (RL) based methods adapt update strength using model-generated rollouts, but often require substantially more sampling and can be unstable on hard tasks. We propose \textbf{Online Self-Weighted Fine-Tuning (OSW-FT)}, a simple method that augments SFT with online, trajectory-level weighting. For each query, OSW-FT estimates the model's current success rate using a small number of inference-only rollouts and rescales the standard SFT loss accordingly. The optimization direction remains anchored to the expert trajectory, while the update magnitude adapts online. For binary-verifiable reasoning, we connect this weighting to SFT and RL at the gradient level, inspired by variance-reduction principles. The resulting estimator is unbiased for the exact OSW-FT surrogate update for any finite rollout count, and we analyze convergence with respect to the corresponding surrogate objective. Evaluated across Qwen3 series ranging from 0.6B to 4B on multiple challenging benchmarks (e.g., AIME), OSW-FT consistently improves over SFT on small-to-medium scale models. OSW-FT offers a favorable compute-performance trade-off as a practical approach for fine-tuning small-to-medium LLMs on binary-verifiable reasoning tasks with only \textbf{2 online rollouts}.

1 Introduction

OSW-FT addresses the mismatch between uniform SFT weighting and changing query difficulty by adapting update magnitude while retaining expert supervision. It targets limited-rollout verifiable reasoning and consistently improves over SFT on small-to-medium models.

  • Motivation: Standard SFT weights every expert demonstration uniformly, despite substantial and evolving differences in query learning value.Already-mastered queries provide limited additional signal, whereas frontier queries warrant concentrated optimization.
  • Motivation: RL adapts update strength from model rollouts but incurs substantial sampling cost and instability on hard reasoning tasks.SFT retains expert supervision but lacks adaptive control of trajectory update magnitude.
  • Method: OSW-FT estimates each query’s current pass rate with inference-only rollouts and reweights its expert-trajectory SFT loss by 1 − ˆp(q).The weight is largest for consistently failed queries and decreases toward zero as the model masters them.
  • Method: OSW-FT uses online rollouts only to estimate a scalar weight, without optimizing on generated trajectories, token-level reward shaping, or distribution changes.This preserves expert-trajectory supervision under a limited online rollout budget.
  • Theory: OSW-FT’s estimator is unbiased for its exact surrogate update at any finite rollout count, with convergence analyzed for the corresponding surrogate objective.The framework connects the weighting to RL control variates at the gradient level for binary-verifiable reasoning.
  • Evaluation: OSW-FT consistently improves over standard SFT on Qwen3 models from 0.6B to 4B across multiple challenging benchmarks.Compared with GRPO, results vary across models and metrics; the method targets limited-rollout settings rather than replacing RL generally.

2 Online Self-Weighted Fine-Tuning

The method derives OSW-FT by combining SFT’s expert-anchored direction with an online, policy-dependent magnitude based on binary success probability. It preserves the data distribution while reallocating gradient magnitude using inference-only rollouts.

  • SFT and motivation: SFT applies a uniform gradient weight of 1 to every ground-truth trajectory, whereas online optimization seeks to reduce effort on mastered queries.The motivation is to emphasize unresolved frontier queries without abandoning expert demonstrations.
  • Control variate: For binary rewards, the MSE-minimizing control variate equals the model’s current success probability ps(q), which also determines the oracle trajectory’s advantage magnitude.This links variance reduction to the proposed self-weighting signal.
  • SFT substitution: SFT avoids RL gradient starvation by substituting a deterministic expert trajectory, guaranteeing a continuous learning signal while nullifying the negative gradient term.The substitution anchors the optimization direction to the expert trajectory.
  • OSW-FT gradient: OSW-FT decouples data-driven direction from policy-driven step size by retaining the expert trajectory while weighting its update by 1 − ps(q).The formulation is presented as a regularized surrogate gradient rather than an algebraic identity.
  • Online weighting: The online weight uses a Monte Carlo estimate of success probability and modulates expert updates without changing the training distribution.Unlike rollout-optimized RFT, OSW-FT anchors gradient direction to expert demonstrations and uses K inference-only rollouts per query.

3 Theoretical Analysis

The analysis establishes finite-rollout unbiasedness and convergence for a surrogate objective whose weights are periodically frozen. It also identifies K = 2 as a practical point where soft discrimination begins while additional rollouts yield diminishing returns.

  • Finite-rollout estimator: OSW-FT estimates success probability with K rollouts and applies the estimator ˆg_K(q) = (1 − ˆp(q)) · g0(q).Here g0(q) is the expert-trajectory log-likelihood gradient.
  • Finite-rollout robustness: For any finite K, the OSW-FT estimator is unbiased for the exact surrogate update, with expectation (1 − p) · g0(q).The result holds for fixed queries with success probability p in (0, 1).
  • Finite-rollout robustness: At K = 2, signal loss for p = 0.5 falls to 25%, while the intermediate weight 0.5 enables soft discrimination; larger K offers diminishing returns.At K = 1, signal loss is 50% and weighting reduces to hard example selection.
  • Convergence analysis: Because weights depend on the current policy but remain fixed within batches, convergence is analyzed using a surrogate objective that freezes weights at each outer iteration.The guarantee assumes standard smoothness and bounded-gradient conditions.
  • Convergence analysis: The convergence bound contains an O(1/√T)-type decaying term and an additional term reflecting changes in success probability between weight updates.The latter term is small under typical learning rates according to the analysis.
  • Second-moment comparison: OSW-FT has a smaller stochastic-gradient second moment than SFT under identical conditions, when the model succeeds on a positive-measure set of queries.This is the stated theoretical basis for its convergence advantage.

4 Experiments

OSW-FT is evaluated against SFT and GRPO across Qwen3 models and reasoning benchmarks, including limited-rollout and weighting ablations. It improves over SFT at small-to-medium scales, while its comparison with GRPO varies by model, metric, and rollout budget.

  • Evaluation Protocol: The experiments compare Qwen3-0.6B, 1.7B, and 4B models using AMC, AIME, MATH-500, and GPQA-Diamond benchmarks against SFT and GRPO.Competition benchmarks report Pass@1 and Pass@16, while MATH-500 and GPQA-Diamond report Pass@1.
  • Main Results: OSW-FT improves over standard SFT across the evaluated small-to-medium model scales, but its comparison with GRPO varies across models and metrics.The results indicate a trade-off rather than a universal ordering between expert-anchored training and reinforcement fine-tuning.
  • Main Results: 36.67% Pass@16 on both AIME 24 and AIME 25 for the 4B model compares with 26.67% for SFT.OSW-FT consistently outperforms or matches SFT on Pass@16 for the 0.6B, 1.7B, and 4B models.
  • The Capacity-Data Mismatch Phenomenon: The advantage weight (1 −p̂) decreases toward zero more rapidly for the 4B model, while the 0.6B model retains a sustained learning signal.This scale-dependent behavior reflects faster dataset mastery by the more capable model.
  • Performance under Limited Rollout Budgets: K = 2 produces 20.20% on GPQA-D, up from 15.15% at K = 1, while performance plateaus for K > 2.The ablation varies OSW-FT rollouts over K ∈ {1, 2, 4, 8}; K = 2 captures the necessary advantage signal with limited overhead.
  • Performance under Limited Rollout Budgets: 0.00 on both AIME 24 and AIME 25 is obtained by GRPO with K = 2, whereas larger group sizes such as K = 8 can restore non-zero AIME performance.The contrast is attributed to sparse advantage signals when limited exploration fails to discover correct long-horizon reasoning paths.
  • Ablation on Weighting Strategies: Alternative weighting heuristics cause severe performance degradation, whereas OSW-FT scales down updates as capability increases.The comparison supports content-aware weighting under a limited online rollout budget.

5 Related Work

Related work addresses SFT inefficiency through offline reweighting and regularization, online rollout-driven alignment, and curriculum learning. OSW-FT differs by tracking evolving model capability online while retaining expert-trajectory supervision.

  • Offline Reweighting and Regularization: Offline approaches reweight SFT examples or add KL-divergence regularization against a static reference model.Examples include SoftDedup, iw-SFT, DFT, and ASFT.
  • Offline Reweighting and Regularization: These offline methods operate without online generation, whereas OSW-FT tracks continuously evolving capabilities through real-time rollouts.OSW-FT is presented as bypassing static heuristics by operating online.
  • Online Rollout-Driven Alignment: Online rollout-driven alignment includes methods that train on self-generated correct rationales or optimize from model-generated trajectories.The related paradigms include STaR, GRPO, and CHORD.
  • Data Selection and Curriculum Learning: Curriculum learning traditionally orders examples by manually defined difficulty, while automatic curriculum learning adapts the task sequence.The passage situates data selection and curriculum learning as approaches to alignment and data efficiency.

6 Conclusion

The paper concludes that OSW-FT augments SFT with online trajectory-level weighting while preserving expert supervision. Across small-to-medium models, it improves over SFT and offers a favorable performance–rollout trade-off under limited online budgets.

  • 6 Conclusion: OSW-FT estimates current success rates with inference-only rollouts to adjust the update strength assigned to each expert trajectory.The optimization direction remains anchored to expert demonstrations.
  • 6 Conclusion: OSW-FT consistently improves over SFT on small-to-medium models and offers a favorable performance–rollout trade-off when online rollout budgets are limited.The conclusion frames this as the empirical outcome of the proposed method.

Limitations

OSW-FT is currently scoped to binary-verifiable reasoning with reliable verification and high-quality expert trajectories. Its experiments focus on Qwen3 models and primarily mathematical reasoning, leaving broader reward formulations for future work.

  • OSW-FT assumes a reliable binary verifier and high-quality expert trajectories.
  • The success-rate weighting has not been validated for continuous rewards or open-ended generation.
  • GPQA-Diamond remains a binary-verifiable multiple-choice task, so it does not establish applicability to non-binary settings.
  • Experiments focus on the Qwen3 model family and primarily mathematical reasoning tasks.
  • Extending OSW-FT to code generation and other reward formulations remains future work.

B Theoretical Proofs

The theoretical analysis assumes binary-verifiable rewards, with each query’s reward restricted to success or failure and its success probability well-defined.

  • The analysis uses p := ps(q) and g0(q) := ∇θ log πθ(o∗|q) for a fixed query satisfying Assumption 1.
  • For each query q, the reward R(o) is binary, taking values in {0, 1}.
  • The success probability ps(q) is defined as Eo∼πθ[R(o)|q] and lies in (0, 1).

B.1 Finite Rollout Analysis (Proof of Theorem 1)

The finite-rollout analysis characterizes OSW-FT’s Monte Carlo weighting estimator, showing unbiasedness for the surrogate update and quantifying signal loss and estimation error.

  • For K i.i.d. rollouts, the estimated success rate satisfies E[ˆp] = p and Var[ˆp] = p(1 − p)/K.
  • Eˆp[ˆgK(q)] = (1 − p)·g0(q), so the estimator is unbiased for the exact OSW-FT surrogate gradient.
  • The zero-weight event occurs exactly when all K rollouts are correct, namely when ˆp(q) = 1.
  • The finite-rollout second-moment expression includes a K-dependent term that vanishes as K →∞.
  • The estimation-induced mean-squared error follows from ˆgK − (1 − p)g0 = (p − ˆp)g0 and Var[ˆp] = p(1 − p)/K.

B.2 Convergence Analysis (Proof of Theorem 2)

The convergence analysis optimizes a sequence of surrogate objectives with weights fixed during inner updates and accounts for stochastic-gradient error and objective switching across outer iterations.

  • Surrogate objective: At outer iteration t, the OSW-FT surrogate objective uses weights computed from πθt and held fixed during inner-loop updates.
  • Surrogate objective: Differentiating the weight would add a ∇θpθ(q) term absent from the OSW-FT estimator, motivating stop-gradient treatment.
  • Regularity assumptions: The convergence proof assumes L-smooth objectives and a bounded second moment for the weighted stochastic gradient.
  • Convergence proof: Within each outer iteration, (1 − ˆp(q))g0(q) is an unbiased estimator of ∇θJ(t)(θt), enabling standard SGD analysis over S inner steps.
  • Convergence proof: Switching surrogate objectives across outer iterations introduces a mismatch whose cost accumulates over M = T/S transitions.
  • Convergence proof: The switching cost is bounded by ϵpB over training horizons satisfying T ≤ S^2.

C Unifying SFT and RL via General Policy Gradient

The paper places SFT and RL within a General Policy Gradient framework, showing SFT as a deterministic, zero-baseline update. It identifies the resulting missing dynamic weighting and derives OSW-FT by setting the baseline to the model’s current success probability.

  • SFT and RL are unified under the General Policy Gradient framework.
  • SFT maximizes JSFT = −LSFT while restricting trajectories to the deterministic ground-truth oracle with reward 1 and a zero baseline.
  • SFT gives trivial, already-mastered samples the same gradient coefficient as completely unsolved samples.
  • RL methods use dynamic baselines to reduce variance and scale updates according to the policy’s evolving capability.
  • OSW-FT sets the baseline to the model’s current success probability, restoring a dynamic advantage while preserving supervised-learning stability.
Loading 2609.00734v1…