PaperlayerView exact revision on arXiv

Source-linked AI summary

On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training

Shufan Shen, Zhongni Hou, Junshu Sun, Yufei Zhang, Wei Lin, Guojun Yin, Qingming Huang, Shuhui Wang

arXiv:2609.36659v1cs.LG

TL;DR

Prior studies of on-policy parameter updates have largely treated them as byproducts, leaving open whether those behaviors could improve SFT generalization. The paper analyzes changing update directions and introduces OPSFT to constrain SFT updates to an on-policy direction; OPSFT outperforms vanilla SFT and reaches performance comparable to on-policy training.

  • Problem

    Prior studies treated on-policy parameter update behaviors as byproducts rather than evaluating them as optimization principles for improving SFT generalization.

    Source: §1 · p. 2

  • Method

    The paper analyzes on-policy update directions and proposes OPSFT, which constrains SFT parameter updates to a direction identified by on-policy training.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    OPSFT substantially outperforms vanilla SFT and achieves performance comparable to GRPO across benchmarks and model scales.

    Source: §3.3 · p. 5

  • Takeaways & Limitations

    A few on-policy steps can identify a direction for efficient OPSFT, and OPSFT can use high-quality trajectories to continue training an on-policy-post-trained model without disrupting learned capabilities.

    Source: §1 · p. 2

    Experiments use high-quality reasoning trajectories from teacher models substantially stronger than the post-trained models, and limited trajectory-quality measurements prevent explicitly accounting for trajectory quality and suitability.

    Source: §C.2 · pp. 16–17

Abstract

from arXiv · show

The strong generalization performance of on-policy post-training paradigms has motivated studies of their parameter update behaviors. However, these studies treat the observed behaviors only as byproducts in on-policy training, overlooking their potential to serve as optimization principles for improving the generalization of other paradigms such as supervised fine-tuning (SFT). To address this limitation, we investigate whether there exists a specific on-policy update behavior that can achieve such improvements. First, our analyses reveal that SFT updates parameters along consistent directions, while the on-policy paradigm continuously adjusts the direction during training. This difference inspires us to focus on the cumulative update direction of each parameter as a promising behavior. Then, we evaluate its effectiveness for improving generalization by proposing On-Policy direction-constrained Supervised Fine-Tuning (OPSFT), which constrains SFT updates to the direction identified by on-policy paradigms. The strong performance of OPSFT indicates that the generalization advantage of on-policy paradigms can be transferred to SFT through the parameter update direction. Once such a direction is identified, even SFT can generalize with its updates constrained to this direction. This finding offers two practical benefits by combining the strong generalization of on-policy paradigms with the advantages of SFT, including the high training efficiency and ability to leverage high-quality trajectories. For efficiency, we identify update directions that support strong generalization using a few on-policy training steps, and subsequently apply OPSFT to achieve high training efficiency. For leveraging high-quality trajectories, OPSFT can utilize these trajectories to continue improving a post-trained model along its update direction without disrupting the ability learned from on-policy training.

1 Introduction

On-policy training generalizes strongly, but prior analyses of its parameter updates largely treated their behavior as a byproduct rather than an optimization principle for SFT. The paper studies update direction and introduces OPSFT, which constrains SFT updates to an on-policy direction and achieves performance comparable to on-policy training.

1 Introduction

  • Prior work examined on-policy update behaviors such as sparse update locations and small spectrum shifts, but treated them as training byproducts.Component-wise comparisons are costly and fragmented because on-policy paradigms and SFT differ in numerous components.Source: §1 · pp. 1–2
  • OPSFT substantially outperforms vanilla SFT while achieving performance comparable to the on-policy paradigm.The paper reports experiments across different models and benchmarks.Source: §1 · p. 2
  • On-policy training continuously adjusts parameter update directions, unlike SFT, which updates along consistent directions.This motivates examining the cumulative on-policy update direction as a potential optimization behavior for SFT.Source: §1 · p. 2
  • OPSFT retains SFT while constraining each parameter's update direction to the direction identified by on-policy training.The paper evaluates whether that direction improves SFT generalization by comparing OPSFT with vanilla SFT and on-policy paradigms.Source: §1 · p. 2
  • A few on-policy training steps can identify the direction for efficient OPSFT, while high-quality trajectories can continue improving an on-policy-post-trained model along that direction.The paper states that this continuation avoids disrupting learned capabilities and repeating the full SFT-then-RL process.Source: §1 · p. 2

2 Related Work

Related work examines on-policy optimization behavior, parameter localization, and ways to improve SFT generalization. This paper distinguishes its approach by using on-policy update directions to guide SFT.

Optimization Behavior of On-Policy Post-Training

  • Prior studies examine optimization geometry, training-objective effects, and localized parameter updates in on-policy training; this work extends the analysis from update locations to directions.The paper states that these directions can transfer on-policy generalization advantage to SFT.Source: §2 · p. 3

Task-Relevant Parameter Localization

  • Task-relevant parameter methods estimate which parameters to adjust, whereas this work uses on-policy training to identify both relevant update locations and directions.The prior approaches described rely on heuristic strategies under off-policy paradigms.Source: §2 · p. 3

Generalization Ability of Supervised Fine-tuning

  • Existing SFT improvements use data selection, logit-weighted losses, or negative samples; this work instead uses on-policy update directions to improve SFT generalization.The paper contrasts its approach with methods that imitate on-policy training through objectives or dataset construction.Source: §2 · p. 3

3 On-Policy Update Direction Underlies Generalization

The section examines whether on-policy update behavior can improve SFT generalization and tests constraining SFT to cumulative on-policy update directions.

3.1 Analysis of Parameter Update Directions

  • SFT follows consistent update directions, whereas on-policy directions change with evolving advantage distributions; on-policy interval updates are nearly orthogonal across stages.Cumulative update cosine similarity is close to 1.0 for SFT and around 0.5 for on-policy methods.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Applying On-Policy Update Directions to SFT

  • OPSFT derives a sign vector from the before-and-after on-policy parameters and retains SFT gradient elements whose signs match that direction.The constrained gradient is passed to the optimizer so updates follow the identified on-policy directions.Source: §3.2 · p. 4

3.3 Experimental Results

  • OPSFT substantially improves over vanilla SFT and reaches performance comparable to GRPO, while location-only constraints perform worse than vanilla SFT.The comparison supports update direction, rather than update location alone, as the transferable behavior.Source: §3.3 · p. 5, Figure 3 · p. 5
  • On out-of-domain benchmarks, OPSFT consistently outperforms vanilla SFT and slightly surpasses GRPO when trained on DeepMath.The reported results cover different benchmarks and model scales.Source: Table 1 · p. 5, §3.3 · p. 6
  • Update directions can be reused across datasets within a domain, but Eurus code-domain directions yield weak OPSFT performance on math.DAPO-derived directions slightly outperform DeepMath-derived directions on DeepMath despite cosine similarity of only around 0.2.Source: §3.3 · p. 6, Figure 4 · p. 6

4 Applications of On-Policy Update Direction

OPSFT uses an on-policy update direction to improve SFT’s generalization, either after a few GRPO steps or when continuing to train an already post-trained model. Ablations examine update behavior, reasoning patterns, parameter precision, and the choice of direction-identification stage.

4.1 Training Efficiency Improvement

  • 41.67 vs. 40.31 mean accuracy on Qwen3-8B, with OPSFT taking 8.9h versus GRPO’s 19.3h; code-task results were consistent with the math results.OPSFT uses 50 GRPO steps to identify a direction, followed by 100 OPSFT steps; it also outperforms SFT and DFT while requiring less training time.Source: §4.1 · p. 7, Table 2 · p. 7, Table 4 · p. 8

4.2 Improve Models After On-Policy Post-Training

  • Qwen3-4B mean accuracy rose from 38.96 to 41.36 with OPSFT, while direct SFT after GRPO lowered it to 36.25.The code-task results were consistent with the math results; OPSFT is presented as a way to incrementally improve post-trained models using accumulated high-quality trajectories.Source: §4.2 · p. 8, Table 3 · p. 8, Table 5 · p. 8

4.3 Ablation Studies and Analysis

  • Later-stage GRPO directions consistently improved OPSFT generalization, but the gain from 50 to 100 steps exceeded the gain from 100 to 150 steps.The marginal benefit of using later-stage directions gradually diminished as GRPO training progressed.Source: §4.3 · p. 8
  • Under BF16, OPSFT achieved 40.94% versus vanilla SFT’s 38.33% while updating 0.408% of parameters; under FP32, it achieved 42.81% versus GRPO’s 42.01%.Under FP32, OPSFT updated 9.45% of parameters, compared with approximately 90% for vanilla SFT.Source: §4.3 · pp. 8–9, Table 6 · p. 9
  • OPSFT updates are dense within the parameter subset selected by the on-policy direction, and its early and later matrix-level update directions differ substantially.This behavior suggests the parameter-level sign constraint redirects SFT toward a matrix-level direction conducive to generalization.Source: Figure 6 · p. 7, §4.3 · p. 9
  • OPSFT and GRPO have similar response lengths, while OPSFT’s accuracy standard deviation and reasoning cues more closely resemble GRPO’s than vanilla SFT’s.The comparison includes case-splitting and logical cues; the findings indicate the direction constraint does not simply induce longer reasoning.Source: §4.3 · p. 9, Figure 7 · p. 9

5 Conclusion

The paper finds that on-policy paradigms continually adjust parameter update directions, and proposes OPSFT to constrain SFT updates to an on-policy direction. Experiments show OPSFT reaches performance comparable to the corresponding on-policy paradigm, supporting update direction as an optimization principle for improving SFT generalization.

5 Conclusion

  • OPSFT constrains SFT parameter updates to an on-policy direction, and extensive experiments show performance comparable to the corresponding on-policy paradigm.The paper identifies efficient training and use of high-quality trajectories as potential advantages of combining OPSFT with SFT.Source: §5 · p. 9

A.1 Training Datasets

The appendix describes math and code datasets for on-policy training and SFT, including selection and verification procedures for examples.

A.1 Training Datasets

  • The math RL data contains 57K DeepMath samples with difficulty at least 6, and the code RL data contains 25K Eurus-RL-Code samples.The datasets are used for on-policy paradigms.Source: §A.1 · p. 13
  • Math SFT trajectories come from DeepMath prompts completed by Qwen3-30B-A3B-Instruct-2507 and are retained only when the mathematical answer verifier judges them correct.The resulting math dataset contains 44,810 training examples and 914 examples.Source: §A.1 · p. 13
  • Code SFT uses the same model and sampling protocol, retaining a response only when its generated program passes the reference tests; the final Eurus code dataset has 9,585 training examples.Each response must contain a Python code block, and execution metadata is stored with the prompt, response, and ground truth.Source: §A.1 · p. 13

A.2 Training Settings

The appendix specifies GRPO rewards and shared SFT/OPSFT training settings, including the two-level OPSFT direction constraint.

A.2 Training Settings

  • GRPO assigns reward 1.0 for a correct math answer or code passing all unit tests, and 0.0 otherwise.Math and code training hyperparameters are listed in Tables A7 and A8, respectively.Source: §A.2 · p. 13
  • SFT and OPSFT use cross-entropy with FP32 training by default, and OPSFT masks gradients and restores parameters to prevent updates opposite to the cumulative GRPO direction.The experiments use AdamW, 8 NVIDIA H20 GPUs, a global batch size of 64, and 700 optimizer steps unless otherwise specified.Source: §A.2 · p. 13

A.3 Evaluation Details

Evaluation measures average accuracy across math and code benchmarks, while the OPSFT pseudocode constrains SFT updates using on-policy update locations and directions.

A.3 Evaluation Details

  • Evaluation averages accuracy over four math and three code benchmarks, sampling eight solutions per math problem and four per code problem.Generation uses temperature 1.0, top-p 1.0, and a maximum length of 16,384; Math-Verify checks math answers.Source: §A.3 · p. 14

A.4 Pseudo Code of OPSFT

  • OPSFT computes a standard SFT gradient, constrains its support and movement direction, then projects post-update parameters to preserve those constraints.The pseudocode defines M as the on-policy update-location mask and D as the element-wise update direction.Source: §A.4 · p. 14
  • Table A9 reports method accuracy for Qwen3-1.7B on DeepMath, while Table A10 reports its training trajectories using Acc@8 (%).Source: Table A9 · p. 15, Table A10 · p. 15

B Additional Experiments

Additional experiments test OPSFT with OPD, ablate its location and direction constraints, and compare its weight-update spectra with on-policy training. They also distinguish OPSFT from prior gradient-projection methods.

B Additional Experiments

  • With Qwen3-30B-A3B teacher and Qwen3-1.7B student trained for 150 steps, OPSFT with OPD location and direction constraints substantially exceeded vanilla SFT: 16.71% vs. 13.83% mean accuracy, similar to OPD at 16.75%.Source: B Additional Experiments · p. 15
  • On-policy update locations outperform random locations, and on-policy directions outperform random direction constraints, indicating gains beyond gradient masking alone.The complete location-and-direction constraints achieve the best performance among the ablations.Source: B Additional Experiments · p. 15
  • OPSFT updates have more evenly distributed singular values, higher stable rank, and smaller principal-singular-vector rotations than vanilla SFT, resembling GRPO.The spectral similarities suggest these properties may follow from the on-policy update direction rather than being intrinsic to on-policy training.Source: B Additional Experiments · p. 16

C.1 Discussions with Gradient Projection Methods

  • Unlike prior projection methods that derive subspaces by decomposing SFT gradients, OPSFT takes its update direction from on-policy training and targets generalization as well as efficiency.The paper presents gradient projection as a means of transferring on-policy generalization to SFT.Source: §C.1 · p. 16

C.2 Limitations

The experiments use high-quality reasoning trajectories from stronger teacher models, and the study does not explicitly account for trajectory properties that may vary across collection settings.

C.2 Limitations

  • Experiments use high-quality reasoning trajectories generated by teachers substantially stronger than the model being post-trained.Source: §C.2 · p. 16
  • Trajectory diversity, correctness, and target suitability are not explicitly accounted for because research measuring these properties remains limited.The authors leave their effects on generalization and OPSFT update directions for future work.Source: §C.2 · p. 17

Related papers

Loading 2609.36659v1…