PaperlayerView exact revision on arXiv

Source-linked AI summary

Proxy OPD: On-Policy Distillation with Transferable Relative Proxy Update

Daocheng Fu, Rong Wu, Yu Yang, Jianbiao Mei, Licheng Wen, Pinlong Cai, Xuemeng Yang, Yong Liu, Botian Shi, Yu Qiao

arXiv:2607.11505v2cs.LGcs.AI

TL;DR

Existing post-training methods make high-reward behaviors difficult to reuse and can provide weak supervision when matching absolute expert distributions. Proxy OPD transfers reward-induced relative policy updates instead, consistently improving strong targets across mathematical reasoning and code generation while supporting adjustable, reusable signals.

  • Problem

    On-policy reinforcement learning couples policy exploration with model training, while absolute-distribution distillation can become unsuitable when target and expert capabilities or priors differ.

  • Method

    Proxy OPD optimizes a proxy with reward feedback, extracts its relative policy change, and transfers that update through the target model’s on-policy trajectories.

  • Results

    Across mathematical reasoning and code generation, P-OPD consistently improves strong target models and generally outperforms conventional OPD, with transfer adjustable by calibration.

  • Takeaways & Limitations

    Relative policy updates can serve as reusable, adjustable signals for reward-based post-training across model scales, generations, variants, and configurations.

  • Takeaways & Limitations

    Transfer efficiency depends on distributional gaps and trajectory-dependent relationships, motivating adaptive, potentially state-dependent calibration beyond a constant coefficient.

Abstract

from arXiv · show

Post-training for large language models typically couples policy exploration with model optimization, hindering the reuse of high-reward behaviors from policy exploration. While on-policy distillation alleviates this by consolidating independently optimized experts, its reliance on matching absolute expert distributions can yield suboptimal supervision, especially when the target model possesses a different prior or already surpasses the expert's capabilities. To alleviate this, we introduce Proxy OPD (P-OPD), an asynchronous post-training framework that transfers reward-induced policy improvements rather than absolute policy distributions. P-OPD first optimizes a proxy policy via reward feedback. It then extracts the relative distributional changes between the proxy's initial and optimized states, transferring these directional updates through the target model's own on-policy trajectories while retaining the target policy as the reference. This decoupled formulation requires the proxy to provide merely a useful direction of improvement rather than superior absolute capability, enabling update signals from older or weaker proxies to remain highly effective. Systematic experiments on Qwen3-family models across mathematical reasoning and code generation demonstrate that P-OPD consistently enhances already strong target models. Furthermore, transfer intensity can be dynamically modulated through signal scaling, making the extracted update signals seamlessly reusable across diverse model variants and training configurations. These results establish relative policy updates as highly reusable, adjustable assets for scalable, reward-based post-training.

1 Introduction

Proxy OPD addresses the limitations of absolute-distribution matching in on-policy distillation by transferring reward-induced relative policy improvements from independently explored proxy models. Across Qwen3-family reasoning and code-generation tasks, it improves capable targets while allowing transfer strength and update signals to be reused.

  • Motivation: Direct policy-based RL couples policy exploration with training each individual model, requiring new trajectory collection and reward optimization when adapting a new model.The limitation applies to approaches such as PPO and GRPO that optimize models using on-policy rollouts and reward feedback.
  • Existing Approach: On-policy distillation separates policy exploration from alignment, enabling parallel expert construction, reduced optimization interference, and cached expert reuse.Experts are independently optimized with task-specific data and reward signals before target alignment through distribution matching.
  • Motivation: Absolute expert-distribution matching can weaken supervision or constrain the target when it approaches or surpasses the expert or has a substantially different prior distribution.Its effectiveness depends on whether the expert distribution remains a suitable supervision target for the target model.
  • Method: Proxy OPD extracts reward-induced relative distributional changes from a proxy policy and transfers them through the target model’s own on-policy trajectories instead of matching absolute expert distributions.The framework comprises proxy exploration, update-signal extraction, and signal transfer.
  • Results: P-OPD consistently enhances highly capable Qwen3-family targets on mathematical reasoning and code generation, while signal scaling controls transfer intensity and supports reuse across model variants.The results indicate that policy-improvement signals remain transferable beyond their source policies.

2 Related Works

Post-training methods are organized into reward optimization and distribution matching, each relying on high-quality target distributions. Reward methods refine feedback granularity, while distribution matching aligns models with hard, soft, implicit, or teacher distributions but can face data-cost and distribution-shift challenges.

  • Overview: Post-training methods are grouped into reward optimization and distribution matching to improve training efficiency and model performance.This taxonomy frames the representative approaches reviewed in the section.
  • Reward Optimization: Reward-optimization methods improve policies with reward-model feedback, increasingly using step-wise and token-level rewards for more precise supervision.Training reliable reward models still requires large-scale, high-quality data, while AI-generated feedback can reduce this cost.
  • Distribution Matching: Distribution-matching methods align models with hard or soft targets through SFT and KD, or construct implicit target distributions from preference pairs through DPO.These offline methods are efficient but may suffer from distribution shift.
  • Distribution Matching: OPD reduces distribution shift by matching teacher distributions on trajectories sampled from the student.OPD belongs to the distribution-matching family rather than explicit reward optimization.
  • Motivation: Both reward optimization and distribution matching depend on high-quality target distributions, whose acquisition becomes increasingly costly as base models strengthen.The proposed scalability direction is to decouple model optimization from reward exploration by asynchronously extracting, storing, and reusing training signals.

3 Preliminary Analysis: Reward Optimization vs. Distribution Matching

The analysis distinguishes reward optimization as a means of discovering high-reward policy improvements from distribution matching as an efficient means of transferring them. In experiments, OPD variants converged faster than GRPO without improving final performance, motivating relative transfer of reward-induced directions.

  • Complementary roles: Reward optimization explores high-reward behaviors, whereas distribution matching aligns a model with a target distribution; GRPO and OPD represent these complementary roles.The comparison uses GRPO as a reward-optimization baseline and OPD as a distribution-matching method.
  • Training comparison: OPD and its reward-informed variants converge substantially faster than GRPO, while all methods achieve statistically comparable final performance.OPD variants converge within 70 steps and are extended to 500 steps based on converged statistics.
  • Training comparison: Neither OPD FT nor OPD URM significantly improves on standard OPD, indicating that the teacher distribution already encodes the effective learning signal.OPD FT filters trajectories, while OPD URM masks token updates whose directions conflict with the trajectory-level reward signal.
  • Motivation: These findings motivate transferring reward-induced policy-improvement directions rather than requiring distribution matching to a superior absolute target distribution.Reward optimization discovers useful improvement directions, while distribution matching transfers the resulting behaviors to the student.

4 Methodology

P-OPD decouples reward exploration from target-model training by optimizing a proxy, extracting its reward-induced relative update, and transferring that direction through the target model’s own base-policy-centered space. The method replaces absolute proxy-distribution imitation with a calibrated signal-transfer objective that balances following the proxy direction against conserving the target policy.

  • Framework overview: P-OPD uses three stages: proxy exploration, update-signal extraction, and signal transfer.The proxy model discovers reward-preferred behavioral changes, which are then transferred to the target model.
  • Proxy exploration: Proxy exploration optimizes the proxy base policy with reward algorithms such as PPO or GRPO, avoiding repeated costly reward optimization on the target model.Multiple proxy policies can be optimized under different rewards or data distributions, enabling diverse update signals for later reuse.
  • Update-signal extraction: The extracted proxy signal is a log-ratio representing a transferable update direction rather than an absolute proxy-specific target distribution.A scaling coefficient α > 0 controls transfer intensity: larger α amplifies the update, while smaller α produces more conservative transfer.
  • Signal transfer: Signal transfer rewards target tokens favored by the proxy update while penalizing changes relative to the frozen target base policy.After calibration, λ is the effective conservatism coefficient: larger λ yields more conservative updates, whereas smaller λ permits more aggressive adoption of the proxy direction.
  • Signal transfer: During transfer, the proxy base, proxy expert, and target base policies remain frozen, and gradients update only the target expert policy.This preserves OPD’s online distribution-matching structure while replacing direct proxy-expert imitation with relative update-signal transfer.

5 Experiments

Experiments across mathematics and code show that P-OPD transfers relative update signals effectively to stronger targets, across model scales and generations. The method also supports sequential reuse and tunable signal strength, while proxy exploration quality depends on model scale and budget.

  • 5.1 Update Signal Transfer: P-OPD consistently improves Qwen3-8B across mathematical and code benchmarks, while outperforming OPD’s absolute-distribution alignment.On AIME24, HMMT25 (Nov), and LCB, target gains even exceed those obtained during proxy exploration.
  • 5.2 Cross-Scale Transfer: Relative update signals from both Qwen3-1.7B and Qwen3-4B proxies transfer effectively to Qwen3-8B, with P-OPD gains comparable to or exceeding proxy-exploration gains.OPD consistently produces smaller improvements across all benchmarks.
  • 5.2 Cross-Scale Transfer: Sequential transfer from Qwen3-4B through Qwen3-1.7B to Qwen3-8B preserves substantial gains, approaching the performance of direct 4B→8B transfer.Both transfer stages use λ = 1.0.
  • 5.2 Calibration Sensitivity: λ∗= 1.08 for Qwen3-4B and λ∗= 1.57 for Qwen3-1.7B are the fitted optimal calibration coefficients, with Qwen3-4B attaining the higher peak score.The results indicate that proxy scale affects update-signal quality, and state-dependent calibration remains future work.
  • 5.3 Cross-Generation Reusability: Update signals from the older-generation Qwen2.5-7B proxy effectively improve Qwen3-8B and outperform direct Qwen3-8B exploration, whereas OPD performs substantially worse.The experiment uses Skywork-OR1-Math-7B as the proxy expert derived from Qwen2.5-7B.
  • 5.3 Target- versus Proxy-Explored Signals: With 1,200 exploration steps, Qwen3-4B transfers performance comparable to or better than direct Qwen3-8B exploration, while smaller proxies discover increasingly useful signals as budgets grow.Under similar budgets, smaller proxies initially produce lower-quality signals, but transfer converges faster than direct exploration.

6 Conclusion

Proxy OPD transfers reward-induced relative policy updates rather than absolute expert distributions, decoupling proxy exploration from target training. Experiments show that it improves strong target models and generally outperforms conventional OPD.

  • P-OPD transfers reward-induced relative policy updates instead of matching an expert’s absolute output distribution.This design decouples proxy exploration from target-model training.
  • The resulting update signals can be reused across models.
  • P-OPD consistently improves strong target models in mathematical reasoning and code generation and generally outperforms conventional OPD.

7 Discussion and Future Works

P-OPD reframes post-training as extracting environmental data into transferable update signals and applying them to a target model. Its main future directions are stronger proxy exploration and trajectory-adaptive calibration to address signal-quality limits and proxy–target distribution gaps.

  • Framework Perspective: P-OPD separates post-training into extracting environmental data into a transferable signal and applying that signal to a target primary model.This abstraction comprises two sequential processes: signal extraction followed by signal transfer.
  • Update-Signal Extraction: Proxy reward optimization compresses task-specific information from data into reusable distributional improvement signals.Signal quality is bounded by the data distribution, proxy exploration strategy, and signal aggregation mechanism.
  • Future Works: Future work should strengthen proxy exploration and develop trajectory-adaptive calibration mechanisms for dynamically adjusting transfer.Suggested exploration improvements include multi-model parallel exploration and ensemble multi-source voting.
  • Adaptive Signal Transfer: Signal transfer performs policy optimization on the primary model, but proxy–primary distributional gaps and trajectory dynamics affect transfer efficiency.Logit overlap and local entropy can change transfer outcomes, while a constant calibration coefficient provides a baseline.

Appendix · A Experiment Details · A.1 Implementation Details

The appendix details P-OPD experiments on non-thinking Qwen3 models, including proxy training, signal transfer, evaluation, and convergence checks. Experiments use GRPO with verifiable rewards and transfer updates under specified training configurations.

  • A.1 Implementation Details: Experiments use non-thinking Qwen3 models at 1.7B, 4B, and 8B scales, implemented in verl and trained on 8×NVIDIA A100 80GB.These settings apply across the reported experiments.
  • A.1 Implementation Details: Proxy experts are trained with GRPO until convergence on filtered DeepMath-103K for math and Eurus-RL-Code for code, using rewards of 1 for correct answers and 0 otherwise.The complete GRPO hyper-parameters are provided in Table 4 and kept consistent across proxy model scales.
  • A.1 Implementation Details: P-OPD signal transfer follows the training hyper-parameters summarized in Table 5, with calibration coefficient λ set according to Section 5.3.The table covers signal transfer in both math and code domains.
  • A.1 Implementation Details: Code evaluation covers HumanEval+, MBPP+, and LCB, reports Mean@8, and strictly follows the evaluation scripts of the cited prior work.The appendix provides the associated benchmark-evaluation details.
  • A.1 Implementation Details: Proxy training runs to full convergence rather than a fixed budget to ensure reliable, high-quality transferred update signals.This convergence policy is used for each proxy expert, with consistent hyper-parameters across scales.
  • A.1 Implementation Details: The Qwen3-1.7B and Qwen3-8B proxy experts show stable GRPO dynamics, with later gradient norms decreasing to and stabilizing near ∼0.03.Response length also grows and gradually saturates during training.

A.2 Effect of the Calibration Coefficient λ

The calibration coefficient λ controls transfer aggressiveness: smaller values follow proxy-induced directions more strongly, while larger values produce more conservative updates. Across 4B→8B and 1.7B→8B transfers, overly aggressive settings hurt performance, and the optimal calibration increases with the proxy–target capability gap.

  • Calibration mechanism: λ controls how strongly the target compensates for already absorbed updates, with smaller values enabling more aggressive transfer and larger values yielding conservative transfer.The coefficient is evaluated using a shared 8B target and 4B or 1.7B proxy experts.
  • 4B→8B transfer: In 4B→8B transfer, λ = 0.5 is too aggressive, causing early validation-accuracy degradation and the lowest entropy, while λ = 0 causes complete performance collapse.The smallest coefficient produces premature over-updating, and the compensation coefficient at λ = 0 is likely too low to prevent over-updating.
  • 1.7B→8B transfer: In 1.7B→8B transfer, λ = 1.5 achieves the best and most stable accuracy, whereas λ = 0.5 directly causes performance collapse.Accuracy is lower overall than with the 4B proxy, consistent with the larger capability gap between proxy and target.
  • Overall findings: Across both settings, training loss and gradient norm converge stably for all λ, while overly aggressive transfer consistently harms performance.The optimal calibration degree increases as the proxy–target capability gap widens.
Loading 2607.11505v2…