Proxy OPD: On-Policy Distillation with Transferable Relative Proxy Update

• Reward-based policy optimization couples exploration to each model’s training, making high-reward behaviors hard to reuse; OPD [4] [5] supports cached specialized experts [7] but relies on distribution matching.Direct RL approaches such as PPO [2] and GRPO [3] use reward-guided on-policy rollouts, while joint heterogeneous-domain optimization may cause cross-domain interference.Source: §1 · pp. 1–2
• Unlike absolute expert matching, P-OPD needs only a useful improvement direction, allowing older or weaker proxies to provide effective signals even when target priors or capabilities differ.Absolute matching can yield diminishing supervision or constrain target behaviors that are already superior.Source: §1 · p. 2
• P-OPD optimizes a proxy, extracts its initial-to-optimized relative distributional changes, and transfers them through the target’s own on-policy trajectories while retaining the target policy as reference.The proxy can be explored independently of target training, and proxy pairs can be cached for transfer across model generations.Source: §1 · pp. 2–3
• P-OPD consistently enhances highly capable Qwen3-family targets on mathematical reasoning and code generation, while signal scaling controls transfer intensity and reuse across model variants and configurations.The introduction describes relative policy updates as reusable and scalable post-training assets.Source: §1 · pp. 2–3

2 Related Works

• Post-training methods are grouped into reward optimization and distribution matching to improve large language model training efficiency and performance.Source: §2 · p. 3

2.1 Reward Optimization

• Reward optimization uses reward-model feedback, with finer-grained step-wise or token-level signals and GRPO reducing overhead, but still requires on-policy sampling and trajectory evaluation.AI-generated feedback can reduce data costs, while most such methods remain PPO-like and require multiple models during training.Source: §2.1 · p. 3

2.2 Distribution Matching

• SFT, KD, and DPO align models with target distributions, while OPD matches teacher distributions on student-sampled trajectories to mitigate offline distribution shift.Distribution matching can provide more direct credit assignment and faster convergence, but relies on access to a stronger target distribution, which can carry substantial hidden costs.Source: §2.2 · p. 3
• The paper proposes asynchronously extracting, storing, and reusing training signals to decouple model optimization from reward exploration as strong target distributions become costlier to obtain.Source: §2.2 · p. 3

3 Preliminary Analysis: Reward Optimization vs. Distribution Matching

• The analysis frames policy exploration and policy alignment as complementary stages: reward optimization discovers useful improvement directions, and distribution matching transfers the resulting behaviors to the student.This distinction motivates decoupling, reusing, and efficiently transferring update signals across models.Source: §3 · pp. 3–4
• On Qwen3-1.7B, the comparison evaluates GRPO, standard OPD, OPD FT with filtered trajectories, and OPD URM masking updates that conflict with trajectory-level reward.GRPO is trained for 500 steps and also serves as the OPD expert teacher.Source: §3 · pp. 3–4
• OPD and its reward-informed variants converge substantially faster than GRPO, but all methods attain statistically comparable final performance, with neither FT nor URM significantly improving standard OPD.Figure 2 reports mean@16 trajectories on AIME24 and AIME25; its caption states that OPD variants converge within 70 steps.Source: Figure 2 · p. 3, §3 · p. 4
• The results indicate that reward-informed modifications add no effective learning signal beyond the teacher distribution, so OPD primarily transfers teacher behavior rather than independently discovering higher-reward policies.OPD’s efficiency comes from distribution matching once a target distribution is available; reward-driven exploration is still needed to obtain that target.Source: §3 · p. 4

4.1 Setup

• The framework distinguishes a proxy base and optimized expert policy from a frozen target base policy and a trainable target expert policy.The optimized proxy expert is produced during exploration; the target base remains the anchor for training.Source: §4.1 · p. 4

4.2 OPD as a Direct Baseline

• Direct OPD aligns the target expert to the proxy expert’s absolute distribution, requiring the proxy to encode consistently better token preferences for an already capable target.When the target is close to or surpasses the proxy in some states, direct imitation may offer little benefit or suppress better target preferences.Source: §4 · p. 4, §4.2 · p. 5

4.3 Stage 1: Proxy Exploration

• Proxy exploration uses reward optimization, such as PPO or GRPO, to produce a proxy expert that captures reward-preferred behavioral changes.The proxy serves as an exploration testbed, avoiding repeated costly reward optimization on the target; multiple proxies can be optimized in parallel under different rewards or data distributions.Source: §4.3 · p. 5

4.4 Stage 2: Update-Signal Extraction

• Update-signal extraction represents the proxy’s change from base to expert as a log-ratio, yielding a transferable direction rather than a proxy-specific target distribution.A positive scaling coefficient α controls transfer intensity: larger values amplify the proxy-discovered update, while smaller values make transfer more conservative.Source: §4.4 · p. 5

4 Methodology

• P-OPD replaces direct imitation of the proxy expert distribution with transfer of its relative, reward-induced update onto the target model’s own base-policy-centered space.This aims to avoid limiting supervision to the proxy’s absolute preferences when the target already has better token preferences in some states.Source: §4 · p. 4, §4.5 · p. 6

4.5 Stage 3: Signal Transfer

• Signal transfer encourages the target to follow the proxy update while regularizing changes relative to its frozen base; only the target expert is trained.The effective coefficient λ controls conservatism: larger λ penalizes deviation more strongly, while smaller λ permits more aggressive updates; the proxy policies and target base remain frozen.Source: §4.5 · pp. 5–6

5.1 Performance Improvement

• P-OPD improves Qwen3-8B across evaluated math and code benchmarks and beats OPD on most benchmarks and average performance in both domains.On AIME24, HMMT25 (Nov), and LCB, target gains even exceed those obtained during proxy exploration.Source: §5.1 · p. 6, Table 1 · p. 7

5.2.1 Cross-Scale and Sequential Transfer

• Update signals from Qwen3-1.7B and Qwen3-4B both transfer effectively to Qwen3-8B, with P-OPD average gains comparable to or greater than proxy exploration.OPD yields smaller improvements across all benchmarks than P-OPD.Source: §5.2.1 · p. 7, Table 2 · p. 7
• Sequentially transferring Qwen3-4B signals through Qwen3-1.7B preserves substantial gains on Qwen3-8B, approaching direct 4B→8B transfer.Both transfer stages use λ = 1.0.Source: §5.2.1 · p. 7, Table 2 · p. 7

5.2.2 Sensitivity to the Calibration Coefficient

• Fitted optimal calibration coefficients are λ∗= 1.08 for Qwen3-4B and λ∗= 1.57 for Qwen3-1.7B; Qwen3-4B reaches a higher peak score.Because proxy-target relationships may vary across sampled states, the paper identifies adaptive, potentially state-dependent calibration as future work.Source: §5.2.2 · p. 8, Figure 4 · p. 8

5.2.3 Cross-Generation Reusability

• Older-generation Qwen2.5-7B-derived signals improve Qwen3-8B more than direct Qwen3-8B exploration, while OPD performs substantially worse than P-OPD.The older proxy expert is Skywork-OR1-Math-7B; the comparison current-generation expert is Qwen3-8B (RL).Source: §5.2.3 · p. 8, Table 3 · p. 8

5.3 Comparison of Exploration Quality

• All P-OPD runs converge within 70 steps; with 1,200 exploration steps, Qwen3-4B transfer matches or exceeds direct Qwen3-8B exploration.Under similar budgets of 400 target steps and 500 proxy steps, Qwen3-8B explores higher-quality signals; the omitted 4B-RL1200→8B run converges slightly above GRPO (8B-RL400).Source: §5.3 · pp. 8–9, Figure 5 · p. 9
• As sampling budgets increase, smaller proxies uncover additional useful solutions and can match larger-model exploration quality, enabling signals to be reused across transfers.The paper frames this as an “explore once, transfer multiple times” workflow for more efficient post-training.Source: §5.3 · p. 9, Figure 5 · p. 9

• P-OPD consistently improves strong targets on mathematical reasoning and code generation, generally outperforming conventional OPD.Source: §6 · p. 9
• P-OPD transfers reward-induced relative policy updates instead of matching absolute expert distributions, decoupling proxy exploration from target-model training.The resulting update signals can be reused across models.Source: §6 · p. 9
• Signals transfer across model scales and generations, remain useful after sequential transfer, and can be adjusted with a calibration coefficient.A larger exploration budget lets smaller proxies discover signals comparable to those explored directly by larger models, supporting an “explore once, transfer multiple times” pipeline.Source: §6 · p. 9

7 Discussion and Future Works

• P-OPD separates post-training into extracting environmental data as a transferable signal and applying that signal to a target primary model.This framing is illustrated in Fig. 6.Source: §7 · p. 9

Update-Signal Extraction

• Proxy reward optimization compresses task-specific information into reusable improvement signals, but signal quality is bounded by data, exploration strategy, and aggregation.Multi-model parallel exploration or ensemble multi-source voting could raise this upper bound, enabling “compressing once and reusing many times.”Source: §7 · p. 10

Adaptive Signal Transfer

• Signal transfer enables robust cross-model reuse, but proxy–primary distribution gaps and changing trajectory properties challenge transfer efficiency.Trajectory-adaptive calibration could adjust transfer intensity to local distributional relationships such as logit overlap and local entropy.Source: §7 · p. 10

7 Discussion and Future Works

• The framework demonstrates the feasibility of asynchronous exploration and reusable update-signal transfer, with stronger proxy exploration and more sophisticated adaptive calibration left for future work.Source: §7 · p. 10

A.1 Implementation Details

• Experiments use non-thinking Qwen3 models at 1.7B, 4B, and 8B scales, implemented with verl and trained on 8×NVIDIA A100 80GB.The paper identifies Qwen3 [9] and verl [12].Source: §A.1 · p. 14
• Proxy experts are trained with GRPO on DeepMath-103K for math and Eurus-RL-Code for code, using verifiable binary rewards for correctness.The math data is filtered following prior work; rewards are 1 for correct answers and 0 otherwise. [31] [32]Source: §A.1 · p. 14
• P-OPD signal-transfer hyperparameters are summarized in Table 5, and calibration coefficient λ is set as described in Section 5.3.The table covers signal transfer in both math and code domains.Source: §A.1 · p. 14, Table 5 · p. 14
• Math evaluation reports Mean@16 across eight datasets; the main text presents four representative benchmarks, with full results for all eight in the appendix.The suite includes AIME24, AIME25, AIME26, CMIMC25, HMMT25(Feb) [38], HMMT25(Nov) [38], HMMT26(Feb), and SMT25 [31].Source: §A.1 · p. 14
• Code evaluation reports Mean@8 on HumanEval+ [39, 40], MBPP+ [41], and LCB [33], following the evaluation scripts of [28].Source: §A.1 · p. 14
• Proxy experts are trained to full convergence rather than a fixed budget, with Table 4 hyperparameters kept consistent across proxy model scales.This is intended to ensure reliable, high-quality transferred update signals.Source: Table 4 · p. 14, §A.1 · p. 15
• For Qwen3-1.7B and Qwen3-8B proxies, gradient norms stabilize near 0.03 and response lengths gradually saturate during GRPO training.The paper interprets these training dynamics as evidence of convergence and stable reasoning behavior.Source: §A.1 · p. 15

A.2 Effect of the Calibration Coefficient λ

• A smaller λ follows the proxy-induced direction more aggressively, while a larger λ yields more conservative transfer.The study sweeps λ for an 8B target guided by either a 4B or 1.7B proxy, reporting validation accuracy, policy entropy, gradient norm, and training loss.Source: §A.2 · p. 15
• In 4B→8B transfer, λ = 0.5 peaks early then degrades, and λ = 0 collapses performance; λ ∈{0.75, 0.95, 1.0, 1.5} remains stable and converges to higher accuracy.The smallest coefficient also produces the lowest entropy, indicating premature over-updating.Source: §A.2 · p. 15, Table 6 · p. 15, Figure 8 · p. 16
• In 1.7B→8B transfer, λ = 1.5 achieves the best and most stable accuracy, while λ = 0.5 causes collapse and overall accuracy is lower than with the 4B proxy.The larger capability gap requires stronger calibration to regularize the noisier proxy direction.Source: §A.2 · p. 15, Table 7 · p. 15, Figure 9 · p. 16
• Across both proxy scales, overly aggressive transfer (small λ) consistently hurts performance, and optimal calibration increases as the proxy–target capability gap widens.Training loss and gradient norm nevertheless converge stably for all λ.Source: §A.2 · p. 15