PaperlayerView exact revision on arXiv

Source-linked AI summary

MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback

Xuyang Zhong, Jiuding Yang, Yiwen Guo, Chen Liu

arXiv:2610.30873v1cs.LGcs.AI

TL;DR

On-policy RL can be constrained by a model’s existing knowledge, while static expert supervision may suppress self-exploration and expert-prefix completions lack correctness guarantees. MOER anneals expert prefixes, rolls failed completions back to full expert traces, and harmonizes the token objectives; it reports gains over strong baselines across math and out-of-distribution general reasoning benchmarks.

  • Problem

    On-policy RL can be constrained by a model’s existing knowledge, while static expert supervision may suppress self-exploration and expert-prefix completions lack correctness guarantees.

    Source: §1 · p. 1

  • Method

    MOER progressively shortens expert CoT prefixes, rolls failed completions back to full expert traces, and uses distinct objectives for on-policy and off-policy tokens.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    +0.5 points on six math benchmarks and +1.6 points on out-of-distribution general reasoning tasks over the strongest baselines; MOER reports results across diverse tasks, model scales, and base-model categories.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    MOER achieves strong general-domain task utility while acquiring specialized skills, according to its reported benchmark comparisons.

    Source: §5.2 · p. 7

Abstract

from arXiv · show

Mix-policy optimization incorporates off-policy expert data into the reinforcement learning (RL) loop to transcend the capability boundary of on-policy RL. However, existing methods either provide static full-trace guidance that suppresses self-exploration, or adopt expert prefix guidance without correctness guarantees, leading to suboptimal performance or even training collapse. In this paper, we propose MOER, a mix-policy optimization framework that harmonizes expert guidance with self-exploration. MOER injects a decaying expert chain-of-thought prefix into the prompt and lets the model complete the remaining reasoning, with the prefix length progressively annealed to encourage self-exploration. A selective rollback mechanism replaces failed completions with the original full expert trace, guaranteeing the correctness of every off-policy trajectory. Furthermore, a harmonized mix-policy objective optimizes on-policy tokens with a GRPO loss while supervising both expert prefix and completion tokens with a trust-region SFT loss, substantially stabilizing training. Extensive experiments show that MOER achieves the highest average scores in mathematical and general domain reasoning tasks, outperforming all basic and mix-policy baselines, and generalizes consistently across diverse tasks, model scales, and base model categories. Codes are available at https://github.com/CityU-MLO/MOER.

1 INTRODUCTION

MOER addresses limitations in mix-policy RL by combining progressively shortened expert-prefix guidance with selective rollback and a harmonized objective. It reports strong mathematical and general-domain reasoning results across benchmarks and model settings.

1 INTRODUCTION

  • Static, indiscriminate expert supervision can suppress self-exploration and prevent models from internalizing reasoning beyond imitation.The introduction contrasts this with mix-policy methods that incorporate expert data into the RL loop.Source: §1 · p. 1
  • Expert-prefix completions lack correctness guarantees, while treating their completion tokens as on-policy can overlook a context-state mismatch and destabilize training.Failed completions can contaminate off-policy supervision, and overly aggressive guidance can trigger training collapse.Source: §1 · pp. 1–2
  • MOER progressively anneals the expert CoT prefix, rolls failed completions back to the full expert trace, and applies distinct objectives to on- and off-policy tokens.Rollback guarantees correct off-policy trajectories; the harmonized objective uses clipped GRPO for on-policy tokens and a trust-region objective for expert-prefix and completion tokens.Source: §1 · p. 2
  • 49.1 across six mathematical reasoning benchmarks and 59.0 on three out-of-distribution general reasoning benchmarks; MOER outperforms all baselines.The reported gains over the strongest baselines are +0.5 points on six math benchmarks and +1.6 points on out-of-distribution general reasoning tasks.Source: §1 · p. 2
  • MOER’s improvements generalize across tasks, model scales, and base-model categories.The introduction reports the best overall results on several Qwen2.5 models and component analyses supporting training stability and final performance.Source: §1 · pp. 2–3

2 RELATED WORK

The related work groups expert-data approaches into direct data mixing, expert-guided generation, and interleaved SFT-RL updates. These methods vary in how they combine expert trajectories or supervision with RL.

2 RELATED WORK

  • These approaches address the constraint that RL exploration can be limited by a model’s initial knowledge, motivating expert data within the RL loop.The section frames this as an alternative to the serial SFT-then-RL paradigm.Source: §2 · p. 3
  • Direct data-mixing methods combine expert trajectories with self-rollouts at the data level or add sample-level and dynamically weighted SFT objectives.Examples include SimpleMix, LUFFY, SRFT, and CHORD, with approaches ranging from dataset sampling to token-wise weighting.Source: §2 · p. 3
  • Expert-guided generation uses expert data to guide on-policy rollouts, including through prefixes, branching from intermediate steps, or trust-region constraints.UFT, BREAD, Prefix-RFT, and TRAPO illustrate different ways to select or adapt the guidance.Source: §2 · p. 3
  • Interleaved SFT-RL methods alternate or adapt between RL updates and SFT on expert data.ReLIFT targets challenging problems, HPT selects updates based on rollout performance, and AMFT learns an adaptive weighting parameter.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe Mix-Policy GRPO, which places expert-derived off-policy rollouts alongside on-policy rollouts for group-based advantage computation. Its clipped objective combines both rollout types, while the section notes an exploration concern.

3 PRELIMINARIES

  • Mix-Policy GRPO injects expert-derived off-policy rollouts into groups of current-model on-policy rollouts, considering both during advantage computation.The method computes an advantage for each rollout in the group.Source: §3 · p. 3
  • The off-policy expert distribution and pre-update on-policy distribution are distinguished, and each group contains on-policy and off-policy instances.The prose also identifies the reward function and input prompt.Source: §3 · p. 3
  • The objective applies a clipped surrogate to on-policy and off-policy rollout tokens, using importance-sampling ratios and a shared token-count normalization.The expert behavioral policy is typically inaccessible, so its probability is set to 1.Source: §3 · p. 4
  • The section identifies entropy collapse as a concern: optimizing the mix-policy objective can make rollouts more deterministic and diminish exploration of diverse reasoning trajectories.This concern is attributed to the objective’s effect during optimization.Source: §3 · p. 4

4 METHODOLOGY

MOER combines progressively shortened expert CoT prefixes and selective rollback with an objective that treats completions as off-policy tokens. Its training dynamics show the highest validation accuracy among the compared prefix-guidance baselines, while TRAPO collapses early.

4.1 DECAYING EXPERT PREFIX GUIDANCE WITH SELECTIVE ROLLBACK

  • MOER progressively shortens an expert CoT prefix and lets the model complete the remaining reasoning, then replaces incorrect completions with the full expert response.The prefix follows cosine annealing; rollback adds no computational overhead and ensures correctness of the off-policy guidance.Source: §4.1 · p. 4
  • MOER reaches the highest validation accuracy among the compared prefix-guidance baselines, while TRAPO collapses early and response length increases stably.The figure reports training dynamics for Qwen2.5-Math-7B.Source: §4.1 · p. 4, Figure 2 · p. 5

4.2 HARMONIZED MIX-POLICY OBJECTIVE

  • The harmonized objective combines modified GRPO with asymmetric clipping for on-policy tokens and a trust-region SFT loss that clips low-probability off-policy token weights.The trust-region coefficient γ is set to 0.1 by default, and the two losses are combined using off-policy loss weight α.Source: §4.2 · pp. 4–6
  • Because prefix conditioning shifts completion tokens off-policy and rollback selects verified completions, MOER optimizes completions with the off-policy objective rather than the standard on-policy objective.The paper reports that this choice stabilizes training against the resulting distribution shift and selection bias.Source: §4.2 · p. 5

5 EXPERIMENTS

MOER outperforms the baselines on the main Qwen2.5-Math-7B benchmarks and remains superior or competitive across additional model scales and base-model categories. Ablations attribute performance and training stability to the harmonized objective, decaying prefix guidance, selective rollback, and off-policy treatment of completion tokens.

5.1 EXPERIMENTAL SETTINGS

  • Experiments use OpenR1-Math-46k-8192 with DeepSeek-R1 expert traces and evaluate math benchmarks alongside ARC-c, GPQA-diamond, and MMLU-Pro general-reasoning benchmarks.The main training base is Qwen2.5-Math-7B, with additional evaluation on Qwen2.5-Math-1.5B, Qwen2.5-Instruct-7B, and Qwen2.5-Instruct-14B.Source: §5.1 · p. 6

5.2 MAIN RESULTS

  • 49.1 average across six mathematical reasoning benchmarks and 59.0 average on OOD general-domain reasoning benchmarks place MOER ahead of all baselines on Qwen2.5-Math-7B.All mix-policy baselines were reproduced on the same dataset and base model; Table 1 reports the main experiment results.Source: Table 1 · p. 7, §5.2 · p. 7

5.3 RESULTS ON MORE MODELS

  • MOER is superior or competitive across model scales and base-model categories, tying LUFFY at 38.1 average math score on Qwen2.5-Math-1.5B and reaching 42.3 math and 63.8 general averages on Qwen2.5-Instruct-7B.On Qwen2.5-Math-1.5B, it also exceeds ReLIFT's 34.1 math average and achieves a domain-leading 40.6 general-domain average.Source: §5.3 · pp. 7–8, Table 2 · p. 8

5.4 ABLATION STUDY AND ANALYSIS

  • On Qwen2.5-Math-7B, adding the harmonized objective to LUFFY raises math/general averages from 48.4/57.3 to 48.6/58.3, while the complete MOER variant reaches 49.1/59.0.Adding decaying prefixes without rollback lowers math performance to 46.2; adding rollback recovers it to 48.9, and removing expert guidance with rollback also reduces performance.Source: §5.4 · p. 8, Table 3 · p. 8
  • The harmonized objective mitigates entropy and response-length fluctuations, preventing late-stage collapse; its asymmetric clipping contrasts with LUFFY's unclipped design and standard GRPO's symmetric clipping.The training-dynamics figure presents MOER and ablations.Source: §5.4 · pp. 8–9, Figure 4 · p. 9
  • Decaying prefix guidance is associated with better OOD performance than full expert guidance and stronger late-stage exploratory behavior, reflected in higher entropy than the +HO variant.The paper describes the schedule as a way to alleviate forgetting from static, full expert supervision and foster self-exploration.Source: §5.4 · p. 9
  • Without selective rollback, the +HO+PG+COff variant collapses, with off-policy score falling to zero and entropy growing explosively; with rollback, the actual prefix ratio still decays.Figure 5 examines the effect of selective rollback.Source: §5.4 · p. 9, Figure 5 · p. 9
  • Treating completion tokens as on-policy yields late-stage instability and excessive response length, supporting their treatment as off-policy supervision.This variant has suboptimal final performance and overlong responses.Source: §5.4 · p. 9

6 CONCLUSION

MOER combines decaying expert-prefix guidance, selective rollback, and distinct losses for on- and off-policy tokens. In objective comparisons, its scaled trust-region SFT objective outperforms alternatives.

6 CONCLUSION

  • MOER achieves state-of-the-art mathematical reasoning performance while preserving strong general-domain capabilities.The conclusion attributes this result to combining decaying expert prefixes, selective rollback, and distinct loss formulations for on- and off-policy tokens.Source: §6 · pp. 9–10

A DISCUSSION ON DIFFERENT OFF-POLICY OBJECTIVES

  • The compared objectives differ in gradient-weight scaling and boundedness, with their curves diverging mainly for low-probability tokens.LUFFY and Prefix-RFT assign larger weights in that region than CHORD and MOER; all are described as softened SFT objectives.Source: A DISCUSSION ON DIFFERENT OFF-POLICY… · p. 13
  • MOER’s scaled trust-region SFT objective outperforms alternative off-policy objectives; Prefix-RFT causes entropy collapse, LUFFY induces high entropy, and CHORD has worse general-domain performance.CHORD’s training dynamics are similar to MOER’s, despite its worse general-domain performance.Source: Table 5 · p. 13, A DISCUSSION ON DIFFERENT OFF-POLICY… · pp. 13–14, Figure 7 · p. 14

B ANALYSIS ON CLIPPING OPERATION FOR ON-POLICY TOKENS

MOER’s on-policy clipping remains stable when updates become more off-policy, while LUFFY collapses; meanwhile, prefix-conditioned completion accuracy stays higher as its gap with on-policy accuracy closes during training.

B ANALYSIS ON CLIPPING OPERATION FOR ON-POLICY TOKENS

  • With rollout/update batches of 64/16, LUFFY collapses mid-training while MOER remains stable.LUFFY also shows late-stage instability at the default 128/64 setting; with more gradient steps, stale-data importance ratios can yield compounding unbounded gradients.Source: Figure 8 · p. 14, B ANALYSIS ON CLIPPING OPERATION FOR… · p. 14
  • LUFFY’s unclipped importance-sampling ratios are unbounded, while MOER clips on-policy ratios; stable entropy requires coefficients 0.2/0.24 rather than GRPO’s 0.2/0.2.The asymmetric coefficients are the low/high clipping coefficients, respectively.Source: B ANALYSIS ON CLIPPING OPERATION FOR… · p. 14

C ACCURACY OF PREFIX-CONDITIONED COMPLETIONS

D CODE GENERATION EXPERIMENTS

On LiveCodeBench, MOER substantially outperforms LUFFY in coding while maintaining smaller, healthy entropy and shorter responses. The comparison uses Qwen2.5-Instruct-7B trained on CodeForces-CoTs.

D CODE GENERATION EXPERIMENTS

  • MOER substantially outperforms LUFFY on LiveCodeBench while maintaining smaller, healthy entropy and shorter responses.The table reports pass@1, entropy, and response length.Source: Table 6 · p. 15, D CODE GENERATION EXPERIMENTS · p. 15
  • CodeForces-CoTs was filtered by prompt and response lengths of 1024 and 8192 tokens, leaving 13,000 training and 1,402 validation samples.Public test cases verified generated code and were removed from prompts to prevent reward hacking.Source: D CODE GENERATION EXPERIMENTS · p. 15
  • The comparison trains MOER and LUFFY for 380 steps on Qwen2.5-Instruct-7B, increasing LUFFY’s policy-shaping coefficient from 0.1 to 0.2 for stability.LiveCodeBench-CPP is the evaluation benchmark.Source: D CODE GENERATION EXPERIMENTS · p. 15

E RESULTS ON LARGER MODEL

On Qwen2.5-Instruct-14B, MOER outperforms LUFFY across most mathematical reasoning tasks and remains stable throughout optimization while LUFFY collapses.

E RESULTS ON LARGER MODEL

  • MOER outperforms LUFFY on most Qwen2.5-Instruct-14B mathematical reasoning tasks, with more pronounced gains than on 1.5B and 7B models.ReLIFT is not included because its checkpoint is unavailable and its computational costs are prohibitive.Source: E RESULTS ON LARGER MODEL · p. 15, Table 7 · p. 16
  • MOER remains stable throughout optimization while LUFFY collapses mid-training.Source: E RESULTS ON LARGER MODEL · p. 16

F ABLATION STUDY ON HYPERPARAMETERS

The ablation examines the minimum prefix ratio, off-policy loss weight, and clipping coefficient, finding that the latter two have clear preferred settings while performance is relatively robust to the minimum prefix ratio.

Minimum prefix ratio rmin

Off-policy loss weight α

Clipping coefficient ϵhigh

G PASS@K RESULTS

On AIME 2025, MOER’s high-budget pass@k exceeds SFT, while both mix-policy methods outperform the base model and standard RL across sampling budgets. The paper presents the crossover as evidence of improved reasoning capability and solution diversity.

G PASS@K RESULTS

  • MOER and LUFFY consistently outperform the base model and standard RL across all sampling budgets on AIME 2025.The evaluation uses Qwen2.5-Math-7B and compares the base model, SFT, GRPO, LUFFY, and MOER.Source: G PASS@K RESULTS · p. 16
  • 66.7% pass@28 for MOER exceeds SFT’s 60.5% as both mix-policy methods’ curves rise past SFT at k ≥ 25.At k ≤ 24, SFT has slightly higher accuracy; Figure 11 reports pass@k for k = 20 through 28.Source: G PASS@K RESULTS · p. 16, Figure 11 · p. 17
  • The crossover is presented as evidence that MOER improves underlying problem-solving capability and solution diversity, rather than merely increasing sampling of known knowledge.The evaluation varies computational budget to test this distinction.Source: G PASS@K RESULTS · pp. 16–17
  • MOER has math scaling similar to LUFFY alongside stronger general-domain utility, shorter responses, and better training stability.The paper describes this as maintaining reasoning efficiency and generalizability without extra generation overhead during sampling.Source: G PASS@K RESULTS · p. 17

Related papers

Loading 2610.30873v1…