Expected Reasoning-Step Return Unifies On-Policy Learning from Rewards and Teachers

1 Introduction

• ERSR compares student and teacher-proposed reasoning steps in a shared expected-return space using Monte Carlo rollouts continued under the student policy.It treats semantic steps as macro-actions and estimates their expected final task reward.Source: §1 · p. 2, Figure 1 · p. 2
• On successful trajectories student steps are more beneficial, while teacher replacements are more beneficial on failed trajectories.Student answer-probe gains track student-step utility and distinguish beneficial from harmful steps.Source: §1 · p. 2
• R2OPL reinforces student reasoning on successful trajectories and distills teacher signals on failed ones, using group success rates and student-probe gains for modulation.The approach uses group success rate for problem-level difficulty scaling and probe gains for step-level modulation.Source: §1 · pp. 2–3
• Across eight mathematical, scientific, and logical reasoning benchmarks, R2OPL consistently outperforms strong reward-based, distillation, and hybrid baselines.Its training dynamics show joint use of reward- and teacher-side utility, while existing hybrids often leave substantial utility in one branch.Source: §1 · pp. 2–3

2.1 Preliminaries

• Reward learning shares trajectory-level advantage across tokens, while OPD uses teacher–student discrepancy as a surrogate that need not indicate greater expected-return utility.Different reasoning steps can contribute differently to the final reward, and teacher preferences may not improve student task reward.Source: §2.1 · pp. 3–4

2.2 Expected Reasoning-Step Return

• ERSR treats semantic reasoning steps as macro-actions and estimates student-step and teacher-replacement advantages using student-policy continuations from a shared prefix.Monte Carlo estimates place both actions in a common expected-return space.Source: §2.2 · p. 4

2.3 Empirical Findings with ERSR

• On successful trajectories, student actions have higher mean advantage than teacher replacements (+0.032 vs +0.007); on failed trajectories, teacher replacements score higher (+0.064 vs −0.021).Sign balances show the same reversal: +19.0% vs +3.7% on successful trajectories, and −12.9% vs +23.7% on failed trajectories.Source: §2.3 · p. 5, Table 1 · p. 5
• Student-probe gain correlates most strongly with student-step utility across outcomes and ranks more beneficial successful steps and more harmful failed steps.Correlations are 0.246 on successful trajectories and 0.211 on failed trajectories, versus 0.178 and 0.131 for teacher-probe gain.Source: §2.3 · p. 6, Figure 2 · p. 6
• Teacher-side discrepancy and likelihood signals correlate weakly with student-step utility, with two signals negatively correlated on successful trajectories.Observed-step probe, likelihood, and discrepancy signals do not meaningfully correlate with teacher-replacement utility; all absolute correlations are below 0.08.Source: §2.3 · p. 6

3.1 Return-Referenced On-Policy Learning

• Successful trajectories use reward reinforcement, while failed trajectories use teacher-guided OPD; group success rate scales both branches according to prompt difficulty.Prompts the student rarely solves receive stronger updates, with both branches progressively attenuated as group success rate increases.Source: §3.1 · p. 7
• R2OPL segments responses into reasoning steps so every token in a step shares its student-probe modulation factor.The probe gain characterizes the utility of the student's own reasoning step.Source: §3.1 · p. 7
• On successful trajectories, higher student-probe gains identify more beneficial steps and increase their reinforcement strength.The binary reward supplies base credit, while the probe modulates its magnitude.Source: §3.1 · p. 7
• On failed trajectories, lower student-probe gains mark more detrimental steps and receive stronger teacher supervision.The method retains the token-level OPD signal and modulates its strength by the reasoning step's probe gain.Source: §3.1 · p. 7

3.2 Efficient Answer-Probe Computation

• Packed probes use prefix-restricted attention and aligned position IDs, enabling probe scores from the optimization forward pass with 2%–8% training overhead over OPD.Probe tokens are excluded from training loss; the reported overhead spans the three training settings.Source: §3.2 · pp. 7–8, Figure 3 · p. 8

4.1 Experimental Setup

• The experiments train on approximately 32K examples across mathematical, scientific, and logical reasoning and evaluate on eight benchmarks.Three teacher–student configurations are compared against reward-only, distillation-only, and reward–distillation hybrid baselines; Avg@16 is mean accuracy over 16 sampled responses per problem.Source: §4.1 · p. 8

4.2 Main Results

• R2OPL reaches Avg@16 scores of 59.4 and 63.1 in the Qwen and Gemma settings, exceeding the strongest baselines by 3.6 and 2.6 points, respectively.It achieves the best or tied-best result on seven of eight benchmarks in both configurations, with gains over reward-only, distillation-only, and hybrid baselines.Source: §4.2 · pp. 9–10, Table 3 · p. 9
• In the additional Qwen3-4B setting, R2OPL reaches Avg@16 of 73.1, exceeds the strongest baseline by 5.1 points, and leads on all eight benchmarks.The reported additional configuration shows the same overall trend.Source: §4.2 · p. 10

4.3 Analysis and Ablations

• Removing either failed-trajectory OPD or successful-trajectory RL substantially degrades performance, while removing difficulty scaling can cause entropy collapse.Removing probe modulation causes the smallest drop, but shifts the best checkpoint from step 250 to step 500.Source: §4.3 · p. 10, Table 4 · p. 10
• R2OPL reduces both student-reinforcement utility from 0.064 to 0.018 and teacher-distillation utility from 0.137 to 0.052, while reaching AMC23 accuracy of 78.1.SRPO and OPDVR primarily exhaust teacher-distillation utility, whereas RLSD primarily reduces student-reinforcement utility; R2OPL reduces both.Source: §4.3 · pp. 10–11, Figure 4 · p. 10

5 Conclusion

• ERSR compares and tracks student-step and teacher-replacement utility in a common expected-return space.Its analysis finds student steps more beneficial on successful trajectories and teacher replacements more beneficial on failed ones.Source: §5 · p. 11
• Student answer-probe gains track student-step utility and distinguish beneficial from harmful reasoning steps.This finding informs R2OPL's use of student-probe modulation alongside outcome-conditioned routing and difficulty scaling.Source: §5 · p. 11
• R2OPL shows strong, consistent gains across reasoning domains and teacher–student configurations while jointly exploiting reward- and teacher-side utility.The conclusion contrasts this joint utility use with existing hybrid methods.Source: §5 · p. 11

On-Policy Reinforcement Learning

• On-policy reinforcement methods refine optimization and stability, but reasoning-oriented RL still relies primarily on final trajectory rewards, leaving step-level credit comparatively coarse.The section traces the paradigm from clipped policy gradients and critic-free methods to reasoning-focused reward optimization.Source: §A.1 · p. 17

On-Policy Distillation

• On-policy distillation and reward–distillation hybrids develop student-state supervision and combine teacher signals with outcome rewards, but do not directly compare their utility in a common student-policy expected-return space.Teacher supervision may remain noisy, mismatched, or overly restrictive.Source: §A.1 · pp. 17–18

Reasoning-Step Credit Assignment

• Reasoning-step methods estimate progress, localize influential steps, or target teacher guidance, but evaluate student and teacher signals differently; ERSR evaluates both from the same prefix under the same downstream student policy.This places the two actions in a common expected-final-reward space.Source: §A.1 · p. 18

• Changing action granularity does not improve worst-case Monte Carlo convergence: with binary final rewards, estimation error remains O(N^-1/2).The budget for identifying the sign of a nonzero return difference instead scales quadratically with the inverse effect size.Source: §A.2 · p. 19
• Under diffuse reasoning credit, resolving token-level effects requires a quadratic Monte Carlo budget in the number of tokens composing a step.Token-level evaluation also considers approximately m times more actions across a response.Source: §A.2 · pp. 18, 20
• Token-level credits vary with lexical realization and tokenization, while the complete semantic-step return depends only on its boundary states.Different internal decompositions can assign different credits to individual actions yet telescope to the same semantic-step return.Source: §A.2 · pp. 20–21
• Semantic steps aggregate tokens into complete reasoning operations, retaining finer credit assignment than trajectory-level rewards while stabilizing return effects across representations.Token-level returns remain formally well defined, but their effects may be smaller and harder to resolve with finite samples.Source: §A.2 · pp. 18, 21

Structured Prompting

• Explicit-Step CoT matches or slightly improves Qwen3-1.7B across evaluated benchmarks, raising Avg@16 from 43.4 to 43.8.The paper adopts this format throughout training, evaluation, and ERSR evaluation.Source: §A.3 · p. 21, Table 5 · p. 22

Reasoning-Step Segmentation

• Qwen3-1.7B uses explicit step headings in 99.6% of sampled trajectories, versus 75.4% for Gemma-4-E2B-it.A segmentation procedure based only on Step N headings would leave many Gemma responses unsplit.Source: §A.3 · p. 22, Table 6 · p. 22
• Valid Step N sequences define reasoning steps; otherwise, repeated top-level semantic headings provide a fallback, and responses without reliable cues remain a single step.An independent Final Answer section is terminal, while a terminal boxed answer within the final reasoning section stays in that step.Source: §A.3 · p. 22, Table 7 · p. 22

Segmentation Statistics

• The segmentation procedure yields compact multi-step trajectories, with median step counts between 4 and 8.5 on most benchmarks.Mean counts reach 8.60 on AIME24 and 8.10 on AIME25, compared with 4.50 on LogicBench and 4.21 on LogiQA2.0.Source: §A.3 · pp. 23–24, Table 8 · p. 24

Data and trajectory sampling

• The evaluation samples five student rollouts for each of 1,200 questions stratified evenly across six within-question success-rate groups.The 1,200 questions are selected from 14,042 English-language DAPO-17K questions; the resulting set contains 6,000 non-truncated trajectories.Source: §A.4.1 · p. 24

Reasoning-step and teacher-replacement sampling

• Approximately five steps are sampled from each of 6,000 trajectories, excluding the final reasoning step, and each receives a teacher replacement conditioned on the same prefix.The teacher is allowed up to 2,048 tokens, after which only its first complete reasoning step is retained.Source: §A.4.1 · p. 24

Monte Carlo return estimation

• Each sampled step is evaluated from the prefix, preserved-step, and teacher-replacement states using 128 independent student-policy continuations per state.This requires 30,000 × 3 × 128 = 11.52 million continuations.Source: §A.4.1 · pp. 24–25

A.4.2 ERSR across Teachers and Problem Difficulty

• Both tested teachers show higher teacher-replacement utility on failed than successful trajectories, while replacement utility peaks at intermediate difficulty.For Qwen3-4B-Instruct-2507, mean b_A^TR rises from +0.007 on successful trajectories to +0.0638 on failed ones; a single-step replacement on a 0/5 problem may not correct multiple erroneous decisions.Source: §A.4.2 · pp. 25–26

A.4.3 Low-Cost ERSR Estimation

• For training-time aggregate ERSR, 500 sampled steps with MC@2 yields smaller deviations than 250 steps with MC@4 at the same nominal sampling budget.The paper uses 500 steps with MC@2 for tracking aggregate remaining branch utility, reserving MC@128 for offline fine-grained analyses.Source: §A.4.3 · p. 26, Table 10 · p. 26

A.5 Answer-Probe Runtime Analysis

• Packed answer-probe computation increases total training time by 2.0% for Gemma-4-E2B-it, 8.1% for Qwen3-1.7B, and 6.0% for Qwen3-4B.Most additional cost appears in actor updates that process packed probe tokens.Source: §A.5 · p. 27, Table 11 · p. 27

A.6 Dataset and Evaluation Details

• The evaluation covers eight mathematical, scientific, and logical reasoning benchmarks, using 1,098 problems in total and shared subsets across methods.Benchmarks with more than 200 examples are subsampled to 200; smaller benchmarks are evaluated in full.Source: §A.6 · pp. 27–28, Table 12 · p. 27

A.8 Results with Qwen3-4B as the Student

• R2OPL with Qwen3-4B achieves Avg@16 of 73.1, 5.1 points above OPDVR, and ranks best among compared methods on all eight benchmarks.It also exceeds its same-scale teacher’s average performance, 73.1 versus 72.5.Source: Table 14 · p. 29, §A.8 · p. 29

A.9 Balancing RL and OPD in R2OPL

• With equal branch weighting, R2OPL follows OPD-like dynamics and leaves student-step utility largely unchanged while sharply reducing teacher-replacement utility.OPD gradients are nearly two orders of magnitude larger than RL gradients initially; the paper attributes persistent OPD dominance in part to more coherent gradients and AdamW moment accumulation.Source: §A.9 · pp. 30–32, Figure 9 · p. 31
• At 100× relative RL-to-OPD scaling, AMC23 Avg@16 reaches 78.1 with both remaining branch utilities relatively low; at 200×, performance drops to 75.8.The 100× configuration uses μ = 10 and λ = 0.1.Source: Figure 10 · p. 31, §A.9 · p. 31