PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards

Xiaobing Chen, Zhiqi Pang

arXiv:2610.11519v1cs.CL

TL;DR

RLVR assigns response-level outcome labels, while OPD’s token-level guidance does not directly reflect teacher–student disagreement across the vocabulary. RA forms a bounded student-relative advantage and centers it within each response; Co-RA also adapts the teacher on scored student attempts. RA improves all matched comparisons, and Co-RA raises accuracy further in the tested settings.

  • Problem

    RLVR gives each response one outcome label for all its steps, while OPD’s pointwise signal does not directly represent vocabulary-wide teacher–student disagreement.

    Source: §1 · p. 1

  • Method

    RA subtracts the student-policy value of a bounded teacher–student probability residual, then centers the resulting local advantages within each response before adding them to the verifier advantage.

    Source: §1 · p. 2

  • Results

    RA improves all 24 three-seed benchmark means over its base estimator, both combinations surpass teacher-only OPD, and Co-RA raises accuracy further with mixed coverage changes.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    Across the tested settings, RA improves the student’s guidance outcomes even though Co-RA’s adapted teacher becomes a worse solver than the frozen teacher.

    Source: §6 · p. 9

    Zero-mean centering preserves the response-level label, but does not make the guidance gradient orthogonal to the verifier’s and a token’s guidance can outweigh it.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) have become two main paradigms for post-training reasoning models. RLVR gives each response a single outcome label, leaving the steps inside it without separate credit. OPD provides token-level guidance at student-visited prefixes, but its pointwise signal does not directly reflect the pattern of teacher--student disagreement across the vocabulary. Dense, unbounded log-ratio supervision can amplify the teacher's influence, yet a strong solver is not necessarily a suitable guide when the student's solution paths depart from the teacher's. We propose Residual Advantage (\RA{}), which treats the teacher--student probability residual as a bounded one-step reward, subtracts the corresponding state value under the student policy to form a standard advantage, and centers the result within each response before adding it to the verifier advantage. The guidance term has zero mean within each response, so the verifier advantage remains the response's mean label and the teacher only redistributes credit among the steps within it. \CoRA{} further updates a teacher LoRA with verifier advantages on the same scored student batch and uses the updated teacher in the next iteration's residual, adapting guidance to the student's attempts. With Qwen3-1.7B-Base and Qwen3-4B-Base students and a Qwen3-8B teacher, \RA{} combined with GRPO or REINFORCE++ improves the underlying sequence-advantage algorithm in all 24 comparisons on three mathematical benchmarks, raising macro Avg@8 by 1.7--3.6 points and Pass@8 by 3.9--6.3 points. Both combinations surpass teacher-only OPD, and \CoRA{} adds a further 1.0--1.5 Avg@8 points.

1 INTRODUCTION

RLVR assigns one outcome-based advantage to every step, while OPD’s pointwise signal does not directly represent how teacher–student disagreement is distributed. Residual Advantage (RA) centers student-relative teacher guidance within each response, preserving the verifier’s response-level label while reallocating credit among steps.

1 INTRODUCTION

  • RLVR gives every step the same response-level label, while OPD’s token-level signal does not directly capture how teacher–student disagreement is distributed.A strong solver may also be a poor guide when student paths depart from its own.Source: §1 · p. 1
  • RA treats the teacher–student probability residual as a bounded one-step reward and subtracts its expectation under the student policy to form a local advantage.The design compares a sampled token with the student’s other alternatives, weighted by their student probabilities.Source: §1 · p. 2
  • Centering local advantages within each response gives the guidance term zero mean, leaving the verifier advantage as the response’s mean label while redistributing step-level credit.The guidance also fades as student and teacher converge and stays small where the student is confident.Source: §1 · p. 2
  • Co-RA updates a teacher LoRA using verifier advantages on the scored student batch, then uses that adapted teacher in the next iteration’s residual.This is intended to adapt guidance to reasoning along the student’s solution paths.Source: §1 · p. 2
  • RA improves all 24 three-seed benchmark means of its base estimator, and both combinations surpass teacher-only OPD.Co-RA raises accuracy further, with mixed changes in coverage.Source: §1 · p. 3

2 RESIDUAL ADVANTAGE

Residual Advantage (RA) adds a student-relative local advantage to the verifier-derived response label. Centering preserves that response mean, while Co-RA adapts a teacher on scored student attempts for the next iteration’s guidance.

2 RESIDUAL ADVANTAGE

  • RA changes only the per-token label in the clipped PPO surrogate; teacher-derived quantities are computed at the rollout policy and held fixed.The base estimator supplies one verifier advantage per response, using GRPO normalization or REINFORCE++ batch statistics.Source: §2 · p. 3

2.1 STUDENT-RELATIVE RESIDUAL ADVANTAGE

  • At each visited prefix, RA uses the teacher–student probability residual as an immediate reward and subtracts its expectation under the student policy as a value baseline.The sampled-token residual alone does not determine the local advantage’s sign.Source: §2.1 · p. 3
  • The local advantage compares the teacher’s preference for the sampled token with its preference across the student’s alternatives, weighted by student probabilities.It vanishes for a token on which the student is settled, and its logit effect is scaled by the student’s own probability.Source: §2.1 · p. 3

2.2 COMBINING WITH VERIFIER FEEDBACK

  • Centering the local advantages and adding them to the verifier advantage preserves the response’s mean label, but does not make the resulting policy gradients orthogonal.A single token’s guidance can outweigh verifier feedback.Source: §2.2 · p. 4
  • The guidance is bounded by student–teacher distance and fades as the distributions converge, unlike an unbounded log-ratio signal.The bound applies at a fixed guidance weight, without a schedule.Source: §2.2 · p. 4

2.3 CO-RA TEACHES THE TEACHER ON THE STUDENT’S ATTEMPTS

  • Co-RA trains a LoRA adapter on the frozen teacher using verifier advantages from the same scored student batch, then uses the adapted teacher for the next residual.The adapter starts at zero, so the first student update coincides with RA.Source: §2.3 · p. 5

3 EXPERIMENTS

Across two student sizes, two estimators, and three benchmarks, RA improves every matched comparison and outperforms teacher-only OPD on macro accuracy and coverage. Co-RA further improves accuracy in every tested setting, while its coverage changes are mixed.

3 EXPERIMENTS

  • Experiments use Qwen3-1.7B-Base and Qwen3-4B-Base students with a post-trained Qwen3-8B teacher, evaluating MATH-500, AIME24, and AIME25.Training uses DAPO-Math-17K for two epochs; evaluation reports Avg@8 and Pass@8 from eight sampled completions per problem.Source: §3 · pp. 5–6
  • RA improves all 24 matched three-seed means, with macro gains of 1.74–3.58 Avg@8 and 3.89–6.33 Pass@8 points.The comparisons span two students, two estimators, three benchmarks, and two metrics.Source: §3 · p. 6
  • GRPO retains higher completion accuracy after guidance, while RA+REINFORCE++ has higher macro Pass@8 at both student sizes.The gains are larger for REINFORCE++, narrowing its macro Avg@8 deficit to GRPO at both sizes.Source: §3 · p. 6
  • Both RA combinations exceed teacher-only OPD in macro Avg@8 and Pass@8 at both student sizes, with one AIME25 Avg@8 exception at 1.7B.On that exception, OPD scores 5.97 versus 5.00 for RA+REINFORCE++, while both RA runs have higher Pass@8.Source: §3 · p. 6
  • Co-RA adds 1.03–1.45 macro Avg@8 points to the four RA runs, with Avg@8 rising in all twelve benchmark means.Pass@8 rises in four benchmark means, is unchanged in two, and falls in six; each adapted teacher scores below its frozen counterpart as a solver.Source: §3 · p. 6
  • From K = 8 to K = 512 for 1.7B REINFORCE++ runs, RA’s macro Pass@K margin over the untrained student grows from 13.4 to 16.9 points.Its margins over REINFORCE++ and OPD also grow, from 5.8 to 10.4 and 3.9 to 6.8 points, while verifier-only training’s margin narrows from 7.6 to 6.6.Source: §3 · p. 6

4 ANALYSIS

The analyses find that RA’s student-value baseline and response centering matter, and that the probability residual outperforms log-ratio alternatives. Guidance concentrates on student-choice forks, while adding denser teacher-peak supervision hurts verifier-trained students.

4.1 ABLATION STUDY

  • Removing the student value baseline lowers macro Avg@8 by 2.09 points and Pass@8 by 3.58 points at the reference weight, with similar losses at weights 5 and 20.Removing both the baseline and centering costs 1.73 Avg@8 and 2.77 Pass@8 points; the state value then shifts the response mean by −0.32 on average and over 0.5 on a quarter of offline responses.Source: §4.1 · pp. 6–7
  • Restoring response centering adds 0.54 Avg@8 and 0.46 Pass@8 points, a gain that holds at all three tested weights.Without centering, 8% of offline responses carry a mean shift beyond 0.5, which can reverse the verifier’s judgment; centering removes this shift for every response.Source: §4.1 · p. 7
  • Doubling the guidance weight leaves macro Avg@8 nearly unchanged at 29.29 versus 29.25, but widens seed SD from 0.25 to 1.68.All tested weights improve both macro scores over verifier-only training, while halving the reference weight lowers both means.Source: §4.1 · p. 7

4.2 PROBABILITY RESIDUAL VERSUS LOG RATIO

  • With both baselines, the probability residual exceeds the best log-ratio variant by 1.21 Avg@8 and 1.70 Pass@8 points.Raw log-ratio guidance falls below verifier-only training from β = 1.2 onward; centered log-ratio variants remain above it across the sweep.Source: §4.2 · pp. 7–8, Table 3 · p. 7
  • A full-vocabulary squared-distance loss beats verifier-only training but stays below RA and the uncentered control at every weight.After clipping and Adam, the teacher takes 0.28 of the update for RA versus 0.44 for direct loss, keeping RA’s update closer to the verifier’s direction (cosine 0.93 versus 0.80).Source: §4.2 · p. 8

4.3 WHERE THE GUIDANCE LANDS

  • The residual assigns 59% of guidance to the 12% of tokens with sampled-token probability 0.1 ≤ p(at) < 0.5, while centered log ratio assigns 50% to tokens the teacher rates below 10^-4.Confident tokens with p(at) ≥ 0.9 are 61% of tokens but receive under 1% of either label.Source: §4.3 · p. 8

4.4 MORE GUIDANCE IS NOT ALWAYS BETTER

  • The teacher-peak branch supervises the teacher’s top token as a pseudo-sample, and its verifier component can only raise a response’s total label.The sampled route and its centering remain unchanged.Source: §4.4 · p. 8
  • Adding the teacher-peak branch to RA lowers macro Avg@8 by 2.05–2.16 points and Pass@8 by 3.41–3.91 points.Every seed falls below the same seed of RA at each tested weight; the branch slightly helps sampled OPD but lowers both verifier-trained variants.Source: Table 4 · p. 8, §4.4 · p. 9

5 RELATED WORK

Related work covers student-sample distillation, combinations of distillation with verifiable rewards, and methods for baselines or token-level credit. RA and Co-RA are distinguished by using the teacher as a critic-like signal within RL and adapting a teacher on the same scored student batch.

Distilling on the student’s own samples

  • RA replaces sampled-token log ratios with a bounded probability residual, evaluates a chosen token against the student’s alternatives, and removes the teacher’s response mean.Unlike the cited distillation methods, RA uses the teacher as a critic-like signal within RL rather than as a distillation target.Source: §5 · p. 9

Combining distillation with verifiable rewards

  • RA keeps the verifier advantage as the response mean while allowing local sign changes, unlike OPDVR’s gating of sampled log-ratio signs to agree with the verifier.At matched budget, OPD followed by RL remains below RA.Source: §5 · p. 9

Baselines and token-level credit

  • RA uses an exact policy expectation as the baseline of its local reward, while Co-RA trains a teacher LoRA with the base estimator’s clipped update on the same scored student batch.The passage contrasts these choices with prior policy-gradient baselines, process supervision, and feedback-conditioned teachers.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents RA as a way to preserve the verifier’s response-level label while using the teacher to allocate credit among steps, and reports gains across the tested settings. The paper also records broad use of generative AI tools and specifies assumptions for its probability identities and update claims.

6 CONCLUSION

  • RA improves every matched three-seed mean across two students, two estimators, and three benchmarks, and Co-RA improves accuracy further while its adapted teacher becomes a worse solver than the frozen teacher.The conclusion also reports that removing either baseline, replacing the residual with a log ratio, or adding direct teacher-top-token supervision lowers performance.Source: §6 · p. 9

AI USE STATEMENT

  • The authors report using generative AI tools across theory, proofs, hypotheses, methodology, experiments, implementation, writing, and literature review, while retaining responsibility for the final content.They state that authors checked derivations, proofs, cited works, code, experiments, and reported numbers.Source: AI USE STATEMENT · p. 10

A PROOFS AND UPDATE IDENTITIES

  • The proof setup assumes probability vectors on the same finite support with at least two coordinates, and uses positive probabilities for logit identities.Probability identities also hold on the simplex boundary; labels are fixed during differentiation and expectations use the policy defining the label.Source: A PROOFS AND UPDATE IDENTITIES · p. 13

A.1 PROOF OF PROPOSITION 1

These appendices analyze residual guidance and teacher adaptation, including their local update behavior, relation to log-ratio guidance, and effects on guidance across a response. They establish local guarantees and bounds while clarifying that these do not guarantee monotone improvement or extend to sampled updates and full training dynamics.

A.1 PROOF OF PROPOSITION 1

  • For the quadratic residual, Ad(a) is a directional improvement signal with zero student mean and unchanged expected update under value-baseline subtraction, although sampled updates can oppose the expectation.If the teacher is one ideal KL-regularized step ahead, its log ratio recovers the score contrast exactly, whereas the residual passes it through a student-weighted saturating map; a frozen solver need not satisfy that condition.Source: §A.1 · pp. 13–14

A.2 A LOCAL OBJECTIVE AND ITS TARGET

  • The local surrogate has a unique simplex optimum given by the Euclidean projection of q + Q/λ, is nonexpansive in q, and yields sparsemax-style support thresholds.This result freezes Q at the rollout policy and does not establish which tokens finite training discovers or cover the actual protocol’s random lengths and batch normalizer.Source: §A.2 · pp. 14–15

A.3 LOCAL PROBABILITY SCALING

  • The residual’s expected logit influence on coordinate b scales as p(b)Ad(b), whose label is bounded, while the log-ratio influence grows as p(b)[log(1/p(b)) + O(1)] for rare student tokens.Both terms are O(p(b)), but their ratio and combined sign are not bounded by that shared order; response-centering also differs between raw and student-centered log labels.Source: §A.3 · p. 15

A.4 THE RESIDUAL AND THE LOG RATIO AS ENDS OF ONE FAMILY

  • The residual and log ratio are endpoints of a Bregman-divergence family, but only the quadratic endpoint has bounded teacher sensitivity; in confidently wrong modes, the residual pushes more gently.For every family member with α < 2, teacher sensitivity is unbounded at teacher-rare tokens.Source: §A.4 · p. 16

A.5 LABEL CENTERING AND GRADIENT VARIANCE

  • The student value baseline preserves the mean update but can incur four times the raw trace variance at a prefix, a cost of the conditional zero mean required for response composition.In the illustrated confident-student case, a rare action supplies most of the expected update despite its low sampling probability.Source: §A.5 · pp. 16–17

A.6 SENSITIVITY, RANGE, AND THE GUIDANCE BOUNDS

  • Quadratic guidance has bounded teacher sensitivity and vanishes uniformly as student and teacher agree uniformly at scored prefixes, while adding label variation without changing the response mean.Small guidance can also result from centering cancellation, and orthogonality of label components does not imply orthogonality of their parameter gradients.Source: §A.6 · pp. 17–18

A.7 COMPARING LOCAL ADVANTAGES ACROSS A RESPONSE

  • Under the student’s sampling, local advantages and scores are martingale differences, so every cross-position term vanishes; without the student value baseline, this cancellation fails.Offline, local advantages average 3 × 10^-4 per token. For responses longer than 1,000 tokens λAd has SD below 0.03; for shorter responses, SD is 0.32 and 8% have |λAd| > 0.5.Source: §A.7 · p. 19

A.8 THE TEACHER-PEAK BRANCH

  • The teacher-peak branch applies the guidance term at every position in response i.The supplied passage does not specify the two routes further.Source: §A.8 · p. 20
  • The teacher-peak branch adds the teacher’s peak as a pseudo-sample, leaving sampled labels and their response mean unchanged while raising total labels for responses with positive verifier advantage.The added labels are not constrained to have zero mean; their verifier component is nonnegative.Source: §A.8 · p. 20
  • The teacher peak’s local residual advantage is nonnegative when its student probability is at most the student collision probability, and cannot be lower than -1/4.The lower bound is sharp.Source: §A.8 · p. 20

A.9 AN ADAPTED TEACHER IN THE RESIDUAL ADVANTAGE

  • Teacher adaptation preserves the residual’s mean identity, bounded guidance, uniform vanishing under distributional agreement, and fixed-prefix expected-gradient identity at each scoring step.The current batch is scored with the pre-update teacher; the updated teacher enters the next iteration’s residual.Source: §A.9 · pp. 20–21
  • Co-RA trains a teacher LoRA on student-generated trajectories using verifier advantages, but its update identities guarantee neither monotone distance reduction nor policy improvement.The teacher gradient uses student-generated trajectories without importance correction to the teacher; clipping and shared LoRA parameters shape the actual update.Source: §A.9 · p. 21

B.1 TRAINING AND EVALUATION CONFIGURATION

The appendix specifies the training rewards, evaluation protocol, and guidance controls used to compare residual guidance with verifier-only and distillation baselines. It also describes how the teacher adapter is updated and identifies limits on the reported computational evidence.

B.2 PROBABILITY AND ACTOR CONVENTIONS

  • Rollouts use full probability support; labels are fixed at the rollout distribution, and guidance is applied at every response token.The student and teacher distributions are evaluated at visited prefixes over aligned output coordinates.Source: §B.2 · p. 22

B.3 TRAINING REWARD AND ADVANTAGE CONSTRUCTION

  • Correct verifier responses receive full reward up to a threshold and then a linear length penalty, while incorrect responses receive zero.GRPO normalizes shaped returns within prompt groups; REINFORCE++ uses terminal returns and token-weighted normalization when lengths differ.Source: §B.3 · p. 23

B.1 TRAINING AND EVALUATION CONFIGURATION

  • Evaluation samples eight completions per problem and reports unshaped correctness as Avg@8 and Pass@8, with macro scores equally weighting three benchmarks.The two-epoch training setup uses a correct-only linear length penalty for verifier runs, while all runs share the response cap.Source: Table 9 · p. 21, §B.4 · p. 23

B.5 VALIDATION SPLIT AND WEIGHT SELECTION

  • Guidance weights were selected on 64 validation prompts drawn from the training distribution, without using evaluation benchmarks for selection.The same 256 sampled training prompts also supply offline responses for an appendix analysis.Source: §B.5 · p. 24

B.6 GUIDANCE CONSTRUCTION

  • RA and Co-RA use the same residual, student value baseline, response centering, and λ = 10; Co-RA substitutes the current teacher-adapter snapshot.Teacher-only OPD uses a fixed teacher and no verifier feedback or length shaping.Source: Table 13 · p. 24, §B.6 · p. 24

B.7 LABEL AND DIRECT-LOSS CONTROLS

  • Raw log-ratio guidance improves both macro scores over plain REINFORCE++ at β ≤ 0.6 but lowers both at β ≥ 1.2, while full RA improves at every tested weight.At its best weight, the raw log ratio remains 1.97 Avg@8 and 2.70 Pass@8 points below full RA; controls also remove components of RA or use a direct-L2 loss.Source: §B.7 · pp. 25–26

B.8 ADAPTER OBJECTIVE AND UPDATE ORDER

  • Co-RA trains a teacher LoRA on the same scored student batch using verifier advantages, then uses the updated adapter to score the next iteration.The teacher backbone remains frozen, and the scored labels are held fixed during the student and adapter updates.Source: §B.8 · pp. 26–27

B.9 COMPUTATION AND EVIDENCE SCOPE

  • Runs train for two epochs on DAPO-Math-17K; RA adds teacher scoring and a full-vocabulary inner product, while Co-RA also trains a LoRA each batch.Wall-clock and peak-memory measurements are not reported.Source: §B.9 · p. 27

C TRAINING DYNAMICS

On Qwen3-4B-Base, RA maintains higher training accuracy than its verifier-only counterpart over most of training. The guidance magnitude declines at fixed weight, while response lengths differ substantially by verifier correctness.

C TRAINING DYNAMICS

  • RA maintains higher training accuracy than its verifier-only estimator over most of training and stays above teacher-only OPD over most of the shared response budget.The comparison with OPD reflects a different recipe because OPD has neither verifier feedback nor verifier-run length shaping.Source: Figure 4 · p. 27, C TRAINING DYNAMICS · p. 27
  • Training accuracy is measured on training prompts under the rollout policy, unlike held-out benchmark evaluation at a different sampling temperature.The sizes of the training and benchmark gains therefore need not agree.Source: C TRAINING DYNAMICS · p. 27

C.1 GUIDANCE MAGNITUDE AND RESPONSE LENGTH

  • At fixed λ = 10, mean absolute guidance falls from about 0.43 to about 0.30 over training, yet still contributes to actor labels at the end.The reported quantity averages over tokens, and response lengths change during training.Source: Figure 5 · p. 28, §C.1 · p. 28
  • Correct-response mean length rises but stays below about 4,000 tokens, while incorrect-response lengths rise and later fluctuate mostly between 9,000 and 12,000.The shaped reward penalizes length only for correct responses, so it does not oppose incorrect-response growth.Source: §C.1 · p. 28

C.2 WHERE THE GUIDANCE LANDS

Offline analyses show that RA and centered log-ratio guidance place their magnitude on different sampled-token probability regions. The student value baseline nearly preserves a zero response mean in practice, while raw residuals shift response means negatively.

C.2 WHERE THE GUIDANCE LANDS

  • On the same 176,628 offline tokens, RA concentrates guidance on student-uncertain tokens, whereas centered log-ratio guidance concentrates it on near-impossible tokens, especially where teacher probability is below 10^-4.Both labels use the student value baseline and response centering; they differ in the reward.Source: §C.2 · p. 28, Table 15 · p. 29
  • The raw residual shifts response means by −0.32 on average, with 26% of responses beyond 0.5 in absolute value.By comparison, 8% of responses have local-advantage means beyond 0.5 in absolute value.Source: §C.2 · pp. 28–29
  • The local advantage averages 3 × 10^-4 across tokens, while its response-mean spread falls from SD 0.32 below 1,000 tokens to 0.03 between 1,000 and 4,000.Only three of the 256 analyzed responses are correct, so the passage reports no correctness split.Source: §C.2 · pp. 28–29, Table 16 · p. 29

C.3 HOW THE SAMPLED LABEL AND THE DIRECT LOSS SHARE THE UPDATE

The sampled RA teacher gradient matches its analytic expectation, but its interaction with the verifier’s sampled path differs from a direct loss. On the next batch, RA’s update improves the verifier objective more than the direct-loss update in most paired comparisons.

C.3 HOW THE SAMPLED LABEL AND THE DIRECT LOSS SHARE THE UPDATE

  • Across 64 resampled tokens, the sampled teacher gradient matches its analytic expectation, with cosine 0.994 for RA and norm ratio 1.001.RA has 64% outcome-compatible teacher-gradient mass versus 49% for direct loss, mainly from lowering incorrect responses.Source: §C.3 · p. 29
  • The direct loss takes a larger share of the optimizer update than RA, 0.44 versus 0.28, despite having the smaller gradient.The comparison is for Qwen3-1.7B-Base with REINFORCE++ at weight 10, aggregated over three seeds.Source: §C.3 · p. 29, Table 17 · p. 30
  • RA’s next-batch step raises the verifier objective 1.10–1.16 times as much as the direct-loss step and wins 66–74% of paired comparisons.The ordering of RA, the uncentered label, and direct loss also holds on every seed for several reported update measures.Source: Table 17 · p. 30, §C.3 · p. 30

D.1 BENCHMARK-LEVEL RESULTS FOR THE CONTROLS

  • Table 19 reports benchmark-level three-seed means for control runs, while reference runs repeat Table 1.Macro means and sample SD are computed from per-seed macro scores.Source: §D.1 · p. 30, D THREE-SEED RESULTS · p. 30

D.2 PER-SEED BENCHMARK RESULTS FOR THE CONTROLS

This section reports per-seed and benchmark-level results for seven REINFORCE++ control variants, with macro scores summarized across three seeds. The controls include changes to the value baseline, response centering, residual or log-ratio guidance, and direct L2 supervision.

D.2 PER-SEED BENCHMARK RESULTS FOR THE CONTROLS

  • The seven controls test removing the student value baseline or response centering, using raw residual or log-ratio guidance, and direct L2 supervision.Table 20 reports per-seed benchmark Avg@8 / Pass@8 entries for the controls.Source: §D.2 · p. 30, Table 20 · p. 32
  • Macro scores weight the three benchmarks equally, and Tables 23 and 24 provide the associated weight sweeps.The listed tables describe control-run results; their cell values are not supplied here.Source: §D.2 · p. 30
  • Table 18 summarizes macro performance across three training seeds, while Table 19 reports benchmark-level means for the control runs.Table 18 reports sample standard deviations across three run-level macro scores.Source: Table 18 · p. 31, Table 19 · p. 31

D.3 TEACHER-PEAK BRANCH RESULTS

Across these appendix subsections, the authors compare teacher-peak branches, guidance-weight controls, OPD/RL schedules, sampled OPD, Pass@K behavior, and trained models against untrained students and the teacher. The reported comparisons include both per-seed patterns and benchmark-level scores.

D.3 TEACHER-PEAK BRANCH RESULTS

  • At every tested weight, each teacher-peak branch seed trails the matching RA seed; with REINFORCE++, the branch also trails verifier-only at weights 5 and 10.At weight 20, two of three branch seeds trail verifier-only.Source: §D.3 · p. 32, Table 21 · p. 32

D.4 WEIGHT SWEEPS

  • The weight sweeps report three runs per tested weight for 1.7B REINFORCE++, with λ = 10 repeating the reference configuration.The residual and log-ratio sweeps list macro Avg@8 / Pass@8 by seed and three-seed means with sample standard deviations.Source: §D.4 · p. 32, Table 23 · p. 33, Table 24 · p. 34

D.5 DISTILLATION COMBINED WITH RL: SEQUENTIAL AND JOINT

  • Sequential OPD followed by REINFORCE++ reaches 28.39 macro Avg@8 and 43.61 Pass@8 but trails matched-budget RA by 0.86 and 1.59 points.At β = 0.2, joint OPD is 0.47 and 0.50 points above verifier-only, but 3.11 and 5.26 points below RA on those respective metrics; it also trails the sequential schedule.Source: §D.5 · pp. 33–34, Table 26 · p. 36

D.6 FULL-VOCABULARY AND SAMPLED OPD

  • Sampled OPD reaches 24.80 macro Avg@8 and 38.60 Pass@8, 2.65 and 2.70 points below full-vocabulary OPD and 0.87 and 0.84 below verifier-only REINFORCE++.The sampled variant is lower than full-vocabulary OPD on every benchmark score.Source: §D.6 · p. 34, Table 27 · p. 36

D.7 PASS@K UP TO 512

  • At Pass@512, RA’s margin over the untrained student is 16.9 points, up from 13.4 at Pass@8; its last-doubling gain is 2.9 points versus 1.1–1.3 for other runs.The table reports macro Pass@K across three benchmarks and three seeds; the figure plots each trained run’s margin over the untrained student.Source: Figure 6 · p. 34, §D.7 · p. 34, Table 28 · p. 36

D.8 UNTRAINED STUDENTS AND THE TEACHER

  • RA improves macro Avg@8 by 1.7–3.6 points and Co-RA by 2.8–5.0 points over the corresponding estimator; every trained run beats the untrained student’s Pass@8 on every benchmark.The most accurate 4B run, Co-RA+GRPO, remains 5.4 macro Avg@8 points below the teacher.Source: §D.8 · p. 37

D.9 THE ADAPTED TEACHER UNDER THE EVALUATION PROTOCOL

The adapted teachers score below the frozen teacher as standalone solvers, yet are better guides for the students that trained them. The performance drop is larger for adapters trained with the 1.7B student than with the 4B student.

D.9 THE ADAPTED TEACHER UNDER THE EVALUATION PROTOCOL

  • Co-RA-adapted teachers score 0.8–2.3 macro Avg@8 points and 0.7–3.5 Pass@8 points below the frozen teacher.The drop is larger for adapters trained with the 1.7B student than for those trained with the 4B student.Source: §D.9 · p. 37, Table 30 · p. 37
  • The adapted teacher is a better guide for the student that trained it, despite being a worse standalone solver than the frozen teacher.The per-seed macro scores of the adapted teachers are listed in Table 31.Source: §D.9 · p. 37, Table 31 · p. 37
Loading 2610.11519v1…