PaperlayerView exact revision on arXiv

Source-linked AI summary

What Outcome RL Buys: Finishing, Reliability, Rarely Reach

Shan Yang

arXiv:2610.34894v1cs.CL

TL;DR

Pass@1 under a token budget can conflate finishing, reliability, and new problem-solving reach, leaving unclear what outcome RL changes. The paper decomposes gains from twenty released RL models against their thinking bases and finds that the dominant component depends on whether the base is budget-bound. New reach is rare, and supplied physical setups did not demonstrate transfer beyond plain RL in a small-scale probe.

  • Problem

    Because pass@1 is scored under a token budget, its value mixes answering in time, answering correctly, and solving problems the base cannot, making behaviour and knowledge gains hard to distinguish.

    Source: §1 · pp. 1–2

  • Method

    The paper compares twenty released RL models with their thinking bases at fixed budgets and decomposes pass@1 changes into finishing, reliability, and net reach.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Where budgets bind, finishing accounts for 82% of the change at 4k and 8k; where they do not, reliability rises, while new reach is rare.

    Source: §6 · p. 8

  • Takeaways & Limitations

    In the 8B probe, supplying physical setups was not shown to improve transfer to held-out problems beyond plain RL.

    Source: §1 · p. 2

    The strict answer matcher makes part of the measured reliability gain reflect which answer forms it accepts.

    Source: §6 · p. 8

Abstract

from arXiv · show

Outcome RL raises pass@1 but not pass@k at large k (Yue et al., 2025). But every pass@1 is scored under a token budget. Every recipe also trains under one. On physics and math problems at fixed token budgets, we compare the pass@1 of 20 released RL models (1.5B to 32B) with that of the thinking models they started from. We ask whether each pass@1 gain is changed behaviour or new knowledge. We divide each change in pass@1 into three parts. Finishing measures answering before the budget runs out. Reliability measures being right once finished. Reach measures problems RL solves that its base answered wrongly in all eight tries, minus problems the base solves that RL answered wrongly in all eight. We find four things. (1) Where the base often hits the token budget before answering, most of the change, gain or loss, is finishing (82% at 4k and 8k). Given more tokens, the base closes most of the finishing gap itself. (2) Where the base has enough tokens, the gain is reliability: the RL model is graded right more often, even on problems both finish. (3) By our definition, RL rarely solves what its base cannot. Reach adds at most 2.1 percentage points on any of our four physics and math sets. We checked 144 physics problems that only the RL model solved. Of these, 3.5% hold up when resampled or read. In a random sample, nine of ten new RL steps a human confirmed also appear in the base's samples. (4) In a small-scale RL run on an 8B model, the gain on new problems is finishing, with reliability flat. Supplying each problem's physical setup was not shown to improve on plain RL.

1. Introduction

Because pass@1 is scored under a token budget, its gains can mix finishing on time, reliability once finished, and solving problems the base cannot. The paper separates these components and compares twenty released RL models with their thinking bases at fixed budgets.

1. Introduction

  • The study asks whether pass@1 gains reflect changed behaviour or new knowledge, separating finishing, reliability, and net reach.Reach counts problems the base answers wrongly in all eight tries, minus problems the base solves that RL answers wrongly in all eight.Source: §1 · pp. 1–2
  • The main study compares twenty released RL models with their thinking bases at fixed budgets; once budgets suffice, physics reliability rises while reach stays small.All twenty bases are thinking models that distillation has already widened.Source: §1 · p. 2
  • In physics outcome-RL training, supplying physical setup to an 8B thinking model changed the model during training, but was not shown to carry over to held-out problems beyond plain RL.Source: §1 · p. 2
  • The paper tests whether supplying missing problem knowledge can install it, using five ways to provide physical setups to an 8B model during RL.The probe tests transfer on held-out text-only twins.Source: §1 · p. 2

2. Related Work

Prior work reports that outcome RL raises pass@1 while bases can outperform RL at large k, with evidence pointing to changes in behaviour and coverage. This paper distinguishes finishing from reliability and measures net reach at fixed budgets.

2. Related Work

  • Prior work finds RL raises pass@1 but the base wins at large k, and reports that RL stays within problems its base can sample (Wu et al., 2025b).Other studies attribute reduced coverage to rare successes dropping out of small sample groups or winner-take-all reinforcement of likely base outputs (Zhou, 2026; Nguyen et al., 2025).Source: §2 · p. 2
  • Studies report that budget mismatch and attempt inflation contribute to measured gains, while RL often changes execution or derivation behaviour more than content (Wu et al., 2025a; Yang & Chen, 2026; Qin et al., 2025).Other work reports fine-tuning broadens policies while RL narrows them, and that fine-tuning rather than RL moves knowledge (Matsutani et al., 2025; Wu et al., 2025c).Source: §2 · pp. 2–3
  • Prior evidence is observational; the paper supplies missing problem content and tests whether RL learns it, while splitting pass@1 into finishing, reliability, and net reach at fixed budgets.Reach is measured per problem at k = 8 and subtracts the mirror count to control for sampling noise.Source: §2 · p. 3
  • Related work also studies methods for crediting rare successes, including pass@k objectives, VinePPO branching, and replay of past successes (Chen et al., 2025d; Kazemnejad et al., 2024; Zhang et al., 2025a).Length and answer rate are recognized confounds in evaluating thinking models, motivating separation of finishing and a held-out set fixed before training (Liu et al., 2025b; Yu et al., 2025b; Aggarwal & Welleck, 2025; Hou et al., 2025).Source: §2 · p. 3

3. Experimental Setup

The study evaluates thinking-model bases and their RL versions on physics and math, using a budget-aware decomposition of pass@1 and a fixed reading protocol for twenty released pairs.

3. Experimental Setup

  • The study focuses on thinking bases, where token budgets make changes in reasoning length, answer commitment, and correctness visible in the score.All studied models are thinking models; models that answer in a few hundred tokens are out of scope except one contrast pair.Source: §3 · p. 3
  • Physics tests transfer beyond recipes’ math training domain, while math measures RL in its own domain; the main sets are 300 text-only physics problems and MATH-500.The physics set is filtered to problems the controlled-run base solves at least sometimes, and 671 unfiltered physics problems check the reach result; the math check contains 276 recent contest problems.Source: §3 · p. 3

3.1. Metrics

  • The decomposition separates finishing, the rate of answering within budget, from reliability, the rate of correctness among problems with finished samples.Reach measures problems the base leaves unsolved in k samples, subtracts the mirror count for problems RL leaves unsolved, and the three parts sum exactly to the pass@1 change.Source: §3.1 · p. 4
  • The finishing-reliability cross term is assigned to finishing; share reports the relative magnitudes of the two terms with a 95% interval, and matched reliability requires both models to finish at least seven of eight samples.The paper also defines robust reach to exclude one-off wins by requiring the other model to solve a problem in at least k/2 samples.Source: §3.1 · p. 4

3.2. Protocol for Reading the Released Pairs

  • Twenty released outcome-RL models are compared with their training bases using the same sampling setup, one prompt, and one answer checker.Each model produces eight answers per problem at temperature 0.6; budgets include 4k, 8k, 16k, each recipe’s training cap, and the next budget above it.Source: §3.2 · p. 4
  • The pass@1 split is read per sample at k = 1, while reach is read at k = 8 and k = 32 in an appendix; predictions for ten pairs at 4k and 8k were registered before outputs existed.Reads at the training cap and the next budget above it were not registered.Source: §3.2 · p. 4

4. Finishing, Reliability, Rarely Reach

At tight budgets, finishing accounts for much of the change; where bases already finish, reliability rises, while measured reach remains small. Checks suggest RL more often uses steps its base already samples than adds new ones.

4.1. Finishing

  • At 4k and 8k, finishing is the larger part of the change for recipes up to 7B, except Open-RS3, whose score moves by at most 0.004.Across score changes, the finishing term tracks the change closely (Spearman ρ = 0.97 over 108 points).Source: §4.1 · p. 4, Figure 4 · p. 5
  • Given 64k tokens, bases close most of the finishing gap on problems where RL finishes and scores higher at 4k or 8k.The authors select problems using four of eight samples, then evaluate on the other four and a separate 64k base run.Source: §4.1 · p. 5

4.2. Reliability, Where the Base Already Finishes

  • Reliability rises in all sixteen recipes that are not cap-bound, whose bases finish 93–100% of derivations; it is larger by size in fourteen.Matched reliability, measured only on problems both models finish, still rises in all sixteen recipes and is clear of zero in twelve.Source: §4.2 · pp. 5–6
  • In the step check, none of the 26 steps judged decisive for RL wins was missing from the base; on 17, the base stated the step but did not carry it through.The check sampled problems where the base finished all eight samples but was right only one to four times, while RL was right more often.Source: §4.2 · p. 6
  • Across all twenty recipes, reliability is the part of the gain that rises consistently and is the largest part of the mean physics gain at the cap.Finishing moves either way, and recipe differences in data, steps and algorithm prevent the authors from explaining why some gain more reliability.Source: §4.2 · p. 6

Of what RL solves on physics, its base already solves

  • Reach stays at or below 0.021 across the four physics and math sets.Of 144 RL-only physics wins checked, five held up as real physics; among 57 checked problems where a step decides, the base’s samples wrote RL’s deciding step in 56.Source: Of what RL solves on physics, its base… · pp. 6–7
  • Nine of ten new RL steps upheld by a human in a random sample were already present in the base’s samples.The authors also found the base’s samples contained RL-side steps about as often as the reverse in a search of four other samples.Source: Of what RL solves on physics, its base… · p. 7

5. A Bounded Probe: Can Supplied Knowledge Be Installed at Small Scale?

The controlled 8B probe found that outcome RL improved held-out performance through finishing, while supplying physical setups was not shown to improve transfer beyond plain RL. Reward raised solve rates on problems it reached, but gains did not extend clearly to other problems or their twins.

5. A Bounded Probe: Can Supplied Knowledge Be Installed at Small Scale?

  • On held-out problems, plain RL’s gain came from finishing, with reliability flat; none of the four evaluable setup channels was shown to improve beyond plain RL.The test could detect a gain one-third the size of plain RL’s; the setup-reward channel could not be evaluated because the policy gamed its scorer.Source: §5 · pp. 7–8
  • Revisiting 60 hard training problems raised their solve rate from 0.10 to 0.28, but did not move the other 293 hard problems or held-out set.On problems the control had earned reward on, both finishing and reliability rose, so the finishing share depends on split order.Source: §5 · p. 8
  • For credited problems’ text-only twins, the planned mean-score excess was above zero, but the measure of solving a twin at least once in eight samples included zero.Finishing accounted for 26–62% of the mean-score excess; adjustment and a post-hoc test leave partial transfer unresolved.Source: §5 · p. 8
  • The probe cannot distinguish whether setup channels failed or supplied knowledge simply does not transfer in this way.The run used one seed per treated arm, a single model family and domain, and did not test prolonged training or large-scale supervised distillation.Source: §5 · p. 8

6. Discussion and Limitations

The paper concludes that released-model gains chiefly reflect finishing when budgets bind and reliability when they do not, while reach remains small. It recommends reporting finishing alongside pass@1 and comparing recipes with their bases at or above the training cap; its released-pair analysis is limited by scope, sampling, reach definitions, and grading.

6. Discussion and Limitations

  • Where budgets bind, finishing accounts for 82% of the change at 4k and 8k; where they do not, gains are reliability, while reach is rare.At the probe’s scale, supplied knowledge was not shown to transfer beyond problems reward lands on.Source: §6 · p. 8
  • Report pass@1 with fin@1(B) at the same budget and compare a recipe with its base at or above its training cap.QuestA and two Polaris recipes score below their bases on physics at 16k but not at their own caps; QuestA still loses 0.07 on MATH-500 at its cap.Source: §6 · p. 8
  • The released-pair and controlled studies cover thinking bases only and are bounded in scale, sampling, and grading.The probe used one model family and domain, one seed per treated arm, and did not test prolonged training or large-scale supervised distillation.Source: §6 · p. 8
  • Reach may be understated on MATH-500 because models were not resampled on problems each alone failed; physics may overlap with pretraining, and recipe training-set overlap was not checked.On the 40 reliability problems checked by readers, 12 RL wins differed only in answer form or grading, so measured reliability partly reflects matcher acceptance.Source: §6 · p. 8

A. The released pairs

  • The released-pair study compares twenty recipes on physics and MATH-500 using eight samples per problem, with outcome-reward recipes and documented exceptions.Physics is at most about 1% of stated RL data for recipes reporting their mix, except ProRL v2; generation protocols agree within 0.01 on cell means.Source: A. The released pairs · p. 13
  • Resampling bases widens at-cap finishing and reliability intervals until they overlap, while reliability’s lead above the cap remains clear of zero at 34 points [19, 62].Across eighteen recipes and both benchmarks, the matched change is 0.014 [0.008, 0.021]; the split’s full-credit change can differ from mean-credit by up to 0.03.Source: A. The released pairs · p. 13
  • At the physics cap, the matched split assigns 45% to finishing, 49% to reliability, and 5% to reach; beyond the cap, it assigns 28%, 62%, and 10%.The matched split shifts shares toward reliability, and for cap-bound recipes the finishing and reliability terms vary by recipe.Source: A. The released pairs · p. 13
  • The registered finishing-share prediction passed for all ten added recipes, while the pooled matched change was +0.017; physics alone averaged +0.042 and QuestA reached +0.108.The pooled result includes MATH-500, where the matched base is near ceiling; the physics-only QuestA value falls outside the registered H2 bound.Source: A. The released pairs · p. 13

A.1. How the traces were read

Trace checks find that many apparent RL-only reasoning steps also appear in base samples, while reliability wins often involve steps present in the base’s own derivations. Regrading and rereading at authors’ temperatures leave the broad split largely intact, with specific caveats from answer form and sparse checks.

A.1. How the traces were read

  • The trace checks sampled RL-only wins, compared derivations, and read some pairs blind; model recipe details and sampling protocols are documented separately.The twenty-pair study includes documented exceptions, and QuestA and KnowRL used guidance in training prompts that was absent from this paper’s prompts.Source: §A.1 · p. 13, Table 6 · pp. 14–15
  • In a blind re-read weighted to 195 wins, new steps appeared on 46% of RL derivations and 50% of base derivations; human checks upheld 19 of 45 decided claims.The human check found no detectable difference between sides, and the authors characterize the readers’ counts as upper bounds.Source: §A.1 · p. 14
  • Of 83 steps flagged in an unblinded reading, 77 appeared in another base sample, and the base’s samples held 9 of the 10 RL-side steps upheld by human checking.The search may lean toward finding steps; the author’s later checks upheld 9 of 11 RL-side steps missed by the forward search on physics.Source: §A.1 · pp. 14–15
  • Among reliability comparisons at 16k, none of the 26 decisive RL steps was absent from base samples; all were used correctly in a base sample graded right.On wrong base samples, the decisive step was stated on all 26 problems, often without being carried through or with another failure.Source: §A.1 · p. 15
  • For a cap-bound comparison, base samples wrote 31 of the 32 deciding RL steps; within the cap, they used 24 correctly and stated 7 without carrying them through.One compared base derivation was cut at the cap on 15 of 40 problems; one step was absent even at 64k.Source: §A.1 · p. 16

A.2. Judge regrade

  • Under judge regrading, net reach moved by less than 0.01 in every cell, while the answer-form share of reliability lies between estimates including an 8% reduction.For the 32B pair, net reach at the cap fell from 0.013 to 0.004; 29 of 68 problems unsolved by the base under the matcher were solved under the judge.Source: §A.2 · p. 16
  • On a regenerated read of sixteen pairs, the judge moved F by at most 0.021, P by at most 0.011, and net reach by at most 0.008; no reliability term changed sign.The second generation moved each split term by at most 0.007 relative to the main read, with only Open-RS3’s reliability term changing sign.Source: §A.2 · pp. 16–17, Table 8 · p. 17

A.3. Authors’ temperature

  • For eight reread pairs, the split did not depend on sampling temperature; each pair reread at authors’ settings split as at 0.6.Skywork-OR1-32B’s net reach fell from 0.013 [−0.001, +0.029] to +0.002 [−0.010, +0.015] at its authors’ temperature.Source: §A.3 · p. 17

A.4. Robust reach by recipe

The robust-reach analysis distinguishes RL wins on problems the base never solves in eight samples from the reverse, and examines whether apparent lone wins survive resampling and trace checks. Most lone wins recover in additional base samples, while many robust cells reflect answer form rather than missing physics knowledge.

A.4. Robust reach by recipe

  • Robust wins count problems one model solves while the other finishes all eight samples without solving; rows are ordered by RL wins meeting that criterion.The appendix’s pooled table uses physics and MATH-500 at each recipe’s cap, with specified exceptions for some recipes.Source: §A.4 · p. 17, Table 9 · p. 18
  • For 13 non-cap-bound recipes, problems the base never got right but sometimes left unfinished account for 10% of summed RL gain; counting them unsolved changes net reach by at most 0.008.Those problems contribute 34% of summed gain for three cap-bound recipes, including 0.066 of ProRL v2’s 0.177; trace checks found base samples writing the RL deciding step on 56 of 57 decisive cases.Source: §A.4 · p. 17
  • With 56 extra samples, four bases solved 87 of 121 physics pairs where an RL recipe had beaten a base that left the problem unsolved in eight samples; only 3 real physics wins remained.The crossing test’s net difference was −0.013.Source: §A.4 · p. 18
  • Across recipes, shared residue problems exceed independent draws on two of three bases with at least three recipes, and one problem is residue for all eight recipes.Of 26 robust cells read, about three fifths were answer form, about a quarter near-misses, and no cell was a physics fact every base lacked.Source: §A.4 · p. 18

A.5. Reach at k = 32

  • After 24 additional samples, RL models solved at least one on 23% of lone-win problems, versus 27% for bases.For recipes trained at 16k and below, the base’s recovery at a larger budget is an upper bound.Source: §A.5 · p. 18

A.6. A harder, unfiltered physics set

Across the harder and unfiltered physics sets, net reach remains small, and the base often solves RL’s apparent lone wins when resampled. The findings persist beyond the base-filtered set, but do not show broad new coverage.

A.6. A harder, unfiltered physics set

  • Net reach is −0.006 to +0.018 on two additional sets, while RL solves at least four of eight samples on at most 10 base-unsolved problems.Bases leave two to three times more finished-but-unsolved problems on these sets than on the paper’s sets.Source: §A.6 · pp. 18–19, Table 10 · p. 19
  • Of 19 robust wins by QuestA and KnowRL, 13 stand after reading; their net reach remains about 0.016 and 0.014, respectively.Three credited answers disagree with the gold, and three more leave only three correct samples out of eight.Source: §A.6 · p. 19
  • Net reach at the cap is −0.009 to +0.014 on the unfiltered set, versus −0.007 to +0.013 on the filtered set.The comparison offers no sign that filtering hid reach.Source: §A.6 · p. 19
  • After resampling, the base solves 244 of 448 RL wins (54%), while RL solves 187 of 289 base wins (65%).The resampled net is 92 problems across 19 pairs, about five per pair.Source: §A.6 · p. 19

A.7. A competition-math set

On competition math, reliability rises where bases usually finish, while net reach remains limited and contamination evidence is narrow. In a separate controlled run, supplying setup information improved training-set solving but did not show a clear held-out gain over control.

A.7. A competition-math set

  • Net reach on 276 competition-math problems is −0.002 to +0.021, clear of zero for nine pairs and touching or straddling zero for eleven.The judge changes net reach by at most +0.001.Source: §A.7 · p. 19, Table 11 · p. 20
  • For all nine recipes whose base finishes at least 90%, reliability rises clear of zero and leads under either split.On matched problems, pass@1(B | fin) rises clear of zero for every such base; selection accounts for part, but not all, of the gains for QuestA and KnowRL.Source: §A.7 · p. 20
  • More reach on 2025 than 2026 is clear of zero only for STILL-3 and Skywork-OR1-7B; the 2026 strata are too small to distinguish contamination from chance.Those strata contain 11 and 13 finished-but-unsolved problems.Source: §A.7 · p. 20
  • The harness reproduces reported AIME 2025 pass@1 within 0.040 for all five models when temperature and budget are comparable.Other gaps range from −0.160 to −0.011 and are attributed to budget or temperature effects.Source: §A.7 · p. 20
  • After resampling, RL solves 44 of its wins again while the base still fails them, versus 17 corresponding base wins—a net of 27, about 1.4 per pair.Initially, the base solves 99 of 159 RL wins (62%).Source: §A.7 · p. 20

B. The controlled run

  • The controlled probe trains Qwen3-VL-8B-Thinking with outcome RL at a 12k-token cap, supplying setup through five channels.The supplied setup is the physical modelling before algebra; one seed was used per treated arm and two for control.Source: §B.1 · p. 21, B. The controlled run · p. 21

B.1. The probe in full

  • The right question nearly triples the training hard set’s share solved at least partly in eight samples, from 0.062 to 0.175, but transfer beyond plain RL is not shown.At 64 samples, 41% is solved only with help and 34% under no tried condition; content transfer is neither shown nor excluded.Source: §B.1 · p. 21
  • No channel raises held-out score clear of zero; the setup-hint leads of +0.009 and +0.019 are below the detection floor.A +2.4-point hint-arm lead on an earlier corpus did not hold at the 16k read.Source: §B.1 · p. 21

B.2. Why the knowledge is not installed

In the controlled run, success is rarely sampled on hard problems, and training credit appears localized to problems where a success is sampled. The setup can help solve training problems, but held-out transfer is not established.

B.2. Why the knowledge is not installed

  • The base solves 24% of the training hard set in 64 samples, rising to 44% at a 16k cap; a group of sixteen sees a success less than half the time at a few percent per sample.Per-visit gains are largest after one to six successes in eight samples: +0.16 to +0.18, versus +0.05 and +0.02 at the extremes.Source: §B.2 · p. 21
  • Finishing explains 48% of within-group reward variance early and 21% once saturated, while problem-specific content is not detected to carry.Finishing’s gradient adds up across the batch.Source: §B.2 · p. 21
  • Setup reward, hinted derivations, self-written setups, and questioner samples mainly change touched problems or format, not success on other problems.Teacher derivations raise touched-problem solving from 12% to 59%, not their neighbours; self-written setups succeed equally often (intraclass correlation 0.03).Source: §B.2 · p. 21
  • On 60 revisited hard problems, solve rate rises from 0.10 to 0.28, or 0.31 with replay; held-out and 293 unrevisited problems do not move.Replay shortens the wait for a success, but transfer to rarely solved problems is not excluded.Source: §B.2 · p. 22, Figure 5 · p. 22

B.3. Setting details

  • The study uses Qwen3-VL-8B-Thinking with GSPO and text-only physics problems; the protocol takes eight samples at temperature 0.6 with a 12,288-token cap.A 16k variant raises the cap to 16,384; greedy decoding is not used because it loops and 17% of items flip between runs.Source: §B.3 · pp. 22–23, Table 12 · p. 22

B.3.1. THE BOTTLENECK IS THE SETUP, AND IT CAN BE SUPPLIED

  • Setup or concept accounts for 41.7% of 120 typed misses, while computation accounts for 1.7%; 36.8% of modelling decisions on held-out parts are wrong.Source: §B.3.1 · p. 23
  • Supplying mined setup raises the share of 116 problems with any scoring sample among four from 0.345 to 0.578; another problem’s claims add only +0.017.A blind grader separates correct from incorrect samples at AUC 0.766.Source: §B.3.1 · p. 23

B.4. The control gains by finishing, and the training cap sets how much

The control’s held-out gain is primarily finishing, with the training cap affecting how much finishing is learned. Across setup-supplying channels, gains are localized, transfer is not established, and some apparent gains reflect scorer or prompt effects.

B.4. The control gains by finishing, and the training cap sets how much

  • The control gains by finishing more: on problems finished at both reads, pass@1(B | fin) is flat as the finished set expands to harder problems.New finishes are right as often as old ones.Source: §B.4 · p. 23
  • The 16k-trained arm learns less finishing and gains a little matched pass@1(B | fin); read at 12k, it scores 0.202 against the control’s 0.242.There is one run per arm, the 16k arm stopped at step 40, and the contrast is not at the same cap.Source: §B.4 · p. 23

B.5. Statistical power

  • Differences smaller than about 0.03 are not treated as claims because intervals omit run-to-run variance; only hint and teacher channels ran at full scale.Adding variance between control seeds puts the minimum detectable effect near 0.03.Source: §B.5 · p. 23

B.6. Channel A: reward the setup

  • The full-scale setup-reward scorer was gamed: appending required units produced a +99% apparent gain and no real one.A powered rerun with a valid scorer was never done.Source: §B.6 · p. 23

B.7. Channel B: supply the setup in the prompt

  • The setup hint helps hinted rollouts in context but shows no matched-data gain; its +0.09 estimate has an interval spanning zero.The hint arm beats an equal-length checklist only on PhysUniBench-OE (+3.9 [+1.1, +6.6]); the PhysOlym-A lead is prompt scaffolding.Source: §B.7 · p. 23, Table 13 · p. 24, Table 14 · p. 24

B.8. Channel C: distil the hinted derivation

  • Hinted derivation distillation raises fully solved rates on early-spliced problems from 12% to 59%, but held-out forced accuracy is 6.2 against control’s 8.1.Untouched training problems decline by −5.1 [−6.8, −3.5], and outputs are about 15% longer.Source: §B.8 · pp. 24–25

B.9. Channel D: make the policy write the setup

  • Policy-written setup blocks yield no held-out gain over no block, and rewarding written sections changes the section rather than accuracy.Under a strict judge, held-out accuracy is 0.202 against control’s 0.238 at 30 updates.Source: §B.9 · p. 25

B.10. Channel E: distil the questioner

  • The question-asking habit is learned but inert: the model asks in 99.5% of samples, while asking samples solve 2.0% versus 2.6% otherwise.The fine-tune failed both gates for starting RL, so no RL run followed.Source: §B.10 · p. 25

B.13. Text-only twins of the training hard set

The text-only-twin analysis finds finishing accounts for part of the credited twins’ pass@1 gain, but its checks do not resolve whether the twins show a content gain. A proposed transfer mechanism is that finishing rewards generalize across prompts, while content rewards act on recurring problems; the setup rubric judges setup claims rather than answers.

B.13. Text-only twins of the training hard set

  • On credited originals, Eq. (2) assigns 71% of the gain to finishing, versus 50% when the cross term is split evenly and 29% when assigned to reliability.The second seed gives corresponding shares of 73%, 51%, and 29%.Source: §B.13 · p. 26
  • Credited twins’ pass@1 excess was +0.093 [+0.005, +0.181], with finishing contributing 0.058 under Eq. (2).The excess mean score was +0.096 [+0.008, +0.182]; the any-solve@8 excess, +0.095 [−0.054, +0.242], included zero.Source: §B.13 · p. 26
  • The twin split finds no content gain but cannot exclude one: reliability contrasts are imprecise, and neither adjusted check clears zero.The credited-minus-other twin reliability contrast was −0.003 [−0.048, +0.037]; adjusted contrasts were +0.061 [−0.029, +0.149] and +0.077 [−0.004, +0.160].Source: §B.13 · p. 26
  • The twins are easier than their originals, and when the base finishes under half of them, finishing dominates; gains needing content only on hardest problems would not show.Source: §B.13 · p. 26

B.14. Why the finishing signal transfers

  • Finishing rewards are shared across prompts and can transfer to held-out prompts, whereas content rewards act only where a problem recurs.The authors connect the latter pattern to replay, which shows success installed on the problem itself.Source: §B.14 · p. 27

C.1. Setup readout rubric

  • The setup readout grades mined setup claims as established, contradicted, or absent, judging setup alone rather than arithmetic or the answer.Source: §C.1 · p. 27

C.2. The judge protocol

The protocol’s grader choice changes absolute scores but leaves differences between systems nearly unchanged. Its matching rules also yield some deterministic decisions, with a model deciding the remainder.

C.2. The judge protocol

  • A deterministic checker decides 43–55% of items; a model given the same rules decides the rest.Graders disagree mostly over whether a bare number matches a gold answer with a unit.Source: §C.2 · p. 27
  • Regrading made every system 6.7–8.9 points stricter, while the between-system margin agreed to 0.11 points.This comparison used six systems on PhysReason held-out (n=526) with a pinned Qwen3-32B grader.Source: §C.2 · p. 27

Related papers

Loading 2610.34894v1…