PaperlayerView exact revision on arXiv

Source-linked AI summary

Probe with Participation Trophies: Random-Reward RL as a Probe of LLM Capability

Yu Mao, Lei Yu, Zining Zhu, Yusheng Zheng, Haohang Li, Freda Shi, Yutong Yin, Zhaoran Wang, Jingcheng Niu

arXiv:2610.01066v1cs.CL

TL;DR

The paper asks what explains improvement under random-reward RL and whether a probe reveals a model’s capability or learns the task from labels. It frames random-reward RL as a reachability probe and reports distinct training-response stages across OLMo checkpoints.

  • Problem

    Random rewards can improve LLM performance despite providing no correctness information, while supervised probes leave open whether they reveal model capability or learn the task from labels.

    Source: §1 · pp. 1, 3

  • Method

    The paper formalizes reachability under specified training constraints and uses random-reward RL as a probe whose reward channel carries no correctness information.

    Source: §1 · pp. 2–3, §4 · p. 8

  • Results

    OLMo checkpoints show three response stages: little early improvement even with ground-truth rewards, later gains with ground-truth but weak random rewards, and mid-training gains under some random-reward runs.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Random-reward reachability offers a way to study what further training can attain beyond current task performance and without correctness feedback in the reward.

    Source: §1 · pp. 2–3

    The paper focuses on RL as the vehicle for reachability probing, though other post-training modes could also support such probes.

    Source: §6 · p. 12

Abstract

from arXiv · show

We connect the spurious-reward paradox to a model's reachability and propose random-reward reinforcement learning (RL) as a useful tool for the probing enterprise, addressing a decade-long debate over what probing performance actually reveals about a model. There are two prevailing explanations for the surprising finding that even random rewards can improve the performance of large language models (LLMs): one attributes the gains to particular mechanisms within RL training; the other to data contamination. Our results motivate a different view: spurious-reward RL can probe a model's reachability, or what further training can attain from its current state under specified constraints, beyond what is reflected in its current performance. Two OLMo checkpoints with the same accuracy on synthetic arithmetic (3.5%), for example, reach 8.5% and 55% in their best runs under the same correctness-rewarded RL. Examining OLMo checkpoints across pre-training and mid-training reveals three distinct regimes of training response: early on, RL produces little improvement even when correct answers are rewarded; later in pre-training, rewarding correct answers becomes effective while random rewards remain weak; and, upon entering mid-training, even random rewards can produce large gains. A similar ordering appears in a number-masked supervised fine-tuning (SFT) analysis of these checkpoints, suggesting that the pattern is not specific to a particular RL mechanism. Moreover, RL with random rewards offers a distinctive perspective on what training can make an LLM do, since its reward signal supplies no information about which answers are correct. By asking what training can attain without correctness feedback, it addresses the label-leakage side of a central problem in decodability-based probing: whether a successful probe reveals the model's capabilities or learns the task itself.

1 Introduction

The paper connects gains from random-reward RL to a model’s reachability: what further training can attain from its current state under specified constraints. Across OLMo checkpoints, training response changes by stage, while random-reward RL offers a probe without correctness feedback.

1 Introduction

  • The paper’s account considers both dormant capability and prompt information: a prompt may elicit useful responses that random rewards can reinforce by chance.This offers an alternative to explanations centered on data contamination or particular RL mechanisms.Source: §1 · p. 2
  • Reachability describes how far and readily a model can improve from its current state under constraints on procedure, supplied information, and training budget.The paper proposes random-reward RL as a probe of this property.Source: §1 · p. 2
  • Random-reward RL supplies no correctness information through its reward channel, isolating the label-leakage question of whether a probe reveals capability or learns the task from supervision (Hewitt and Liang, 2019).The authors frame it as a correctness-free probe of dormant capability.Source: §1 · p. 3
  • Early OLMo-2 checkpoints improve little under either reward; later checkpoints respond to ground-truth rewards, and some mid-training random-reward runs reach high accuracy.The authors describe these as dormant, receptive, and autodidactic stages, respectively.Source: §1 · p. 3
  • The authors present reachability as a framework for probing further-training potential and identify training-response changes that current task performance may not fully reflect.They also characterize this as an alternative account of the spurious-reward paradox.Source: §1 · p. 3

2 Background, Related Work, and Experimental Setup

This section reviews spurious-reward RL and the probing debate over whether a successful probe reveals model knowledge or learns the task from labels. It positions random-reward RL as a way to examine training potential without correctness information in the reward.

Background, Related Work, and Experimental Setup

  • In RLVR, a verifier rewards generated responses, and methods such as GRPO can update a model using relative rewards from multiple sampled responses without worked solutions.Prior work found that spurious rewards, including rewards uncorrelated with correctness, can sometimes improve reasoning performance.Source: §2 · p. 3
  • Prior explanations for spurious-reward gains include reduced reward-misalignment damage and entropy minimization, while this paper treats random rewards as a probe of dormant capabilities.Random rewards are presented as having zero correlation with correctness.Source: §2 · p. 3
  • Probing typically freezes a model’s representations and trains a classifier to predict a property, raising whether the probe learned that property or the task from its labels (Hewitt and Liang, 2019).Accounts of decodability consider what is recoverable and usable under different learner assumptions and conditions.Source: §2 · pp. 3–4
  • Because LLM capability probes still largely use task labels, random-reward RL is proposed to test what training can attain while excluding correctness information from the reward signal.This targets the label-leakage concern in probing.Source: §2 · p. 4

3 Explaining the Spurious-Reward Paradox: Learner and Input

Random-reward outcomes vary with both the learner checkpoint and task information retained in the input, beyond rollout quality alone.

Different Models Yield Different Learning Outcomes from the Same Rollouts

  • P2 reached 80.0% from 13.5% using P1 responses, while P1 reached at most 9.5%; with P2 responses, P2’s highest terminal score was 57.0% versus P1’s 14.0%.The comparison uses the same rollouts and random rewards within each source condition.Source: Table 1 · p. 5, §3 · p. 5
  • With P1 rollouts, P2’s mean gain was 4.5 percentage points versus P1’s 0.6; with P2 rollouts, gains were 13.7 versus 2.9.Figure 2 summarizes changes after 500 steps; each group contains 16 runs.Source: Figure 2 · p. 5, §3 · p. 5
  • Because P1 and P2 receive identical responses and rewards within each source condition, rollout quality alone cannot explain their different learning outcomes.The cited account may still explain variation across rollout sources, but the learner contrast requires an additional property of the learner itself.Source: §3 · pp. 4–5

Reachability and an Account of the Spurious-Reward Paradox

  • The proposed account is that a starting model and its input can elicit useful behavior that random updates may reinforce, improving task performance without correctness feedback.The paper presents reachability as a way to study what further training can attain from the current model state.Source: §3 · pp. 6–7

Random Inputs without Task Cues Yield Little Improvement

  • Replacing questions with random tokens while retaining instructions and template markers preserves task cues, but supplies neither the mathematical problem nor its answer.The two random-token input conditions therefore differ in task information despite sharing the absence of a valid problem.Source: §3 · pp. 5–7

4 Reachability Probing with Random-Reward RL

The section defines reachability as constrained training potential and uses random-reward RL to probe it without correctness feedback through the reward channel.

What Do We Mean by Information and Dormant Capability?

  • The paper distinguishes task information from computational accessibility: training can make the same information easier for a model to use.The paper connects this distinction to probing dormant capability through post-training task performance.Source: §4 · p. 7

Reachability

  • Reachability is the highest expected task performance attainable from a checkpoint within a training budget using a prespecified family of allowed adaptation procedures.Its improvement is measured over the checkpoint’s current performance.Source: §4 · p. 8
  • Reachability depends on the task, intervention family, and budget; comparing checkpoints under the same constraints can reveal different training potential despite similar current scores.The budget also describes how much training is needed for a given expected performance.Source: §4 · pp. 7–8

No Correctness Information through Random Rewards

  • When random rewards are independent of the input, response, and training history, replacing the verifier leaves the training-trajectory distribution unchanged.Performance gains can still arise as the model learns to use task information in its inputs more effectively.Source: §4 · p. 8

Random-Reward RL as a Reachability Probe

  • Random-reward RL is an allowed procedure whose expected gain provides a lower bound—not an optimization—on improvement attainable within the procedure family.Repeated runs estimate the gain through their mean terminal performance change.Source: §4 · pp. 8–9

Dormant, Receptive, and Autodidactic — Three Stages of Training Response Revealed by Reachability Probing

  • Figure 5 compares GT and Random reward traces for checkpoints starting at frozen GSM accuracies of 0.5%, 2.5%, and 13.5%.The figure’s thin traces show sampled rollout accuracy and its dashed lines show frozen greedy held-out accuracy.Source: Figure 5 · p. 9, §4 · p. 9

5 LLM Capability Emergence through the Lens of Reachability

Across OLMo-2 checkpoints, training response moves from little gain under either reward to gains under correct rewards, then to large gains under random rewards in mid-training. Similar ordering appears under SFT and across other tasks and models, although transition timing varies.

Dormant, Receptive, and Autodidactic — Three Stages of Training Response Revealed by Reachability Probing

  • At P1: 34B, mean gains are 0.4 points under GT and 0.5 under Random; at P1: 3,896B, GT gains 79.5 points while Random gains 4.3; at P2: +5B, Random gains 26.8 points.At P1: 3,896B, runs end at 61.5–90.5% from a 2.5% start; at P2: +5B, four of eight runs reach 59.0–77.5% from a 13.5% start.Source: §5 · pp. 9–10
  • Across twelve P2 checkpoints, mean Random gains range from −11.8 to +26.8 points and exceed 10 points at only three checkpoints.The checkpoints with gains above 10 points are +5B, +21B, and +42B.Source: §5 · p. 10
  • Full SFT becomes effective earlier than number-masked SFT, whose strongest response occurs later, following an ordering similar to RL.Number-masked SFT masks numbers in questions and solutions but retains wording and solution structure.Source: §5 · p. 10, Figure 6 · p. 10

Similar Patterns across Tasks and Models

  • On code and lookup, random-reward gains are larger at later checkpoints, but transition timing differs by task and model.At 462B tokens, GT training reaches 97.5% on lookup but 8.5% on GSM; in OLMo-3-7B, large random-reward gains appear later in mid-training than in OLMo-2, while tested SmolLM2 checkpoints reach only the receptive stage.Source: §5 · p. 11, Figure 7 · p. 11

6 Discussion and Conclusion

The paper presents random-reward RL as a way to probe training reachability and concludes that other post-training modes could support related probes. Its preliminaries describe how language-model policies, reward signals, GRPO updates, clipping, reference policies, and entropy are used in the training setup.

6 Discussion and Conclusion

  • The paper proposes random-reward RL as a probe of LLM capability development across training regimes.It frames this as a new explanation of the spurious-reward paradox and a way to study reachability.Source: §6 · p. 11
  • The study focuses on RL as the vehicle for reachability-based probing, while other post-training modes could support a broader suite of probes..Source: §6 · p. 12

A.1 Language Models as Policies

  • A language model’s policy is its distribution over next tokens, and RL updates parameters to increase expected reward for generated responses.The response is a rollout whose length depends on the stopping rule and generation cap.Source: §A.1 · p. 20

A.2 Verifiable and Random Rewards

  • GT rewards use a correctness checker, whereas Random assigns independent fair coin flips so correct and incorrect responses have equal reward chances.Random rewards therefore differ from incorrect-answer rewards, which still use correctness information.Source: §A.2 · p. 20

A.3 Group-Relative Advantages

  • GRPO advantages compare each response’s reward with the group’s others and apply that advantage to all generated tokens, not individual reasoning steps.Under Random rewards, advantage signs depend only on coin flips.Source: §A.3 · p. 21

A.4 The GRPO Objective and Clipping

  • GRPO clips token probability ratios to limit incentives to move from the old policy, but clipping does not impose a hard bound on parameter updates.When old and new probabilities match at the same parameters, ratios equal one and clipping leaves the gradient unchanged.Source: §A.4 · p. 21

A.5 Reference Policies and Entropy

  • The reference policy is usually a frozen copy of the starting model; entropy measures concentration of the policy’s probabilities, not response correctness.The paper’s token-entropy diagnostic differs from response-level and extracted-answer entropy.Source: §A.5 · pp. 21–22

A.6 Training Rewards and Evaluation Scores

The section distinguishes fixed-checkpoint performance from what training can attain and situates reachability among work on contamination, memorization, and generalization. It notes that synthetic problems reduce retrieval of specific question–answer pairs but cannot certify absence from a large training corpus.

A.6 Training Rewards and Evaluation Scores

  • Reachability measures how far training can improve a checkpoint under specified constraints, unlike scores that describe what a fixed checkpoint produces.The best observed run establishes an attained score, while estimating how often training attains it requires a separate estimate.Source: §A.6 · p. 22

B Extended Related Work

  • The spurious-reward paradox links where an improved answer comes from with what the improvement reveals about the model.The discussion contrasts contamination, RL training, and probing’s question of whether a measurement reveals knowledge or supplies what was missing.Source: B Extended Related Work · p. 22

B.1 Data Contamination and Memorized Answers

  • Synthetic instances reduce opportunities to retrieve a specific question–answer pair, but template generation alone cannot certify absence from a large training corpus.The authors treat prior exposure to the underlying skill as part of what they want to measure.Source: §B.1 · p. 23

B.2 RL Training and Pre-existing Capabilities

The related work connects training response to starting models, rewards, and inputs, and positions reachability alongside probing methods that decode or intervene on representations. The theoretical section states that pure random-reward training is unchanged by the answer verifier and that updates can depend on the model and its inputs.

B.2 RL Training and Pre-existing Capabilities

  • Prior work relates random-reward outcomes to clipping, entropy, rollout quality, prompt choice, and reward information, while the paper’s shared-rollout comparison holds responses and rewards fixed to examine learner differences.The authors note that this offline comparison does not contradict Chen et al.’s on-policy analysis because their assumptions and object of study differ.Source: §B.2 · pp. 23–24

B.3 Probing, Information, and the Use of Knowledge

  • Reachability complements representation probes by asking what task behavior training can make attainable under specified information and optimization constraints, rather than prespecifying an internal feature.Random rewards remove correctness feedback, but the measurement still depends on the adaptation procedure and does not show that a complete successful procedure existed unchanged in the starting checkpoint.Source: §B.2 · p. 25, §B.3 · pp. 26–27
  • The probing literature distinguishes preserved information from ease of extraction and from whether the model uses decoded information; reachability instead measures what adaptation can attain.The authors also note that random-reward training requires the model itself to perform the task after adaptation.Source: §B.3 · pp. 25–27

C Reachability: Definitions and Theoretical Foundations

  • The theoretical results state that changing the answer verifier leaves pure random-reward training unchanged, bound information from selecting a trained candidate, and show update distributions can depend on the model and inputs..Source: C Reachability: Definitions and… · p. 27

C.1 Reachability under Specified Training Constraints

Reachability measures what a checkpoint can attain under a specified task, adaptation family, and training budget. The appendix formalizes when verifier changes leave training trajectories unchanged, bounds information from selecting runs, and describes the random-reward gradient and checkpoint-scan setup.

C.1 Reachability under Specified Training Constraints

  • Reachability is defined relative to a task, adaptation family, and budget, whose constraints specify inputs, supervision, updates, selection, and stopping rules.Varying the budget describes how much training is needed to attain a given expected score; random-reward probing restricts procedures to correctness-free ones.Source: §C.1 · p. 27

C.2 Verifier-Intervention Invariance

  • Changing the verifier leaves the training-state trajectory law unchanged when the initial state and exogenous inputs are jointly invariant and updates do not depend on the verifier.This does not imply the capability existed before adaptation; GT and steering controls fall outside the result's assumptions, and label-based evaluation must not affect training.Source: §C.2 · pp. 28–29

C.3 Information Introduced by Selecting a Run

  • Selecting one of N fixed candidates can introduce at most log2 N bits of verifier information, although that bound does not limit the selected run's improvement.Under the stated independence assumptions, choosing among eight runs introduces at most three bits; reporting full scores or earlier label-dependent protocol selection is excluded.Source: §C.3 · p. 29

C.4 Moments of the Random-Reward Gradient

  • Random-reward gradients have zero conditional mean, but their covariance can still depend on sampled responses and alter behavior through curvature.For fair-coin rewards with exact population-standard-deviation normalization, the group-size multiplier is about 1.764 times larger at G = 4 than G = 8; implementation details prevent this factor alone from predicting performance ordering.Source: §C.4 · pp. 29–30
  • Changing prompts changes sampled responses and updated parameter directions even when the reward law is fixed, but this calculation does not predict which prompts improve GSM performance.The curvature expansion is only intuition for smooth behavioral quantities, not a formula for experimental accuracy changes.Source: §C.4 · p. 30

D.1.1 OLMo-2 training checkpoints

  • The OLMo-2 scan covers 25 pre-training and 12 mid-training GSM checkpoints, with additional code and lookup scans at specified checkpoints.Mid-training changes task exposure, including GSM-style material, so generated arithmetic evaluation reduces dependence on benchmark answers without excluding familiarity with underlying procedures.Source: §D.1.1 · p. 31, §D.1.2 · p. 31

D.1.2 Synthetic arithmetic

  • Synthetic arithmetic uses 1,000 generated template-based problems, with 200 held out; training and evaluation may share template families.The worked answer is not a random-reward training target, and lenient and strict scoring distinguish numerical correctness from requested answer formatting.Source: §D.1.2 · p. 31

D.1.3 Code generation; D.1.4 Compositional lookup

  • Code evaluation uses 964 MBPP problems and executable tests, while compositional lookup tests one- and two-hop questions over shuffled facts.Visible tests supply task information in code prompts even when random rewards withhold pass/fail feedback; lookup pools 104 one-hop and 96 two-hop items.Source: §D.1.3 · pp. 31–32, §D.1.4 · p. 32

D.3 Training-Input Controls

This section examines input controls and other training conditions for OLMo-2 Random runs, and qualifies how the results and gradient mechanisms should be interpreted. It includes a report of synthetic GSM random terminal accuracy at step 500.

D.3 Training-Input Controls

  • Random-reward D/E controls changed LoRA adapters, but their measured norms do not establish equivalent effective updates.Mean terminal LoRA B-matrix Frobenius norms were 2.099 in D and 2.298 in E. D and E also differ in random strings, vocabulary filters, length targets, and template, so the comparison tests those input conditions together; input lengths were closely matched, with varying deviations by condition.Source: §D.3 · pp. 33–34

D.4 Additional Model Families; D.5 Group-Size and Early-Steering Controls

  • The additional-family comparison uses four checkpoints each from SmolLM2-1.7B and OLMo-3-7B, while earlier controls vary group size and steer the first 50 steps at P2 +5B.The family comparison uses the same 800 training and 200 evaluation GSM problems as OLMo-2, with two GT and eight Random seeds per checkpoint. The earlier controls use GT or inverted-GT rewards for 50 steps before Random for 450, so those trajectories are not zero-label-information probes.Source: §D.4 · p. 34, §D.5 · p. 35, Table 5 · p. 35

D.6 Protocol Selection

  • The exploratory 1,224-run sweep found marginal gain rates of 11.4% at G = 4 and 10.9% at G = 8, so later group-size results apply only to tested conditions.The sweep informed selection of learning rate 3×10−5 and reference coefficient 0.01; reused code/lookup runs from the task-level search do not independently validate those settings.Source: §D.6 · p. 35

D.7.1 Terminal scores and threshold crossings; D.7.2 Gains relative to the starting checkpoint

  • Terminal maxima measure the best observed run, not mean performance or a capability ceiling; scheduled threshold visits can miss between-evaluation crossings.The earlier study defines a jackpot as terminal improvement greater than ten percentage points, whereas the scan’s absolute 50% threshold asks a different question; high intermediate scores lost before step 500 do not count as terminal successes.Source: §D.7.1 · pp. 35–36, §D.7.2 · p. 36

D.8.1 Advantage normalization; D.8.2 Token averaging and response length

  • With Bernoulli(1/2) rewards, the unstabilized advantage factors are 0.875 at G = 4 and 0.9921875 at G = 8; random-reward raw gradients have zero conditional expectation.With random response lengths, the denominator remains inside the expectation because length and score can depend on one another; longer responses alone do not determine gradient-covariance changes.Source: §D.8.1 · p. 36, §D.8.2 · pp. 36–37

D.8.3 LoRA and the optimizer; D.8.4 Clipping in the observed runs; D.8.5 The reference term

  • Only 1,057 of 434,000 newer-experiment steps were flagged for PPO clipping (about 0.24%), but the diagnostic does not measure gradient-altering events or their magnitude.LoRA coordinates, AdamW, clipping, and other optimizer components transform raw gradients; the reference term is not a force toward the reference policy, and the complete Random update need not have zero mean.Source: §D.8.4 · pp. 37–38, Table 6 · p. 37, §D.8.3 · p. 37

D.9 Statistical Analysis

  • With zero successes in eight runs, the marginal 95% success-rate interval’s upper endpoint is about 0.369, leaving substantial uncertainty about the population rate.The analyses use paired seed-level comparisons and Holm correction within seven test families and across 290 evaluable tests, but correction does not account for prior protocol and hypothesis choices.Source: §D.9 · pp. 38–39

E.1 OLMo-2 Checkpoint Development

  • None of 200 P1 Random runs ended at ≥50%, versus 15 of 96 P2 runs; 35 P2 runs reached 50% transiently, while equal-3.5% checkpoints peaked at 8.5% and 55.0% under GT.The 462B and 839B checkpoints had the same frozen greedy accuracy; the later checkpoint’s best GT outcome was higher. All 15 Random runs ending above 50% lenient accuracy fell below that threshold under strict extraction, so output format contributes to measured behavior.Source: §E.1 · p. 39, Table 8 · p. 41
  • Table 9 reports synthetic GSM random terminal accuracy at step 500; seed headings omit 3100, “Max” is the row maximum, and dashes indicate unrun configurations.The supplied passage does not include the table’s cell values, so no accuracy comparison can be summarized.Source: Table 9 · p. 42

E.2 Development across Tasks: Code Generation and Compositional Lookup

Across tasks and checkpoints, the point at which training becomes effective differs: lookup responds to correctness-guided training before arithmetic, while code responds later. Additional models show a similar ordering, and the qualitative SFT comparison follows it too.

E.2 Development across Tasks: Code Generation and Compositional Lookup

  • Lookup becomes responsive to correctness-guided training before arithmetic, while consistently higher code scores appear later, making the three-stage account task-specific.Figure 8 compares code generation and invented-fact lookup across 12 OLMo-2 checkpoints.Source: §E.2 · p. 43, Figure 8 · p. 45
  • All 16 P2 code GT runs score at least 16.5%, with checkpoint maxima of 22.0%–27.0%; most P1 runs remain near zero.P1 Random never exceeds 1.5%, and P1 GT stays at or below 0.5% except at 2,098B, where it reaches 11.0%.Source: §E.2 · p. 43, Table 10 · p. 44, Table 11 · p. 44
  • At 462B, the two lookup GT runs reach 97.0% and 97.5% from a 3.0% frozen score, while the best GSM GT score is 8.5%.The 97.5% lookup run answers 94 of 96 two-hop questions correctly; lookup Random reaches at most 26.0% at this checkpoint, while later P2 maxima reach 81.0% except at +26B.Source: §E.2 · p. 43, Table 12 · p. 44, Table 13 · p. 46

E.3 Checkpoint Development in SmolLM2 and OLMo-3

  • At SmolLM2 checkpoint S4, GT reaches 54% and 78%, while all eight Random runs remain between 2% and 7.5%.Neither protocol reaches 10% terminal accuracy at S1–S3; no Random run gains more than ten points in the four-checkpoint comparison.Source: §E.3 · p. 47, Figure 9 · p. 47, Table 14 · p. 47
  • At OLMo-3 checkpoint O4, both GT runs reach 100% and four of eight Random runs gain over ten points, with a maximum of 90%.At the P1 endpoint O2, GT reaches 90.5% and 93.5% while Random ends between zero and 15%; at O4, Random peaks coexist with variable outcomes across runs.Source: Figure 9 · p. 47, Table 14 · p. 47, §E.3 · p. 48
  • The O2 frozen score changes from 15.5% to 53% under a first-block diagnostic, but that rule can truncate valid solutions and is unavailable throughout the scan.The comparison retains the original scorer before and after training rather than replacing only the baseline.Source: §E.3 · p. 48, Table 16 · p. 48

E.4 The Developmental Pattern under Supervised Fine-Tuning

  • Number-masked form-only SFT becomes effective after fuller-supervision SFT in a pattern resembling RL, despite using neither policy-ratio clipping nor group-relative advantages.The comparison is qualitative, not a numerical equivalence between the training procedures.Source: §E.4 · pp. 48–49
  • The SFT ordering is based on recovered figure values without per-seed results, and its budgets, coverage, selection rules, and evaluation details are not established as matched to RL.The comparison supports examining the order of developmental changes, not numerical equivalence.Source: §E.4 · p. 49

E.5 Training-Input Controls on OLMo-2

At P2 +50B, large gains occur with several task-formatted inputs, but random tokens without the math template leave scores near their starting level. The outcomes vary across seeds, can reverse during continued training, and differ in strength of statistical support.

E.5 Training-Input Controls on OLMo-2

  • Large gains occur with original GSM, shuffled GSM, MMLU questions, and random tokens inside the math template, while random tokens without the template leave scores near baseline.The five conditions are compared at P2 +50B.Source: §E.5 · p. 49, Figure 10 · p. 49, Table 17 · p. 49
  • With random tokens inside the template, seed 31004 rises from 19.0% at step 100 to 68.5% at step 500, but the condition's terminal mean is 17.50% versus a 20.5% frozen score.Without the template, all 160 scheduled scores lie between 14.5% and 23%.Source: Figure 10 · p. 49, §E.5 · p. 50, Table 18 · p. 50
  • Random-reward gains can be transient: MMLU seed 31016 rises to 86.5% at step 500, while shuffled-GSM seed 31016 peaks at 50.0% and ends at 4.5%.Another MMLU seed peaks at 86.0% at step 300 and ends at 22.5%.Source: §E.5 · p. 50, Figure 11 · p. 51
  • Across inputs, the samples do not establish an ordering of input quality, and the same random-reward procedure produces different outcome distributions depending on training input.Original GSM has three terminal gains above ten points and 15 collapses out of 32 runs; MMLU has two gains and one collapse out of 16.Source: Table 17 · p. 49, §E.5 · p. 51
  • Sixteen of 32 D runs change by more than ten points in either direction, versus none in E; positive gains alone, four versus zero, yield p = 0.125.The D/E conditions also differ in token construction, so the comparison does not isolate the instruction effect.Source: Table 17 · p. 49, §E.5 · p. 51

E.6 Training-Input Controls on Qwen2.5 and Llama

The random-token input comparison shows that task-formatted inputs can produce large outcome changes in Qwen2.5 and Llama, while their patterns and statistical support differ. In Llama, high-scoring runs coexist with substantial deterioration.

E.6 Training-Input Controls on Qwen2.5 and Llama

  • With the math template, Qwen’s terminal scores range from zero to 91.5%, including four gains above ten points; without it, scores remain between 43% and 51%.Figure 12 presents the terminal scores for both models and conditions.Source: §E.6 · p. 51, Figure 12 · p. 52, Table 19 · p. 52
  • Llama has one D gain to 64%, but three D runs collapse and its mean falls below its starting score; six of eight D runs lose over ten points from peak to endpoint.Neither model’s D/E comparison of absolute change survives the 15-test correction.Source: §E.6 · p. 52, Figure 12 · p. 52, Table 19 · p. 52

E.7 Shared-Rollout Cross-Training

Shared-rollout experiments compare checkpoint learners on common response streams, while additional controls test group size and early steering. The supplementary statistical comparisons include substantial uncertainty and treat most analyses as exploratory.

E.7 Shared-Rollout Cross-Training

  • 13.56 versus 1.75 points in mean absolute change with P1 responses was the only shared-rollout learner comparison to pass the six-test correction.With P2 responses, mean absolute changes were 14.28 for P2 and 3.63 for P1; these comparisons distinguish response magnitude from whether runs improved.Source: §E.7 · p. 54, Table 21 · p. 54
  • P2 learners responded more strongly across shared sources, but response source also mattered: P2 learners had six improved runs with P2 responses versus three with P1 responses.The exploratory interaction was +6.84 points, with 13 positive and three negative block differences; its tests were outside the planned six-test family.Source: §E.7 · pp. 52, 54, Table 23 · p. 55
  • Prefix scoring reversed the primary-score starting order, and eight P2 runs improved by more than ten points under each source, while P1 had ten or eleven declines.The prefix rule was unvalidated and had no planned confirmatory tests; output-format changes also mean neither extraction rule alone separates mathematical capability from answer presentation.Source: §E.7 · p. 55, Table 22 · p. 55

E.8 Group Size and Early Steering

  • At G = 4, one of five P1 runs and two of five P2 runs gained more than ten points, while none of the corresponding G = 8 runs did; the study does not establish a general group-size effect.The exploratory sweep did not show the same sharp ordering.Source: §E.8 · p. 55, Table 25 · p. 56
  • Three of five GT-steered runs ended with jackpots versus none under inverted-GT steering, but early gains could collapse and the paired comparison was not significant after correction.Two other GT-steered runs reached 78.5% and 90.5% at step 100 before ending at 1.0% and 3.0%, respectively.Source: §E.8 · p. 56, Table 26 · p. 56

E.9 Statistical Comparisons and Uncertainty

  • Except for the separately planned shared-rollout tests, the statistical analyses were exploratory and report nonsignificant comparisons as well as significant ones.Five-seed controls leave substantial uncertainty about jackpot rates.Source: §E.8 · p. 56, §E.9 · p. 57

E.9.1 Training inputs

Training-input comparisons find larger OLMo-2 changes under task-formatted inputs, while checkpoint contrasts show differences in response across development. Earlier controls offer limited evidence, and selected-checkpoint contrasts do not locate a universal transition.

E.9.1 Training inputs

  • 12.59 versus 1.11 points in mean absolute change under D and E, respectively, differed by 11.48 points on OLMo-2; this finding survives correction across all 290 tests.The strongest result is greater change under D, including both gains and losses, not more frequent improvement.Source: Table 27 · p. 57, §E.9.1 · p. 57

E.9.2 Checkpoint development

  • Across eight Random seed blocks, P2 exceeded P1 by 17.32 points (p = 0.0078, pH = 0.0938), summarizing selected checkpoints rather than locating a universal developmental transition.The frequency difference in terminal scores of at least 50% was 15.63 points (p = 0.0156, pH = 0.1719).Source: §E.9.2 · p. 59, Table 30 · p. 59
  • The 462B/839B comparison showed a 25.88-point difference in mean GT terminal score, with all four paired differences positive, but four pairs leave its population magnitude uncertain.Additional-model GT comparisons had only two paired seeds, and code-phase comparisons had one common GT seed and two common Random seeds.Source: §E.9.2 · p. 59, Table 30 · p. 59, Table 31 · p. 59

E.9.3 Earlier controls and individual conditions

  • None of the earlier group-size or steering comparisons survived the ten-test correction; the steering mean-gain difference and jackpot comparison each had unadjusted p = 0.25.Averaging group-size differences over three checkpoints leaves five seed blocks, not fifteen independent observations.Source: §E.9.3 · p. 60, Table 32 · p. 60
  • Within-condition sign tests assess whether nonzero changes are more often positive or negative, not mean gain; SFT has no per-seed results for these tests or interval estimates.Tables 33–36 report within-condition changes and paired GT-minus-Random differences; Tables 38–40 give checkpoint-specific Random-run intervals.Source: §E.9.3 · p. 60, Table 33 · p. 61, Table 34 · p. 62

Related papers

Loading 2610.01066v1…