PaperlayerView exact revision on arXiv

Source-linked AI summary

Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning

Zhuohan Wang, Haoran Ma, Tianyu Wu, Yuanlin Duan, Zichun Liao, Jieming Yu

arXiv:2609.32235v1cs.CL

TL;DR

Multi-step math accuracy can hide whether models fail at planning, executing steps, or combining them. OracleLadder tests milestone solving alongside a fixed ladder of oracle help, and finds the composition gap largest for all six evaluated models. The roadmap effect also appears on other math benchmarks and a code-generation pilot.

  • Problem

    Accuracy compresses distinct reasoning failures, making it unclear which parts of reasoning differ across models or training runs.

    Source: §1 · p. 1

  • Method

    OracleLadder crosses a milestone-alone test with parent-problem probes that progressively provide no help, a fixed roadmap, and the roadmap plus gold milestone answers, using symbolic verification.

    Source: §2 · p. 3

  • Results

    The composition gap is largest for all six models, covering 33–48% of problems and 24–37% after removing cases flagged as grading or format errors.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The diagnostic distinguishes failure structures that aggregate accuracy groups together and tracks how training moves problems between gap states.

    Source: §1 · p. 2, §6 · p. 10

    Category shares depend on the anchor model, source dataset, and student, and the set is a stress set screened by Qwen-base.

    Source: §3 · p. 5, §6 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) can solve every intermediate step of a multi-step math problem on its own and still fail the full problem, even when given a roadmap of the steps and all of their answers. We introduce OracleLadder, a diagnostic evaluation that locates where LLM math reasoning fails by giving the model increasing levels of oracle help. For each problem, a teacher model writes a fixed roadmap of intermediate sub-goals (milestones), and a deterministic symbolic verifier grades every answer. Testing the model with no help, with the roadmap, with the roadmap plus the milestone answers, and on each milestone alone sorts each failure into one of five reasoning gaps. On 354 NuminaMath problems and six models from 8B to 671B parameters (Qwen3, gpt-oss, Llama 3.3, DeepSeek-V3.1), the largest gap for every model is the composition gap, a stricter form of the compositionality gap. It covers 33-48% of problems, and 24-37% after removing problems that an LLM review flags as grading errors. Accuracy and milestone-help recovery rank the two strongest models differently, and two RLVR runs with similar accuracy gains move problems differently. The roadmap effect replicates on MATH500 and AIME 2024/25, per-problem recovery agrees for 83-87% of problems under an independent second teacher, and the help ladder carries over to code generation. We release the data, roadmaps, prompts, and code at https://github.com/slark-prime/OracleLadder.

1 Introduction

Accuracy compresses distinct failures in multi-step math reasoning, leaving unclear whether a model or training run struggles with planning, executing, combining steps, or capability. OracleLadder holds problems and roadmaps fixed while varying oracle help to diagnose these gaps.

1 Introduction

  • Accuracy can group together models or training runs whose planning, step execution, composition, or capability failures differ.This is especially relevant to RLVR, where rewards check final answers and post-training evaluation still relies on final accuracy.Source: §1 · pp. 1–2
  • OracleLadder crosses a milestone-alone test with parent-problem probes under no help, a roadmap, and a roadmap plus milestone answers.A teacher fixes each problem’s roadmap across probes; a deterministic symbolic verifier grades answers.Source: §1 · p. 2
  • The composition gap is largest for all six models, affecting 33–48% of problems and 24–37% after removing cases flagged as grading or format errors.This stricter form of the compositionality gap describes models that solve every milestone alone but fail the parent problem at every help level.Source: §1 · p. 2
  • On 354 NuminaMath problems, the composition gap remains largest across six models from 8B to 671B parameters, including models from other lineages.The reported category shares remain 24–37% after an LLM rubric review flags possible verifier or format errors.Source: §1 · p. 2
  • The roadmap effect transfers to MATH500, AIME 2024/25, and a code-generation pilot, while per-problem recovery agrees for 83–87% under an independent teacher.The contribution summary also reports stable model rankings under different screening models and stability when roadmaps are made four times finer.Source: §1 · p. 3

2 Method

OracleLadder measures whether a model can solve each intermediate milestone alone and how much help it needs to solve the fixed parent problem. Crossing these measurements yields a reasoning-gap diagnosis.

2 Method

  • The method combines a per-family test of solving every milestone alone with the smallest parent-help level that yields a correct rollout.The integration milestone is withheld because its answer is essentially the parent answer and would leak the solution.Source: §2 · p. 3
  • Each milestone family is compiled from a parent problem, its reference solution, and a teacher model, then reused unchanged across probes and evaluated students.Milestones have standalone descriptions, types, answer schemas, and gold answers; the evaluated student does not participate in compilation.Source: §2 · p. 3
  • Each remaining milestone receives eight direct attempts, and a family passes Stage 0 only if every tested milestone is solved in at least one attempt.Because finer roadmaps make this test stricter, the paper reports milestone counts, compares five-way labels only at matched granularity, and also reports fraction passed.Source: §2 · p. 3
  • In the worked example, Qwen-base solves all tested milestones alone but solves the parent only when milestone answers are supplied.This yields a milestone-execution gap in the example shown.Source: Figure 2 · p. 4
  • The three parent probes provide no help, the roadmap, or the roadmap plus gold milestone answers; the outcome is the first level with a correct rollout.The roadmap keeps the integration step private, and symbolic-equivalence checks flag cases where a milestone answer could reveal the parent answer.Source: §2 · p. 4
  • Three matched controls replace the roadmap with another family’s milestones, a generic decomposition prompt, or wrong answers paired with the correct roadmap.These controls test whether recovery depends on problem-matched intermediate information.Source: §2 · p. 4

3 Experimental Setup

The diagnostic set is selected from NuminaMath using a fixed Qwen-base screen, then evaluated across six model instances under a shared rollout and verification protocol. Its category shares describe an anchor- and source-dependent stress set.

3 Experimental Setup

  • The main diagnostic set contains 354 families selected from NuminaMath after teacher compilation and Qwen-base screening.From 2,000 candidate parents, 1,024 yielded valid families, 731 passed the screen, and requiring at least two non-INTEGRATE milestones produced the 354-family set.Source: §3 · pp. 4–5
  • Category shares describe a stress set shaped by the anchor model and source data, rather than a dataset-independent population.A Qwen3-8B direct-failure screen selects 7% of MATH500 but 63% of AIME 2024/25; re-screening with panel models retains positive ranking comparisons.Source: §3 · p. 5
  • The panel includes six model instances from four lineages, including two Qwen RL endpoints trained on a separate 2,000-parent pool.OutcomeRL-2K rewards parent correctness; MilestoneRL-2K adds milestone-level rollouts and also differs in rollout budget, SFT warmup, and curriculum.Source: §3 · p. 5
  • Main parent probes use eight rollouts per condition, a 16,384-token limit, identical prompts, and the same symbolic verifier.A condition counts as solved if at least one rollout verifies; the external MATH hard-100 comparator reports mean single-rollout correctness over eight samples.Source: §3 · p. 5

4 Results

Across the results, problem-matched milestone help supports recovery, but the largest reasoning gap for every model remains solving the parent problem despite passing every milestone alone. The readout also distinguishes models and training runs that aggregate accuracy alone groups together.

4.1 Recovery Comes from Problem-Matched Milestones

  • Correct roadmaps and roadmap-plus-answer help solve substantially more families than direct prompting, while random and generic roadmaps do not improve on the direct baseline.The mismatched-answer control lifts over direct prompting, but falls short of the gold-answer condition.Source: §4.1 · pp. 5–6
  • The matched-roadmap effect also appears on MATH500 and AIME 2024/25, and a code-generation pilot shows increasing pass rates across the help ladder.In the pilot, 6.5%, 41.2%, and 88.8% of attempts pass with no help, a roadmap, and a roadmap plus verified helper functions, respectively.Source: §4.1 · p. 6

4.2 Cross-Model Readout: Direct Accuracy and Diagnostic Recovery Decouple

  • DeepSeek-V3.1 has the highest per-rollout direct accuracy (28.4%), but gpt-oss-20b solves more families directly (45.5% vs. 40.4%) and by any probe (60.5% vs. 51.7%).Table 1 separates direct solving, diagnostic recovery, and external MATH performance on the same 354 milestone families.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Reasoning-Gap Taxonomy: Composition Is the Largest Gap

  • 33–48% of families fall into the COMPOSITION-GAP for every model: each milestone is solved alone, but no parent probe solves the problem.The category remains 24–37% after removing families flagged by an LLM rubric review as verifier or format errors.Source: §4.3 · p. 7, §4.5 · p. 9
  • The taxonomy separates five failure patterns; outside the composition gap, capability gaps vary from 6–16%, roadmap gaps from 7–12%, and milestone-execution gaps from 2–7%.Families that pass the milestone test divide into roadmap, milestone-execution, and composition gaps; those that fail it divide into missing-milestone and capability gaps.Source: §4.3 · p. 7
  • Under an independent teacher, per-problem recovery agrees on the C2 outcome for 83% and the C3 outcome for 87% of 53 families.Five-way labels are more sensitive to teacher changes than these parent-probe outcomes.Source: §4.3 · p. 8

4.4 Longitudinal Readout: Similar RL Gains, Different State Movement

  • The two RL runs achieve similar MATH hard-100 gains (+20.0 and +18.0 points) but change parent-probe outcomes differently.The runs differ in several ways and are not matched in compute, so the comparison does not rank the training methods.Source: §4.4 · p. 8

4.5 What the Composition Gap Contains

  • An LLM rubric review flags 22–29% of each model’s composition-gap families as verifier or format errors; removing them leaves 24–37% in the category.The audit also identifies roadmaps that omit a bridging step and genuine failures to combine steps the model can already do.Source: §4.5 · p. 9

5 Related Work

Prior math evaluations include benchmark scores, structured tests, step judgments, and interventions that alter problems, traces, or hints. OracleLadder instead holds each problem and teacher roadmap fixed while varying the intermediate information shown to the model.

Math benchmarks

  • Aggregate benchmark scores compress different failure types, while OracleLadder adds a diagnostic layer to existing math problems.The cited benchmark landscape spans grade-school, competition, olympiad, and research-level tasks.Source: §5 · p. 9

Structured and intervention-based evaluation

  • Prior evaluations use structured test suites, LLM-judged step quality, problem or trace edits, prompting, and problem-specific hints to assess reasoning.OracleLadder combines a fixed roadmap, graded help, and separate milestone tests to distinguish missing steps from steps available but not combined.Source: §5 · p. 9

Process supervision and RL training

  • Step-level feedback may best fit roadmap and milestone-execution gaps, while composition-gap families already have milestone answers to combine.The longitudinal readout tracks which gap each problem moves through during RLVR training.Source: §5 · p. 10

6 Discussion and Limitations

OracleLadder supports fixed-problem comparisons of models and training runs, while separating grading errors, roadmap omissions, and failures to combine correct steps. The appendix specifies the workflow’s implementation algorithms.

6 Discussion and Limitations

  • On a fixed problem set, OracleLadder supports comparing models by their failure families and tracking how training moves families between states.Counts of families solved by any probe describe the stress set and support comparisons between models and conditions on it.Source: §6 · p. 10
  • The largest gap mixes grading errors, omitted bridging steps, and failures to combine available steps, each pointing developers toward a different intervention.The proposed responses are to improve the verifier and reference answers, expand roadmap coverage, or train for combining already-correct steps.Source: §6 · p. 10
  • Category shares depend on the anchor, source dataset, and student; the 354-family set was screened by Qwen-base.Compiling new roadmaps requires a strong teacher model, while new students can be evaluated on released roadmaps.Source: §6 · p. 10

A Algorithms

  • The appendix specifies milestone-family compilation, Stage 0 testing, and assignment to DIRECT, ROADMAP-NEEDED, ANSWERS-NEEDED, or UNRECOVERED.Compilation validates milestone schemas and family constraints; the test evaluates non-INTEGRATE milestones, and parent-probe outcomes determine the assigned state.Source: A Algorithms · p. 19

B Dataset Funnel and Milestone Types

The diagnostic set is built through sequential filters and is conditioned on Qwen-base screening. The section also describes anchor sensitivity, milestone-type patterns, recovered integration packets, and the exact prompt variations.

B Dataset Funnel and Milestone Types

  • 1,024/2,000 candidates passed teacher compilation, 731/1,024 passed anchor-solvability and milestone-test screening, and 354/731 had at least two non-INTEGRATE milestones.The resulting set is anchor-conditioned on Qwen-base screening.Source: B Dataset Funnel and Milestone Types · pp. 19–20
  • Across 15 anchor pairs, composition-gap rankings had positive Kendall’s τ with a mean of +0.80 under the 16K milestone test.The 4K screening-budget rankings were also positive across all 15 pairs, with mean +0.62.Source: B Dataset Funnel and Milestone Types · p. 20, Table 4 · p. 20
  • Recovered INTEGRATE packets are released as reading aids, not used in main-paper numbers, and their gold answers are not claimed to be universally correct.Strict checks can fail on unusual parent-answer formats even when the teacher’s answer is mathematically defensible.Source: B Dataset Funnel and Milestone Types · pp. 20–21
  • MODEL milestones appeared in 57% and 59% of ROADMAP-NEEDED and ANSWERS-NEEDED families, versus 44% of DIRECT families; the breakdown is descriptive.Milestone types are teacher-assigned and not independently validated; rows are not mutually exclusive.Source: B Dataset Funnel and Milestone Types · p. 21, Table 8 · p. 21

C Exact Oracle Prompts

  • Across prompts, the system prompt and boxed-answer instruction stay fixed while appended context varies across direct, roadmap, answer-aided, and control conditions.Controls use another family’s milestones, a generic decomposition prompt, or correct milestones paired with answers sampled from other families.Source: C Exact Oracle Prompts · pp. 21–22

D Threshold Sensitivity Analysis

The appendix checks whether the diagnostic readout holds across thresholds, controls, model rankings, and milestone-test settings. The reported ordering and key comparisons persist across these checks, with stated qualifications.

D Threshold Sensitivity Analysis

  • 53% of families were UNRECOVERED at τ=1 and 81% at τ=5, and UNRECOVERED remained the largest non-DIRECT bucket at every threshold.ROADMAP-NEEDED and ANSWERS-NEEDED families together declined from 69 at τ=1 to 42 at τ=5; the state ordering stayed the same.Source: D Threshold Sensitivity Analysis · p. 22, Table 9 · p. 22

E Specificity Tables and Per-Contrast Interpretation

F Bootstrap CIs and Pairwise Ranking Stability

  • gpt-oss-20b dominated and Llama-3.3-70B was dominated in 100% of bootstrap resamples; OutcomeRL-2K exceeded MilestoneRL-2K in 81%.The two RL runs were not clearly separated on any-probe count alone, motivating comparison of redistribution shape.Source: F Bootstrap CIs and Pairwise Ranking… · p. 23

G Stage 0 Milestone Solvability and Milestone-Test-Pass Fingerprints

  • 87%–95% per-milestone solve rates show that most unrecovered cases are not explained by missing milestone-level solvability alone.Changing the Stage 0 budget from 4K to 16K shifted per-model pass counts by at most 8 families and composition-gap counts by at most 7.Source: G Stage 0 Milestone Solvability and… · p. 24

H Aggregate-Comparator Evaluation Settings and External Reference Table

The external comparator uses a separate set of public MATH level-5 problems, so it provides supporting context rather than the primary same-set comparison. The appendix specifies shared evaluation settings and notes a formatting caveat.

H Aggregate-Comparator Evaluation Settings and External Reference Table

  • MATH hard-100 is a different problem distribution from the 354-family diagnostic set and serves as supporting context, not the primary comparison.The comparator uses the same 100 level-5 problems from the MATH500 test split for all models.Source: H Aggregate-Comparator Evaluation… · p. 24
  • All six models use the same prompt, decoding budget, eight rollouts, and symbolic verifier on the 100 problems.The models use native chat templates; decoding uses max_tokens=16384 per rollout.Source: H Aggregate-Comparator Evaluation… · pp. 24–25
  • pass@1 is mean single-rollout correctness, while pass@8 is the solve rate with at least one correct result across eight rollouts.Both metrics use the same sample of 100 problems.Source: H Aggregate-Comparator Evaluation… · p. 25
  • A correct answer may lose credit if it omits boxed formatting and cannot be extracted.The appendices measure the resulting verifier and format errors.Source: H Aggregate-Comparator Evaluation… · p. 25

I Training Dynamics: Endpoint Numbers and Trajectory Tomography

Across training checkpoints, diagnostic fingerprints expose changes that aggregate accuracy misses. The appendix also shows that decoding budget can alter model rankings and interpretations of training trajectories, while the held-out specificity pattern recurs directionally.

I Training Dynamics: Endpoint Numbers and Trajectory Tomography

  • MilestoneRL-2K’s any-probe count fell from 152 to 114 at step 30, remained about 30 families below Qwen-base for roughly 120 steps, then recovered to 165 at step 180.OutcomeRL-2K stayed near baseline through step 120, dipped to 129 at step 150, and reached 170 at step 180; both trajectories include below-baseline states as MATH hard-100 pass@1 trends upward.Source: I Training Dynamics: Endpoint Numbers… · pp. 25–26

J RL Training-Recipe Disclosure for OutcomeRL-2K and MilestoneRL-2K

  • OutcomeRL-2K and MilestoneRL-2K reached near-identical MATH hard-100 accuracy, 0.654 and 0.634 pass@1, while their diagnostic fingerprints differed.The runs differ in multiple training dimensions beyond reward construction, so they are a longitudinal diagnostic case study, not a controlled ablation of milestone reward.Source: J RL Training-Recipe Disclosure for… · p. 27

K Cross-Model Panel Details

  • Figure 8 partitions each model’s 354-family outcomes by the smallest parent-probe help level that unlocks a solution: DIRECT, ROADMAP-NEEDED, ANSWERS-NEEDED, or UNRECOVERED.Table 21 reports the aggregate counts behind the bars.Source: K Cross-Model Panel Details · p. 27, Figure 8 · p. 28

L Decoding-Budget Sensitivity and Format Audit

  • At 16K rather than 4K, OutcomeRL-2K, gpt-oss-20b, and DeepSeek-V3.1 gained 14–42 any-probe families, while the other three models changed by at most 7.At 4K DeepSeek ranked fifth on any-probe count (141/354), but at 16K it ranked second (183/354); reading the Qwen training trajectory at 4K also changes its interpretation. DeepSeek truncation was 53.8% at C1 and 44% at C3-gold at 4K.Source: Table 22 · p. 29, L Decoding-Budget Sensitivity and… · p. 29

M Held-out Replication Table

  • On a 32-family held-out slice, correct roadmaps and roadmap-plus-answers lifted recovery, while random and generic controls did not; the authors treat this as directional replication, not fully powered confirmation.The mismatched-answer control lifted above C1 but stayed below C3-gold.Source: Table 25 · p. 30, M Held-out Replication Table · p. 30

N Replication on MATH500 and AIME 2024/25

On MATH500 and AIME 2024/25, matched roadmaps recover more families than corrupted or generic roadmaps, although the five-way gap distribution differs from the main set. A small LiveCodeBench pilot shows a similar ladder of help.

N Replication on MATH500 and AIME 2024/25

  • On AIME 2024/25, the matched roadmap recovered 17 of 38 families (45%), versus 5 of 38 (13%) for each control; on MATH500 it achieved 106 of 280 attempts (37.9%), versus 10.0% for C1.On MATH500, random and generic roadmaps reached 7.9% and 9.6%; paired exact tests on AIME gave p=0.004 and p=0.002 against the controls.Source: N Replication on MATH500 and AIME… · p. 31
  • Gold milestone answers added little over matched roadmaps on these datasets: success was 107 versus 106 of 280 attempts on MATH500 and 84 versus 74 of 304 on AIME 2024/25.On the 354-family set, C3-gold solved more families than C2-correct for four of six panel models.Source: N Replication on MATH500 and AIME… · p. 32
  • The five-way category shares differed from the 354-family set: Qwen3-8B passed the milestone test on only 0 of 35 MATH500, 1 of 38 AIME 2024/25, and 2 of 21 AIME 2025 families.Almost all unrecovered families in these datasets failed the milestone test and fell into CAPABILITY-GAP; more tested milestones and lower solo milestone solve rates contributed to low pass counts.Source: N Replication on MATH500 and AIME… · p. 32
  • Category shares depend on dataset and student, so the authors report COMPOSITION-GAP counts alongside milestone-test pass counts; the matched-versus-corrupted roadmap contrast is what replicates.The screen retained 35 MATH500, 38 AIME 2024/25, and 21 AIME 2025 families.Source: N Replication on MATH500 and AIME… · pp. 31–32

O Code-Generation Pilot on LiveCodeBench

  • With Qwen3-8B on 10 LiveCodeBench problems, per-attempt success rose from 6.5% with no help to 41.2% with a helper plan and 88.8% with verified helpers.Problem-level recovery rose from 1 to 6 to 10 of 10; the pilot uses test execution as its verifier and helper functions in place of math milestones.Source: O Code-Generation Pilot on LiveCodeBench · pp. 33–34

P Teacher and Granularity Robustness

Roadmap changes leave per-family recovery mostly consistent, but can alter five-way labels because the milestone test depends on the roadmap. The authors therefore distinguish recovery comparisons from label comparisons.

P.1 A Second Teacher

  • Inkling’s roadmaps averaged 6.8 tested milestones per family versus 2.1 in the released set, making step-by-step alignment ill-defined when a coarse milestone spans several fine ones.Across all milestones, Inkling wrote 8.1 per family against 5.8 for GPT-5.4.Source: §P.1 · p. 35

P Teacher and Granularity Robustness

  • Under a second teacher, per-family C2 and C3 recovery agreed for 83% and 87% of families; under roadmaps about four times finer, C2 and C3 outcomes each agreed for 85%.Aggregate recovery shifted slightly under the second teacher, so the authors recommend comparing recovery levels under one teacher.Source: §P.1 · p. 35, §P.3 · p. 36

P.2 Five-Way Labels Under the Second Teacher

  • Five-way labels agreed for 22 of 39 families under the strict test, rising to 29 of 39 when passing meant solving at least half the tested milestones.Under the strict test, 11 of 17 disagreements occurred despite matching parent-probe outcomes, because the milestone test alone flipped the label.Source: §P.2 · p. 36

P.3 Controlled Granularity Ablation

  • Coarse and fine roadmaps produced the same C2 outcome for 28 of 33 families (85%) and the same C3 outcome for 28 (85%), while aggregate recovery shifted by one family.C2 recovery was 11 versus 10 families, and C3 recovery was 13 versus 12.Source: §P.3 · p. 36

P.4 How to Compare Readouts Across Roadmaps

  • Because the strict milestone test requires solving every roadmap milestone, the authors compare five-way labels only across matched-granularity roadmaps and report milestone counts and solve rates alongside the results.They use fraction-of-milestones passing as a secondary readout when roadmap lengths differ.Source: §P.4 · p. 37

Q Milestone Answers in the Models’ Own Solutions

Milestone answers appear more often in unaided solutions to their own problems than in solutions to other problems, and an independent roadmap fits those solutions similarly when answer length is matched. The appendix also reports training-state transitions and audits of unrecovered families, including the effects of reruns and possible answer leakage.

Q Milestone Answers in the Models’ Own Solutions

  • 13.6–17.2% of milestone–attempt pairs contain the gold answer, versus 3.8–5.2% for the cross-problem control; for statement-novel answers, the rates are 9.4–12.5% versus 0.8–1.7%.The comparison uses 50 families with eight unaided attempts each; 104 of 110 gold answers are statement-novel.Source: Q Milestone Answers in the Models’ Own… · p. 37, Table 35 · p. 37
  • For three models, milestones solved alone appear more often in unaided attempts than unsolved milestones: 14.6% vs. 0.0% for Qwen-base, 18.6% vs. 1.4% for MilestoneRL-2K, and 17.7% vs. 2.8% for DeepSeek-V3.1.Llama-3.3-70B-Instruct shows no such link (14.3% vs. 17.2%); only 8 of its 110 milestones are ones it cannot solve alone.Source: Q Milestone Answers in the Models’ Own… · p. 37, Table 36 · p. 38
  • At matched answer lengths, released and Inkling roadmaps fit the same unaided solutions similarly: containment-rate ratios are 0.76–1.10 for answers up to 25 characters and 0.80–0.93 for answers up to 15.At raw length, released answers are shorter (median 15 vs. 19 characters) and appear 1.02–1.51 times as often.Source: Q Milestone Answers in the Models’ Own… · p. 38, Table 37 · p. 38
  • The authors read these findings as evidence that roadmaps capture problem structure: their milestones appear in same-problem solutions more often than in other-problem solutions, and an independent roadmap fits similarly at matched answer length.The teacher compiles each family from the parent problem and reference solution without seeing evaluated-model outputs.Source: Q Milestone Answers in the Models’ Own… · pp. 37–38

R Transition Matrix Counts

  • The OutcomeRL-2K → MilestoneRL-2K transition endpoints differ most on families not directly solved by both arms, indicating that the training arms reshape residual recovery structure as well as direct accuracy.Source: R Transition Matrix Counts · p. 38, Table 38 · p. 38

S Audit Procedure Details

  • On the 100-family audit, a separate validation pass reached κ=0.64 and 76% raw agreement; it found fewer genuine composition failures (12% vs. 18%) and more verifier artifacts (34% vs. 26%).The audit uses mutually exclusive categories and applies a fixed priority rule to resolve disagreements.Source: S Audit Procedure Details · p. 39, Table 39 · p. 39
  • Of 18 families initially labeled genuine composition, 11 produced at least one success in a fresh eight-rollout pass and only 7 remained unrecovered, exposing sampling variation in the original triage.The original UNRECOVERED designation meant zero successes in eight attempts; the audit reports both the original classification and fresh rerun fingerprint.Source: S Audit Procedure Details · p. 40
  • Applying the broader 16-family leak subset moves at most two families per model from ANSWERS-NEEDED to UNRECOVERED, leaving the 33–48% composition-gap range and cross-model ordering unchanged.The audit counts 10 strict, 13 strict-plus-trivial, and 16 verifier-accepted cases where an intermediate answer could equal the parent answer.Source: S Audit Procedure Details · pp. 40–41

S.1 Audit Provenance, Automated Review, and Raw-Output Retest

The appendix checks how author audits, automated review, raw-output retests, and leak-safe repairs affect the composition-gap interpretation. It finds grading errors in a substantial share of flagged families, while also documenting a direct failure to combine supplied steps and limits on what the audited subset establishes.

S.1 Audit Provenance, Automated Review, and Raw-Output Retest

  • Removing verifier or format flags lowers the 16K composition-gap range from 33–48% to 24–37%; the six-model ranking changes only between models within one percentage point.The automated review flags 22.2–29.1% of each model’s composition-gap families; it labels 4K snapshot families and carries those labels to the 16K sets.Source: §S.1 · p. 41, Table 42 · p. 41
  • On ff548f26, Qwen3-8B solves both milestones in 8 of 8 attempts, receives both correct answers, and still fails all eight parent attempts, while gpt-oss-20b solves the parent in 6 of 8.The retest used a stand-in 8B student because the audited base checkpoint was no longer available.Source: §S.1 · p. 42, Table 43 · p. 42
  • With every milestone answer supplied, Qwen3-8B solved at least one attempt on 16 of 18 audited families, while gpt-oss-20b did so on all 18.The two families separating the models include ff548f26 and 0ad0ce27; the former is the direct failure-to-compose example.Source: §S.1 · p. 42, Table 43 · p. 42
  • The 18 genuine-composition families were author-audit-selected, so their counts show the failure exists but do not estimate its per-model grading share, which comes from the review above.Two author-team raters independently labeled 100 unrecovered MilestoneRL-2K families and knew every sampled family was unrecovered.Source: §S.1 · pp. 41–42

T Leak-Safe Repair Rubric and Examples

  • The natural repair is excluded because all 47 decomposition-repaired families gained a milestone answer equivalent to the parent answer, making C3 success uninformative about composition.Source: T Leak-Safe Repair Rubric and Examples · p. 43
  • A leak-safe repair excludes regenerated milestones whose answers equal or trivially transform the parent answer, or reduce the parent solve to a final formatting step.Symbolic equivalence is mechanically checked; the other two conditions require human review, stronger normalization, or teacher-side constraints.Source: T Leak-Safe Repair Rubric and Examples · p. 43

Repair leakage examples

  • In the allowed verifier-repair example, tightening answer normalization changes oracle counts to C1=4, C2=2, and C3=7 without adding milestones or exposing the parent answer.By contrast, the leaky decomposition example yields C3=2 because its final-synthesis milestone gives the parent answer verbatim.Source: Repair leakage examples · p. 44

U Teacher Prompt (Excerpt)

  • The teacher prompt requires individually solvable milestones that cover the solution path, each with a specified answer schema and canonical answer; schema-invalid milestones are regenerated.The teacher selects the schema before writing the milestone description.Source: U Teacher Prompt (Excerpt) · p. 44

V Verifier Cascade: Extended Characterization

The verifier audit finds no observed false positives in its paired sample, but adjudication also identifies false negatives involving unusual answers or defective references. A frontier-judge comparison shows a one-sided disagreement pattern that differs sharply between NuminaMath and AIME; the pipeline assigns deterministic parsing and equivalence checks to each answer schema.

V Verifier Cascade: Extended Characterization

  • 0 strict-cascade false positives were observed in 400 paired grades, with a 95% upper bound of 0.75%; adjudication estimates cascade grading errors at about 8.7% of graded responses.The cascade’s errors include correct responses it cannot parse or match and responses graded against defective references.Source: V Verifier Cascade: Extended… · pp. 44–45

V.1 Frontier-Judge Comparison

  • The frontier judge accepted 127 responses (12.7%) that the cascade rejected and rejected 1 (0.1%) that it accepted; disagreements were 17.9% on NuminaMath parent problems versus 0.7% on AIME.A grader-blind adjudication of 34 cases found 18 correct answers the cascade could not parse or match, 6 defective references, and 10 judge over-acceptances.Source: §V.1 · pp. 45–46, Table 46 · p. 46
  • About 68% of the 997-response frontier-judge disagreement sample reflects cascade strictness or defective references, while about 32% reflects judge over-acceptance.The judge’s over-acceptance covers about 4.0% of graded responses in that sample.Source: §V.1 · p. 46, Table 47 · p. 46

W Answer Schema Definitions

Related papers

Loading 2609.32235v1…