PaperlayerView exact revision on arXiv

Source-linked AI summary

InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning

Matthew Y. R. Yang, Hao Bai, Ian Wu, Gene Yang, Amrith Setlur, Aviral Kumar

arXiv:2601.14209v1cs.LGcs.AIcs.CL

TL;DR

Outcome-based RL struggles to distinguish correct reasoning steps from errors, making credit assignment difficult in long reasoning traces. InT trains models to propose single-step corrections to failed traces; it yields a ∼14% gain on IMO-AnswerBench [30] and nearly 10% average improvement across four challenging math benchmarks.

  • Problem

    Outcome-based RL does not distinguish correct intermediate reasoning from critical mistakes, while reliably training process reward models to assign step-level credit remains difficult.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    InT has the model compare failed reasoning with reference solutions, propose a single-step correction, and fine-tunes on the intervention before RL post-training.

    Source: §1 · p. 2, §3.1 · p. 5, §7 · p. 16

  • Results

    A ∼14% gain on IMO-AnswerBench [30] accompanies nearly 10% average improvement across four challenging mathematical reasoning benchmarks; the 4B model also surpasses gpt-oss-20b.

    Source: §1 · p. 2, §5.5 · p. 14

  • Takeaways & Limitations

    InT lets off-policy reference solutions improve already capable models while avoiding typical drawbacks of off-policy supervised fine-tuning.

    Source: §7 · p. 16

    The study does not include a PRM-based comparison because high-quality PRMs were unavailable for its Qwen3 models and training them requires substantial compute.

    Source: §5.1 · p. 11

Abstract

from arXiv · show

Outcome-reward reinforcement learning (RL) has proven effective at improving the reasoning capabilities of large language models (LLMs). However, standard RL assigns credit only at the level of the final answer, penalizing entire reasoning traces when the outcome is incorrect and uniformly reinforcing all steps when it is correct. As a result, correct intermediate steps may be discouraged in failed traces, while spurious steps may be reinforced in successful ones. We refer to this failure mode as the problem of credit assignment. While a natural remedy is to train a process reward model, accurately optimizing such models to identify corrective reasoning steps remains challenging. We introduce Intervention Training (InT), a training paradigm in which the model performs fine-grained credit assignment on its own reasoning traces by proposing short, targeted corrections that steer trajectories toward higher reward. Using reference solutions commonly available in mathematical reasoning datasets and exploiting the fact that verifying a model-generated solution is easier than generating a correct one from scratch, the model identifies the first error in its reasoning and proposes a single-step intervention to redirect the trajectory toward the correct solution. We then apply supervised fine-tuning (SFT) to the on-policy rollout up to the point of error concatenated with the intervention, localizing error to the specific step that caused failure. We show that the resulting model serves as a far better initialization for RL training. After running InT and subsequent fine-tuning with RL, we improve accuracy by nearly 14% over a 4B-parameter base model on IMO-AnswerBench, outperforming larger open-source models such as gpt-oss-20b.

1. Introduction

Intervention Training (InT) uses a model’s comparison of failed reasoning traces with reference solutions to propose targeted corrections, addressing credit-assignment problems in outcome-reward RL without explicit value-function training. After SFT on these interventions and subsequent RL, InT improves performance on challenging mathematical reasoning benchmarks.

Introduction

  • Outcome-reward RL treats correct steps like critical mistakes in incorrect long-horizon rollouts, potentially increasing verbosity or premature reasoning shifts and providing no signal when all rollouts fail [5].On sufficiently difficult problems with no correct rollout, advantages collapse to zero.Source: §1 · pp. 1–2
  • Credit assignment is difficult because step-value estimation can require prohibitively expensive branched rollouts, while training and optimizing value functions over long traces remains challenging [18] [39] [63].Even an accurate value function may not identify a useful alternate step among the vast space of possible future steps.Source: §1 · p. 2
  • InT compares an incorrect trace with a reference solution to identify its first error and propose a single-step corrective intervention, enabling successful counterfactual continuations from before failure.The method uses textual comparison as an easier verification task than generating a complete correct solution.Source: §1 · p. 2
  • SFT on targeted interventions followed by RL shifts probability away from incorrect steps; InT avoids branched rollouts, explicit value-function training, and step-level changes to the RL objective.The approach relies on the same model’s instruction-following, verification, and generation abilities rather than a larger model [41].Source: §1 · p. 2
  • After online RL, InT improves performance by nearly 10% on average across four challenging mathematical reasoning benchmarks, including ∼14% on IMO-AnswerBench [30].IMO-AnswerBench consists of IMO-level problems curated by former medalists.Source: §1 · p. 2
  • With strong base models such as Qwen3 and a relatively small (≈1K-example) SFT dataset, InT particularly improves pass@k and yields more on-policy trajectories than reference-solution or reflection-based alternatives.The resulting fine-tuned model is positioned for online RL.Source: §1 · p. 2

2. Preliminaries, Notation, and Problem Statement

The section frames reasoning training as maximizing binary final-answer reward over model-generated traces, then defines credit assignment as identifying how intermediate steps contribute to outcomes. It argues that outcome-level rewards obscure those contributions, motivating a focus on locating errors in incorrect rollouts.

Credit assignment

  • Outcome-reward RL uniformly reinforces every step in successful traces and discourages every step in failed traces, obscuring each step’s contribution to the answer.This can upweight irrelevant steps and suppress locally correct steps, creating noisy learning signals in long trajectories.Source: §2 · p. 3
  • Outcome-reward RL can plateau early or produce excessively verbose or prematurely truncated responses that fail to maximize reward [41] [54] [5] [36].The section argues that avoiding these outcomes requires identifying which steps cause success or failure.Source: §2 · p. 3
  • Process reward models could score individual steps, but their value estimates are costly and unstable to train, and penalizing errors does not find correct replacements.Efficiently searching for replacement steps remains difficult even when an accurate process reward model is available.Source: §2 · p. 3

Problem statement

  • The problem statement targets incorrect rollouts, pinpointing the steps that derail solutions and assuming access to a reference solution for each training problem.The authors note that such references are commonly available in open-source math datasets.Source: §2 · p. 4
  • More than 80% of rollout groups on a curated Olympiad-level math set contain no successful trajectories at training start, making failed rollouts a potential source of otherwise discarded supervision.In those groups, outcome-reward advantages collapse to zero.Source: §2 · p. 4
  • Failed attempts average 10,000 to 15,000 more tokens than successful ones [11] [45], while negative advantages signal suboptimality without specifying a correction [41].The section presents incorrect rollouts as a difficult credit-assignment regime and argues that learning from them can improve sample efficiency when successes are rare.Source: §2 · p. 4
  • More than 60% of first errors in incorrect trajectories arise after the 50th step, so credit assignment must locate errors throughout the trajectory.Figure 2 shows that incorrect rollouts may contain many correct steps before the first mistake.Source: Figure 2 · p. 4, §2 · p. 4

3. Credit Assignment via Self-Proposed Interventions

Intervention Training uses self-verification against reference solutions to identify an error and propose a corrective step, then tests whether continuing from that intervention improves rollouts. The section also compares this approach with hint-guided rollouts and examines factors affecting intervention quality.

3. Credit Assignment via Self-Proposed Interventions

  • The approach uses the same model to compare its reasoning trace with a reference solution, identify where it goes astray, and propose a single-step correction.It exploits the model’s greater reliability at verifying individual steps than generating a correct solution from scratch, avoiding the need to optimize an explicit process reward model.Source: §3 · pp. 4–5, §3.1 · p. 5

3.1. Proposing Interventions through Self-Verification

  • The two-stage procedure verifies and indexes potentially incorrect steps, then proposes an alternative for the first error so the rollout can continue from that correction.Fixing the initial error may make corrections to later errors unnecessary because they can follow from the earlier mistake.Source: §3.1 · p. 5

3.2. Do Interventions Improve Success Rates?

  • Interventions raise average reward from 0.0713% to 1.56% (22×) and solve 80/334 unique problems versus 29/334 when continuing from the error.The figure compares intervention continuations with continuing from the erroneous step or from the preceding prefix without an intervention.Source: §3.2 · pp. 6–7, Figure 4 · p. 7
  • Interventions alone solve 18/176 problems versus 11/176 with hint-guided rollouts, while combining hints and interventions reaches 25/176.Hints guide the initial direction, while interventions provide fine-grained credit assignment when errors arise later in a rollout.Source: Table 1 · p. 6, §3.2 · p. 7
  • Interventions generated without reference solutions outperform naïve baselines but are significantly weaker than interventions generated with reference solutions.The reference-free variant doubles accuracy and increases coverage by 7 over naïve baselines.Source: Table 1 · p. 6, §3.2 · p. 7
  • The instruction-tuned 4B-Instruct model solves 80/334 problems, compared with 73/334 for the reasoning-focused 4B model.The passage attributes the gap to instruction-following failures that can make proposed interventions unusable.Source: §3.2 · pp. 7–8
  • Using Qwen3-30B-A3B-Instruct-2507 to propose interventions yields an approximately 2× improvement in accuracy and solves 21 additional problems out of 334.This comparison suggests intervention quality scales with the capacity of the model proposing interventions.Source: §3.2 · p. 8

4. InT: Intervention Training for Credit Assignment

InT uses supervised fine-tuning on targeted interventions to patch a model’s reasoning before standard outcome-reward RL. Its SFT objective reinforces the original prefix and corrective step, while excluding the suffix and filtering for interventions with a successful continuation.

4. InT: Intervention Training for Credit Assignment

  • The intervention-data SFT patch prepares the base model for subsequent RL, which can then learn from challenging problems that otherwise largely produce incorrect rollouts.The paper describes this patching step as sufficient to enable solutions to previously unsolvable problems and improve performance.Source: §4 · p. 8

4.1. Training via Supervised Fine-Tuning on Intervention Data

  • Cloning the original prefix with the intervention solved 40 more problems out of 235 than omitting the prefix.The authors argue that reinforcing the prefix keeps the intervention relevant when the model encounters the same problem at test time.Source: Table 2 · p. 8, §4.1 · p. 9
  • Including the suffix in SFT nearly halves unique problems solved; filtering for interventions with at least one correct continuation adds 6 out of 235 problems.The resulting configuration clones the prefix and intervention, excludes the suffix, and applies the correctness filter.Source: Table 2 · p. 8, §4.1 · p. 9

4.2. Fine-Tuning via Reinforcement Learning

  • InT patches the model with SFT on intervention data, then applies standard outcome-reward RL to amplify corrective behaviors and learn from previously unproductive rollouts.The data pairs each problem with the rollout prefix up to the first error concatenated with the proposed intervention.Source: §4.2 · pp. 9–10

5. Experimental Evaluation of InT

The experiments evaluate InT as a way to improve credit assignment, comparing intervention-patched models with alternative approaches before and after RL. InT improves pass@k, reduces persistent training failures, and achieves the strongest overall performance on the evaluated benchmarks.

5.1. Experimental Setup, Comparisons, and Evaluation Metrics

  • The training pool comprised about 4,500 hard problems filtered for zero accuracy under 64 or 128 rollouts; intervention filtering yielded 1,076 problems, with 334 selected for naïve-rollout comparisons.Comparisons included reference-solution SFT + RL, standard RL, Self-Reflection SFT + RL, and hint-guided RL.Source: §5.1 · p. 10
  • Evaluation covered IMO-AnswerBench, AMO-Bench, Apex Shortlist, and November 2025 HMMT; the study omitted PRM comparisons because suitable models were unavailable and training required substantial data and compute.The benchmarks were chosen as difficult post-release evaluations, and IMO-AnswerBench and AMO-Bench were filtered for automatically verifiable answers.Source: §5.1 · p. 11

5.2. Results: SFT with Interventions

  • Intervention SFT raised pass@k across k=16–1024 on both training and test splits and increased the likelihood of intervention tokens after fine-tuning.The test evaluation used IMO-AnswerBench; the paper interprets the gains as a stronger initialization for subsequent RL.Source: §5.2 · pp. 11–12, Figure 6 · p. 11, Figure 7 · p. 11
  • InT improved pass@1 and pass@k over Self-Reflection, reference-solution, and DeepSeek-R1 trace SFT, while its training traces had the highest likelihood under the base model.Across trace sources, NLL ranked InT as most on-policy, followed by self-reflection, R1 outputs, and reference solutions; performance ordering tracked NLL, though the paper notes this correlation is not necessarily causal.Source: §5.2 · p. 12, Figure 9 · p. 12
  • Because interventions are short relative to full rollouts, InT retains mostly base-model tokens and keeps next-token entropy roughly comparable to the base model, supporting exploration in later RL.The paper contrasts this with off-policy trace cloning, which raises entropy and can disrupt existing reasoning patterns if fine-tuning continues for longer.Source: §5.2 · pp. 12–13, Figure 8 · p. 12

5.3. Results: InT + RL Solves Most Hard Training Problems

  • InT initialization produced the largest training-reward increase and greatest reduction in the zero-advantage ratio, the fraction of problems with no correct rollouts.The pattern also appeared on test problems; the paper illustrates corrections on two unseen IMO-shortlist problems qualitatively in Appendix F.Source: Figure 11 · p. 13, §5.3 · p. 13

5.4. Results: InT Expands the pass@𝑘Frontier on Test Problems

  • Across RL training iterations, InT achieved the highest test pass@k across all evaluated k, outperforming both the base model and reference-solution SFT.The comparison tracks pass@k from 0 to 400 RL iterations.Source: §5.4 · p. 13, Figure 12 · p. 14

5.5. Results: Performance on Standardized Benchmarks

  • 33.72 average score across four benchmarks for InT exceeded standard SFT+RL (28.26) and the base model (21.17); on IMO-AnswerBench, 25.62 topped standard RL (23.46).The paper also reports that InT’s IMO-AnswerBench score was more than double the base model’s 11.68.Source: §5.5 · p. 14, Table 4 · p. 15

6. Related Work

Related work addresses reasoning credit assignment with outcome rewards, process reward models, feedback, and reference hints. InT distinguishes itself by using the model’s own trace verification to propose targeted corrections for training.

Credit assignment in LLM reasoning

  • Long-horizon RL with outcome rewards faces credit-assignment bottlenecks [7] [16] [46]; PRMs provide dense step rewards [24] [36], but remain difficult to define and optimize [63].Branched rollouts can be costly [18] [39], while human-annotated PRMs can suffer reward hacking [6]. InT uses trace verification to construct a corrective step, amortizing value estimation and policy optimization.Source: §6 · p. 15

Learning from natural language feedback

  • Unlike refinement and text-gradient work aimed at improved outputs [61] [22] [31] or human-provided interventions [37], InT self-proposes short, targeted corrections for on-policy training.Prior intervention work also extends to robotic manipulation and long-horizon control [13] [32] [51]. InT uses the same model that produced the trace, without human feedback or stronger external models; reference solutions improve corrections over base-model priors.Source: §6 · pp. 15–16

Hint-guided RL

  • The paper reports that over 70% of errors appear after the first 50 steps; existing hint-guided methods use reference-solution prefixes and cannot directly tackle errors arising later in a trace.Hint-guided methods can substantially improve exploration on otherwise unsolvable problems, but do not directly address credit assignment on incorrect traces; InT is complementary and can be combined with them [1] [23].Source: §6 · p. 16

7. Discussion and Perspectives on Future Work … Detailed Verification Log

InT uses self-verification to identify errors and propose corrective interventions, then trains on the resulting patched reasoning. The discussion outlines future work on more autonomous self-improvement and credit assignment in continual learning, while the appended prompts specify verification and intervention procedures.

7. Discussion and Perspectives on Future Work

  • InT targets credit assignment by having models verify incorrect rollouts, propose single-step interventions, and train on patched traces before RL post-training.The authors report strong performance in training and test settings, including using off-policy reference solutions to improve capable models without the typical drawbacks of off-policy supervised fine-tuning.Source: §7 · p. 16
  • Future work could train stronger, continually improving verifiers and problem generators to reduce reliance on reference solutions and enable more autonomous self-improvement.The proposed system would combine problem proposal, solution generation, and verifier-provided reward signals; the discussion also names IMO-ProofBench [30] and describes concurrent meta-verifier work by Shao et al. [43].Source: §7 · pp. 16–17
  • Extending credit assignment to continual learning raises questions about tracing errors through evolving summaries or memory and jointly improving memory and generation.The paper frames its current study as single-turn math problem solving and asks how to distinguish generation errors from memory errors when decisions persist through memory representations.Source: §7 · p. 17

High-Level Reference Solution

  • The intervention-generation prompt compares each numbered student substep with a high-level reference solution and seeks the first unrecovered factual or logical error that disrupts reasoning.It requires step-by-step checks, brief justifications for correct substeps, and detailed explanations for errors or gaps.Source: High-Level Reference Solution · pp. 24–25

3. Propose an Intervention Step

  • After identifying the critical error, the prompt asks for one student-perspective intervention inserted before it, offering guidance without giving away the exact answer.The intervention should sketch next steps that lead toward a correct solution while leaving room for learning from the error.Source: §3 · p. 25, §4.3 · p. 26

4. Output Format

  • The prompt requires a detailed verification log and critical-error report, then checks that the intervention neither reveals the answer nor comes after the first critical error.Drafts that fail any self-check must be revised before producing the final intervention.Source: §4 · p. 25, §4.2 · p. 25, Self-check · p. 26

3. Propose Replacement Steps

  • A separate replacement-step prompt allows rewriting existing steps before an error to avoid it or after it to recognize and recover from it.It requests all potential recovery locations and replacement steps in the student’s perspective, followed by a parser-ready list; a correct solution with no critical errors leaves that list empty.Source: §3 · pp. 27–28, §4.3 · pp. 28–29

Detailed Verification Log

  • The sample verification log flags the inference that finitely many 2025-tuples imply eventual periodicity as a critical error and notes that finite-alphabet non-eventually-periodic sequences can have finitely many such blocks.It identifies this inference as invalidating the student’s line of reasoning and says the reference solution demonstrates the stated possibility.Source: Detailed Verification Log · pp. 30–31

Critical Error Report · Intervention Step · B. Intervention Examples

The example identifies an unrecoverable error in inferring eventual periodicity from finite block counts, then inserts a guiding question at the error location. The section also presents four figures labeled as Qwen3-4B-Instruct error-and-intervention examples.

Critical Error Report

  • The student’s Substep 14 claim that a finite set S makes the sequence eventually periodic is false, including for sequences over an infinite alphabet such as Z.The error report describes this as a factual and logical error.Source: Critical Error Report · p. 31
  • A non-eventually-periodic Fibonacci word over {0, 1} has exactly k + 1 distinct blocks of length k, countering the periodicity assumption.The report uses this finite-alphabet example to show that finitely many blocks need not imply eventual periodicity.Source: Critical Error Report · p. 31, Intervention Step · p. 31
  • The error invalidates the argument that S must be infinite and is unrecoverable because the student later accepts infinity based on the flawed premise.The report cites Substep 145 as a later step that accepts ∞.Source: Critical Error Report · p. 31

Intervention Step

  • The intervention at location 14 avoids giving the reference answer 2026 and instead asks the student to explore finite-alphabet sequences and block complexity.The review says it supplies the key insight that non-eventually-periodic sequences can have finite block complexity and is placed at the Substep 14 error.Source: Intervention Step · p. 32

B. Intervention Examples

C. Evaluation Configuration

Experiments with Qwen3-4B-Instruct use the official recommended sampling settings unless specified otherwise.

C. Evaluation Configuration

D. Training Hyperparameters for InT · E. Self-Reflection Baseline

InT consistently outperforms the self-reflection baseline, which rewrites entire solutions rather than proposing single-step interventions. The appendix also identifies the hyperparameter tables used for RL and SFT runs.

D. Training Hyperparameters for InT

  • Table 5 lists the PipelineRL [33] hyperparameters used for RL runs.Source: Table 5 · p. 37
  • Table 6 lists the LLaMa Factory [44] hyperparameters used for SFT runs.Source: Table 6 · p. 37

E. Self-Reflection Baseline

F. Are Interventions Memorized? · G. InT with Interventions From a Stronger Model (Gemini 2.5 Pro)

On two IMO Shortlist problems outside the training set, InT exhibits distinct reasoning behavior from the base model; the paper also evaluates Gemini-generated interventions and full-trace distillation as ways to improve a smaller reasoning model.

F. Are Interventions Memorized?

  • On two IMO Shortlist 2024 problems outside training, the authors compare InT with the Qwen3-4B-Instruct base model to test whether interventions generalize.The examples are Problems C1 and C2.Source: F. Are Interventions Memorized? · p. 38
  • On Problem C1, after getting 3 for n=3, the InT model questioned its initial assumption and updated its hypothesis, unlike the base model’s path.Both models began with an incorrect assumption about the formula; the passage does not specify the revised hypothesis.Source: F. Are Interventions Memorized? · p. 38, Figure 17 · p. 39
  • On Problem C2, both models concluded even cool numbers must be multiples of four, but only InT tested n=12 and reached the more selective correct hypothesis.The n=12 case led the InT model to consider that not every multiple of four is cool.Source: F. Are Interventions Memorized? · p. 38, Figure 18 · p. 39
  • Across 150 RL iterations, the InT-patched initialization improved pass@k consistently, while oracle-trace distillation and the base-model initialization mainly sharpened.The figure compares a base model patched with InT, a base model distilled on oracle traces, and the base model directly.Source: Figure 19 · p. 40

G. InT with Interventions From a Stronger Model (Gemini 2.5 Pro)

  • The Gemini 2.5 Pro experiments ask whether SFT on Gemini-generated interventions improves correct-trace sampling on hard problems.Gemini-generated interventions are referred to as Gemini interventions in this section.Source: G. InT with Interventions From a… · p. 40
  • The experiments compare InT with standard RL and distillation of Gemini’s full expert reasoning traces to patch e3-1.7B on difficult math reasoning problems.e3-1.7B is described as a strong open-source reasoning LLM fine-tuned on Qwen3-1.7B.Source: G. InT with Interventions From a… · p. 40

G.1. Inference-Only Results · G.2. Experimental Setup and Evaluation Metrics · G.3. InT uniformly pushes the pass@𝑘frontier upwards on test problems

The section evaluates inference-time interventions and InT-based RL on hard math problems. On held-out hard problems, pass@k improves throughout RL only for the InT-patched model, while the base and distilled models show no improvement.

G.1. Inference-Only Results

  • Using Gemini solutions instead of references without double-checking interventions yielded 3.8× higher accuracy and 47 more solved problems than naive rollouts.These inference-time results use a different subset of 235 DeepScaleR problems with pass@32 = 0.Source: §G.1 · p. 40
  • Intervention-conditioned rollouts double the number of problems with at least one correct rollout and improve rollout accuracy by more than an order of magnitude.The table compares rollouts conditioned on differently generated interventions.Source: Table 7 · p. 40
  • Using the intervention to isolate localization benefits, pinpointing the mistake location alone led to superior accuracy gains of 2.4× and coverage gains of 22 problems compared to the baseline.Source: §G.1 · p. 40

G.2. Experimental Setup and Evaluation Metrics

  • The hard training set comprises 472 problems the e3-1.7B model could not solve in 32 rollouts; Gemini 2.5 Pro solved 16% in one attempt.The source pool is a 2.5K subset of DeepScaleR problems, and the starting model is e3-1.7B [41].Source: §G.2 · pp. 40–41
  • The study compares InT-related approaches with distillation plus RL, standard RL, and SFT-only patching on hard training problems and standardized holdout tests.Distillation plus RL first trains on full reference solutions; standard RL continues from the same base checkpoint.Source: §G.2 · p. 41
  • Evaluation covers 2025 competition benchmarks and 64 held-out i.i.d. hard problems, with performance also tracked on training problems at 32,768 output tokens.Named benchmarks include AIME2025, HMMT2025, BRUMO2025, and CMIMC2025.Source: §G.2 · p. 41

G.3. InT uniformly pushes the pass@𝑘frontier upwards on test problems

  • On held-out hard problems, RL consistently improves pass@k for InT-patched e3-1.7B, while RL on the base and distilled models makes no pass@k gains.The comparison tracks three models across RL iterations 0–150.Source: §G.3 · p. 41

G.4. InT outperforms distillation on standardized evaluations

On standardized evaluations, InT gains over distillation are attributed to interventions preserving the base model’s distribution better than distilling entire traces; InT followed by RL also leads on in-distribution hard-test pass@8.

G.4. InT outperforms distillation on standardized evaluations

  • RL from the base model did not improve training reward, whereas RL from distilled and InT models did; continued RL on the distilled model reduced held-out pass@1 as training progressed.The distilled model continued making training-set progress, as indicated by a decreasing zero-advantage ratio.Source: Figure 20 · p. 42
  • The comparison used an InT checkpoint trained on 64 hard problems, while RL from distilled and base checkpoints used an expanded set of about 1.3k DAPO problems, including those 64.The expanded sets were used to boost the baselines; RL from the base model otherwise did not raise the reward curve, and distillation on only 64 problems had sharply reduced standardized-evaluation capability after RL.Source: §G.4 · p. 42
  • InT followed by RL achieved the highest pass@8 on the in-distribution hard test set and improved averaged hard-test performance versus distillation despite using only 64 problems.The paper reports gains over distillation on both in-distribution hard tests and standardized hard-problem benchmarks, mainly because interventions alter the base model’s distribution less than distilling the entire trace.Source: Table 8 · p. 42, §G.4 · p. 42

Related papers

Loading 2601.14209v1…