PaperlayerView exact revision on arXiv

Source-linked AI summary

Inject a Turn, Not a Token: Learning Where to Redirect LLM Reasoning for Self-Correction

Jiwon Jeon, Myungsik Cho, Jeonghye Kim, Youngchul Sung

arXiv:2610.13672v1cs.CLcs.AI

TL;DR

A single autoregressive mistake can derail reasoning, and self-correction from a rollout alone may be unreliable, motivating the search for where and how to intervene. UPTURN learns a cut and redirects a failed rollout with a new user-turn instruction; across four reasoners and six math benchmarks, it consistently improves average accuracy and outperforms correction baselines under matched budgets.

  • Problem

    A single mistake can steer autoregressive reasoning off course, while a model may struggle to recognize its own error from a rollout alone.

    Source: §1 · pp. 1–2

  • Method

    UPTURN uses lightweight learned heads to select a cut in a failed rollout and injects a fixed instruction as a new user turn before regenerating the suffix, without fine-tuning the reasoner.

    Source: §1 · p. 2, §5.1 · p. 5

  • Results

    Across four reasoners and six math benchmarks, UPTURN consistently improves average accuracy and outperforms self-correction and reflection-token injection baselines under matched correction budgets.

    Source: §7 · p. 9

  • Takeaways & Limitations

    UPTURN transfers across reasoners and to GPQA-Diamond without retraining, and complements GRPO and self-correction methods.

    Source: §1 · p. 2

    The study focuses on reasoning tasks with automatically checkable outcomes and a single deliberately minimal intervention; richer interventions or harder-to-verify settings may require different choices.

    Source: B LIMITATIONS · p. 13

Abstract

from arXiv · show

Large language models struggle to reliably identify and correct errors in their own reasoning. Unlike prior training-based approaches that fine-tune an LLM to generate revisions or feedback, we learn where to intervene in failed rollouts and determine how to redirect them. To choose where to intervene, we train a 130K-parameter MLP head on a frozen encoder using only final-answer correctness. For redirection, we find that under a fixed generation budget, the tested inline injections fail to reliably redirect reasoning, whereas presenting the same instruction in a new user turn changes the model's course. The resulting correction framework, UPTURN (User-Prompt TURN injection at a learned position), requires no LLM fine-tuning: it retains a selected prefix, injects an instruction as a new user turn, and regenerates the suffix. Across four reasoners and six math benchmarks, UPTURN improves average accuracy in every setting over a 32-rollout baseline. It outperforms self-correction and reflection-token injection baselines under matched correction budgets, transfers across reasoners and to GPQA-Diamond without retraining, and complements GRPO and self-correction methods.

1 INTRODUCTION

A single reasoning mistake can derail an otherwise reachable solution, while self-correction from a rollout alone may be unreliable. The paper asks where to intervene and how to redirect failed rollouts, finding that delivery through a new user turn can redirect reasoning when inline injection does not reliably do so.

1 INTRODUCTION

  • Because autoregressive generation conditions each step on preceding reasoning, one mistake can steer a failed rollout away from a solution the model may already be able to reach.The paper frames correction as redirecting an existing trajectory rather than teaching reasoning de novo.Source: §1 · p. 2
  • Under a fixed generation budget, tested inline tokens and instructions fail to reliably redirect reasoning, whereas the same instruction can redirect it when delivered in a new user turn.The authors call the transition from an assistant continuation to a new user turn a turn boundary.Source: §1 · p. 2
  • Across four reasoners and six math benchmarks, UPTURN raises average accuracy from 37.1% to 55.3% and outperforms self-correction and reflection-token baselines under matched correction budgets.It also outperforms heuristic cuts and inline injection.Source: §1 · p. 2
  • UPTURN learns where to intervene with a 130K-parameter head on a frozen encoder, using final-answer correctness rather than step-level annotations or reasoner fine-tuning.It retains a selected prefix, injects a fixed instruction across a turn boundary, and regenerates the suffix.Source: §1 · p. 2
  • UPTURN transfers across reasoners and to GPQA-Diamond without retraining, and complements GRPO and existing self-correction methods.These are among the paper’s stated contributions.Source: §1 · p. 2

2 RELATED WORK

Prior work corrects reasoning through prompting, reasoner fine-tuning, external critics, or reflection-token injection. UPTURN instead selects where to intervene in a completed rollout and delivers a fixed instruction as a separate user turn.

2 RELATED WORK

  • Prompting-based self-correction asks a model to critique or reflect on an attempt and revise it without updating weights, but relies on the reasoner to diagnose errors reliably.Reflexion generates a verbal reflection to guide a fresh attempt.Source: §2 · p. 2
  • Training-based approaches fine-tune reasoners to revise answers or train critics and process reward models to score steps.These approaches typically use correction data and train an LLM-scale component; UPTURN instead uses lightweight heads and keeps the reasoner frozen.Source: §2 · pp. 2–3
  • Reflection-token methods inject or promote tokens during ongoing generation, while UPTURN corrects completed rollouts flagged as likely incorrect.Under a matched generation budget, tested inline injections fail to reliably redirect reasoning even at learned cut positions, whereas the same instruction as a separate user turn does.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe generation as autoregressive across user and assistant turns, then define the math-answer reward and introduce the group-relative optimization method used to train the position policy.

3 PRELIMINARIES

  • The conversation is represented as concatenated user and assistant turns, and the autoregressive model conditions each generated token on preceding tokens.A question and response form a two-turn conversation; the assistant turn remains open during response generation.Source: §3 · p. 3
  • In the math setup, a chain-of-thought rollout is generated in one assistant turn, and its last boxed expression is scored by exact match with the ground-truth answer.The setup uses the reward defined in Equation 2.Source: §3 · p. 3
  • GRPO forms response advantages by centering and scaling rewards within each input’s sampled group, then applies PPO’s clipped surrogate objective, typically with a KL penalty.The paper adapts this group-relative construction to train the position policy.Source: §3 · p. 3

4 MOTIVATION

The section asks where to intervene in failed reasoning and how to redirect it. It finds that turn-boundary delivery of an instruction yields higher continuation success than inline injection, while recovery also depends on the retained prefix.

4 MOTIVATION

  • RLVR’s concentration of updates at high-entropy decision points motivates testing whether analogous inference-time interventions can redirect failed traces.The section frames correction as choosing where to intervene and how to redirect, rather than assuming that each failed rollout requires reasoning from scratch.Source: §4 · p. 3

4.1 RECOVERY DEPENDS ON THE RETAINED PREFIX

  • Continuation success on failed Qwen3-4B traces is high at earlier prefixes but drops sharply and stays low at later prefixes.The study samples 16 continuations from each of ten equally spaced prefixes per trace; this motivates cutting before the transition to preserve a prefix from which recovery remains possible.Source: §4.1 · p. 4

4.2 INLINE VS. MULTI-TURN INJECTION

  • Multi-turn injection yields substantially higher continuation success than inline injection, although tested inline texts remain close to uninjected regeneration.The comparison varies the injection format while holding the model and injected text fixed; the same text is presented either within the assistant turn or as a new user turn.Source: §4.2 · pp. 4–5, Figure 2 · p. 4

5 METHOD

UPTURN selects a cut in a failed rollout, retains its prefix, inserts a fixed instruction as a new user turn, and regenerates the suffix. Lightweight heads learn cut positions from regenerated-answer correctness and gate which rollouts to correct, while the reasoner and encoder remain frozen.

5 METHOD

  • UPTURN retains a selected prefix, injects a fixed instruction as a new user turn, and has the frozen reasoner regenerate the suffix.The reasoner produces the original and revised rollouts; only lightweight heads are trained, with no LLM fine-tuning.Source: §5 · p. 5, Figure 3 · p. 5

5.1 POSITION POLICY

  • A 130K-parameter MLP scores paragraph-boundary cuts using causal encoder states that depend only on the question and retained prefix.The normalized scores define a policy over candidate cuts, avoiding evaluation at every token.Source: §5.1 · p. 5

5.2 TRAINING THE POSITION POLICY

  • The position policy is trained to favor cuts whose regenerated suffixes more often produce correct final answers.For each failed rollout, sampled cuts form a GRPO group; averaging correctness over multiple suffixes estimates each cut’s reward, and within-rollout advantages compare positions on the same trajectory.Source: §5.2 · p. 6

5.3 GATED INFERENCE

  • A 66K-parameter gate estimates rollout correctness; scores below the fixed threshold γ=0.4 trigger one correction, while retained rollouts stay unchanged.The gate uses the frozen encoder’s hidden state at the final boxed answer and is trained once on 2,874 correctness-labeled rollouts, then reused across reasoners.Source: §5.3 · p. 6

6 EXPERIMENTS

The experiments evaluate UPTURN across reasoners and benchmarks, compare it with correction baselines, and test its transfer and design choices. The results assess learned-position, turn-boundary correction within fixed generation budgets.

6.1 SETUP

  • The evaluation covers six math benchmarks disjoint from training and four frozen reasoners, with Base@32 and UPTURN@16 allowing up to 32 generations per question.Each correction reuses a retained prefix and generates at most the remaining response-token allowance.Source: §6.1 · pp. 6–7

6.2 MAIN RESULTS

  • UPTURN@16 improves average accuracy over Base@32 on every benchmark, despite Base receiving 32 full rollouts.Pass rate improves in all but one setting across four reasoners and six math benchmarks.Source: Table 1 · p. 7, §6.2 · p. 7

6.3 COMPARISON WITH SELF-CORRECTION BASELINES

  • At Lmax, UPTURN exceeds the strongest baseline, Reflexion, by +16.2 points in average accuracy and +15.9 in pass rate.At 2Lmax, it retains +4.9 points in average accuracy and +0.8 in pass rate over the best baseline.Source: §6.3 · p. 7, Table 2 · p. 8

6.4 GENERALIZABILITY OF UPTURN

  • Transferred position policies gain +17.2 Avg points over Base, versus +18.2 for target-trained policies, with no transfer more than 2.6 points below its target-trained counterpart.Applying a Base-trained policy to GRPO-post-trained reasoners adds +6.1 Avg and +4.9 Pass points without retraining.Source: §6.4 · p. 8, Figure 4 · p. 8
  • Without retraining, retuning, or changing the gate threshold on science data, UPTURN outperforms Base for every reasoner on GPQA-Diamond.The benchmark contains 198 graduate-level multiple-choice questions in physics, chemistry, and biology.Source: §6.4 · p. 8, Table 3 · p. 8
  • At a total budget of 2Lmax, UPTURN adds +8.8 Avg and +3.0 Pass points to Reflexion, and +11.8 Avg and +1.8 Pass points to SCoRe.The comparison averages across six benchmarks and three reasoners.Source: §6.4 · p. 8, Table 4 · p. 8

6.5 ABLATION

  • Among fresh regeneration and start, random, or learned-position injection variants, the learned position performs best; separately trained inline injection yields smaller gains than multi-turn injection for every reasoner.The position variants use the same gate and replacement rule.Source: §6.5 · p. 9, Figure 5 · p. 9
  • Base@32 reaches 52.0 GS versus 53.5 Pass, and replacing the learned gate with a ground-truth gate changes all metrics by less than one point.UPTURN@16 retains a +11.0 point advantage under gate selection.Source: §6.5 · p. 9

7 CONCLUSION

UPTURN redirects failed reasoning by retaining a selected prefix and injecting an instruction as a new user turn, without retraining the reasoner. The conclusion reports gains across reasoners and benchmarks, while noting that broader tasks and interventions remain open.

7 CONCLUSION

  • UPTURN consistently improves average accuracy across four reasoners and six math benchmarks, outperforming correction baselines under matched budgets and transferring without retraining.It also provides additional gains on top of GRPO and existing self-correction methods.Source: §7 · p. 9

A ILLUSTRATIVE CORRECTION EXAMPLE

  • From the same failed rollout cut, an unmodified continuation stays in re-verification, inline injection resumes the loop, and a new user turn redirects toward the correct answer.Source: Figure 6 · p. 13

B LIMITATIONS

  • The study covers automatically checkable reasoning tasks and one deliberately minimal intervention; richer interventions and harder-to-verify settings remain open.Source: B LIMITATIONS · p. 13

C IMPLEMENTATION DETAILS

Implementation keeps the encoder and reasoners frozen while a lightweight position head selects cuts for suffix regeneration. Training, inference, evaluation separation, and model-family chat templates define the protocol.

C IMPLEMENTATION DETAILS

  • The position head scores candidate cuts from frozen Qwen3-0.6B encoder states using LayerNorm, a 1024→128 linear layer, GELU, and a 128→1 output layer.It has 133,377 parameters, about 130K; the encoder receives the reasoner’s exact chat-templated prompt and rollout tokens without re-tokenization.Source: C IMPLEMENTATION DETAILS · p. 13
  • Training uses 30 steps of 64 prompts with failed rollouts, for 1,920 kept prompts total, and updates the head once per step for one epoch.Data come from the 17K-prompt DAPO-17K subset; rollouts and regenerations use identical sampling settings.Source: C IMPLEMENTATION DETAILS · p. 13
  • At inference, each flagged rollout is corrected once, with cut positions sampled from the learned policy and a gate threshold of γ=0.4.When a cut falls inside an unclosed <think> block, the system appends </think> before injection.Source: C IMPLEMENTATION DETAILS · p. 13
  • Evaluation benchmarks are held-out test sets with no overlap with the DAPO training pool; the gate’s 1,197-prompt pool is independently sampled from that pool.Source: C IMPLEMENTATION DETAILS · p. 13
  • Experiments ran on NVIDIA B200 GPUs using a fork of verl, vLLM 0.15.1, transformers 4.57.1, and PyTorch 2.9.1.Source: C IMPLEMENTATION DETAILS · p. 14
  • The fixed instruction is “Resolve this concisely.”, and turn boundaries use each reasoner family’s own chat-template tokens.Table 7 lists the injected sequence by model family.Source: C IMPLEMENTATION DETAILS · p. 14, Table 7 · p. 14

D GATE DETAILS

A separate lightweight gate identifies likely failed rollouts and is reused across reasoners. Its ranking is strong, and the reported accuracy is stable across gate thresholds and close to the oracle-gate result.

D GATE DETAILS

  • The gate is trained separately on correctness-labeled rollouts from 1,197 DAPO prompts and reused unchanged across experiments, including different-scale thinking models.Class balancing yields 2,874 examples; post-hoc temperature calibration uses a held-out prompt split, and the learned gate is a separate 66K-parameter MLP.Source: D GATE DETAILS · p. 15
  • Mean gate AUC is .962 across four reasoners, with benchmark-wise AUC at least .920.Table 8 reports gate AUC by benchmark and the mean across reasoners.Source: D GATE DETAILS · p. 15, Table 8 · p. 15
  • Replacing the learned gate with ground-truth error flags changes Avg from 55.3 to 55.9 and leaves Pass at 71.3, averaged across four reasoners.The oracle changes only which rollouts are corrected; the rest of the setup is held fixed.Source: D GATE DETAILS · p. 15, Table 9 · p. 15
  • From γ=0.1 onward, Avg varies by at most 0.4 points and Pass by at most 0.1 as the corrected fraction rises from 64% to 100%.The authors choose γ=0.4 as a cost–coverage compromise; γ=0 corrects no rollouts.Source: D GATE DETAILS · p. 16, Table 10 · p. 16
  • UPTURN exceeds Base@32 in 21 of 24 benchmark cells, ties on AMC23 for both Qwen3 reasoners, and is lower only on HMMT for QwQ-32B.The comparison uses Base@32 and UPTURN@16 within the Table 1 budget; UPTURN’s GS gain is smaller than its Pass gain.Source: D GATE DETAILS · p. 16, Table 11 · p. 16

E WALL-CLOCK COST

The wall-clock comparison measures Base@32 and UPTURN end to end on AIME24 with Qwen3-4B on one B200. UPTURN uses fewer decoded tokens and is faster overall despite added processing overhead.

E WALL-CLOCK COST

  • The timing comparison reports generated calls, decoded tokens per question, and total benchmark wall-clock time, excluding a shared 32-second engine startup cost.Each pipeline ran twice with seeded sampling; reported time is the mean plus or minus half the absolute difference between measurements.Source: E WALL-CLOCK COST · p. 17
  • UPTURN reduces decoded tokens by approximately 36% and wall-clock time by 15% relative to Base@32.The comparison is on AIME24 with Qwen3-4B on one B200.Source: E WALL-CLOCK COST · p. 17, Table 12 · p. 17
  • Additional gating, position selection, and prefix processing make the wall-clock gain smaller than the token reduction, but cost less than the decoding time saved.Source: E WALL-CLOCK COST · p. 17

F DOES A SECOND CORRECTION ROUND HELP?

Across the tested correction-round rules, the first correction accounts for most improvement; after a turn boundary is introduced, the instruction wording also affects results.

F DOES A SECOND CORRECTION ROUND HELP?

  • The first correction accounts for most improvement; gated round 1 beats naive on Avg and Pass for both reasoners, while later naive gains vary by reasoner.Further gated rounds add no gains and slightly reduce Avg or Pass in some settings; naive performance continues improving on Qwen3-4B, while Qwen3-8B peaks at round 2.Source: Figure 7 · p. 18, F DOES A SECOND CORRECTION ROUND HELP? · p. 18

G SENSITIVITY TO THE INJECTION INSTRUCTION

  • The instruction sensitivity test varies only the injected text, keeping the phase-1 pool, gate, position policy, budget, and seed identical.It compares four task-relevant instructions with the controls “Wait.”, “Turtle.”, and None, an empty user turn.Source: G SENSITIVITY TO THE INJECTION… · p. 19
  • 4.7 and 4.1 points: the default instruction raises Avg over None on Qwen3-4B and Qwen3-8B, achieving the highest Avg and rescue rate on both.Every multi-turn variant improves Avg over Base, including an empty user turn.Source: Figure 8 · p. 19, G SENSITIVITY TO THE INJECTION… · p. 19

H PARAGRAPH BOUNDARIES ARE FORKING POINTS

Paragraph boundaries are plausible cut points because they coincide with high next-token entropy and capture a substantial share of the most uncertain positions.

H PARAGRAPH BOUNDARIES ARE FORKING POINTS

I SCALING THE GENERATION BUDGET

UPTURN retains advantages over self-correction and reflection-token injection at matched correction budgets, while its gains over Base@32 narrow as generation budgets grow.

I SCALING THE GENERATION BUDGET

  • The position policy trained at Lmax is used unchanged at longer budgets, without retraining or adjusting the gate threshold.Base@32 and UPTURN allow up to 32 generations per question; the stated constraints are allowed budgets, not actual token usage.Source: I SCALING THE GENERATION BUDGET · p. 21
  • At Lmax, UPTURN adds 14.1 to 30.3 Avg points; by 3Lmax, Base@32 matches or edges Qwen3, while UPTURN retains +7.0 on DeepSeek-R1-Qwen3-8B.At 2Lmax, the gains are +5.3, +5.0, and +18.2 Avg points on Qwen3-4B, Qwen3-8B, and DeepSeek-R1-Qwen3-8B.Source: Figure 9 · p. 21, I SCALING THE GENERATION BUDGET · p. 21

J FULL COMPARISON WITH SELF-CORRECTION BASELINES

K COMPARISON WITH REFLECTION-TOKEN INJECTION

  • At Lmax, inline injection methods stay close to Base@32; at 2Lmax they improve by more than twenty points, yet UPTURN leads at both budgets.The comparison places reflection-token methods in UPTURN’s correction slot, regenerating flagged rollouts under the same aggregation rule.Source: Table 15 · p. 23, K COMPARISON WITH REFLECTION-TOKEN… · p. 23

L COMPLEMENTARY WITH OTHER BASELINES

Adding UPTURN to existing self-correction outputs improves every baseline on every reasoner, with the largest gains on weaker pools and occasional slight losses on strong ones.

  • UPTURN takes each baseline’s 16 corrected solutions as its phase-1 pool, flags weak ones, and redirects each flagged solution once without retraining.The gate, position policy, and instruction remain unchanged from Table 1.Source: L COMPLEMENTARY WITH OTHER BASELINES · p. 24, Table 16 · p. 24
  • The gain is largest where the corrected pool is weakest, but can turn slightly negative on already-strong cases.The paper notes slight negatives for DeepSeek-R1-Qwen3-8B on AMC23 and MATH500 with Reflexion at 2Lmax.Source: L COMPLEMENTARY WITH OTHER BASELINES · p. 24

M SEED ROBUSTNESS

Across three seeds used for both position-policy training and rollout sampling, Avg’s standard deviation stays within about two points—well below every UPTURN gain over Base.

M SEED ROBUSTNESS

N RESULTS ON OTHER MODELS

The section reports that UPTURN improves average accuracy across every benchmark on two additional RL-finetuned reasoners, and presents rollout examples of multi-turn injection reaching a correct answer and of unresolved geometric reasoning. The latter examples trace unsuccessful attempts to relate a hexagon to a triangle, including a rollout that ends with an unsupported answer.

N RESULTS ON OTHER MODELS

  • +1.8 to +15.2 points in average accuracy on every benchmark; pass rate improves on all but two benchmarks, where UPTURN ties Base.The results use the same setup as Table 1 on AceReason-Nemotron-1.1-7B and Skywork-OR1-7B.Source: Table 18 · p. 26, N RESULTS ON OTHER MODELS · p. 26

O FULL ROLLOUTS FOR FIGURE 6

  • At the same learned cut, the original and inline continuations are wrong, while the multi-turn user-message continuation answers correctly (80).The shared prefix is followed by three full continuations: original, inline injection, and multi-turn injection.Source: O FULL ROLLOUTS FOR FIGURE 6 · p. 28
  • The model repeatedly calls AB and CD parallel, then contradicts itself about which sides are parallel and whether the lines can form the stated triangle.It later reasserts the parallelism while continuing to question how the triangle is formed.Source: O FULL ROLLOUTS FOR FIGURE 6 · pp. 29–30
  • Before settling on the reciprocal-sum formula, the model considers a harmonic-mean alternative and says it needs to think more carefully.The harmonic-mean calculation gives 240, conflicting with the final answer of 80.Source: O FULL ROLLOUTS FOR FIGURE 6 · pp. 29–30
  • The rollout’s final answer is 80, obtained from the asserted formula 1/s = 1/200 + 1/240 + 1/300.The passage states that the reciprocal sum is 1/80, so s = 80, but does not establish the formula’s validity.Source: O FULL ROLLOUTS FOR FIGURE 6 · p. 30
  • The rollout finds no intersection from its attempted line-extension equations, which require non-collinear vectors to be scalar multiples.It first derives (t + s)a = a + b + c and later (t + s)a = b + c, then says these have no solution in the stated general case.Source: O FULL ROLLOUTS FOR FIGURE 6 · p. 31
  • The rollout considers similarity, homothety, vector relations, and area as possible ways to connect the triangle’s sides to the hexagon, but reaches no relation.The triangle sides are given as 200, 240, and 300; the excerpt ends while exploring a possible vector-based connection.Source: O FULL ROLLOUTS FOR FIGURE 6 · p. 31
Loading 2610.13672v1…