PaperlayerView exact revision on arXiv

Source-linked AI summary

The Thinking Leakage in Reinforcement Learning for LLMs

Wenxuan Zhu, Shuming Liu, Bang An, Aznaur Aliev, Bing Li, Yibo Yang, Bernard Ghanem

arXiv:2610.14270v1cs.CL

TL;DR

Hybrid models can answer in think or no-think mode, and reinforcement learning in no-think mode is commonly used to improve reasoning. This paper tests whether those gains reflect new problem-solving behavior or reactivation of think-mode reasoning, finding that much of the gain is associated with thinking leakage. Interventions and analysis of the GRPO signal provide evidence for that interpretation.

  • Problem

    No-think RL gains in hybrid models could reflect either new problem-solving behavior or greater use of reasoning already present in think mode, and few studies have examined whether RL in one mode draws on the other.

    Source: §1 · p. 1

  • Method

    The authors train Qwen3-1.7B, 4B, and 8B with GRPO in no-think mode and examine behavior, weights, activations, attention heads, and the training signal.

    Source: §1 · p. 2

  • Results

    Much of the no-think RL gain is attributed to thinking leakage, with interventions on weights, activations, and attention heads supporting reuse of think-mode behavior.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    Higher no-think accuracy does not necessarily indicate better or new problem-solving strategies; it may reflect think-mode behavior becoming easier to express in no-think mode.

    Source: §7 · p. 12

Abstract

from arXiv · show

Hybrid reasoning models such as the Qwen3 series switch between a THINK mode and a NO-THINK mode through a few chat-template tokens. A popular recipe trains such models with reinforcement learning in NO-THINK mode and reports improved reasoning performance. We provide evidence that much of this gain can be explained by thinking leakage: the gain comes largely from partially reactivating the reasoning abilities of the original model's THINK mode instead of learning enhanced reasoning behavior. We train Qwen3-1.7B, 4B, and 8B with GRPO in NO-THINK mode. As accuracy improves, responses become longer and more reflective, are increasingly judged by the original model to resemble its own thinking, and the model's internal activations shift toward its THINK mode. We then investigate at the level of weights, activations, and attention heads: scaling the RL update further elicits THINK-mode behavior and approaches THINK-mode accuracy but does not exceed it; in Qwen3-8B, replacing a low-dimensional THINK-related activation subspace of the RL-trained model with its original-model value removes most of the RL gain, and the reverse exchange transfers a considerable fraction of the gain to the original model; and the RL-induced shift passes through the attention heads that read the empty think block. We further identify a possible mechanism in the GRPO training signal: responses favored by the advantage-weighted update are more THINK-like, and a gradient step computed from this signal moves the model toward its THINK state. Thus, gains in NO-THINK mode can largely reflect reactivation of reasoning abilities already present in the original model.

1 Introduction

The paper asks whether reinforcement learning (RL) in a hybrid model’s no-think mode teaches new reasoning or reactivates behavior already available in think mode. It presents evidence that thinking leakage accounts for much of the gain and examines its behavioral, internal, and training-signal correlates.

1 Introduction

  • The study asks whether no-think RL teaches new problem-solving behavior or draws on reasoning already present in think mode, a distinction few studies have examined.The two modes share weights and differ in the prompt; prior work has instead examined RLVR’s added reasoning ability and how models learn when to think.Source: §1 · p. 1
  • Across Qwen3-1.7B, 4B, and 8B, no-think RL makes responses longer and more reflective, more think-like to the original model, and shifts internal activations toward think mode.At 8B, the original model prefers RL-trained responses to its own no-think responses 94% of the time, and the activation shift reaches 0.56 of the original mode difference.Source: §1 · p. 2
  • In Qwen3-8B, replacing the RL-trained model’s think-related activation subspace removes 75% of its improvement, while the reverse patch transfers 50% of that gain to the original model.Weight and attention-head interventions also support the interpretation that RL-induced shifts reuse think-mode behavior.Source: §1 · p. 2
  • A possible source of leakage is the GRPO signal: advantage-favored responses are more think-like, and a gradient step from that signal moves the model toward its think state.The authors describe this as optimization pressure toward thinking leakage in the training signal itself.Source: §1 · p. 2
  • The paper studies thinking leakage through observed behavior, interventions on weights, activations, and attention heads, and the possible source in the GRPO training signal.The interventions support the claim that reuse of think-mode reasoning accounts for a large part of the no-think RL gain.Source: §1 · p. 2

2 Related work

Related work covers RL with verifiable rewards, hybrid reasoning modes, changes induced by post-training, and causal interventions on model activations. This paper connects these lines by testing whether no-think RL gains draw on think-mode behavior.

2 Related work

  • Prior RLVR studies report that gains may reflect abilities already latent in base models, including results on pass@k, recovery from limited or spurious rewards, and amplification of pretrained behaviors [44] [39] [30] [48] [51].One study finds that penalizing wrong answers alone is nearly as effective as the full signal [51].Source: §2 · p. 2
  • Hybrid-reasoning work studies how models use think and no-think modes, learn when or how long to think, and trade off reasoning length and performance [41] [23] [7] [16] [47] [1] [8] [25] [33] [4] [42].Other work considers hiding reasoning in latent tokens or continuous latent states [45] [12].Source: §2 · pp. 2–3
  • Work on post-training finds structured, often low-rank weight changes and studies whether post-training elicits existing capabilities rather than adding new ones [26] [50] [32].The paper uses scaled weight differences to ask where the no-think RL update points.Source: §2 · p. 3
  • Activation steering and patching studies represent behavior with residual-stream directions and test whether activations mediate it [52] [35].This paper uses matched-token activation replacement to link an original-model think subspace and attention heads to no-think RL accuracy gains [37] [36] [40].Source: §2 · p. 3

3 Experimental setup

The experiments use Qwen3 models in think and no-think prompt configurations, train them with GRPO-style updates, and evaluate on AIME problems. The setup also defines behavioral and activation-based measures of resemblance to think mode.

3 Experimental setup

  • The experiments use released Qwen3-1.7B, 4B, and 8B checkpoints, with think mode prompting for a reasoning block and no-think mode appending an empty think block.The checkpoints before RL training are called the original models.Source: §3 · p. 3
  • Training samples 256 prompts and five responses per prompt per step, using a PPO-style clipped objective, KL penalty, rank-64 LoRA on every linear layer, and AdamW.The clip ratio is 0.2, the KL penalty is 10^-3, learning rate is 10^-5, and gradient-norm clipping is 1.0.Source: §3 · p. 3
  • Evaluation uses 90 AIME 2022–2024 problems at temperature 1, grades the final boxed answer, and allows up to 32,768 generated tokens per response.Responses without a boxed answer count as incorrect.Source: §3 · p. 3
  • Think-likeness measures include response length, reflection-word counts, and think win rate, which compares responses using likelihood under think versus no-think prefixes.The original model is the fixed judge; its own no-think responses have a 50% think win rate by construction, while its think-mode text has 100%.Source: §3 · p. 4
  • The thinking activation shift is the RL-trained model’s activation change projected along the original model’s think direction on fixed original no-think responses.The coordinate is normalized by the full activation difference between the original model’s think and no-think modes; cross-fitting changes it by at most 0.01.Source: §3 · p. 4

4 RL in no-think mode shifts the model toward think mode

During no-think RL, all three model sizes show accuracy gains alongside increasingly think-like responses and internal activations. The section tracks these changes over training on AIME.

4 RL in no-think mode shifts the model toward think mode

  • Across all three model sizes, accuracy improves as responses become longer and more reflective, judge scores rise, and activations move toward the original think direction.Figure 1 tracks these measures during training, with all quantities evaluated in no-think mode.Source: §4 · pp. 4, 6, Figure 1 · p. 5
  • For Qwen3-8B, accuracy rises from 0.17 to 0.43, while mean response length grows from 2.7k to 7.2k tokens and reflection words from 1.3 to 13.0.In original think mode, accuracy is 0.74, 0.69, and 0.43 for 8B, 4B, and 1.7B, respectively; responses contain 16k–18k tokens and about 120–130 reflection words on average.Source: §4 · p. 5
  • Think win rate reaches 94% for 8B, 86% for 4B, and 84% for 1.7B, with 8B rising from 50%.The original model’s own think-block text scores 100%.Source: §4 · p. 5
  • Activation shift α reaches 0.56, 0.40, and 0.21 for 8B, 4B, and 1.7B, with positive alignment to the think direction at every checkpoint.Cosine similarity ranges from 0.33 to 0.70; for 8B it increases from 0.54 at step 5 to 0.70 at step 67.Source: §4 · pp. 5–6

5 Evidence that the RL gain comes from thinking leakage

Weight scaling, activation exchange, and attention-head interventions test whether RL gains in no-think mode reflect reuse of think-mode behavior. The interventions show that think-related activations and heads account for substantial parts of the gain and mode shift.

5.1 Weights: scaling the RL update brings the model close to think mode

  • Scaled RL updates raised accuracy to 0.72 for step 67 and 0.68 for step 15, but neither exceeded the original think-mode accuracy of 0.74.The comparison was tested on Qwen3-8B in no-think mode on AIME.Source: §5.1 · p. 6, Figure 2 · p. 7
  • Scaled updates elicited </think> despite the closed think block; for step 67 it appeared in 51% of responses at c=3, near its think-mode position.The original model never generated the token in 1,260 no-think responses; scaled responses placed it at a median 91% of response length, versus 92% in think mode.Source: §5.1 · p. 6, Figure 2 · p. 7

5.2 Activations: patching an 8-dimensional think subspace removes or transfers much of the gain

  • Exchanging the 8-dimensional think subspace removed 75% of the RL gain, while the reverse exchange transferred 50% to the original model.PCA and random subspaces removed 22% and −3% of the gain, respectively; they transferred 25% and 10%.Source: Figure 3 · p. 8, §5.2 · p. 8

5.3 Attention heads: the RL-induced shift passes through the heads that attend to the empty think block

  • Knocking out the 5% of heads attending most to the empty block shifted activations toward think mode (α=0.78, 0.88, 0.66) and yielded R_KL of 72%, 74%, and 77%.Randomly selected sets of the same size produced α of at most 0.04 and R_KL of at most 6%.Source: Table 1 · p. 9, §5.3 · p. 9
  • The top patching heads also attended strongly to the empty think block, and their single-head effects reached 44% for 8B, 37% for 4B, and 54% for 1.7B.The same effects held on held-out responses, and all three top patching heads at each size were among the top 5% by attention to the block.Source: Table 2 · p. 10, §5.3 · p. 10

6 A possible cause: GRPO advantages favor think-like responses

The section examines how GRPO advantages relate to response think-likeness and whether a gradient step using those advantages moves the model toward think mode. Within groups, reinforced responses are more think-like than penalized responses, and the full training signal increases the thinking activation shift.

Within-group comparison

  • Within each group, correct responses had higher think win rates than wrong responses by 5.9, 2.1, 1.6, and 2.5 percentage points for k = 1–4.Because advantages within each group sum to zero, the group’s gradient contribution contrasts its correct and wrong responses.Source: Figure 4 · p. 11, §6 · p. 12

Think-likeness across groups

  • Correct responses receiving larger advantages were more think-like across groups, with think win rates declining from 57.8% at k = 1 to 40.5% at k = 4.This is an association across groups, not a direct effect of think-likeness on advantage; advantage is determined by k.Source: §6 · p. 11
  • Among wrong responses, smaller penalties were associated with greater think-likeness, with think win rates declining from 51.9% at k = 1 to 38.0% at k = 4.This is an association across groups because the advantage is determined by k, not directly by think-likeness.Source: §6 · p. 11

One gradient step moves the model toward think mode

  • A full GRPO gradient step increased the thinking activation shift by +0.0075; shuffling advantages within groups changed it by −0.0022, while equal weighting changed it by only +0.0006.Positive advantages alone changed the shift by +0.0072, and negative advantages alone by +0.0057.Source: Table 4 · p. 12, §6 · p. 12

7 Conclusion

The authors conclude that much of the no-think RL improvement reflects thinking leakage rather than a separate non-thinking policy. They describe how responses and activations were measured, including the prompts, think-likeness judge, and activation-shift procedure.

7 Conclusion

  • The experiments suggest that much of the no-think RL gain comes from reusing behavior associated with think mode, rather than improving a separate non-thinking policy.The authors argue that evaluations should distinguish new capability from changed access to existing capability in hybrid models.Source: §7 · p. 12

A Measurement details

  • The no-think prompt ends with an empty think block, whereas the think prompt opens a think block before the response.Measures use generated token IDs, including the end-of-turn token when a response finishes within budget.Source: A Measurement details · p. 17
  • Reflection words are counted with case-insensitive whole-word patterns, with overlapping matches counted once and counts reported per response.The patterns include terms such as “wait,” “hmm,” and specified “let me” phrases.Source: A Measurement details · p. 17
  • Think win rate uses the original model to score responses against its no-think responses, with ties counting half in the Mann–Whitney statistic.The judge separates the original model’s think-mode text from its no-think responses completely.Source: A Measurement details · p. 17
  • The activation-shift measure averages activations over fixed response tokens and reports the middle layer; at 8B step 67, the orthogonal shift norm is 0.58 ∥v∥.The fixed responses are the original model’s no-think responses, and the reported layer is 18 of 36 for 8B.Source: A Measurement details · p. 17

B Scaling the RL update

Scaling RL updates further continues the training-associated relation between accuracy and response length only up to a point, while eliciting think-mode behavior. The cross-model patching experiment reports that a 16-dimensional think subspace transfers part of the RL gain to the original model.

B Scaling the RL update

  • The step-15 update at c = 7 and step-67 update at c = 2 had the same activation shifts (α = 0.96 and 0.98), accuracy (0.68 and 0.69), and </think> share (19% each), despite different weight-change sizes.This matched pair suggests activation position along the think direction may be more informative than weight-update size, but two points cannot establish this generally.Source: B Scaling the RL update · p. 17, Table 5 · p. 18
  • Across 14 scaled models, 443 of 2,520 responses contained </think> exactly once; its median position was 0.91 of response length, versus 0.92 in think mode.The original model produced </think> in none of its 1,260 no-think responses.Source: B Scaling the RL update · p. 18
  • Up to c = 5 for step 15 and c = 2 for step 67, scaled models follow slightly below the checkpoints’ accuracy–length relation; beyond these factors, responses hit the 32k budget and accuracy falls far below it.The figure compares accuracy with mean response length; its sets differ in mode and sample count, so only their shapes are comparable.Source: B Scaling the RL update · p. 18, Figure 5 · p. 19

C Think subspace and cross-model patching

  • Cross-model patching replaces the target’s subspace component at layers 1–35 and every position, while the source processes the same tokens without patching.Both models retain their own key–value caches during generation.Source: C Think subspace and cross-model… · p. 19
  • A 16-dimensional RL-to-original think-subspace patch reached accuracy 0.322 and transferred +0.133, or 60% of RL gain G, to the original model.The dimension, patched layers, and complete replacement were fixed before AIME results were seen.Source: C Think subspace and cross-model… · p. 20, Table 6 · p. 20

D Steering the original model

Steering the original 8B model along its think direction in no-think mode increased reflection words, response length, and accuracy as the steering coefficient rose.

D Steering the original model

  • Accuracy rose from 0.19 at a = 0 to 0.60 at a = 1, where 24% of responses reached the budget; at a = 0.5, reflection words matched the RL-trained model at 12.4, while accuracy was 0.29 versus 0.43.The experiment steers layer 18 of the original model; response length also rises with the steering coefficient.Source: D Steering the original model · p. 20, Table 7 · p. 21

E Attention-head experiments

Patching identifies attention heads that carry much of the RL-induced shift toward think mode, while a separate decomposition finds that the shift is written mainly through frozen pretrained weights. The head analyses use fixed responses, and the activation measurements are subject to truncation and teacher-forcing conditions.

E Attention-head experiments

  • The Section 5.3 analyses use teacher-forced fixed responses truncated to 2,048 tokens, yielding α = 0.52 rather than 0.56 for the 8B model at the middle layer.The reported difference is attributed to truncation.Source: E Attention-head experiments · p. 21
  • Across models, 90–91% of heads place less than 1% of their attention on the empty think block, while only 5–7 heads place at least 10%.At 8B, the three largest attention shares are 64% (L4H23), 25% (L18H19), and 18% (L12H4).Source: E Attention-head experiments · p. 21
  • The attention knockout zeros selected heads’ attention to the empty think block only at response positions; random-head controls average five draws.The first response token is excluded because it is predicted at the last token of the block, which the knockout does not affect.Source: E Attention-head experiments · p. 21
  • Head patching replaces each original-model head output with the corresponding RL-trained output on fixed tokens, then measures the shift at three-quarters depth.The ten heads with the largest effects were rechecked on 60 texts excluded from ranking, and their effects agreed with the ranking set.Source: E Attention-head experiments · pp. 21–22
  • All three heads with the largest patching effects at each model size belong to the 5% of heads that attend most to the empty think block, versus 0.1 expected by chance.The top-head patching effects are 44%, 37%, and 56% for 8B, 4B, and 1.7B; single-head effects are not additive.Source: E Attention-head experiments · p. 22

F Where the activation shift is written

  • More than 99% of both the think-direction and orthogonal activation shifts comes from frozen weights acting on changed inputs, not directly from the RL update.The authors caution that this decomposition alone does not distinguish reuse of pretrained computation from new computation.Source: F Where the activation shift is written · p. 22

G Training-signal analysis

The training-signal analysis samples original-model responses, scores them with the training reward and a judge, and compares response preferences within GRPO groups. It also tests how different advantage-weighted gradient signals move the model’s thinking activation shift.

Training-signal analysis

  • The analysis draws 256 questions, samples 20 original-model responses per question, scores each with DAPO reward and a judge, and groups responses in fives for GRPO advantages.Groups with equal rewards receive zero advantage; the reward is −1 or +1 and requires the final answer on a line beginning “Answer:”.Source: G Training-signal analysis · pp. 22–23
  • Correct responses exceed wrong responses by an average of 3.0 percentile points within nonzero-advantage groups.The analysis compares mean response-score percentiles within each group; win rates use all 5,120 sampled responses as the reference pool.Source: G Training-signal analysis · p. 23
  • First-step gradients compare full, positive-only, negative-only, shuffled, and equal-response weighting, then measure the updated model’s thinking activation shift.The 2,990 responses with nonzero advantages contribute to the gradients; each Adam step changes parameters by 10^-5 in the sign direction.Source: G Training-signal analysis · p. 23

H Training curves

The training curves track training-batch correctness and mean response length across three runs. Response length grows with reward at every model size, with some late-training responses reaching the imposed length limits.

H Training curves

  • Response length grows with reward at all three model sizes during training.Figure 6 plots training-batch correct-response fraction and mean response length at each RL step.Source: H Training curves · p. 23, Figure 6 · p. 24
  • At 8B, up to 14% of late-training responses reach the 8k-token limit, versus at most 3% at 4B and 1.7B under 16k-token limits.The response limits are 8k tokens at 8B and 16k tokens at the two smaller sizes.Source: H Training curves · p. 23
  • Figure 6 presents the fraction of correct training-batch responses and mean response length against RL step for the three runs.The figure’s panels encode correctness in (a) and response length in (b).Source: Figure 6 · p. 24

Related papers

Loading 2610.14270v1…