PaperlayerView exact revision on arXiv

Source-linked AI summary

On Language Drift during RLVR Post-Training

Michael Sullivan, Alexander Koller

arXiv:2610.02015v1cs.LGcs.AI

TL;DR

Language drift in reasoning traces is increasingly observed, but its causes remain poorly understood. The paper proves contrasting drift bounds for RLVR and SFT, then examines drift during RLVR on novel reasoning tasks. It finds greater drift when behavior sharpening is not viable and shows that constraining drift can constrain expected reward.

  • Problem

    Language drift in LLM reasoning traces is increasingly observed, but its causes and relation to RLVR optimization pressure remain poorly understood.

    Source: §1 · pp. 1–2

  • Method

    The paper proves theoretical bounds for drift under SFT and RLVR and empirically compares drift and trace legibility across training conditions and runs.

    Source: §1 · p. 2, §5.1.2 · p. 7, §6 · p. 9

  • Results

    RLVR permits unbounded drift while SFT has a finite upper bound, and empirical drift is greater under RLVR than SFT when behavior sharpening is not viable.

    Source: §1 · p. 1, §6 · p. 9

  • Takeaways & Limitations

    Constraining language drift during RLVR necessarily limits maximum expected reward, complicating efforts to maintain CoT monitorability without hindering performance.

    Source: §1 · pp. 1–2

    Experiments used small 1–1.5B models and only GSM8K, so observed drift did not reach some frontier-trace levels and broader dataset evaluation remains future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent advances in LLM reasoning models---driven primarily by the paradigm of post-training via reinforcement learning with verifiable reward (RLVR)---have enabled them to accomplish impressively complex tasks. However, in parallel with their rising capabilities, LLMs have increasingly displayed signs of language drift in their chains of thought (CoTs): unusual, non-standard, and seemingly nonsensical language use. Although it is well-documented---and can potentially impair CoT monitorability---the causes of language drift are thus far poorly understood. In this paper, we identify the conditions under which language drift occurs: we prove theoretically that RLVR optimization pressure permits unbounded language drift, while supervised fine-tuning does not. We then show empirically that language drift specifically arises during RLVR on novel reasoning tasks---i.e. when the target behavior cannot be drawn out of the base model. Finally, we prove that it is not possible to constrain language drift without constraining expected reward, suggesting that CoT monitorability cannot be improved without harming performance during RLVR post-training at the frontier.

1. Introduction

Language drift makes reasoning traces difficult to read, but its causes have been poorly understood. The paper links drift to RLVR on novel reasoning tasks and proves that constraining it can limit expected reward.

1. Introduction

  • Language drift includes non-standard or ungrammatical CoT language that can become illegible and impair monitoring of intent and future actions.The introduction illustrates drift with atypical style and altered word reference, including conflating a dog’s cost with grooming cost.Source: §1 · pp. 1–2
  • RLVR permits unbounded language drift, whereas SFT limits drift to a finite upper bound.The theoretical result concerns optimization pressure alone; the figure’s examples use Llama-3.2-1B models trained on GSM8K.Source: §1 · pp. 1–2
  • Constraining language drift during RLVR necessarily limits the maximum expected reward the model can attain.The paper connects this trade-off to whether CoT monitorability can be maintained without hindering downstream performance.Source: §1 · p. 2
  • When behavior sharpening is not possible on novel tasks, models display greater language drift during RLVR than during SFT.The paper identifies this setting as one in which RLVR discovers novel behavior.Source: §1 · p. 2
  • Different RLVR runs from the same model and dataset develop distinct linguistic behavior, potentially reducing mutual CoT monitorability across models.The paper notes that models trained on the same tasks may not be able to monitor a sufficiently drifted model’s CoTs.Source: §1 · p. 2

2. Related Work

Prior work reports linguistic changes under reinforcement learning, communication training, and reasoning-model post-training. This paper places language drift in modern reasoning models trained with RLVR on novel tasks.

2. Related Work

  • Prior RL optimization can produce repeated-token reward hacking, while this paper focuses on novel, internally coherent linguistic behavior [2022].The discussion distinguishes narrow reward-hacking cases from the linguistic behavior analyzed here.Source: §2 · pp. 2–3
  • Multi-agent reinforcement learning has produced language-like compositional codes and shifting word meanings in communication tasks [2018; 2020].The studies use limited-action agents or referential games, rather than modern reasoning models trained with RLVR.Source: §2 · p. 3
  • Earlier communication studies target communicative behavior, whereas this work extends the evidence to reasoning settings, transformer architectures, and RLVR algorithms.The cited studies demonstrate linguistic invention under optimization for communication ability.Source: §2 · p. 3
  • DeepSeek-R1-Zero’s poor readability and language mixing improved under a language-specific penalty, but performance was harmed [2025].The paper notes uncertainty about whether language mixing qualifies as drift, since bilingual code-switching is well documented [1980].Source: §2 · p. 3
  • Other reports describe illegible reasoning traces in frontier models, and removing illegible portions can considerably harm performance [2025; 2026].The paper suggests these traces may remain useful and legible to the model despite being human-illegible.Source: §2 · p. 3
  • Prompting can elicit human-illegible but LLM-legible text, while DPO-style preference optimization has been used to induce ad-hoc linguistic conventions.The latter work deliberately pressures models to develop unique linguistic behavior; its objective is agent-human communication, not arbitrary drift.Source: §2 · p. 3

3. Preliminaries

The paper formalizes text-based reasoning models, measures their language drift relative to a human-language reference, and models SFT and RLVR runs as improving checkpoint sequences under stated assumptions.

3. Preliminaries

  • The analysis covers text-based reasoning LLMs that produce a CoT and answer from a prompt, modeling each as a conditional distribution over CoTs and answers.Multimodal models are excluded for analytical tractability.Source: §3 · p. 3

3.1. Quantifying Language Drift

  • Language drift is measured relative to a fixed, unknown human-language distribution, using expected prompt-conditioned KL divergence.The reference distribution may be conditioned on context, such as math reasoning; computing it in practice is not the paper’s aim.Source: §3.1 · p. 3

3.2. Modeling LLM Training

  • An SFT run is modeled as checkpoints with non-increasing cross-entropy on its target data distribution; RLVR checkpoints have non-decreasing expected reward on their target distribution.The RLVR setup assumes outcome-only reward: one for a correct final answer and zero otherwise.Source: §3.2 · p. 4

3.3. Assumptions

  • The proofs assume full support for the human-language, SFT, and model distributions, non-uniform conditional RL targets, and finite vocabulary and sequence lengths.The paper describes finite vocabulary and context length as reflecting LLMs’ finite tokens and context windows.Source: §3.3 · p. 4

4. When Does Language Drift Occur?

The section contrasts RLVR’s unbounded potential for language drift with SFT’s finite bound, then argues that constraining drift limits maximum expected reward. It hypothesizes that drift arises especially when RLVR must discover novel reasoning behavior rather than sharpen behavior already present in the base model.

4.1. RLVR Permits Language Drift, SFT Does Not

  • RLVR optimization permits unbounded language drift, whereas SFT imposes a finite upper bound under the stated assumptions.No SFT run can exceed a fixed degree γ of drift while minimizing cross-entropy with its target distribution.Source: §4.1 · pp. 4–5
  • RLVR can drift because it optimizes answer correctness without constraining the reasoning-trace distribution, while SFT’s fixed training distribution bounds drift.The divergence between the human-language and SFT target distributions is constant, yielding a finite bound for SFT checkpoints.Source: §4.1 · pp. 4–5

4.2. Constraining Language Drift Constrains Expected Reward

  • For every drift radius ε, a larger radius δ permits greater expected reward, so constraining drift necessarily limits the maximum reward attainable during RLVR.As maximum expected reward approaches its supremum, minimum possible language drift approaches infinity.Source: §4.2 · pp. 5–6
  • The argument links reward gains to concentrating probability on the most likely answer, which can increase drift; on frontier-level tasks, previously incorrect traces may precede correct answers.The paper notes that drift can also arise from shifting the answer distribution without changing the CoT distribution.Source: §4.2 · p. 5

4.3. Hypothesis: Novel Tasks Induce Language Drift

  • Unbounded possible drift does not mean every RLVR run drifts more than SFT, and the theorems do not establish how much drift practical training requires.Some task-model pairs may need only minimal drift for practically acceptable expected reward.Source: §4.3 · p. 6
  • The authors hypothesize that RLVR on novel reasoning tasks produces language drift as a side effect of discovering novel behavior when sharpening behavior from the base model is not possible.For non-novel tasks, they argue that behavior sharpening is preferred and meaningful drift will not occur.Source: §4.3 · p. 6

5. Experiments

The experiments compare SFT and RLVR on three base models, testing whether drift is greater when RLVR must discover novel behavior rather than sharpen existing behavior. Legibility results show more drift for Llama and Gemma under RLVR, but virtually none for Qwen, which primarily learned behavior sharpening.

5. Experiments

  • The experiments used three base models on GSM8K, with base models chosen to avoid possible contamination and emulate difficult novel tasks economically.Source: §5 · p. 6

5.1.1. TRAINING AND VALIDATION

  • SFT checkpoints were matched to same-seed RLVR validation scores when SFT performed better; Qwen also used matched RLVR checkpoints because its RLVR runs performed better.The study selected best-performing checkpoints and replicated the optimal configuration across five seeds per model and training method.Source: §5.1.1 · pp. 6–7

5.1.2. EVALUATION

  • Trace legibility estimates drift by asking instruct models to guess a target model’s submitted answer from partial reasoning traces.The experiment samples ten partial traces per trace and measures answer agreement, not answer correctness.Source: §5.1.2 · pp. 7–8

5.2. Results

  • Llama and Gemma showed greater language drift under RLVR than SFT across trace- and self-legibility, best-performing and iso-performance comparisons.Figure 7 plots the legibility differences; Table 2 summarizes them with AUC values.Source: Figure 7 · p. 8, Table 2 · p. 8, §5.2 · p. 9
  • Qwen, the only model showing behavior sharpening during RLVR, showed virtually no language drift relative to SFT.Around two-thirds of Qwen’s RLVR accuracy gains came from learning to emit EOS after submitting an answer.Source: §5.2 · pp. 8–9
  • RLVR-trained Llama and Gemma runs were less mutually legible than their SFT counterparts, indicating that each run developed unique linguistic behavior.Source: §5.2 · p. 9

6. Discussion and Conclusion

The paper reports that RLVR on novel tasks produces language drift, unlike SFT when behavior sharpening is not viable, and proves that drift cannot be constrained without constraining expected reward. It also finds that drift varies across runs, raising concerns about mutual CoT monitorability.

6. Discussion and Conclusion

  • Constraining language drift during RLVR necessarily constrains expected reward.The paper states that experiments did not empirically demonstrate this theoretical result.Source: §6 · p. 9
  • RLVR produces more language drift than SFT when behavior sharpening is not viable—that is, when the task is novel to the base model.The paper also reports that drift differs across runs initialized from the same model and trained on the same dataset.Source: §6 · p. 9

Limitations

The experiments used small models and a restricted set of reasoning datasets, limiting how far the findings can be generalized to frontier-model traces. They also did not empirically test the theorem that constraining language drift constrains expected reward.

Limitations

  • Resource constraints limited experiments to 1–1.5B models, whose observed drift did not reach the illegibility of some frontier-model traces.The paper says the smaller models also limit the size and difficulty of reasoning datasets they can be trained on.Source: Limitations · p. 9
  • Only GSM8K met the criteria of annotated CoTs and learnability through RLVR and SFT for all three base models.The authors plan to extend the experiments to more reasoning datasets.Source: Limitations · p. 9
  • The experiments did not empirically demonstrate the theoretical finding that constraining language drift constrains expected reward.The authors identify this as a result for future experimental work.Source: Limitations · pp. 9–10

A.1. Theorem 1

The proof material constructs a policy concentrated on a low-probability human-language trace and a reward-maximizing answer, and analyzes limits involving policy divergence. The supplied proof passages also bound a human-language divergence term using KL divergence and finite terms.

A.1. Theorem 1

  • Theorem 1's construction concentrates π* on z*(x), minimizing P(HL)(z | x), and y*(x), maximizing D(RL)(y | x).The proof identifies π* as a conditional distribution assigning probability one to that pair.Source: §A.1 · p. 12
  • For traces z other than z*(x), the proof states that π_n(z | x) tends to zero as n grows.It then derives divergence of the log ratio between P(HL)(z, y | x) and π_n(z, y | x).Source: §A.1 · p. 12

A.2. Theorem 2

  • Theorem 2's proof bounds a human-language divergence term using KL divergence between D(SFT) and π(SFT), together with bounded terms.The proof says the bound's constant is finite under its full-support assumption.Source: §A.2 · pp. 14–15

A.3. Theorem 3

Theorem 3's proof identifies a reward-optimal policy concentrated on the reward-maximizing answer and shows it has infinite language drift. Interpolating toward that policy from a finite-drift policy increases reward and yields greater drift.

A.3. Theorem 3

  • The policy π* is optimal for D(RL) because it concentrates probability on y*(x), the reward-maximizing answer.The proof states that this construction has infinite language drift.Source: §A.3 · p. 15
  • Interpolating a finite-drift policy toward π* increases expected reward, while the resulting policy has greater language drift.The proof compares the interpolated policy with the finite-drift maximizer for its drift constraint.Source: §A.3 · pp. 15–16

B.1. Training and Validation

The appendix records the training setup and evaluation prompts, then describes figures and a table used to assess trace legibility and whether CoTs contribute to answer predictions.

B.1. Training and Validation

  • Training swept hyperparameters by training type, used the same learning rates for DAPO and SFT, and ran each training run on one H100 GPU.All runs used the Figure 8 few-shot prompt; DAPO used fixed update iterations of 1, ϵlow of 0.2, and ϵhigh of 0.28.Source: Table 3 · p. 17, §B.1 · p. 17, Figure 8 · p. 17

B.2. Trace Legibility

C. CoT Utility Sanity Check

  • Figure 12 compares answer probabilities conditioned on partial traces with those conditioned on full traces; the difference at a full trace is 0.0 by design.It plots mean differences across partial-trace fractions with 95% confidence intervals; Table 4 reports original-versus-swapped-CoT probability differences, all significant at p < 0.001 under paired t-tests.Source: Figure 12 · p. 19, Table 4 · p. 19

C.1. Performative CoT

The sanity checks compare answer probabilities using partial or swapped reasoning traces against probabilities using the original full trace. Results indicate that models generally use their CoTs rather than relying on performative or invisible reasoning.

C.1. Performative CoT

  • Across nearly all models and seeds except Gemma Iso-SFT, final-answer probability rose as partial traces approached full length, indicating use of CoTs.The test compared answer probability conditioned on each partial trace with probability conditioned on the full trace.Source: §C.1 · p. 19

C.2. Invisible Reasoning

  • The invisible-reasoning test compared each answer probability conditioned on its original trace with probabilities conditioned on five other-query traces from the same length decile.The experiment collected 128 trace/answer pairs and divided traces into ten length buckets.Source: §C.2 · p. 20
  • Table 4 reports the mean difference between original answer probability and swapped answer probability; it is positive for all models, indicating they are not performing invisible reasoning in their CoTs.Source: §C.2 · p. 20

D.1. Llama

The appendix presents example reasoning traces for Llama and Gemma on arithmetic word problems, including correct answers and inconsistent intermediate calculations.

D.1. Llama

  • In Llama examples, the sheep-count trace gives 260 total sheep, matching both the direct and expanded calculations shown.Source: §D.1 · p. 20
  • In the cited examples, Siobhan is reported to have 23 jewels after Aaron’s 25; a seller sold 7 DVDs across three customer groups, and Dylan bought 82 sausages: 38 chicken and 44 fish.Source: §D.1 · pp. 20–21
  • Other Llama examples calculate a $70 grooming cost, 225 pounds of potato salad, and $360 for six pairs of shoes.Source: §D.1 · pp. 20–21

D.2. Gemma

  • Gemma's traffic-jam examples give incompatible accounts of the cars passing during the first 15 minutes.One trace concludes five cars passed; another instead describes 30 cars in the first 15 minutes and five remaining after the later interval.Source: §D.2 · p. 22
  • Gemma's sausage example totals 82, while its friends example reaches 120 despite misstating Lily's total as 100 in one trace.Source: §D.2 · p. 22
Loading 2610.02015v1…