PaperlayerView exact revision on arXiv

Source-linked AI summary

Base Models Can Reason By Taking a Cue From Training Data

Sophie L. Wang, Amil Dravid, Rulin Shao, Kevin Farhat, Sewon Min, Alexei A. Efros

arXiv:2610.06851v1cs.LGcs.AIcs.CL

TL;DR

The paper asks whether RL teaches new reasoning abilities or makes capabilities already present in base models more likely to appear. It evaluates short response cues and edits training data to study their effects, finding that cues can bring base-model performance near RL levels and that learned data associations help shape cue effects.

  • Problem

    It remains open whether RL teaches novel reasoning capabilities or exploits capabilities acquired during base-model training, since observing behavior after post-training does not establish that it was absent beforehand.

    Source: §1 · p. 2

  • Method

    The paper evaluates prefixed response cues, intervenes on training-data associations, and compares cue-induced hidden states with training-data states.

    Source: §3 · p. 6, §3.1 · p. 6

  • Results

    Across tested settings, cues approach RL performance; Olmo-3-7B reaches 78% MATH-500 pass@1 from 42%, versus 75% after RL, and training edits can create effective cues.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings distinguish what a model can do from what RL makes it likely to do.

    Source: §7 · p. 11

    The analysis focuses on R1-Zero-style RL applied directly to base models, and cue effectiveness is model-dependent; no tested opening was effective for Llama-3.1-8B.

    Source: §6 · p. 11

Abstract

from arXiv · show

In this paper, we study how training data creates associations between the tokens at the start of a base model's response and the reasoning behavior that follows. First, we demonstrate that fixing particular starting token cues makes a base model's performance competitive with that of its reinforcement learning (RL)-trained counterparts on math and coding. For instance, the cue ".\n\nOkay" raises Olmo-3-7B's MATH-500 pass@1 accuracy from 42% to 78%, while "Alright," raises Qwen3-14B's from 72% to 87%. Second, RL makes these cues more likely, while fixing them recovers much of its performance gain over the base model. Third, we trace the reasoning effects of token cues to the training data. We perform causal data interventions to turn an arbitrary word, such as "chicken", into an effective reasoning cue, or remove an existing cue's effect. A similar edit makes the prompt instruction "Think duck duck goose" as effective as "Think step by step" at eliciting reasoning. We also find that the hidden state representations induced by different cues correlate with different document types from the training set. Finally, we extend our study of token cues with a case study in language model safety, finding that different cues elicit distinct refusal and compliance behaviors that correspond to different types of training data.

1 Introduction

The paper asks whether reinforcement learning (RL) creates reasoning abilities or makes abilities already present in base models more likely to appear. It argues that short response cues can elicit reasoning, that training-data associations help explain these effects, and that cues also shape safety behavior.

1 Introduction

  • The open question is whether RL teaches novel reasoning capabilities or exploits capabilities acquired during base-model training.Prior work observes reasoning after post-training, but the paper notes this alone does not show the ability was absent beforehand.Source: §1 · p. 2
  • Fixing only the first two response tokens can bring base-model reasoning performance to the level of its RL-trained counterpart.For Olmo-3-7B, the cue raises MATH-500 pass@1 from 42% to 78%, compared with 75% for its RL counterpart.Source: §1 · p. 2
  • Replacing “okay” with “chicken” in training makes “chicken” an effective reasoning cue, while changing training associations can remove an existing cue’s effect.The “chicken” opening raises MATH-500 accuracy from 2.4% to 37.2%.Source: §1 · p. 2
  • Different cues elicit distinct safety responses: the reasoning cue produces more selective refusal, while “Okay,” reduces refusal and increases harmful responses to unsafe requests.The reasoning cue shifts the base model toward the behavior of its post-trained Instruct and Think-SFT variants.Source: §1 · p. 2

2 Token Cues Elicit Reasoning

Fixing selected response openings can improve base-model reasoning without weight updates, and RL increases the probability of effective cues. The section also tests how cues are selected, whether gains reflect response length, and how cue-forced training affects RL progress.

2.2 Token Cues Can Match RL

  • Olmo-3-7B rises from 42% to 78% MATH-500 pass@1 with its cue, versus 75% after RL; Qwen3-14B rises from 72% to 87%, matching RL.The cues also improve performance on other math benchmarks; on HumanEval, Olmo’s cue matches RL, while Qwen remains near its already high base accuracy.Source: Figure 2 · p. 3, §2.2 · p. 4

2 Token Cues Elicit Reasoning

  • The study separates the effect of a fixed opening on its continuation from how likely the model is to generate that opening, by prefilling cues without changing weights.The motivating observation is that 50% of correct Olmo-3-7B MATH-500 answers begin with a period and two newlines, versus 14% of incorrect answers.Source: §2 · pp. 2–3

2.1 Discovering Token Cues

  • The cue search ranks likely openings, then selects the candidate with the lowest average answer entropy, using answer agreement as a label-free proxy for accuracy.On Olmo-3-7B, this criterion selects a cue within 1.1 percentage points of the highest MATH-500 pass@1 among 500 tested openings.Source: §2.1 · p. 3

2.3 RL Makes Token Cues More Likely

  • RL concentrates its next-token distribution changes in the first two positions and increases the probability of the effective cues; prefilling recovers RL-level accuracy in the reported runs.Cue-forced training matches about 100 RL steps on Olmo and 50 on Qwen, while further cue-forced training adds little.Source: §2.3 · pp. 4–5

2.4 Token Cues Improve Accuracy beyond Longer Generation

  • The cue’s accuracy advantage is not explained by longer responses alone: some openings produce roughly twice as many tokens yet score below no cue.At matched budgets, the cue outperforms no cue from 1k tokens onward and reaches RL-Zero accuracy at roughly half its token cap.Source: Figure 4 · p. 5, §2.4 · p. 6

3 Connecting Token Cues to Training Data

Counterfactual edits to mid-training data create or redirect token-cue effects, while hidden-state comparisons link different cues to different training sources. Cue timing also matters: early insertions sustain reasoning-trace similarity, unlike later ones.

3.1 Editing Training Data Determines Which Tokens Elicit Reasoning

  • 37.2% MATH-500 accuracy with .\n\n Chicken on the rename mix versus 2.4% on the base mix; .\n\n Okay reaches 38.3% on the rename mix.The edit also raises .\n\n Chicken-cued GSM8K accuracy from 2.2% to 60.6%.Source: §3.1 · p. 6, Figure 5 · p. 7
  • On the same factorization problem, the renamed cue makes the model solve and check the prompt instead of inventing a chicken-nugget problem, while ordinary bird and food uses remain.Figure 6 contrasts the base mix and rename mix on the problem and shows ordinary uses elsewhere.Source: §3.1 · p. 6, Figure 6 · p. 7
  • Removing “okay” from mid-training leaves its reasoning effect intact, but lowers its probability after the prompt from 0.15 to approximately 5 × 10−7; on the redirect mix, accuracy falls to 0.2%.On the redirect mix, next-token preferences change from let and so to what and which.Source: §3.1 · p. 6

3.2 Different Cues Shift Hidden States toward Different Training Sources

  • Okay inserted at paragraphs 1–2 yields 69–71% pass@1 versus 38% with no cue, and only those early insertions sustain increased reasoning-trace similarity through the next 128 tokens.Insertions at paragraphs 3–6 yield 23–30% pass@1, below the early positions.Source: Figure 9 · p. 8, §3.2 · p. 9

3.1 Editing Training Data Determines Which Tokens Elicit Reasoning

  • After replacing “step by step” with “duck duck goose” in training, MATH-500 accuracy with “Think duck duck goose” rises from 6% to 17%, comparable to “Think step by step” at 15%.Evaluation uses the prompt phrase without response prefill.Source: Figure 7 · p. 7, §3.1 · p. 8

3.2 Different Cues Shift Hidden States toward Different Training Sources

  • In the safety case study, I’m sorry shifts hidden-state similarity toward refusal text, Okay toward reasoning traces, and How toward code and science text.The cues also produce distinct refusal patterns on harmful and harmless requests.Source: Figure 10 · p. 9
  • Cue-induced hidden states shift toward different training sources: Okay toward synthetic reasoning traces, To toward expository math, and Answer toward short question-answer sources.Problem instead shifts toward meta-reasoning sources but scores 6.3% pass@1 and does not calculate an answer, so this similarity alone does not imply successful reasoning.Source: §3.2 · pp. 8–9, Figure 8 · p. 8

4 Token Cues Shape Refusal and Compliance

The safety evaluation shows that response-opening cues change refusal and compliance behavior, and that the resulting hidden-state similarities differ by cue. The math cue shifts refusal behavior toward greater selectivity, while a punctuation change to Okay, is associated with much more harmful compliance.

4 Token Cues Shape Refusal and Compliance

  • The study evaluates Olmo-3-7B on XSTest, using WildGuard to classify refusals and harmful content, with Instruct and Think-SFT variants as references.XSTest includes 250 safe and 200 unsafe prompts.Source: §4 · pp. 9–10
  • I’m sorry elicits refusals of both unsafe and safe requests, while .\n\n Okay shifts the base model toward more selective refusal behavior.Okay, instead elicits direct instructions in the harmful example.Source: §4 · p. 10
  • 42.4% of unsafe-request responses contain harmful content with Okay, versus 0.2% with .\n\n Okay.The authors report similar behavior with Sure, as a compliance cue.Source: §4 · p. 10
  • The hidden-state shifts differ by cue: I’m sorry moves similarity toward refusal text, both Okay forms toward reasoning traces, and How toward code and science text.The two Okay forms therefore align with reasoning-trace similarity despite differing refusal and harmful-response rates.Source: §4 · p. 10

5 Related Work

Related work studies reasoning elicitation, control over reasoning length, training-data influences, and refusal behavior. This paper positions its contribution as testing response-opening cues in base models and directly editing training data to examine their effects.

5 Related Work

  • Prior work elicits reasoning with demonstrations and instructions, decoding alternatives, or likelihood-guided resampling; this paper tests fixed one- or two-token openings across problems.It selects cues using answer agreement without reference answers (Wang et al., 2023; Zhang et al., 2025).Source: §5 · p. 10
  • Prior token interventions extend or suppress reasoning in post-trained models (Muennighoff et al., 2025; Wang et al., 2025a; Yang et al., 2026), whereas this work targets elicitation in base models.The paper states that its effective openings improve accuracy beyond simply increasing response length.Source: §5 · p. 10
  • Prior accounts propose that RL shifts probability toward existing reasoning trajectories or amplifies pretraining-favored patterns; this work tests whether RL promotes effective openings..Source: §5 · p. 10
  • Training-data associations have been studied through explicit control labels and attribution; this work directly edits data and retrains models to test how associations create or redirect cue effects (Keskar et al., 2019; Grosse et al., 2023).Source: §5 · pp. 10–11
  • Prior work links jailbreaks and safety post-training to early output tokens (Zou et al., 2023); this paper connects reasoning and refusal through shared response openings.Its results suggest that a cue selected for mathematical reasoning can also produce more selective refusal on safety prompts.Source: §5 · p. 11

6 Limitations

The analysis focuses on direct-from-base RL and selected models and training mixes, so the demonstrated cue effects have a bounded scope. The authors note that multi-stage post-training may add capabilities beyond those elicited by cues.

6 Limitations

  • The RL setting applies RL directly to a base model without an explicit prior post-training stage; complex multi-stage pipelines may add capabilities beyond cue-elicited behavior.Source: §6 · p. 11
  • Cue effectiveness is model-dependent: the tested openings yielded no effective cue for Llama-3.1-8B.Source: §6 · p. 11
  • Counterfactual experiments study Olmo and SmolLM3, whose training mixes include synthetic reasoning traces, where repeated cue-trace pairings may make effects especially strong.Source: §6 · p. 11

7 Discussion

The discussion treats token-cue effects as evidence that learned data associations can shape reasoning and instruction responses. It highlights implications for post-training design and synthetic data, while noting that these associations may arise unintentionally.

7 Discussion

  • Replacing “step by step” with “duck duck goose” in training provides an initial example of data associations shaping an instruction’s effect.The authors suggest richer interventions could help disentangle semantic content, surface form, and learned data associations.Source: §7 · p. 11
  • RL can increase the likelihood of cues that already elicit reasoning, prompting a distinction between what a model can do and what it is likely to do.The discussion asks whether making one behavior more likely through weight updates could make other useful behaviors harder to elicit.Source: §7 · p. 11
  • Synthetic-data biases can amplify cue-driven behaviors, and deliberately pairing cues with useful behaviors may make those behaviors easier to elicit.The authors also warn that such associations can arise unintentionally as synthetic data becomes more common in pretraining.Source: §7 · p. 11

A.1 Method

The appendix details how openings are proposed and selected, and tests larger searches, random openings, longer cues, and GEPA-optimized prefills. In the reported GEPA runs, short token cues score higher on all three evaluated benchmarks.

A.1 Method

  • The method proposes likely openings by beam search, then selects among them by answer agreement across sampled continuations.It retains 20 openings and chooses the candidate with the lowest mean answer entropy, subject to a missing-answer-rate filter.Source: §A.1 · p. 19, Table 1 · p. 20

A.2.1 Larger Search

  • On Olmo-3-7B under RL-Zero, with four rollouts per MATH-500 problem and a 31,744-token budget, the selected cue scored 0.773 versus 0.784 for the best of 500 two-token openings.Source: §A.2.1 · p. 19

A.2.3 Longer Openings

  • Extending the selected cue adds little accuracy, then reduces it once the opening becomes problem-specific.The reported extension follows the model’s most likely continuation one token at a time.Source: §A.2.3 · p. 19

A.2.2 Random Openings

  • Random tokens after a paragraph break score 0.333–0.471, below the paragraph break alone at 0.766 and below the reasoning cue.All ten random openings produce longer responses and reach the token budget more often than no cue.Source: §A.2.2 · p. 19, Table 2

A.3 Comparing with Prompt Optimization

  • Short token cues score higher than GEPA-selected prefills on all three benchmarks in these runs.GEPA uses correctness feedback and an external reflection model, whereas cue search uses no answer labels.Source: §A.3 · p. 22, Table 3 · p. 23

B.1 Sampling and Metrics

The appendix specifies response generation, grading, and metrics, then describes the prompts and benchmarks used for evaluation. It defines how correctness, sampling variation, and response-budget use are reported.

B.1 Sampling and Metrics

  • Generation stops at end-of-text, a new Problem block, or the token budget, and math grading checks extracted answers against references.Extraction, parsing, or verification failures count as incorrect; HumanEval responses must pass all unit tests.Source: §B.1 · p. 23
  • Pass@1 is the fraction of correct responses, while pass@16 estimates the chance that at least one of 16 sampled responses is correct.Pass@1 variation is measured across rollout indices; pass@16 uses bootstrap resamples of problems.Source: §B.1 · p. 23
  • Benchmark-table cells are reported only when every problem has all 32 responses generated with the stated token budget.Cap rate is the fraction of responses that reach the maximum generated response length.Source: §B.1 · pp. 23–24

B.2 Prompts

  • The evaluation uses plain-text zero-shot and few-shot prompts, with each token cue placed at the start of the response.The prompt set includes boxed, RL-Zero, and Minerva 4-shot formats.Source: §B.2 · p. 24

B.3 Benchmarks

  • The listed benchmarks span math problem sets and HumanEval coding problems.The appendix gives benchmark sizes, including 500 MATH-500 problems, 1,319 GSM8K problems, and 164 HumanEval problems.Source: §B.3 · p. 24

C.1 Benchmark Results

Across the evaluated models, short response cues improve reasoning accuracy, and much of the gain seen during reinforcement learning is available when the cue is fixed. The results also show that cue effects depend on the prompt, model, and response opening.

C.1 Benchmark Results

  • Short cues improve reasoning accuracy across multiple models and benchmarks; the reported tables compare base and RL-trained models with and without cues.The SmolLM3 boxed-prompt comparison transfers the RL-Zero cue because cue search finds no opening under that prompt.Source: Table 5 · p. 25, §C.1 · pp. 25, 28, Table 6 · p. 26

C.4 Prompts and Post-Trained Models

  • Cue gains vary with prompt and post-training: Minerva cues improve four Olmo and Qwen3 base models by 16–43 points, while SmolLM3 changes from 0.380 to 0.382.For post-trained Qwen3, cues add 6–7 points with thinking off but change accuracy by less than 0.1 point with thinking on; base-model RL-Zero cues can instead lower accuracy with thinking on.Source: §C.4 · pp. 29–30

C.7 Cue Variants

  • On Olmo-3-7B, either the paragraph break or Okay alone recovers most of the full cue’s MATH-500 gain; the paragraph break has a different effect on refusal.The paragraph break alone scores 0.766, close to 0.769 for the full cue and above 0.415 with no cue.Source: §C.7 · p. 31, Table 10 · p. 32

C.8 Paragraph Breaks in Uncued Responses

  • Without a cue, paragraph breaks begin 50% of correct responses versus 14% of incorrect ones, and responses beginning with that break are 47 points more often correct on matched problems.After a paragraph break, 95% of responses continue with Okay; bare Okay is unlikely after the prompt but ranks first after the break.Source: §C.8 · p. 32, Table 11

D.1 Training Objective and Setup

  • The RL comparisons use GRPO with binary correctness rewards on MATH; training runs use 7,500 training problems and evaluate on held-out MATH-500.Cue-forced runs include the cue in the prompt, and the figure reports MATH-500 pass@1 over 300 GRPO steps.Source: §D.1 · pp. 32–33, Figure 14 · p. 34

D.2 Learning Curves and Opening Probabilities

  • All four models start cue-forced RL at higher MATH-500 accuracy, and much of the gain within 300 training steps is already available with the cue.Qwen3-4B reaches the accuracy of 300 standard steps after 100 cue-forced steps; Qwen3-14B gains little from RL beyond the cued base model.Source: §D.2 · p. 34, Figure 14 · p. 34

D.3 Swapping Openings between Base and RL Models

  • In two MATH-500 examples, cued base-model responses solve problems that uncued responses miss; an RL-generated opening lets the base model reach 0.777 pass@1, while an RL model with a base opening reaches only 0.437.The opening swap compares with 0.751 for RL alone and 0.417 for the base model with a base-generated opening.Source: §C.2 · p. 28, §C.3 · p. 29, §D.3 · p. 36

D.4 Weight-Update Analysis

RL-related weight changes favor cue tokens at the response opening, and a targeted weight edit lets base models generate a cue and approach prefilled-cue accuracy. The analysis also finds that these updates are concentrated in a few active directions.

D.4 Weight-Update Analysis

  • The analysis merges RL-trained LoRA changes in residual-writing projections, decomposes them by SVD, and reads singular output directions through the unembedding.The signed contributions of the two most activated modes are projected into vocabulary space; Figure 18 shows cue variants becoming prominent in later layers.Source: §D.4 · p. 36, Figure 18 · p. 38
  • The vocabulary-space readout measures token-logit alignment, not the exact logit change, because it omits state-dependent RMS normalization and later computation.Either sign is valid for a singular vector, so the analysis uses its input-dependent coefficient to determine the signed contribution.Source: §D.4 · p. 36
  • At the final MLP output projection, the two leading modes account for 55.9% of the update’s squared Frobenius norm and align with cue-related tokens.Their signed readouts favor Okay variants and paragraph-break tokens, with contributions determined using the modes’ input-dependent coefficients.Source: §D.4 · pp. 36–37
  • Across layers, average participation ratios are 1.62 for MLP output projections and 1.56 for attention output projections, indicating concentrated input-conditioned update energy.Although LoRA updates can have rank 64, their response-boundary energy is concentrated in a small number of singular directions.Source: §D.4 · p. 37

D.5 Increasing Cue Probability with a Weight Edit

  • The weight edit uses the cue log-likelihood gradient and constrains expected squared output change on ordinary text, with C−1 weighting reducing changes along common text directions.The edit is applied to the final MLP projection, with its scale chosen to reach mean cue probability 0.95 on the sampled prompts.Source: §D.5 · p. 38
  • The weight edit brings accuracy close to the prefilled cue on both models without GRPO updates.The edit targets cue likelihood while limiting ordinary-text output changes; C4 perplexity changes little with C−1 weighting and increases more without it.Source: §D.5 · p. 38, Figure 19 · p. 39

E Test-Time Compute

Across test-time sampling and data-edit experiments, cues can improve base-model responses, while cue effectiveness depends on the opening and the training mix. Repeated no-cue sampling often requires substantially more tokens to match a cued response.

E.1 Length and Token Caps

  • At larger matched token caps, cue gains over no cue persist, though Qwen3-14B performs worse with its cue at the smallest caps.The comparison grades held-out responses after truncation rather than regenerating them.Source: §E.1 · p. 39, Figure 20 · p. 40

E.2 Repeated Sampling

  • No-cue voting needs 3.6–12 times as many tokens to match one cued response across the four evaluated models.At matched sample counts, RL matches or exceeds cued-base accuracy on three models from k=2, but remains below it on Qwen3-4B.Source: §E.2 · p. 39, Figure 21 · p. 40

F.1 Olmo-3-7B

  • On the rename mix, Chicken-cued accuracy rises across displayed math comparisons; on the redirect mix, Okay-cued accuracy is at most 0.011 and below no cue on every displayed comparison.On HumanEval, Chicken-cued accuracy is slightly above the no-cue baseline on the rename mix but below it on the redirect mix.Source: §F.1 · p. 41

F.1.1 Renaming Across the Full 100B-Token Mix

  • After 100B tokens, the Chicken cue reaches accuracy comparable to Okay, which remains effective across all three checkpoints.No-cue accuracy also increases after the edit.Source: §F.1.1 · p. 41, Table 16 · p. 42

F.1.2 Response Examples

  • In a redirect example, Okay generates an unrelated question and answer instead of solving the problem, while Chicken elicits a correct solution from the same checkpoint.The compared responses use the same MATH-500 problem and checkpoint.Source: §F.1.2 · p. 41

F.2 SmolLM3-3B

  • At the published trace share, renaming raises accuracy when cued with Chicken; redirecting lowers Okay below no cue, while Chicken, Hmm, and Alright remain effective (Tab. 17).Source: §F.2 · p. 43

F.2.1 Upsampling Reasoning Traces

Training-data edits can create or redirect cue effects, and the resulting behavior varies with cue, prompt boundary, and training context. Related edits also transfer an instruction’s reasoning effect to an arbitrary phrase.

F.2.1 Upsampling Reasoning Traces

  • Upsampling reasoning traces to 10% raises accuracy across evaluated openings and mixes while preserving the cue-specific pattern.Okay remains effective after renaming but falls below no cue after redirecting.Source: §F.2.1 · p. 44, Table 18
  • With 10% reasoning-trace tokens, the cue-specific pattern extends across MATH-500, GSM8K, AMC 23, and HumanEval: renaming raises Chicken above no cue, while redirecting lowers Okay below no cue.On the redirect mix, Chicken remains effective; both AIME sets remain at or below 0.003, too low to assess the pattern.Source: Table 19 · p. 44, §F.2.1 · p. 44

F.2.2 Opening Context

  • On the redirect mix, Okay reaches 0.250 pass@1 after a think tag, close to Chicken’s 0.247, despite scoring 0.024 after a paragraph break.This contrast supports cue effects depending on the preceding context.Source: §F.2.2 · pp. 44–45, Table 20 · p. 45

F.3 Additional Cue Counterfactuals

  • In Olmo, renaming Alright to Duck creates a Duck cue and redirecting Alright lowers it below no cue; in SmolLM3, Duck gains accuracy but remains below Okay and Hmm.For SmolLM3, Alright stays effective after removal from traces, then falls below no cue after redirecting.Source: §F.3 · p. 45, Table 21 · p. 45, Table 22

F.4 Prompt-Instruction Edit

  • After replacing “step by step” with “duck duck goose” in training data, “Think duck duck goose” achieves accuracy comparable to “Think step by step” across tested benchmarks.On MATH-500, the edited phrase rises from 6.4% to 16.5%, while “Think step by step” rises from 12.2% to 14.9%.Source: Table 23 · p. 46, §F.4 · p. 46

F.5.1 Opening Frequency and Continuations

  • Among 148,607 traces, Okay opens 82.5%, Alright 6.2%, and Hmm only 13 documents, yet all three reach 30–32% accuracy on the base mix.Source: §F.5.1 · p. 46

F.5.2 Ordinary Word Uses

  • After the rename, Chicken still refers to a bird or food in objects and definitions, but begins deliberative text at paragraph breaks and about half of sentence openings.After the redirect, paragraph-initial Okay begins a question.Source: §F.5.2 · pp. 46–47, Table 26 · p. 47

F.6.1 Earlier Checkpoints

  • The cue improves accuracy over no cue at final pretraining checkpoints, and this advantage becomes larger during mid-training on both Olmo models.No-cue accuracy also rises from 13.8% on the base mix to 23.3% on the rename mix and 26.7% on the redirect mix; opening changes accompany but do not establish the cause of this gain.Source: §F.6.1 · p. 47, §F.6.2 · p. 47, Figure 22 · p. 48

G.1 Method and Data

The paper compares response representations with a sampled bank of training-document states and examines wording markers, response styles, and the persistence of cue-associated similarity. These measurements characterize associations rather than causal contributions from training sources.

G.1 Method and Data

  • Generated states are compared by cosine similarity with a balanced bank of training-document states, and neighbor shares indicate similarity within that bank rather than causal source contributions.The bank covers eight source groups; layer 24 is selected because group-prediction accuracy there is 67%, versus 63% at layer 16.Source: §G.1 · pp. 48–49

G.2 Phrase Markers

  • Phrase-marker counts are wording measures, not correctness measures, and their prevalence can depend on response length.The cue increases marker counts for RL-trained Olmo-3-32B under the boxed prompt but has little effect under RL-Zero; for Olmo-3-7B, cued-base and RL marker rates are similar overall.Source: §G.2 · p. 49, Table 30 · p. 50

G.3 Response Styles

  • The response-style comparison separates accuracy, median length, phrase-detector matches, and answer-line frequency across openings.Answer-line frequency describes format separately from accuracy.Source: Table 31 · p. 51, §G.3 · p. 51

G.4 Data Edits

  • Renaming Okay to Chicken brings Chicken close to Okay in reasoning-trace neighbor share and accuracy: 0.36 to 0.79 and 0.020 to 0.375, respectively.Source: Table 32 · p. 52, §G.4 · p. 52

G.5 Cue Position

  • At paragraph two, inserting the cue raises reasoning-trace neighbor share from 0.45 to 0.73 within 64 tokens, with a paired difference of 0.21 still present at 256 tokens.For paragraph four, every measured mean difference from 16 to 256 tokens lies within two standard errors of zero.Source: §G.5 · p. 52, Table 33 · p. 53

G.6 Sustained Representational Shifts

  • In math responses, reasoning-trace neighbor share reaches 1.00 at paragraph-initial Okay and remains 0.77 after 255 tokens; in web text, it falls from 0.79 to 0.05 within eight tokens.Random paragraph starts stay at 0.57–0.61 in math and 0.02–0.05 in web text.Source: §G.6 · p. 53, Figure 23 · p. 53

H.1 Evaluation Protocol

The safety evaluation compares how response-opening cues affect refusal and harmful responses, then examines cue representations, other models, and example outputs. Results show that punctuation and whitespace can alter refusal behavior, while cue effects vary across models.

H.1 Evaluation Protocol

  • The evaluation uses 250 safe and 200 unsafe XSTest prompts, eight fixed-seed rollouts per prompt, and separate WildGuard labels for refusal and harm.Safe prompts resemble unsafe requests; at most 4 of 1,600–2,000 rollouts per cell are excluded for unparsable verdicts.Source: §H.1 · pp. 53–54
  • Among non-refusing unsafe responses, harmful labels range from 0 of 7 with I ’m sorry to 89% with How ,; refusal and harm are judged separately.The harmful label also includes responses that begin to comply or repeat the request.Source: §H.1 · p. 54

H.2 Token Cues

  • 38.3% to 81.9% refusal of unsafe requests and 42.4% to 0.2% harmful responses follow changing Okay, to .\n\n Okay, while removing the comma makes little difference.Safe-request refusal changes from 4.3% to 4.2% when changing Okay, to .\n\n Okay.Source: §H.2 · p. 54

H.3 Hidden States and Other Models

  • At the cue boundary, all nearest neighbors of paragraph-break Okay are reasoning-trace states, whose share stays 0.11–0.16 in later bins; space-prefixed Okay falls from 0.25 to 0.04.Responses with space-prefixed Okay contain about 140 tokens, compared with 600 for paragraph-break Okay.Source: §H.3 · p. 55
  • On Qwen3-4B-Base and Qwen3-14B-Base, Sure and I ’m sorry show similar refusal patterns to Olmo, but paragraph-break Okay lowers unsafe-prompt refusal.The selective-refusal benefit of Olmo’s reasoning cue is specific to Olmo among these models.Source: §H.3 · p. 55

H.4 Response Examples

  • Example tables show one safe and one unsafe prompt across cue conditions, with refusal counts and response lengths; selected unsafe examples show responses that follow the request.The displayed responses use Olmo-3-7B-Base and are cut to their first two sentences, except selected unsafe responses, which show three.Source: §H.4 · p. 56, Table 35 · p. 56, Table 36 · p. 56
Loading 2610.06851v1…