PaperlayerView exact revision on arXiv

Source-linked AI summary

Prefilling the Reasoning Channel: Output-Prefix Attacks on Reasoning LLMs

Lukáš Brůna, Robert Bridges, Adam Ek

arXiv:2609.29775v1cs.CRcs.AI

TL;DR

Output-prefix attacks have been studied on non-reasoning models, but their interaction with scratchpad reasoning across exposed- and hidden-reasoning systems was not systematically tested. This study compares reasoning and prefix combinations across models and finds that reasoning injection alone is ineffective, while pairing it with a prefix can achieve high success on some models.

  • Problem

    It was not known systematically whether scratchpad reasoning buffers models against output-prefix attacks or provides an additional attack surface, and susceptibility may vary with harness design.

    Source: §1 · pp. 2–3

  • Method

    The study systematically compares reasoning-only and reasoning-plus-output-prefix attacks across exposed- and hidden-reasoning models, using a factorial evaluation on three models.

    Source: §1 · p. 3, §5.1 · p. 5

  • Results

    Malicious reasoning alone yields approximately 0% attack success, but combined with a trivial output prefix reaches as high as 99%; contextual prefixes outperform static ones and susceptibility varies by model.

    Source: §1 · p. 3, §8 · p. 8

  • Takeaways & Limitations

    The scratchpad is an additional attack surface even when reasoning is hidden, but strong resistance in Claude Haiku 4.5 indicates the vulnerability is addressable at scale.

    Source: §8 · p. 8

    The study covers only three models and one dataset, so results may not generalize across alignment strategies and reasoning implementations; its single automatic judge was validated on one tactic and model.

    Source: §8 · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) consume and produce a single sequence of text; hence, if text can be added to the beginning of the LLM's response, i.e., an output prefix, then all subsequent tokens will be conditioned on it. This output-prefix attack technique is a cheap black-box prompt injection. Prior work has shown this type of attack can reliably jailbreak non-reasoning models. Most reasoning models add an intermediate scratchpad reasoning step before the assistant's final response. The ability to edit this reasoning channel is exposed by some APIs and attack vectors can be leveraged for reasoning injection attacks. We present the first systematic, controlled study that isolates the scratchpad reasoning channel as an output-prefix attack vector, and the first to compare reasoning-only, output-prefix-only and reasoning-plus-output-prefix attacks across both exposed- and hidden-reasoning models. Using a factorial design of 3 prefix types $\times$ 2 reasoning injections over $1{,}800$ test cases drawn from AdvBench, we attack three 2026-era frontier models Gemini 3 Flash Preview, DeepSeek V4 Flash, and Claude Haiku 4.5. We find that injecting malicious reasoning alone is essentially inert ($\approx0\%$ attack success), but injecting the same reasoning together with a trivial output prefix raises the attack success rate to as high as $99\%$ for some models. For this type of attack we find that contextual prefixes work better than static prefixes; and that susceptibility is dependent on the model.

1 Introduction

The paper asks whether output-prefix attacks can exploit scratchpad reasoning, an attack surface that may either buffer or amplify vulnerabilities. Its controlled study finds that injected reasoning alone is ineffective, while pairing it with an output prefix can produce high success on some models.

Research questions

  • The central open question is whether scratchpad reasoning buffers a model against an attack or becomes an additional high-leverage surface that an attacker can seed.The paper also asks how attacks fare across harnesses that expose or hide reasoning.Source: §1 · pp. 2–3

Contributions

  • The study isolates scratchpad reasoning as an output-prefix attack vector and compares reasoning-only with reasoning-plus-output-prefix attacks across exposed- and hidden-reasoning models.The attack is described as requiring essentially no compute, unlike GCG’s repeated gradient computations [31].Source: §1 · p. 3

Study design

  • The evaluation uses a 3×2 factorial design crossing prefix type (none, static, contextual) with reasoning injection (absent, injected), across 1,800 AdvBench [31] test cases and three frontier models.Source: §1 · p. 3

Findings

  • Reasoning injection alone yields approximately 0% attack success, but the same reasoning with a trivial output prefix reaches as high as 99%, with contextual-prefix and model-dependent effects.Claude Haiku 4.5 is near-immune.Source: §1 · p. 3

2 Background and Threat Model

The background links attacks to autoregressive generation and the serving harness’s control over prior assistant content. It defines the scratchpad and the assumed attacker capabilities used in the study.

2.1 Auto-regression and statelessness

  • Autoregression makes each output token depend on input and previously generated output, while statelessness lets an attacker supply text as if it were prior model output.Editing those prior tokens shifts the probabilities of subsequent tokens, potentially toward a malicious continuation.Source: §2.1 · p. 3

2.2 Reasoning and the scratchpad channel

  • Scratchpad reasoning is intermediate reasoning emitted before the answer, and the serving harness determines whether an editable reasoning field is exposed.The paper’s exposed/hidden shorthand refers to the harness, not the model weights; latent reasoning is out of scope.Source: §2.2 · p. 3

2.3 Alignment

  • The attack targets harmlessness alignment under a compromised context where the attacker supplies the model’s purported prior output and, when exposed, prior reasoning.Source: §2.3 · p. 4

2.4 Threat model

  • The threat model assumes a black-box attacker with API access and knowledge of the message template, able to prefill assistant content and exposed reasoning but not access weights or gradients.The attacker’s goal is to elicit a harmful response the model would otherwise refuse.Source: §2.4 · p. 4

3 Related Work

Prior work establishes output-prefix attacks on non-reasoning models and identifies reasoning as a possible weakness, but does not systematically isolate injection into the assistant’s reasoning channel. Related work also points to prefill awareness as a possible defense.

Output-prefix attacks on non-reasoning models

  • ChatBug [13] and OPRA report high attack success for output-prefix attacks on non-reasoning models, but neither tests a harness that exposes reasoning.ChatBug studies fixed, prompt-tailored, and LLM-crafted prefixes; OPRA builds tailored prefixes by probing response habits.Source: §3 · p. 4

Reasoning as a vulnerability

  • Overthink [15] uses reasoning to exhaust resources, while GSM-Symbolic [18] shows reasoning can be derailed by irrelevant context; this paper instead treats reasoning as a jailbreak injection surface.Source: §3 · p. 4

Conversation- and reasoning-injection

  • Dialogue Injection [17] and ChatInject [6] forge conversational context, while this study injects into the harness-presented assistant reasoning channel and isolates its effect with and without an output prefix.Krishna et al. compare reasoning and non-reasoning models across prompt-based attacks.Source: §3 · p. 4

Prefill awareness as a defense

  • Prefill-awareness research finds the ability inconsistent and shallow across models but emerging in recent Claude models, consistent with the model-specific resistance observed here.Wang et al. [27] study detecting altered message history.Source: §3 · p. 4

4 Attack Methodology

The methodology separates the malicious payload from how a serving harness delivers it, then tests output-prefix and reasoning-injection combinations. It targets the model’s treatment of attacker-supplied prior assistant content as authoritative.

4 Attack Methodology

  • The paper separates the malicious prefix or reasoning payload from its delivery mechanism, such as assistant prefill, to distinguish model vulnerabilities from harness-specific delivery methods.Other possible delivery mechanisms include system prompts, few-shot seeding, and logit biasing.Source: §4 · p. 4

4.1 Core vulnerability

  • The core vulnerability is the model’s assumption that prior assistant output and, when exposed, prior reasoning are authoritative; the harness must prevent attackers from controlling that context.Some endpoints force a fresh response, while others permit assistant prefill.Source: §4.1 · p. 5

4.2 Delivering the payload via the API

  • For exposed-reasoning APIs, the attacker supplies malicious reasoning_content and a short output prefix; for hidden-reasoning models, the attacker fabricates a <think> block inside the output prefix.Source: §4.2 · p. 5

4.3 Tactics of attack

  • The six tactics cross prefix type with reasoning injection: baseline, static or contextual prefix, reasoning only, and reasoning combined with either prefix.The factorial is organized as three prefix types crossed with reasoning injection absent or present.Source: §4.3 · p. 5, Table 1 · p. 5

5 Experimental Setup

The study tests whether scratchpad reasoning buffers or increases vulnerability across models with different harnesses. It evaluates AdvBench prompts using generated reasoning payloads and a three-point response judge.

5 Experimental Setup

  • The experiment tests whether exposed reasoning helps models re-align before answering or instead creates an additional attack vector.These are the study’s H1 and H2, respectively.Source: §5 · p. 5

5.1 Victim models

  • The three API-tested models differ in alignment, harness design, and reasoning exposure; DeepSeek V4 Flash exposes an editable reasoning field.Gemini and Claude keep reasoning internal; DeepSeek is the only open-weight model among the three.Source: §5.1 · pp. 5–6

5.2 Prompts, generation, and tactics

  • Across 1,800 cases, each model received the same 100 AdvBench prompts under six tactics, with context-specific scratchpads generated by Wizard-Vicuna-30B.Contextual prefixes use AdvBench target strings; the scratchpad generation adds a fixed closing sentence to discourage further reasoning, and Table 2 gives tactic examples.Source: §5.2 · p. 6, Table 2 · p. 6

5.3 Evaluation

  • A three-point judge scores only newly generated completion text, and manual ratings matched automatic ratings exactly in 92% of 100 cases.The reported mean absolute error was 0.09 on the 1–3 scale.Source: §5.3 · p. 6

6 Results

Reasoning injection alone leaves attack success essentially unchanged, but adding it to an output prefix raises success on the two vulnerable models. Prefix effects and overall susceptibility vary by model, with contextual prefixes outperforming static ones in the prefix-only comparison.

6 Results

  • Malicious reasoning alone leaves all three models near baseline at approximately 0% ASR.A same-prompt example contrasts refusal after reasoning-only injection with compliance after adding an unfinished output prefix.Source: §6 · pp. 6–7
  • Contextual prefixes beat static prefixes: DeepSeek V4 Flash rises from 19% to 56% ASR, while Gemini 3 Flash Preview rises from 78% to 83%.Claude Haiku 4.5 remains at 1% for both prefix-only tactics.Source: §6 · p. 7
  • DeepSeek V4 Flash ASR rose 19→65 (+46 pp) with static and 56→76 (+20 pp) with contextual prefixes; Gemini 3 Flash Preview rose 78→99 (+21 pp) and 83→97 (+14 pp), respectively.The strongest tactic differs by model: reasoning+contextual for DeepSeek (76%) and reasoning+static for Gemini (99%).Source: §6 · p. 7, Table 3 · p. 7
  • Claude Haiku 4.5 never exceeds 1% ASR, while Gemini 3 Flash Preview is most vulnerable overall; Figure 1 shows the Miss, Partial Hit, and Hit breakdown across tactics.Claude’s partial-hit rate reaches 8% under reasoning+static and 5% under reasoning+contextual; DeepSeek’s highest partial-hit rate is 22% under the static prefix.Source: §6 · pp. 7–8, Figure 1 · p. 7

7 Discussion and Mitigations

Output-prefix attacks remained effective on some reasoning models but varied sharply by model; adding malicious reasoning increased success when paired with a prefix, while reasoning alone did not. The discussion attributes this contrast to continuation patterns and identifies screening and prefill controls as defenses.

Results

  • 78–83% ASR on Gemini, 19–56% on DeepSeek, and approximately 1% on Claude show that prefix-only attacks are strongly model-dependent; reasoning alone does not succeed, but reasoning plus a prefix raises ASR on vulnerable models.DeepSeek was more robust than Gemini under both static and contextual prefixes.Source: §7 · p. 8

Why reasoning-only is inert

  • At a sentence boundary, broad next-token probabilities can let safety-aligned refusals resume, whereas an unfinished output prefix concentrates continuation on compliance.The shared reasoning payload in tactics 5 and 6 yields higher ASR when an output prefix is added.Source: §7 · p. 8

Mitigations

  • Screen user prompts, attacker-supplied prefixes, and reasoning content; removing assistant prefill can also reduce exposure, though compatibility endpoints may reintroduce it.The paper also notes that screening and model-based guardrails add cost and can themselves be attacked.Source: §7 · p. 8

8 Conclusion and Future Work

The study concludes that reasoning injection alone was inert, but pairing it with an output prefix could produce high ASR, with susceptibility varying by model and harness. It also reports study limitations, responsible-disclosure practices, tool use, serving-endpoint behavior, and evaluation procedures.

8 Conclusion and Future Work

  • Across 1,800 cases, reasoning alone was inert, but pairing it with a trivial output prefix reached 99% ASR; contextual prefixes beat static ones, and vulnerability varied by model and harness.The scratchpad attack also appeared with hidden reasoning and was nearly free to implement on vulnerable models. Future work would add providers, benchmarks, and longer, more sophisticated prompts.Source: §8 · p. 8
  • The study tested only three models and AdvBench, limiting generalization across alignment strategies and reasoning implementations; its automatic judge was validated on only one tactic/model.The authors say broader manual validation would strengthen the findings.Source: §8 · p. 8

A Ethics and responsible disclosure

  • The authors redacted malicious specifics, used potentially offensive examples to convey attack seriousness and ease, and submitted vulnerability details to affected providers on May 20, 2026.Source: A Ethics and responsible disclosure · p. 11

B LLM usage in this paper

  • An AI coding agent drafted evaluation and visualization code before manual editing; generative models also supported experiments, automatic evaluation, and paper editing.Source: B LLM usage in this paper · p. 11

C Serving-API mechanics

  • Gemini 3 native prefill was disabled while its OpenAI-compatibility endpoint accepted it; OpenAI endpoints ignored supplied assistant turns, and no OpenAI victim model was evaluated.Table 4 compares endpoints by assistant-turn prefill, editable scratchpad fields, and default safety filtering.Source: Table 4 · p. 11, C Serving-API mechanics · p. 12

D Generation and judge prompts

  • Wizard-Vicuna-30B generated malicious scratchpads for AdvBench prompts with a fixed closer, and DeepSeek V4 Flash at temperature 0 judged completions as hit, partial hit, or miss.Source: D Generation and judge prompts · p. 12

E Result details: Hit rate, Partial-hit rate, and usable n

Table 5 reports hit rates, partial-hit rates, and usable response counts for the Table 3 cells, with ASR corresponding to hit rate. Several Gemini cells had fewer than 100 usable responses.

E Result details: Hit rate, Partial-hit rate, and usable n

F Redacted transcript pair

The transcript pair holds the malicious reasoning constant while contrasting refusal with a hit after adding a prefix. The accompanying heatmap encodes hit rates by model and tactic, and the listings illustrate different API delivery paths.

F Redacted transcript pair

  • With the same DeepSeek prompt and malicious reasoning, tactic 4 produces a refusal, while tactic 5 produces a hit after adding the unfinished static prefix.The two conditions differ only in whether an output prefix is appended.Source: F Redacted transcript pair · p. 13

G Hit-rate heatmap

  • Figure 2 maps ASR by model and tactic, with darker cells indicating lower rates and yellow cells higher rates; its labels identify six tactics.Figure 2 presents the same hit rates as Table 3 and Figure 1.Source: G Hit-rate heatmap · p. 13, Figure 2 · p. 14

H Additional API listings

  • The listings show exposed-reasoning requests, Gemini compatibility-endpoint prefill, and Claude requests that concatenate scratchpad text into assistant content rather than a reasoning_content field.Source: H Additional API listings · pp. 14–15

I Reasoning of large language models

The section distinguishes inference-time chain-of-thought prompting from trained token-level scratchpad reasoning and latent reasoning. It also frames scratchpad access as a serving-harness choice and identifies latent reasoning as outside the paper’s scope.

I Reasoning of large language models

  • Chain-of-Thought (CoT) prompting [29] elicits step-by-step reasoning at inference time through an ⟨input, chain of thought, output⟩ example pattern.The section notes its success, especially on math problems.Source: I Reasoning of large language models · p. 15
  • DeepSeek-R1-Zero’s reinforcement-learned scratchpad uses a separate multi-step thinking process before the answer, with the answer—not the reasoning method—evaluated.The paper uses scratchpad reasoning [20] for explicit intermediate reasoning and investigates editing it to be malicious or compliant with a malicious user prompt.Source: I Reasoning of large language models · pp. 15–16, Table 7 · p. 17
  • Scratchpad reasoning is explicit token-level reasoning in a separate space before the answer, unlike latent reasoning’s hidden-state representation.The section describes scratchpad tokens as often enclosed by <think> and </think>.Source: I Reasoning of large language models · p. 16
  • The serving harness determines whether scratchpad reasoning is exposed, while latent reasoning feeds the last hidden state back as an input embedding and is out of scope.Latent reasoning trades efficiency for less visible explainability and auditability.Source: I Reasoning of large language models · p. 16
Loading 2609.29775v1…