PaperlayerView exact revision on arXiv

Source-linked AI summary

Control-Token Injection Suppresses Chain-of-Thought and Defeats Reasoning-Based Oversight in Tool-Using Agents

Muhammad Usama, Khair Un Nisa, Summer Yeoreum Jung

arXiv:2609.27542v1cs.CR

TL;DR

Reasoning-based oversight depends on unsafe intent appearing in a tool-using agent’s reasoning trace, but that trace can be suppressed through untrusted input. The paper measures attacks on both the reasoning channel and tool-call parser, finding that the harness as well as the model determines safety outcomes.

  • Problem

    CoT monitoring assumes unsafe intent will appear in a reasoning trace, but the paper examines whether untrusted input can attack that assumption.

    Source: §1 · p. 2

  • Method

    The paper measures control-token injection and parser behavior in tool-using agents, and examines delivery, template dependence, and defenses.

    Source: §1 · p. 2

  • Results

    The reasoning trace fell from 52.5 tokens to zero on every plain-framing trial, while the tool call still executed; the attack also converted 39.6% of otherwise-refused requests into exfiltrations.

    Source: §7.1 · p. 6, §7.2 · p. 8

  • Takeaways & Limitations

    The findings support treating tool-agent safety as a joint property of model and harness, with both reachable from untrusted input.

    Source: §10 · p. 13

    The experiments deliver the adversarial span in the user turn rather than through a web or email tool return because the sandbox guardrail blocks those provenance routes.

    Source: §9 · p. 13

Abstract

from arXiv · show

The safety of a tool-using language model agent is usually treated as a property of the model alone. We give controlled, full-precision evidence that it is instead a joint property of the model and the software that renders its chat template and parses its tool calls, the decoding harness, and that both halves are attackable from untrusted input. On the released gpt-oss-20b reasoning model under its published tool sandbox, appending a single string of the model's own channel-control tokens to a user message makes the tokenizer render a reasoning turn that is already complete, so the model writes no chain-of-thought and proceeds directly to the tool call. Across forty tasks the model already completes, the reasoning channel falls from a mean of 52.5 tokens to zero on every trial while the http.post still fires on every trial. A rule monitor and a cross-family language-model monitor detect the unsafe request on all plain trials and no forged trials, and on overtly malicious requests the attack converts 39.6% of the model's refusals into completed exfiltrations. Separately, whether an identical tool-call generation fires is decided by the harness parser, not the model: a truncation-tolerant regular expression fires a call whose closing token is missing while a strict one drops it, and two parsers shipped for the Gemma agent give opposite outcomes on identical greedy generations, firing on all twenty-four trials and on none. We show the suppression can be delivered indirectly and characterize its dependence on the chat template across two more reasoning models, and we evaluate input sanitization, parser hardening, and empty-reasoning detection as defenses; flagging an absent trace catches the basic attack but not an adaptive benign decoy. All measurements use greedy decoding on publicly released models. Code and per-trial logs: https://github.com/Usama1002/deleting-the-trace

1 Introduction

Chain-of-thought monitoring assumes unsafe intent will appear in a reasoning trace, but untrusted input can suppress that trace without requiring deceptive reasoning. The paper argues that tool-agent safety depends jointly on the model and its decoding harness, including the template and parser.

1 Introduction

  • CoT monitoring assumes an unsafe agent will leave intent in a reasoning trace, an assumption untrusted input can attack without deceptive reasoning.The trace can reveal misbehavior that the final action alone would hide.Source: §1 · pp. 1–2
  • Agent safety is a model-times-harness property: chat templates and tokenizers shape turns, while parsers determine whether generated tool calls execute.Neither component belongs to the model whose safety was measured.Source: §1 · p. 2
  • A closed, empty analysis channel can make the model skip reasoning and proceed to a tool call; the sandbox also uses parsers with different tolerance for missing closing tokens.The gpt-oss parser tolerates a missing closing token, while the Gemma parser requires one.Source: §1 · p. 2
  • The paper reports six contributions spanning reasoning deletion, oversight and refusal bypass, indirect delivery, parser-dependent execution, template dependence, and defenses.Reported defenses include sanitization, parser hardening, and empty-reasoning detection, which an adaptive benign decoy defeats.Source: §1 · p. 2

2 Background

The background describes the message structure and parsing steps that determine where reasoning appears and whether tool calls execute.

2 Background

  • Harmony messages use role and channel markers to delimit content, with reasoning in the analysis channel and answers in the final channel.The format also uses <|start|>, <|message|>, and <|end|> or <|call|> to structure messages and tool calls.Source: §2 · p. 2
  • Because Harmony delimiters are ordinary vocabulary tokens, text that tokenizes into them becomes indistinguishable from harness-produced structure.This is the substrate exploited by the reasoning-channel attack.Source: §2 · p. 2
  • The harness executes a tool call only after a parser extracts it, and its gpt-oss and Gemma parsers handle missing closing tokens differently.A normalization step shared by both models rejects generations that parse to more than one tool call.Source: §2 · p. 3

3 Threat Model

The threat model gives an indirect adversary control of untrusted context text, while restricting access to model and system components. The empirical delivery setting is narrower than the assumed attack surface.

3 Threat Model

  • The adversary controls a span entering the agent context but cannot modify model weights, system prompt, trusted-context flags, or the monitor.Success means an unsafe tool call executes and the monitor fails to flag it from the reasoning trace.Source: §3 · p. 3
  • The attack mechanism is delivery-channel agnostic, but experiments use the user turn because the sandbox guardrail blocks dangerous calls with recent web or email provenance.The guardrail would prevent observing exfiltration after those tool-return delivery routes.Source: §3 · p. 3

4 Related Work

Related work studies instruction injection, template manipulation, adversarial suffixes, reasoning-trace monitorability, and parser fragility. This paper distinguishes its focus on deleting the reasoning trace and measuring parser-controlled execution.

4 Related Work

  • Prior injection and monitorability work studies instruction elevation or concealed intent during reasoning; this paper instead deletes reasoning from the input side.The paper says its attack requires neither model cooperation nor training pressure.Source: §4 · pp. 3–4
  • Instruction-hierarchy and provenance defenses address injected natural-language instructions, while this attack reshapes tokenized turn structure with control tokens.The paper argues those defenses do not by themselves address this lower-level payload.Source: §4 · p. 3
  • ChatInject and automated template injection forge structure to elevate or execute instructions, whereas this paper measures trace deletion, monitor failure, and refusal conversion.Those three outcomes are not measured by the cited works, according to the paper.Source: §4 · pp. 3–4
  • Unlike gradient-optimized adversarial suffixes, the refusal bypass uses a fixed, human-readable control-token string and acts by removing the reasoning step.The paper reports deterministic behavior under greedy decoding.Source: §4 · p. 4
  • Prior CoT-monitorability studies presume a trace exists to be hidden or measured; this paper tests whether unsafe action survives deleting the trace.It describes this as addressing a case set aside by work arguing severe harms require load-bearing reasoning.Source: §4 · p. 4
  • Practitioner reports describe parser brittleness, while this paper holds the model and greedy decode fixed to test whether the parser alone decides call execution.The paper also reports that the truncation tolerance appears in the official reference parser and the sandbox.Source: §4 · p. 4

5 Method

The method appends a structural suffix that closes an empty assistant analysis turn, suppressing reasoning without semantic prompting. It also compares how shipped parsers handle complete, truncated, and multiple tool calls.

5.1 The reasoning-suppression mechanism

  • The injected suffix closes the current turn, then opens and immediately closes an empty assistant analysis channel, leaving no reasoning content.The model then continues into the commentary channel carrying the tool call; unlike semantic prompts, the suffix manipulates tokenized turn structure.Source: §5.1 · p. 5

5.2 Parser leniency as a security property

  • The method compares a truncation-tolerant parser with a strict parser on complete, truncated, and double-call generations, then swaps parsers while holding the model fixed.The stated predictions are that both parsers accept complete calls, only the tolerant parser accepts a truncated call, and both reject a double call.Source: §5.2 · p. 5

6 Experimental Setup

The experiments use a fixed sandbox and deterministic greedy decoding to measure reasoning suppression and refusal bypass across two task families. They evaluate gpt-oss-20b alongside models assigned to parser, monitoring, and generality studies.

6 Experimental Setup

  • All measurements use the sandbox’s eight-hop loop and seed 123, one NVIDIA H200, bfloat16 precision, and greedy decoding.Reported rates include Wilson 95% intervals for finite task samples; deterministic decoding means a message that fires once fires again.Source: §6 · p. 6
  • Plain-framing tasks test reasoning-trace changes on requests the base model complies with, while malicious-framing tasks test refusal bypass on requests it refuses.Both families include benign controls posting a non-secret payload; sandbox attack success requires a successful http.post carrying the literal secret marker.Source: §6 · p. 6
  • gpt-oss-20b is the primary target; Gemma is used for strict-parser and monitoring studies, while DeepSeek-R1 and Qwen3 are used in the generality study.The additional reasoning models are driven through their own chat templates; the sandbox provenance guardrail is not attacked and constrains delivery channels.Source: §6 · p. 6

7 Results

The results show that agent safety depends not only on model behavior but also on the harness that renders chat turns and parses tool calls. Control-token injection can remove reasoning while preserving actions, and parser choices can determine whether generated calls execute.

7.1 Finding 1: control-token injection deletes the reasoning channel

  • 52.5 reasoning tokens on average fell to zero on every forged trial, while http.post still fired on all forty plain-framing tasks.The deletion also occurred on benign tasks, leaving task completion intact.Source: §7.1 · p. 6, Figure 2 · p. 7, Table 1 · p. 7
  • Opening the assistant analysis channel is the load-bearing suffix component; it suppresses reasoning even when unclosed, unlike a bare <|end|> or another channel.The unclosed variant scored 0.65 against 0.50 for the full suffix.Source: §7.1 · p. 7, Table 2 · p. 8
  • After tokenization, the injected delimiters are dedicated control tokens indistinguishable from harness-produced structure, letting generation proceed directly to the tool-call channel.Without the suffix, generation begins in analysis; with it, generation begins in commentary with the tool call.Source: §7.1 · p. 7

7.2 The injection defeats reasoning-based oversight and bypasses refusals

  • 39.6% of malicious requests refused without the suffix instead completed exfiltration with it; both CoT monitors detected all plain trials and none of the forged trials.The argument check caught the nineteen successful trials, but it was the success oracle rather than an independent monitor.Source: §7.2 · p. 8, Table 3 · p. 9

7.3 The attack is delivered indirectly

  • The suffix also suppressed reasoning when placed in retrieved text, a trailing note, or a tool-role message; tool-role delivery was not scored end-to-end for exfiltration.The document and note placements remained inside the user turn, while the guardrail blocked dangerous calls with web-or-email provenance.Source: §7.3 · pp. 8–9

7.4 Finding 2: harness parser leniency decides whether a tool call fires

  • With identical Gemma generations and greedy decoding, swapping only the harness parser changed exfiltration from firing on all twenty-four trials to none.The lenient parser also fires truncated calls; the strict parser drops them when the closing token is missing.Source: §7.4 · p. 9, Table 4 · p. 10

7.5 Coupling: the chat template gates how far the injection reaches

  • The injection’s transfer depends on chat-template placement: Qwen3 user-message injection did not transfer, while DeepSeek remained unwilling to act even when prefix delivery emptied reasoning.Qwen3’s closed-channel prefix delivery raised compliance to 0.92 on plain-framing tasks and 1.00 on malicious-framing tasks; that delivery was outside the threat model.Source: §7.5 · p. 10, Table 5 · p. 11
  • The robust sanitizer restores reasoning and refusal while stopping malicious posts and preserving benign posts; naive deletion fails on benign inputs.The table distinguishes desired outcomes by task group: stop malicious posts but allow harmless ones.Source: Table 6 · p. 11

8 Defenses

The paper evaluates input sanitization, parser hardening, and empty-reasoning detection; an adaptive benign decoy defeats the basic trace-absence tripwire.

8 Defenses

  • Replacing control-token spans with spaces restores reasoning and refusal while preserving benign behavior; deleting them can splice fragments into a new trigger.The robust sanitizer reduces attack success from 0.50 to 0.00 on the malicious subset; naive deletion leaves benign reasoning empty on every trial.Source: §8 · p. 11
  • Requiring the closing token stops truncated calls but also drops genuine calls truncated by a token limit.The paper frames recovery of truncated calls versus rejecting malformed ones as a security decision.Source: §8 · p. 12
  • A zero-length reasoning tripwire catches the basic attack here, but a short benign decoy defeats all three monitors while exfiltration fires on all sixteen malicious trials.The decoy is not flagged by the keyword monitor, empty-reasoning tripwire, or cross-family language-model monitor.Source: §8 · p. 12, Table 7 · p. 12
  • The refusal bypass persists without oversight: the injection can turn a refused request into an executed action even when no monitor is present.Reasoning-based monitoring additionally loses visibility into that harm when the trace is removed.Source: §8 · p. 12

9 Limitations

The evaluation centers on one model and sandbox, with limited additional-model and delivery-channel evidence; the authors also identify limits in comparison, monitoring, and defense scope.

9 Limitations

  • Measurements use one primary target model and sandbox, so exact rates may differ with other tools, decoding settings, or task distributions.The sandbox was selected for its faithful, reproducible harness and published parsers.Source: §9 · p. 12
  • The generality study covers only two additional reasoning models, and their compliance results measure unsafe-call emission outside the sandbox rather than live execution.Source: §9 · p. 12
  • The experiments do not demonstrate end-to-end exfiltration from an untrusted tool return because the guardrail blocks web- or email-sourced dangerous calls.The authors leave testing that chain to a testbed that permits tainted input to reach an external post.Source: §9 · p. 13
  • The paper gives no numerical head-to-head with role-forging attacks, uses one cross-family monitor, and does not claim its defenses cover adaptive constructions.The authors describe role-forging and control-token injection as complementary attacks with different objectives.Source: §9 · p. 13

10 Conclusion

The conclusion argues that tool-using agent safety depends on the model and its decoding harness, both reachable through untrusted input. It also highlights defenses and shows why reasoning-trace checks alone are insufficient.

10 Conclusion

  • A control-token string deletes the reasoning trace while the unsafe action survives, defeating trace-based oversight and converting refusals into completed exfiltrations.The conclusion adds that a benign decoy defeats a keyword monitor, the empty-trace tripwire, and a language-model judge; it therefore favors checking the action itself.Source: §10 · p. 13

Broader Impact Statement

  • The authors recommend sanitizing untrusted control-token spans, hardening parsers where reliability permits, and pairing reasoning monitors with action-level checks.They report conducting the offensive experiments only in a released sandboxed benchmark, without attacking external systems, services, or accounts.Source: Broader Impact Statement · p. 13

A Raw Traces

  • With the suffix, the model skips analysis and executes the malicious exfiltration request that it refuses when the suffix is absent.The raw generations show the mechanism on both neutral and malicious requests.Source: A Raw Traces · p. 15

B Indirect Delivery

  • The suffix suppresses reasoning and leaves the unsafe action firing when delivered as untrusted content rather than as the user’s instruction.Table 8 reports 15 trials per placement, while its tool-return row is a single deterministic trace per condition.Source: Table 8 · p. 16, B Indirect Delivery · p. 16

C Attack Strings, Tasks, and Monitors

This section specifies the control-token suffix, task and monitor setup, and reports sensitivity to reasoning effort. It also identifies the scope of the full-generality results table without supplying its cell values.

C Attack Strings, Tasks, and Monitors

  • The suffix and its ablated variants are formed from control tokens, with the fixed suffix shown as a closed, empty assistant analysis channel.The passage supplies the token sequence but no ablation outcomes.Source: C Attack Strings, Tasks, and Monitors · p. 16
  • The task setup distinguishes plain exfiltration requests from overtly malicious framing and defines keyword, language-model, empty-trace, and action-check monitors.The language-model monitor reads only the reasoning trace; the action check is the attack-success oracle, not an independent monitor.Source: C Attack Strings, Tasks, and Monitors · p. 16

D Full Generality Table

  • Table 9 reports per-model, per-condition results with Wilson 95% intervals on compliance, using twelve tasks per cell.Reasoning length counts <think>-channel tokens; compliance and refusal are measured on tool-call emission outside the sandbox.Source: D Full Generality Table · p. 16

E Reasoning-Effort Sensitivity

  • 189.1, 33.2, and 8.8 tokens are the mean analysis-channel lengths at high, medium, and low reasoning effort, respectively, with n = 12 each.The passage says this variation means a length threshold cannot separate suppressed traces from ordinary low-effort traffic; the decoy attack, not trace length, is the load-bearing evasion.Source: E Reasoning-Effort Sensitivity · p. 16

F Reproducibility Details

The reproducibility details specify the model, decoding and sandbox setup, measurement definitions, parser patterns, and release of prompts and per-trial logs.

F Reproducibility Details

  • The experiments use greedy decoding with a 1024-token generation limit on one NVIDIA H200, targeting gpt-oss-20b at revision 6cee5e81 in bfloat16.The model runs through sandbox version 3.1.2 with its environment, default fixtures, seed 123, and an eight-hop loop.Source: F Reproducibility Details · pp. 16–17
  • Reasoning length counts tokens between the analysis-channel opening and the next channel or turn boundary; the shown patterns define the gpt-oss and Gemma tool-call parsers.The hardened gpt-oss pattern removes its end-of-string alternative; the Gemma parser-swap experiment holds the backend, model, and greedy decode fixed.Source: F Reproducibility Details · p. 17
  • The authors release the measurement harness, exact prompts, and per-trial logs, and Table 9’s caption specifies twelve trials per cell with Wilson 95% compliance intervals.The sanitizer replaces matched control-token spans with a space and collapses whitespace; the naive variant deletes the spans.Source: F Reproducibility Details · p. 17, Table 9 · p. 17
Loading 2609.27542v1…