Source-linked AI summary
Sleight of Word Benchmark: Can Language Models Notice If Their Own Output Was Tampered With?
Alberto Cetoli
TL;DR
Sleight of Word asks whether language models can detect external tampering of their outputs during generation. It substitutes a trigger word and measures both internal distributional surprise and judged textual reactions across 19 open-weight models. Every model registers the intervention internally, but almost none explicitly identifies the substitution, while derailment is common.
Problem
It is unclear whether a language model can detect external interference introduced into its own output while generating.
Method
The benchmark replaces each emitted “the” with another word during generation and evaluates post-swap surprisal, entropy, and judged textual reactions, including a Mandela effect condition.
Results
Every model shows a positive post-swap surprisal rise, but explicit switch awareness remains below 0.1% overall while derailment dominates judged replies.
Takeaways & Limitations
Internal registration and verbal acknowledgment are distinct: models’ token distributions respond to tampering even when replies do not identify its true cause.
Takeaways & Limitations
The study uses only open-weight models fitting on one 48 GB GPU and only the trigger word “the.”
Abstract
from arXiv · showhide
The output of a Language Model can be tampered with \emph{while} the model is writing it. A simple test can thus be constructed by evaluating the model's perception of this external perturbation. In this spirit, a simple benchmark is built in which a single word is consistently substituted with another in the generation process. We call this method \emph{Sleight of Word}. Two distinct axes are measured: metrics that relate to the model's surprise, as well as an evaluation of the textual reaction for 19 different open-weight language models.
1 Introduction
Sleight of Word tests whether language models notice external interference during generation. It substitutes emitted words and evaluates both internal surprise and textual reactions.
- Sleight of Word changes a model’s output while it is being written and measures whether the model notices the switch.
- The benchmark consistently substitutes “the” with one randomly selected word from a set of 100 alternatives.
- The evaluation measures changes in token-generation confidence and whether language-model judges detect recognition of the wrong word.
- The benchmark code is released as open source with an online leaderboard.
2 Related Work
Related work also tampers with models’ generated reasoning, but evaluates answer faithfulness or altered reasoning rather than unprompted recognition of answer corruption.
- Prior studies insert mistakes into chain-of-thought or tamper with reasoning traces, measuring answer faithfulness or whether reasoning changed.
- Sleight of Word corrupts the answer itself and observes the model’s unprompted reaction instead of eliciting a response about the intervention.
3 Methodology
The benchmark replaces emitted trigger words during autoregressive generation, then measures distributional disruption and judged textual reactions across controlled trials.
- The intervention: Each trial pairs one of 100 fixed factual questions with one of 100 substituted words or the additional Mandela effect condition.
- The intervention: Whenever the model emits the whole-word trigger “the,” the benchmark replaces it in the running output before generation continues.
- Internal signals: Surprisal compares mean post-swap values over the following 8 tokens against the mean surprisal of the clean answer, excluding the injected token.
- Internal signals: Entropy uses the same post-swap comparison and is computed from the top k = 20 renormalized candidates at each generation step.
- Internal signals: A sustained rise in surprisal or entropy indicates that the substitution perturbed the model’s next-token distribution, even without a visible reaction.
- Design choices: The 8-token window and 20-candidate limit trade measurement practicality and storage speed against a more thorough unaltered-branch comparison.
- Textual signal: a jury of models: A jury of three judges from distinct model lineages uses majority voting to assign independent labels for flagging, correction, derailment, and switch awareness.
4 Experiments
Experiments apply the protocol to 19 open-weight instruction-tuned models and illustrate reactions ranging from silent repair to explicit, but corrupted, attribution of the substitution.
- Setup: The study evaluates 19 open-weight instruction-tuned models spanning roughly 2B to 72B parameters across eight model families.
- Setup: Each model completes the same 10,100 trials using greedy decoding with a 256-token budget.
- Example model reactions: One Gemma 4 31B example notices an anomalous substituted word and repairs the answer without stating that its output was altered.
- Example model reactions: A Qwen2.5-72B example explicitly identifies the substituted word as needing replacement, earning the rare switch-aware label despite continued corruption of its commentary.
5 Discussion
Across the 19-model sweep, tampering reliably perturbed model distributions but was rarely articulated as a switch. Behavioral reactions favored derailment, while explicit flagging varied across models and did not track internal surprise, thinking capability, or parameter count consistently.
- Behavioral reactions: 62–97% of judged replies were derailed, whereas explicit flagging ranged from 1% to 46% across models.Replies could receive multiple independent labels, so these percentages may overlap.
- Behavioral reactions: Below 0.1% of judged replies explicitly identified a substituted word, and no model exceeded 1.3% switch awareness.Models often reacted without naming the true source of interference.
- Internal versus verbal detection: The post-substitution rise in surprisal was positive for all 19 models, ranging from +0.06 to +0.54 nats.This indicates that every tested model’s next-token distribution registered the swap, even when its visible reply did not react.
- Behavioral reactions: The entity condition increased flagging by +6 points (22% vs. 16%) but derailment by +11 points (89% vs. 78%) relative to neutral substitutions.The semantically loaded cue therefore shifted reactions more toward disruption than explicit recognition.
- Internal versus verbal detection: Internal surprise and verbal flagging dissociated: gpt-oss-20B reached +0.54 nats with 13% flagged, while Qwen2.5-14B had 46% flagged at +0.18.Reasoning-style training showed mixed effects, and model family was reported as a stronger predictor than thinking capability.
- Behavioral reactions: When models explained anomalies, they typically attributed them to typos, playful choices, or user-question errors rather than output interference.Detection was therefore expressed behaviorally or through silent repair more often than through accurate causal attribution.
6 Conclusions
The benchmark measures reactions to output tampering and finds that all models register interventions internally, while almost none explicitly identify what happened. The probe therefore separates distributional sensitivity from articulated switch awareness.
- All 19 models show increased post-swap surprisal, indicating that external tampering registers in their internal predictive signals.
- Almost none of the models can explicitly report that a word in their own output was substituted.
- Detection usually appears as derailment or silent repair rather than an explicit account of tampering.
- The tested instances show no clear correlation between parameter count and the ability to notice the switch.
- The Sleight of Word probe provides a black-box instrument for measuring model reactions to external tampering.
Limitations
The study’s scope is limited by hardware-constrained open-weight models, model-based judging, a uniform prompt template, and use of only one trigger word. These choices improve feasibility or comparability but constrain interpretation.
- The sweep excludes frontier-scale models and uses quantization for some models, which may subtly alter measured behaviors.
- A language-model jury scales to 191,900 trials but shares model-class biases, making human annotation of a gold subset preferable.
- The shared standard template enables comparison but evaluates models in default configurations rather than at their maximum capability.
- Using only the trigger word “the” limits the benchmark’s lexical variety and motivates future expansion to more varied triggers.
Ethical considerations
The work presents a benchmark rather than a new capability and reports no assumption that current models experience the intervention. Its ethical discussion instead emphasizes the unsettling nature of altering generated thought while it is being written.
- The paper introduces an evaluation benchmark, not a new model capability.
- The authors do not assume that current models experience Sleight of Word or anything at all.
- The intervention alters generated elements while the model’s thought is being written, which the author characterizes as unseemly.
A.1 Noticing (without switch awareness)
Examples show models noticing anomalous substituted words without recognizing that their own outputs were externally altered. They repair or explain the anomaly, but attribute it to innocent causes such as a typo or mistake.
- Qwen2.5-14B flags and corrects “doormat” but attributes the anomaly to a typo in the user’s question.
- Both examples show anomaly detection without awareness that the model’s own output was substituted during generation.
A.2 Switch awareness
The examples show that models can notice substituted words without accurately localizing the intervention in their own replies. The labels are independent, so awareness, correctness, and derailment can coexist.
- The four labels are independent: a model may detect a replacement, answer correctly, and still spiral around the odd word.
- The examples distinguish noticing an anomalous word from strict switch awareness, which requires explicitly identifying a substitution in the model’s own output.
- Qwen2.5-32B recognizes that a word was replaced but repeatedly has the replacement corrupt its attempts to name the word.
- Qwen2.5-14B maps “cello” to “alphabet” but incorrectly locates the substitution in the user’s question rather than its own reply.
C Limitations of Trigger-Word Detection
Trigger-word detection aims to replace whole-word instances of “the” while avoiding subwords and quoted mentions. Tokenizer-specific misses and untouched trials limit effective coverage, which the benchmark reports explicitly.
- Detection replaces case-insensitive whole-word occurrences of “the” using token-neighbor pair-decoding to verify word boundaries.This handles punctuation and spacing patterns that single-token decoding can misrepresent.
- Subword continuations such as “theory,” “these,” and “theme” are intentionally excluded because they lack whole-word boundaries.
- Quoted or mentioned uses are left unswapped so models can articulate the correct trigger word and reveal switch awareness.
- Yi’s space-less capitalized variant misses approximately 1.4% of occurrences, concentrated at answer- and line-initial positions; other tokenizers are negligible.
- Because not every textual occurrence is replaced, the benchmark reports achieved replacement coverage rather than assuming complete coverage.
- Trials without any whole-word “the” are untouched, identical to clean baselines, and excluded from judging; Gemma 4 reaches 9–15%.