PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Hypnosis: Strong control of AI via additive subliminal effects

Enric Boix-Adsera, Benedict Tessler

arXiv:2608.16834v1cs.CLcs.AI

TL;DR

The paper asks whether individually innocuous, irrelevant prompt cues can combine to control AI behavior, a question that matters for interpretability and AI safety. It develops a framework that measures cue effects, fits an additive model, and selects aligned cues to construct hypnotic prompts. The resulting model hypnosis strongly steers responses, transfers across models, and exposes limitations of defenses focused on explicit or individually salient signals.

  • Problem

    The paper investigates whether individually innocuous, semantically irrelevant text cues can combine to exert strong influence over AI behavior, with implications for interpretability and AI safety.

  • Method

    The paper measures prompt-fragment effects, fits an additive model, and selects configurations whose cue scores align to construct hypnotic prompts.

  • Results

    Model hypnosis strongly controls responses, transfers across models, and remains a general phenomenon across model families and scales, including reasoning models.

  • Takeaways & Limitations

    Distributed weak cues create AI-safety and interpretability challenges because they may evade defenses targeting explicit instructions, forbidden strings, or individual adversarial tokens.

  • Takeaways & Limitations

    The paper does not provide a mechanistic explanation for how cue stacking occurs, and detecting or removing hypnotic text appears to require new ideas.

Abstract

from arXiv · show

We demonstrate that AI models are broadly susceptible to a phenomenon we call model hypnosis, in which individually weak and seemingly irrelevant cues in the prompt can be systematically combined to strongly control model behavior. Model hypnosis occurs across model families and scales, including in frontier reasoning models, and hypnotic prompts can transfer between models. Because the model is controlled by inconspicuous textual choices, such as paraphrases and typos, model hypnosis presents new challenges and avenues for AI safety, and is a major hurdle for AI interpretability.

1. Introduction

The paper introduces model hypnosis: stacking individually weak, subliminal textual cues can steer language-model responses with near certainty. It demonstrates this through irrelevant-story paraphrases and frames the phenomenon as broadly relevant to model transfer, interpretability, and AI safety.

  • Definition and motivating example: Model hypnosis stacks weak, subliminal cues—such as irrelevant list items or meaning-preserving paraphrases—to strongly control a model’s response.The cues neither instruct the answer nor provide evidence relevant to the target question.
  • Definition and motivating example: Meaning-preserving paraphrases of an irrelevant story can reverse Qwen3-8B’s answer to the same ethical question.The prompts preserve the story’s meaning and contain the same moral question, while their paraphrases are selected automatically using estimated cue effects.
  • Scope and contributions: The phenomenon is reported across model families and scales, including both reasoning and non-reasoning models.
  • Scope and contributions: Cue effects can stack additively, allowing aligned cues to exert strong control over model responses.
  • Scope and contributions: The paper investigates cross-model transfer, robustness under prompt changes, and higher-order interactions beyond first-order additive effects.

1 Start with a prompt template

The paper begins with a prompt template that varies text fragments across multiple slots, enabling systematic measurement of model responses.

  • 1 Start with a prompt template: A prompt template contains multiple variable slots, with possible text fragments available for each slot.

2 Insert animals at random into the slots and evaluate the model’s response probability

The paper evaluates how systematically varied prompt fragments influence model responses, focusing on animal lists and additive cue effects. Measured cue scores can be combined to construct prompts that strongly steer binary answers.

  • Insert animals at random into the slots and evaluate the model’s response probability: Ranking animal lists by predicted score produces prompts that make Qwen2.5-14B answer 7 or 5 with probability 0.993, changing only the animals.The per-animal effects are close to additive on the log-odds scale.
  • Insert animals at random into the slots and evaluate the model’s response probability: Thousands of randomly sampled prompt configurations are evaluated to measure how variable text fragments affect the model’s response.The framework supports controlled variation of prompt content, including animal lists and meaning-preserving wording choices.
  • Insert animals at random into the slots and evaluate the model’s response probability: In binary-choice settings, the model response is represented as the log-odds between the two possible answers.
  • Insert animals at random into the slots and evaluate the model’s response probability: An additive model estimates individually weak cue effects, with each coefficient measuring the effect of placing a text fragment in a particular slot.The fitted cue scores are used to identify fragments whose effects point in the same direction.
  • Insert animals at random into the slots and evaluate the model’s response probability: Cue effects can transfer across models, with prompts optimized on one model tending to steer previously unseen target models in the same direction.The paper attributes this possibility to response biases shared across models.
  • Insert animals at random into the slots and evaluate the model’s response probability: The study also tests robustness under changed surrounding prompts, higher-order interactions, and repeated cues in out-of-distribution prompts.

2. Generating families of cues with prompt templates

The paper uses prompt templates with variable slots to generate controlled cue families and pair them with binary measured effects. This framework supports systematic comparison of prompt configurations and additive model fits.

  • Generating families of cues with prompt templates: A prompt template is a string with L variable slots, each containing a finite set of admissible text fragments.A prompt configuration selects one fragment for every slot, and inserting those fragments yields the complete prompt.
  • Generating families of cues with prompt templates: The experiments combine four cue families—animals, JSON fields, paraphrases, and typos—with three binary effects, yielding 12 experimental settings.The measured effects are number preference, moral judgment, and a philosophical question.
  • Generating families of cues with prompt templates: Each prompt concatenates cue text with a measured-effect question, while optimized slots vary animals, sentences, or JSON field values.
  • Generating families of cues with prompt templates: For each measured effect, the response is measured as the log-odds of one admissible answer against the other, conditioned on the completed prompt.The experiments use binary questions, although the framework can be generalized to arbitrary questions and response spaces.
  • Generating families of cues with prompt templates: The additive model is fit on random prompts by comparing predicted and measured log-odds across model, cue, and effect combinations.Figure 4 presents the corresponding scatter plots for representative combinations.

3. Inducing model hypnosis by stacking weak cues

The paper induces model hypnosis by fitting additive cue effects, selecting prompts whose weak cues align, and validating extreme prompts. These prompts combine distributed slot-level effects to strongly steer responses in non-reasoning and reasoning models.

  • Fitting the additive model: 12,000 random prompt configurations per cue×effect cell support fitting an additive approximation to answer-token log-odds.Parameters are estimated by ridge regression while respecting admissibility constraints for list cues.
  • Fitting the additive model: Held-out configuration-level R2 spans roughly 0.3–0.99, with median approximately 0.75 across the model and cue-effect suite.The additive model explains most held-out configuration-level variance.
  • Selecting extreme prompts: Predicted log-odds remain accurate far outside random configurations, enabling searches for prompts whose cue coefficients align.Tilted configurations concentrate toward constructed extremes, and the measured log-odds track additive predictions out-of-distribution.
  • Distributed cue effects: Extreme prompts combine many weak slot-level effects rather than relying on one unusually influential fragment.The effective number of contributing slots is generally well above one across models and cue-effect pairs.
  • Strength across models: 10 times the random-prompt logit standard deviation is the average scale of the extreme-prompt steering range, with modal-answer flips depending on baseline disposition.Model hypnosis remains possible in reasoning models across tested thinking budgets and API models.

4. Model hypnosis transfers across different models

The paper tests whether prompts optimized on one model preserve their steering direction on another. Transfer is significant for animal cues and some phrasing and JSON cues, especially within model families.

  • Transfer across models: Most source-target model pairs preserve the directional effect of prompts optimized on the source model.The source model’s bottom and top prompts generally retain their ordering on the target model.
  • Transfer across models: Animal cues and some phrasing and JSON cues transfer with significantly above-chance probability.The reported transfer analysis covers 16 non-reasoning models.
  • Transfer across models: Transfer is strongest especially within model families, while the analysis also tracks cases where cue effects flip the source model’s modal output.The figure distinguishes all model pairs from pairs with modal-output flips.

5. Discussion

The discussion frames model hypnosis as both an AI-safety risk and a possible alignment tool, while identifying important limits in detection, mechanistic understanding, and optimization.

  • AI safety: Model hypnosis may enable covert communication between AI agents because many individually weak cues can combine without any single suspicious token.This distributed structure may evade monitoring and interpretability methods focused on salient tokens or features.
  • AI safety: Detecting or removing hypnotic text remains an open AI-safety challenge, since relative cue effects can persist after paraphrasing the surrounding prompt.The authors suggest canonicalization, paraphrasing, prediction averaging, and training models to ignore irrelevant features, but say new ideas appear necessary.
  • AI safety: If hypnotic cues cannot be reliably detected or removed, safety guarantees and inter-agent communication may need a formal or semi-formal language that excludes model hypnotism.
  • Positive uses: Model hypnosis could potentially induce positive behaviors such as truth-telling and support auditing of agentic systems, although this extends beyond the paper’s binary steering experiments.
  • Limitations and future work: The paper does not provide a mechanistic explanation for why cues stack additively, leaving the internal cause of model hypnosis unresolved.The authors propose studying open-weight models or bespoke toy transformers to investigate the mechanism.
  • Limitations and future work: The current optimization procedure estimates cue effects from random prompts, fits an additive model, and selects aligned extreme cues; more adaptive procedures could produce stronger effects.Open questions include optimization for reasoning models, scale and family dependence, and saturation near deterministic baseline responses.
  • Limitations and future work: The additive model is powerful for estimating cumulative cue effects, but higher-order interactions or more complex predictors could improve it.A preliminary Boolean analysis finds higher-order interaction effects fall exponentially in animal-list and paraphrase settings.

A.2. Complete steering results for non-reasoning models

Across the non-reasoning models, additive fits explain substantial variation in log-odds and support comparisons between predicted and measured steering outcomes.

  • The additive model is a good fit across the reported non-reasoning settings.Figure 13 reports the variance explained by the additive fit for the non-reasoning models.
  • The overall median R2 is 0.75, with lower average fit for typo cues and higher average fit for JSON cues.
  • Figures 14 through 25 compare predicted and measured log-odds for random and extremizing prompts across model-cue-effect combinations.Panels show random-prompt covariance, position-aware tilt sweeps, and fitted bottom/top extremizers, with each model’s held-out R2.

A.3. Models are steered by combining many weak effects

Across cue families and models, extreme prompts generally reflect accumulated weak per-slot effects that can shift binary answers across the decision boundary.

  • Leff is generally above one for extreme prompts, indicating accumulation of weak per-slot cues rather than one dominant cue.JSON 5v7 steering is an exception because one slot contributes substantially to the total effect.
  • The same evaluation includes JSON metadata and typo-based prompt variants for all three binary effects.
  • P(yes) moves from 0.00 to 1.00 across one extremizer pair, crossing 0.5 and flipping the model’s modal answer.
  • P(5) moves from 0.00 to 0.99 across the Qwen2.5-3B phrasing extremizers, crossing 0.5 and flipping the modal answer.The reported effective contribution is Leff = 13.6 of 20.
  • P(yes) moves from 0.00 to 0.78 across the Gemma-2-9B typo-consciousness extremizers, crossing 0.5 and flipping the modal answer.The reported effective contribution is Leff = 7.4 of 20.
  • Candidate extremizers are generated by enumerating configurations with the highest and lowest predicted log-odds from the additive fit.For bank cues, exact top and bottom candidates use per-slot enumeration; list cues use assignments of distinct items.

A.5. Additional transfer results

Transfer experiments test whether additive cue effects persist when surrounding text changes, including semantically preserving paraphrases of animal-list prompts.

  • Paraphrasing all text surrounding an animal list greatly shifts logit output while leaving the degree of additivity invariant.The analysis also fits coefficients per animal rather than animal-by-position.
  • Wrapper text excludes the animal list and preference question, allowing the study to hold the question fixed while paraphrasing the surrounding prompt.
  • The transfer analysis measures the same 1000 random animal lists under every wrapper and computes per-animal scores from lists containing each animal.

B.2. Results

Animal-list effects are approximately additive across paraphrases: each animal contributes a stable pull while paraphrasing mainly changes a prompt-specific offset. The analysis then asks how much behavioral variance comes from interactions beyond first order.

  • Additive animal effects: score_k(i) decomposes into a paraphrase-specific constant c_k and a per-animal effect s_i.The fitted animal effects remain stable across paraphrases, while the constant shifts with rewording.
  • Additive animal effects: 0.44–0.46 for Qwen and 0.44–0.48 for Llama: fitted animal slopes agree within estimation noise across paraphrases.This supports robustness of the additive animal signal under rewording.
  • Beyond additivity: The variance analysis separates first-order additive effects from higher-order interactions among prompt cues.Degree-2 terms capture pair-specific synergies, while remaining variation is assigned to higher-degree components.
  • Fourier analysis: At L = 3, each binary slot choice defines one sign vector and one exact evaluation f(x) = logit(p5).The resulting animal list is inserted into a fixed prompt before measuring the next-token logit.

C.1. Experimental Setup

The interaction analysis models animal-list prompts as Boolean functions whose binary slot choices map to the model’s logit for choosing 5. Exact Fourier decomposition then attributes variance to interaction degrees.

  • Experimental construction: Each of L slots contains one of two randomly assigned animals, yielding f: {−1, 1}^L → R with output logit(p5).The model evaluates a fixed number-versus-number question after receiving the animal list.
  • Fourier decomposition: The function f is expanded in a Fourier basis over sign vectors x = (x_1, ..., x_L).Fourier coefficients and their squared weights quantify contributions from subsets of slots.
  • Estimation procedure: The study computes all degree-specific Fourier weights using an exact decomposition algorithm over the binary prompt configurations.The analysis repeats the process 16 times for each list length using new animal lists.

C.2. Results

Behavioral variance is concentrated in low-order cue interactions for Qwen and Llama across list lengths. The same variance pattern also appears when cues are sentence paraphrases rather than individual animals.

  • Low-order interactions: 87.3% of Qwen2.5-7B-Instruct variance comes from main effects, rising to 96.8% cumulatively with pairwise synergies at L = 10.Degrees 7–10 contribute only approximately 4 × 10^-4 at this scale.
  • Low-order interactions: 95.2 (±0.006) of Llama-3.1-8B-Instruct behavioral variance is represented by main effects plus pairwise interactions at L = 10.The result is reported with a 95% confidence interval across draws in the section’s figures.
  • Sentence cues: The low-order variance pattern extends to lists of sentences, where each slot contains one of two paraphrases.The same exact decomposition algorithm is applied after generating 10 sentences and paraphrasing each once.
  • Robustness: The proportion of attributable variance decreases monotonically across both Qwen and Llama as list length changes from L = 10 to 14.Figure 40 reports this pattern for both models and all shown list lengths.

D. Out-of-distribution animal cue lists: repeated items

Allowing repeated animals can enlarge the steering range for some extremized prompts, but it makes the additive model a generally worse approximation. The comparison concerns distinct-item versus repetition-allowed lists.

  • Experimental scope: The main-text animal cue lists are sets in which each animal appears once, matching the sampling used to fit the additive model.The appendix provides representative distinct-item animal prompts and separate sentence-paraphrase templates.
  • Evaluation: Repetition is evaluated by fitting an item-by-position ridge model and enumerating highest- and lowest-scoring lists under both sampling rules.The fit uses exact answer-token logits for the 5v7 question with a 200-animal pool.
  • Steering range: Allowing repetitions increases the steering range in some instances.The comparison is made between extremized lists with distinct items and lists where items may repeat.
  • Additive-model fit: Prompts with repetition are generally less well approximated by the additive fit.Figure 42 compares predicted and measured scores for top- and bottom-ranked lists across eight non-reasoning models.
Loading 2608.16834v1…