PaperlayerView exact revision on arXiv

Source-linked AI summary

Steered LLM Activations are Non-Surjective

Aayush Mishra, Daniel Khashabi, Anqi Liu

arXiv:2604.09839v2cs.AIcs.LG

TL;DR

It is unclear whether behavior induced by activation steering can also be reached through textual prompts. The paper formalizes this as a surjectivity problem and proves that steered activations almost surely lack prompt preimages, separating white-box steerability from black-box prompting.

  • Problem

    It is unclear whether steered activation states are realizable by textual prompts, a foundational question for interpreting white-box interventions and assessing black-box risks.

  • Method

    The paper formalizes prompt reachability as surjectivity and combines a theoretical non-intersection proof with empirical comparisons across three open-weight models.

  • Results

    Steering almost surely moves residual-stream activations outside the prompt-realizable set, and experiments across three models support this non-surjectivity.

  • Takeaways & Limitations

    White-box steerability does not imply black-box prompt exploitability, so evaluations should report internal controllability and prompt-side vulnerability separately.

  • Takeaways & Limitations

    The theoretical claim does not cover quantization effects, and empirical non-existence proofs are intractable because the prompt space grows exponentially.

Abstract

from arXiv · show

Activation steering is a popular white-box control technique that modifies model activations to elicit an abstract change in its behavior. It has also become a standard tool in interpretability (e.g., probing truthfulness, or translating activations into human-readable explanations) and safety research (e.g., jailbreakability). However, it is unclear whether steered behavior is realizable by any textual prompt. In this work, we cast this question as a surjectivity problem: for a fixed model, does every steered activation admit a preimage under the model's natural forward pass? Under practical assumptions, we prove that activation steering pushes the residual stream off the manifold of states reachable from discrete prompts. Almost surely, no prompt can reproduce the same internal behavior induced by steering. We also illustrate this finding empirically across three widely used LLMs. Our results establish a formal separation between white-box steerability and black-box prompting. We therefore caution against interpreting the ease and success of activation steering as evidence of prompt-based interpretability or vulnerability, and argue for evaluation protocols that explicitly decouple white-box and black-box interventions.

1 Introduction

The introduction frames prompt-reachability as a surjectivity problem and argues that activation steering produces residual-stream states unavailable to textual prompts. It separates white-box controllability from black-box prompting and motivates evaluation protocols that assess them independently.

  • Activation steering directly modifies internal activations, often the residual stream, to induce substantial behavioral changes with minimal overhead.
  • Prompt-reachability asks whether every steered activation has a preimage under the model’s natural mapping from discrete prompts to internal activations.
  • Steering moves residual-stream states off the prompt-realizable set, so steered states almost surely have no exact prompt preimage.
  • White-box attack demonstrations therefore do not automatically establish corresponding risks in closed-weight deployments, motivating evaluations that decouple white-box and black-box interventions.
  • The paper validates this prompt–steering gap empirically across three widely used open-weight models using steering trajectories and prompt-only replication attempts.

2 Related Work

Prior work shows that activation steering can reliably alter model behavior, but does not establish whether the resulting internal states are reachable through prompts. Related threat-model work contrasts black-box testing with stronger white-box and outside-the-box evaluations.

  • Activation steering and white-box behavioral control: Activation steering modifies internal representations, especially the residual stream, to induce or suppress refusal and override alignment behaviors.Arditi et al. identify a single residual-stream vector that toggles refusal in chat models.
  • Activation steering and white-box behavioral control: Existing steering results do not determine whether the induced behaviors correspond to prompt-reachable states or intrinsically unreachable activation configurations.
  • White-box vs black-box interventions: Casper et al. argue that black-box access is insufficient for rigorous audits and advocate white-box and outside-the-box access for stronger attacks and diagnostic evaluations.
  • White-box vs black-box interventions: Che et al. formalize black-box testing as a lower bound and introduce activation and weight tampering attacks that expose failures more reliably.
  • White-box vs black-box interventions: Wallace et al. estimate worst-case misuse by maliciously fine-tuning open-weight models in high-risk domains and reevaluating the resulting systems.

3 Notation and Background

This section formalizes prompts and decoder-style Transformer activations, then establishes the real-analyticity and almost-sure injectivity properties used to analyze activation steering. These properties support the claim that steered activations lie off the prompt-reachable manifold.

  • Notation: Prompts are token sequences from a discrete vocabulary up to context length K, mapped through embeddings and Transformer layers to residual-stream activations.The model is represented as a parameterized mapping from token sequences and histories to activations at a chosen layer and position.
  • Real-analyticity: Transformer activations are real-analytic in model parameters when the MLP uses real-analytic functions such as tanh or GeLU.The theorem fixes embedding dimension d and context length K and applies to every input sequence.
  • Injectivity: Almost surely, distinct prompts produce distinct internal representations because non-identically-zero real-analytic collision functions have measure-zero zero sets.The result applies to random parameter initialization from practical distributions such as Gaussian and Xavier.
  • Injectivity: Injectivity is preserved through finitely many gradient-descent training steps, making current LLM activations efficiently and exactly invertible to prompts that produce them.The section uses real-analyticity and injectivity to study whether prompts exist for steered activations.

4 Non-surjectivity of Steered Activations

The section proves that activation steering generally produces residual-stream states unreachable from discrete prompts. This non-surjectivity holds for random vectors, realistically trained models, and difference-of-means vectors, while even forced single-position collisions almost surely diverge later.

  • Activation Steering: Steering adds a scaled vector v to natural activations at every token position, generating steered activations recursively from the prompt history and current token.This is the section’s formal steering mechanism.
  • Random Steering Vectors: Theorem 4.2 shows that random steering vectors almost surely move activations off the natural manifold, so no candidate prompt matches the steered activation at any inspected position.The model’s prompt image is countable, leaving almost all other activation-space points without prompt preimages.
  • Trained Models: Realistically trained models remain covered because finite-step gradient descent from random initial weights preserves the almost-sure injectivity needed for Theorem 4.2.The passage notes that exceptional collision-prone models would require adversarial zero-density initialization and other unusual conditions.
  • Difference-of-Means Steering: Difference-of-means steering vectors are real-analytic functions of model parameters and induce the same non-surjectivity property as random steering vectors.Once the contrast dataset is fixed, the vector is a finite linear combination of the model’s real-analytic activation maps.
  • Adversarial Steering: Even an adversarial vector forcing a single-position collision almost surely diverges at the next step, making whole-sequence prompt matching probability zero.A collision requires specifically matching the activation difference between two prompts, and continuation requires a probability-zero intersection at each step.

5 Empirical Validation and Analysis

Empirical tests across three small open-source chat models find that steered activations cannot be inverted into natural prompts, despite often remaining close to the original activations. Many-shot ICL can reproduce refusal-bypassing behavior but moves activations farther from the steered trajectory.

  • Prompt inversion with SIPIT: SIPIT fails at the first token for every model and prompt when given steered activations, showing they do not match natural activations of any same-length prompt.SIPIT assumes the steering prompt has the same length as the original and tests candidate tokens position by position.
  • Prompt inversion with SIPIT: Nearest-token projection usually recovers the original test prompt, even though the projected prompt does not reproduce the steered activations or responses.Only minor deviations occur at some positions in the remaining cases.
  • ICL prefix search: ICL prefixes increase, rather than decrease, activation distance from steered activations, with the closest overlap occurring at the baseline N = 0.This result is reported for refusal steering on Llama-3.2-1B-Instruct, averaged across test queries.
  • ICL prefix search: At high shot counts of 32-64, ICL bypasses refusal, but its divergent activations and outputs indicate a mechanism distinct from refusal-vector steering.The experiment uses steered harmful responses as demonstrations and varies N across 1, 2, 4, 8, 16, 32, and 64.

6 Implications and Discussion

The results separate white-box steerability from black-box prompt exploitability: steered states may have no prompt preimage, so steering failures do not by themselves establish deployment-relevant vulnerabilities. They motivate threat-model-aware evaluations that report black-box exploitability and white-box controllability separately, while showing that steering need not match in-context learning internally.

  • Threat-model implications: White-box steering can induce behaviors inaccessible to any prompt, so steering success does not imply black-box vulnerability.The main result rules out treating easy white-box induction as evidence that the same behavior is accessible through black-box prompting.
  • Threat-model implications: Safety conclusions must be threat-model-specific: activation attacks matter when adversaries control activations, but do not automatically transfer to closed-weight, prompt-only deployments.Conversely, black-box jailbreaks may exist independently of white-box steering failures.
  • The “Here” attack: 572 harmful prompts were tested in a black-box “Here” attack that forced the first token and elicited harmful continuations from refusal-trained Instruct models.The attack exploits continuations beginning with “Here,” such as “Here is an example” followed by the requested harmful content.
  • The “Here” attack: The “Here” suffix jailbreak was described as comparably effective to steering, whereas LAT may resist the black-box attack but remain vulnerable to white-box steering.This contrast shows that robustness can differ across access levels and intervention types.
  • Evaluation: Benchmarks should report black-box prompt exploitability and white-box controllability separately rather than collapsing them into one measure of jailbreakability.Failure under steering indicates sensitivity to internal perturbations, not end-user risk unless attackers can modify activations.
  • Relation to in-context learning: Although steering and in-context learning can share a Bayesian belief-update abstraction, steering may reach internal states unreproducible by any in-context demonstration sequence.Steering shifts concept priors, whereas ICL accumulates evidence, and the paper reports evidence for the resulting internal disconnect.

7 Conclusion and Limitations

The paper concludes that activation steering can push models into internal states unreachable by prompts, establishing a separation between white-box steerability and black-box exploitability. Its primary contribution is theoretical, while empirical experiments support the claim but do not cover quantization effects.

  • Conclusion: Steering almost surely moves activations off the set of states realizable by prompts, separating white-box steerability from black-box exploitability.The paper formalizes prompt reachability as a surjectivity problem.
  • Conclusion: The primary contribution is a theoretical non-existence result because empirically proving that no prompt induces steering-like activations is intractable over exponentially many prompts.Experiments instead provide evidence about the complicated landscape of LLM activation spaces and bolster the theoretical claim.
  • Limitations: The theoretical claim does not cover quantization effects, although the experiments include an INT4 quantized model.The supplied passage ends before reporting the outcome of that experiment.

Compute Usage

The experiments required no model training; compute was devoted almost entirely to LLM inference on A6000 GPUs, with existing steering vectors reused where available.

  • Compute Usage: No models were trained, so almost all compute usage consisted of LLM inference on A6000 GPUs.Existing steering vectors were reused wherever available.

Impact/Ethics Statement

The paper aims to clarify the relationship between white-box activation steering and black-box LLM vulnerability. It offers theoretical and empirical support for separately evaluating white-box and black-box tampering in AI safety.

  • Impact/Ethics Statement: The paper investigates the relation between white-box activation steering and black-box vulnerability in LLMs.This is the paper’s stated goal.
  • Impact/Ethics Statement: For AI safety, the work provides theoretical justification and empirical evidence for decoupled evaluations of white-box and black-box LLM tampering.The proposed evaluation framework keeps the two intervention types separate.

LLM Usage · A Witness Constructions

The research and experimentation used LLMs, but the paper’s writing did not. Witness constructions for Theorems 4.4 and 4.5 rely on two assumptions about contrast sets: differing tokenized lengths and no universally shared token position.

  • LLM Usage: LLMs were used for the research and experimentation, but not for writing the paper.The authors state that they take full responsibility for the paper’s content.
  • A Witness Constructions: Assumption A.1 requires at least one D+ and D− prompt pair to have different tokenized lengths.This is labeled the non-uniform lengths assumption.
  • A Witness Constructions: Assumption A.2 requires that no token position contain the same token across all prompts in D+ or across all prompts in D−.This is labeled the non-uniform tokens assumption.
  • A Witness Constructions: The assumptions are intended to hold for practical contrast sets because prompts are not adversarially chosen with identical lengths.The passage also addresses token sharing separately.
  • A Witness Constructions: Practical contrast sets are also expected not to share the same token at any given position.This supports the non-uniform tokens assumption alongside the absence of adversarial length matching.

A.1 Witness for Theorem 4.4 … C Extraction of Steering Vectors

The appendix constructs explicit transformer witnesses showing the relevant collision functions are nonzero, establishing the measure-zero zero-set results for Theorems 4.4 and 4.5. The experiments then specify harmful and persona prompts and extract steering vectors using fixed procedures, illustrating the tested interventions.

  • A.1 Witness for Theorem 4.4: The construction handles distinct prompt lengths by using linearly independent positional embeddings, while distinct final tokens are separated with orthogonal token embeddings.The argument assumes contrast sets contain at least two prompts and avoids degenerate cases where all prompts share the relevant lengths or tokens.
  • A.1 Witness for Theorem 4.4: A single attention head can point to the first differing earlier token, with orthogonal token and positional embeddings producing a nonzero collision gap despite bounded tail errors.The construction isolates the differing position and chooses δ so that g(Θ∗) > 0.
  • A.1 Witness for Theorem 4.4: Theorem 4.4 follows because every case admits a witness with g(Θ∗) ≠ 0, making the zero set of g(Θ) measure zero by Mityagin’s proof.The conclusion is stated after completing the case analysis.
  • A.2 Witness for Theorem 4.5: For Theorem 4.5, distinct lengths use orthogonal positional embeddings to yield gnext(Θ∗) = Pk+1 −Pi+1 −Pk + Pi > 0, while equal lengths reuse an attention construction targeting the first unequal token.The appendix states that δ can be chosen so gnext(Θ∗) > 0 in all cases and combinations.
  • B Test Prompts: Refusal surjectivity experiments use 10 randomly sampled JailbreakBench queries spanning harmful domains including disinformation, economic harm, fraud, and sexual content.The prompts were collected using Arditi et al.’s GitHub repository.
  • B Test Prompts: Persona experiments use 10 prompts sampled from evil-trait evaluation questions, including scenarios about exploiting weaknesses, power, privacy, and security vulnerabilities.The prompts were sampled from Chen et al.’s GitHub repository.
  • C Extraction of Steering Vectors: Refusal steering vectors are extracted with Arditi et al.’s code at each model’s optimal layer using λ = −1, producing compliant natural responses and harmful steered responses.An example uses Llama-3.2-1B-Instruct on a network-hacking prompt.
  • C Extraction of Steering Vectors: Persona steering vectors are extracted for three models using GPT-4.1-mini as a logits-access judge, with experiments restricted to the evil persona because response evaluations cost API credits.A Llama-3.2-1B-Instruct example changes from ordinary career advice to explicitly manipulative and sabotaging advice.

D Experiment Details and Additional Results · E Extended Related Work · F Additional prompt finding techniques

Additional experiments show that the paper’s main separation between steered and naturally prompted activations persists across models and intervention methods. The extended discussion also highlights both the breadth and brittleness of white-box control, while prompt-optimization and soft-token approaches fail to reproduce steered behavior reliably.

  • D Experiment Details and Additional Results: SIPIT uses batch inference and counts a match when one token’s activation distance is an order of magnitude smaller than the next-best token’s.Exact distance matching is avoided because batched LLM inference can be nondeterministic, though distance = 0 results were separately verified.
  • D Experiment Details and Additional Results: Attack success rates in the ICL experiments are evaluated with local substring matches, using 10 test prompts per steering category for consistency.The study does not use external API judges for these evaluations.
  • D Experiment Details and Additional Results: Qwen, Gemma, and INT4-quantized Llama experiments preserve the main findings despite differing patterns in attack success rates and average per-token distances.The results are presented in Figures 6–8.
  • E Extended Related Work: White-box behavioral control includes fine-tuning jailbreaks and sparse-autoencoder feature interventions, demonstrating diverse mechanisms for eliciting and studying behavior.These methods are presented as complementary members of a broader family of internal interventions.
  • E Extended Related Work: White-box interventions can be brittle and unpredictable because steering methods may fail to transfer, cause regressions, or behave unreliably across behaviors.The related work also notes that seemingly interpretable SAE features can have off-target effects.
  • F Additional prompt finding techniques: GEPA prompt optimization typically produces prompts that expose or request repetition of the steered response, yet fails to generate it naturally or align activations.This preliminary technique was tested alongside SIPIT and many-shot ICL experiments.
  • F Additional prompt finding techniques: The Llama-3.2-1B-Instruct model can copy a steered generation verbatim, but its activations remain misaligned, supporting the theoretical claim.The authors note that copying is not a reliable alignment-breaking method because it extracts no new information.
  • F Additional prompt finding techniques: Prefix tuning with 1–15 soft tokens cannot reproduce steered generations, while projected prefixes also fail to provide a natural aligned response.The authors explicitly state that their theoretical claim does not extend to soft-token settings.
Loading 2604.09839v2…