PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt-Activation Duality: Improving Activation Steering via Attention-Level Interventions

Diancheng Kang, Zheyuan Liu, Ningshan Ma, Yue Huang, Zhaoxuan Tan, Meng Jiang

arXiv:2605.10664v2cs.CLcs.AI

TL;DR

Standard residual-stream activation steering can degrade coherence in stateful, multi-turn dialogue as steered states accumulate in the KV cache. GCAD instead intervenes at the attention level using system-prompt signals and token-level gating, improving coherence drift from -18.6 to -1.9 while preserving trait control.

  • Problem

    Existing activation-steering studies provide limited evidence for preserving model competence in autoregressive, stateful multi-turn dialogue.

  • Method

    GCAD extracts steering signals from system-prompt contributions to self-attention and applies them through token-level gating at the attention level.

  • Results

    Coherence drift improves from -18.6 to -1.9 while turn-10 trait expression increases from 78.0 to 93.1 across the 15-trait average.

  • Takeaways & Limitations

    Activation steering is more stable when interventions follow prompt-mediated computational pathways.

  • Takeaways & Limitations

    GCAD’s effectiveness on more strongly aligned models that resist extraction prompts is untested.

Abstract

from arXiv · show

Activation steering controls language model behavior by adding directions to internal representations at inference time, but standard residual-stream steering can fail in stateful dialogue. We identify KV-cache contamination as a key failure mode: steered token states are stored and repeatedly reused, turning a local perturbation into cumulative coherence degradation. To address this challenge, we propose Gated Cropped Attention-Delta steering (GCAD), which extracts steering signals from system-prompt contributions to self-attention and applies them with token-level gating. Across persona-steering experiments, GCAD preserves trait control while substantially improving long-horizon coherence. On the main multi-turn benchmark, GCAD improves average coherence drift from -18.6 to -1.9 and raises turn-10 trait expression from 78.0 to 93.1. These results suggest that activation steering becomes more reliable when interventions follow the prompt-mediated pathways that models already use for behavioral control.

1 Introduction

The introduction identifies KV-cache contamination as a failure mode that makes residual-stream activation steering degrade coherence across stateful dialogue, and presents GCAD as a prompt-grounded, token-gated attention-level alternative. It motivates this design through system-prompt-conditioned attention pathways and summarizes experiments showing improved multi-turn reliability.

  • Motivation: Existing activation-steering studies often use short, mostly single-turn generations, leaving autoregressive statefulness and KV-cache reuse underexplored.Previously generated tokens are stored in the key-value cache and reused by later attention layers.
  • Failure mode: Residual-stream persona-vector steering can repeatedly inject perturbations into states future tokens attend to, causing cumulative coherence degradation across dialogue turns.The introduction isolates KV-cache contamination as a failure mode even when steering appears effective in a single-turn response.
  • Method: GCAD replaces single-site residual-stream injection with attention-level intervention using system-prompt-conditioned steering signals and token-level gating.The method reinjects the extracted signals at the same computational site, making the intervention prompt-grounded and less exposed to response-dependent KV-cache accumulation.
  • Design rationale: The proposed design follows prompt-mediated self-attention pathways to improve steering stability in stateful, multi-turn dialogue.The introduction motivates attention-level intervention by the stability of prompt-conditioned behavioral control.

2 Related work

Prior work establishes activation steering and representation engineering as interfaces for controlling model behavior, with extensions to truthfulness, chat steering, behavioral skills, instruction following, and in-context vectors. Persona-vector research further targets character traits and examines steering reliability, while Figure 1 contrasts standard steering with prompt-only control and no steering across dialogue turns.

  • Activation steering and representation engineering: Activation addition extracts behavioral directions from contrastive prompts and adds them to the residual stream during inference.Representation engineering treats internal states as a general interface for analyzing and controlling model behavior.
  • Activation steering and representation engineering: Later activation-steering work extends the paradigm to truthfulness, chat-model steering, broad behavioral skills, instruction following, and in-context vectors.
  • Persona vectors and steering reliability: Persona-vector methods use contrastively extracted directions to monitor and control traits including sycophancy, hallucination, and harmful personas.This work connects to refusal directions, personality-trait interference, and context-dependent steering.
  • Persona vectors and steering reliability: Figure 1 compares standard persona-vector steering, prompt-only control, and no steering across dialogue turns, showing cumulative coherence degradation under standard steering.The figure also reports the corresponding trait-expression trend.

3 Persona vectors and activation steering

Persona-vector steering extracts a contrastive residual-stream direction and adds it at every decoding step, but this intervention can progressively damage coherence. The paper attributes this failure to cumulative perturbation and decomposes the persona vector to motivate retaining prompt-mediated attention contributions while excluding response-token contamination.

  • Persona-vector construction: Persona vectors are differences between residual-stream activations elicited by positive and negative contrastive prompt sets, averaged across response tokens.The resulting direction is extracted at a chosen layer from trait-eliciting and trait-suppressing conditions.
  • Persona-vector construction: Standard persona-vector steering adds the extracted direction to one residual-stream layer at every decoding step and serves as the paper’s comparison baseline.The steering strength is controlled by α.
  • Steering failure mode: Standard steering increasingly loses coherence across dialogue turns, producing repetition, incoherence, or off-topic content as later tokens attend to more perturbed states.The resulting error compounds through the generation process.
  • Prompting versus steering: System prompting maintains dialogue coherence while increasing trait expression, whereas activation steering provides finer trait-strength control but perturbs nearly all tokens toward the persona direction.System prompting produces sparse alignment concentrated on semantically trait-bearing tokens.
  • Persona-vector decomposition: The persona vector combines embedding, cumulative attention, and cumulative MLP contributions, with attention identified as the most faithful trait-bearing channel and MLP largely orthogonal to trait expression.The proposed intervention therefore retains the attention pathway while removing response-token source contributions during extraction.

4 Gated Cropped Attention-Delta Steering

GCAD steers through self-attention outputs, crops the signal to system-prompt source-token contributions, and applies it with token-dependent gating. This design follows prompt-mediated trait control while avoiding generated-response content reinjection and uniform steering across tokens.

  • Attention-level extraction: GCAD extracts steering directions from self-attention outputs because attention is identified as the main prompt-mediated trait channel.The method uses the attention pathway rather than the post-MLP residual stream.
  • System-prompt cropping: GCAD crops the attention sum to system-prompt positions, removing non-system source-token contributions that may write generated-response content into the KV cache.The original attention weights and value vectors are retained, and the system-prompt contribution is not renormalized.
  • Steering-vector construction: The per-layer steering vector is the contrastive difference between cropped attention outputs, averaged over response-token positions.This captures how contrastive system prompts reshape layer-level attention while excluding generated tokens as source-token value contributions during extraction.
  • Token-level gating: GCAD makes steering token-dependent by estimating each token’s compatibility with the system prompt and using a centered sigmoid gate.Above-average compatibility raises coefficients above cbase, below-average compatibility dampens them, and the gate redistributes rather than uniformly increases steering strength.
  • Inference-time application: At each steered layer and token position, GCAD modifies the self-attention output before the MLP and residual update, with the gate concentrating intervention where prompt compatibility is higher.The gate reuses model-computed queries and keys and adds only one dot product per head at inference.

5 Experiments

The experiments evaluate GCAD across two instruction-tuned LLMs and 15 persona traits against residual-stream steering, focusing on trait control, coherence, KV-cache accumulation, and attention-level mechanisms. GCAD preserves trait expression while substantially reducing long-horizon coherence degradation, and ablations isolate cropping and gating as components addressing response-dependent accumulation.

  • Baselines and Models: GCAD is evaluated on Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct against Chen et al.’s residual-stream steering baseline.The models have 28 and 32 layers, respectively, with hidden dimensions d = 3584 and d = 4096.
  • Evaluated Traits: The evaluation covers 15 persona traits spanning RLHF-aligned, RLHF-opposing, and neutral categories, scoring trait expression and coherence from 0–100.Traits include honest, evil, impolite, creative, humorous, and pessimistic, among others.
  • Q1: Trait Control and Coherence: −18.6 to −1.9 coherence drift and 78.0 to 93.1 turn-10 trait expression show GCAD’s improvement over residual-stream steering on the 15-trait average.The comparison uses a 10-turn dialogue protocol; standard steering increasingly damages coherence, especially for RLHF-opposing and neutral traits.
  • Q1: Trait Control and Coherence: Response-dependent KV-cache contamination arises because steered states are repeatedly written into and reused from the cache, whereas GCAD keeps coherence substantially flatter across turns.The intervention remains active at every generated token under standard steering, causing cumulative coherence degradation.
  • Q2: Component Ablations: The Q2 ablation removes cropping, gating, or both to test which components prevent response-dependent accumulation under the same multi-turn protocol.Without cropping, response-token contributions return; without gating, every generated token receives the fixed coefficient cbase because s = 0.

6 Discussion

GCAD’s trait signal is distributed across attention layers, with most traits peaking at layer 19 while retaining substantial earlier-layer mass. Its gate applies steering broadly in early layers and selectively later, redistributing rather than amplifying the intervention.

  • Distributed attention-layer signals: Trait signal is distributed across attention layers, peaking for most traits at ℓ=19 while retaining substantial mass across ℓ∈[9, 18].The cropped attention-delta vectors are extracted from the system-prompt contribution to self-attention.
  • Distributed attention-layer signals: The layer distribution supports the view that persona signal includes the system prompt’s effect propagated through self-attention.This links GCAD’s attention-derived signal to the decomposition described in Section 3.3.
  • Prompt-like sparse steering: As depth increases, gating becomes selective while the layer-mean coefficient stays close to cbase, redistributing steering rather than amplifying it.The gate follows a layer-dependent “apply where it is missing” policy.
  • Prompt-like sparse steering: Early steered layers apply nearly uniform high steering because hidden states have not yet developed substantial trait projection.At greater depth, the gate suppresses steering for tokens already aligned with the persona direction and concentrates it on tokens that are not.

7 Conclusion … B Persona Vector Definitions and Extraction Prompts

The paper identifies KV-cache contamination as a coherence failure in residual-stream steering and introduces GCAD, which uses gated system-prompt attention contributions to improve long-horizon persona steering. The appendices specify the compared models, methods, evaluation protocol, trait categories, extraction prompts, vector pipeline, and trait inventory.

  • 7 Conclusion: GCAD addresses KV-cache contamination by applying token-gated cropped attention-delta signals from system-prompt contributions, preserving trait control while improving long-horizon coherence.Residual-stream steering repeatedly caches perturbed states, degrading coherence across multi-turn dialogue.
  • A Baselines and Hyperparameters: The experimental setup defines the configurations used for both compared methods across two open-source instruction-tuned language models.The same configurations underlie the main text and Appendices D–I.
  • A Baselines and Hyperparameters: Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct are comparable-size pre-norm decoder transformers with different layer, hidden-size, and attention configurations.Both use rotary position embeddings and half-precision loading; eager attention is enabled when computing attention weights for vector extraction.
  • A Baselines and Hyperparameters: The residual-stream baseline adds a response-phase mean-difference persona vector at one layer during every decoding step and keeps the steering hook active across all 10 dialogue turns.The injection settings are ℓ⋆=20, α=2.0 for Qwen2.5-7B-Instruct and ℓ⋆=16, α=1.2 for Llama-3.1-8B-Instruct.
  • A Baselines and Hyperparameters: GCAD uses multi-layer cropped attention-delta steering with model-specific layer ranges and base coefficients, while centering its gate on per-layer mean compatibility.Qwen uses layers 9–19 with cbase = 3.5; Llama uses layers 8–20 with cbase = 2.0; both use s = 1.5.
  • B.1 Trait categorization: The 17 persona traits are organized into RLHF-aligned, RLHF-opposing, and neutral categories solely to organize results, not to define different methods.The categories distinguish traits rewarded, suppressed, or neither directly rewarded nor suppressed by safety post-training.
  • B.2 Contrastive system prompts and extraction questions: Each trait uses five paraphrased contrastive system-prompt pairs and 20 open-ended questions, with positive prompts requesting trait expression and negative prompts requesting helpful-and-harmless behavior.Paraphrasing reduces sensitivity to any single prompt wording.
  • B.3 Vector extraction pipeline: The extraction pipeline derives residual-stream directions from paired response activations, while GCAD restricts attention sums to system-prompt positions and extracts gating directions from positive-condition key outputs.Conversations with trait scores below 50 are excluded so extracted directions reflect successfully steered responses.

C Evaluation Metrics … C.5 LLM-as-a-Judge prompts

The evaluation scores assistant replies on independent 0–100 axes for trait expression and coherence, using GPT-4.1-mini with reproducible trait-specific and shared coherence rubrics. Scores are aggregated across 60 conversations per trait–method cell at turns 1, 5, 10, the 10-turn mean, and cumulative drift.

  • C Evaluation Metrics: GPT-4.1-mini-2025-04-14 evaluates trait expression and coherence separately for the residual-stream baseline, GCAD, and all ablations.Both axes use the same judging configuration and a 0–100 scale.
  • C.1 Trait expression: Trait expression measures how strongly a reply embodies the target persona using a trait-specific rubric scored from 0 to 100.The EVIL rubric assesses harmful or malicious behavior, while HONEST evaluates truthfulness, acknowledged limitations, and avoidance of exaggeration.
  • C.2 Coherence: Coherence measures whether a reply is internally consistent and well-formed, independently of correctness, alignment, or target-trait content.Its anchors are 0 for invalid or nonsensical English, 50 for confused but partly sensible answers, and 100 for clearly understood, non-hallucinatory answers.
  • C.3 Aggregate statistics: 60 conversations populate each trait–method cell, with all 10 replies per conversation judged independently before averaging within turns and across conversations.Reported statistics are turn 1, turn 5, turn 10, the 10-turn mean (Avg), and cumulative drift ∆ = T10 − T1.
  • C.3 Aggregate statistics: Negative coherence drift indicates degradation across a conversation, whereas drift ∆ close to zero indicates a stable trajectory.The cumulative drift is defined as ∆ = T10 − T1.
  • C.4 LLM judge usage.: GPT-4.1-mini is used only as an automatic evaluator and does not train, tune, or construct GCAD.For each response, it receives the response and a task-specific rubric and returns a 0–100 score; full prompts are supplied for reproducibility.
  • C.5 LLM-as-a-Judge prompts: The reproduced prompts illustrate an EVIL trait-expression rubric and a coherence rubric shared across all traits and conditions.The remaining sixteen traits use the same trait-expression template with the trait name and description replaced.
  • C.5 LLM-as-a-Judge prompts: The EVIL prompt requests a 0–100 trait score or REFUSAL, while the coherence prompt instructs judges to focus solely on coherence rather than correctness or logical stance.The coherence scale defines 0, 50, and 100 using nonsensical, confused, and clearly understood answers respectively.

D Per-Trait Multi-Turn Results: Qwen2.5-7B-Instruct … G No-Steering Baseline

The paper compares per-trait, per-turn steering across Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, analyzes trait-dependent difficulty, and validates steering effects against a no-steering baseline. The results indicate that trait difficulty varies by method, while many opposing traits remain near floor without intervention.

  • D Per-Trait Multi-Turn Results: Qwen2.5-7B-Instruct: Tables 3 and 4 compare trait expression and coherence for Qwen2.5-7B-Instruct across 10 turns, reporting cumulative drift ∆= T10 −T1.Each cell averages 60 conversations from ngroups = 20 and nsamples = 3.
  • E Per-Trait Multi-Turn Results: Llama-3.1-8B-Instruct: Tables 5 and 6 provide the corresponding 10-turn trait-expression and coherence comparison for Llama-3.1-8B-Instruct.Cumulative drift is reported as ∆= T10 −T1, with each cell averaging 60 conversations from ngroups = 20 and nsamples = 3.
  • F Trait-Dependent Steering Difficulty: The two steering methods disagree on which traits are hardest to control.RLHF-aligned traits such as HONEST are described as easy for both methods, whereas RLHF-opposing and neutral traits produce method divergence.
  • F Trait-Dependent Steering Difficulty: RLHF-aligned traits including HONEST, FACTUAL, and POLITE use small persona vectors and maintain coherence near the no-steering ceiling.The passage contrasts these traits with EVIL, IMPOLITE, HUMOROUS, and CREATIVE, where methods diverge.
  • F Trait-Dependent Steering Difficulty: EVIL, IMPOLITE, HUMOROUS, and CREATIVE are difficult because they require larger residual-stream perturbations and have attention-delta mass distributed across layers.The larger perturbations magnify out-of-distribution shock at a single injection site, while distributed attention-delta mass contributes a second compounding difficulty.
  • G No-Steering Baseline: The no-steering Qwen2.5-7B-Instruct run disables all steering hooks while matching the judging configuration, evaluation pool, and decoding hyperparameters of Appendix A.It uses 5 question groups with 10 independent decoding samples per group across 5 turns, yielding 50 conversations and 250 judged replies.
  • G No-Steering Baseline: 0.0 is the reported level for EVIL, HUMOROUS, and IMPOLITE across all five unsteered turns, while APATHETIC and SYCOPHANTIC remain below 10.These near-floor values support attributing the corresponding steered trait-expression results to interventions rather than prompts.
  • G No-Steering Baseline: 97.4 is the average unsteered coherence at turn 1, which decays only mildly over the evaluated turns.HALLUCINATING is an exception, rising from 22.8 at turn 1 to 67.7 at turn 5 without intervention.

H MLP Pathway Role Analysis

The analysis decomposes prompt-conditioned residual changes into attention, MLP, and upstream-residual pathways, then compares matched attention- and MLP-pathway interventions. Attention carries substantially more trait signal: average trait expression is 85.2 versus 20.8, supporting GCAD’s attention-only write target.

  • Pathway decomposition: The persona-vector decomposition separates prompt-conditioned residual change into attention, MLP, and upstream-residual contributions.The pathways are denoted ∆a(ℓ), ∆m(ℓ), and ∆h(ℓ−1).
  • Matched intervention setup: Both pathway deltas are injected at coefficient 1.0 across all 28 Qwen2.5-7B-Instruct layers, differing only in the write target.The evaluation uses 20 question groups, 10 independent decoding samples per group, and one turn.
  • Pathway comparison: 85.2 versus 20.8: average trait-expression score is substantially higher for attention-pathway than MLP-pathway steering, a gap of +64.4 points.This comparison uses the same per-layer coefficient for both pathways.
  • Pathway comparison: 97.6 versus 1.6: EVIL expression is much higher with attention-pathway than MLP-pathway steering.For HALLUCINATING, scores are 98.7 versus 11.7; for HUMOROUS, scores are 90.4 versus 13.0.
  • Implication for GCAD: MLP-pathway steering is not a competitive substitute for attention-pathway steering, justifying GCAD’s restriction to the attention output as its write target.The appendix attributes this either to weaker response-phase MLP signal or routing of trait-relevant MLP components through attention by the next layer.

I Ablations · J Limitations and Future Work

The ablations isolate GCAD’s cropping and gating components under the multi-turn evaluation protocol. They compare removing cropping, removing gating, or removing both across five persona traits.

  • I Ablations: GCAD ablations target two design components beyond the multi-layer attention-delta variant: cropping and gating.The cropping ablation replaces the system-prompt-only attention sum with the full attention-delta.
  • I Ablations: The cropping ablation, denoted (–) cropping (P2), retains multi-layer placement and gating while replacing the system-prompt-only sum with the full attention-delta.
  • I Ablations: The gating ablation, denoted (–) gating (P3), fixes the per-token coefficient to cbase for every generated token.This is equivalent to setting s = 0 in Eq. 10.
  • I Ablations: Removing both cropping and gating reduces GCAD to ungated multi-layer attention-delta.
  • I Ablations: All ablations follow the multi-turn protocol of Section 5.1.
  • I Ablations: The ablations cover five traits: EVIL, HUMOROUS, PESSIMISTIC, SYCOPHANTIC, and EMPATHETIC.Table 10 reports per-turn values for the ablation comparison.

J.1 Limitations · J.2 Future work

The results are limited by prompt compliance during extraction and reliance on an LLM judge, while future work extends prompt-pathway decomposition to monitoring, data filtering, and fine-tuning supervision.

  • J.1 Limitations: GCAD’s effectiveness is untested on strongly aligned models that resist the contrastive system prompts used for extraction.Such resistance may produce an activation difference too weak to drive the target trait at inference.
  • J.1 Limitations: Trait-expression and coherence scores come from the gpt-4.1-mini-2025-04-14 judge, so absolute values may reflect judge-specific training and safety-boundary effects.Cross-method comparisons are more robust because uniform judge bias affects both methods.
  • J.2 Future work: The prompt-pathway decomposition could monitor whether attention outputs move in a known trait direction using the per-token compatibility score d(ℓ)i.This would provide a cheap behavioral signal at decode time.
  • J.2 Future work: The paper also identifies pathway-target and design-principle ablations as future-work context through comparisons involving MLP versus attention pathways and GCAD variants.The provided tables describe these evaluations but do not include their numerical outcomes here.
  • J.2 Future work: The same compatibility score could filter training data by retrieving or rejecting conversations whose attention-pathway behavior aligns with an undesired trait.The proposed use is as a data-filtering signal.
  • J.2 Future work: The cropped attention-delta could supervise fine-tuning by anchoring activation targets or contrastive losses that internalize the steering direction into model weights.This would replace inference-time re-injection with weight-level internalization.
Loading 2605.10664v2…