PaperlayerView exact revision on arXiv

Source-linked AI summary

Steering Language Models With Activation Engineering

Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, Monte MacDiarmid

arXiv:2308.10248v5cs.CLcs.LG

TL;DR

Language models contain capabilities that prompting and finetuning do not fully elicit, motivating inference-time activation steering. The paper introduces ActAdd, which contrasts prompt activations to form a steering vector and adds it during the forward pass. ActAdd achieves SOTA on toxicity reduction and sentiment shift while retaining overall model capabilities, and the authors present it as lightweight and promising for flexible behavioral control.

  • Problem

    Prompting and finetuning do not fully elicit language models’ capabilities, leaving an elicitation gap.

  • Method

    ActAdd contrasts intermediate activations from a target-property prompt and its opposite, then adds their difference during inference to steer outputs.

  • Results

    ActAdd achieves SOTA on toxicity reduction and sentiment shift while retaining overall model capabilities.

  • Takeaways & Limitations

    ActAdd’s success provides experimental evidence for feature linearity and suggests steering vectors can establish causality in examined contexts.

  • Takeaways & Limitations

    ActAdd is not immediately applicable through typical APIs because it requires models to cache and expose intermediate activations.

Abstract

from arXiv · show

Prompt engineering and finetuning aim to maximize language model performance on a given metric (like toxicity reduction). However, these methods do not fully elicit a model's capabilities. To reduce this gap, we introduce activation engineering: the inference-time modification of activations in order to control (or steer) model outputs. Specifically, we introduce the Activation Addition (ActAdd) technique, which contrasts the intermediate activations on prompt pairs (such as "Love" versus "Hate") to compute a steering vector (Subramani et al. 2022). By tactically adding in e.g. the "Love" - "Hate" steering vector during the forward pass, we achieve SOTA on negative-to-positive sentiment shift and detoxification using models including LLaMA-3 and OPT. ActAdd yields inference-time control over high-level output properties (like topic and sentiment) while preserving performance on off-target tasks. ActAdd is lightweight: it does not require any machine optimization and works with a single pair of data points, which enables rapid iteration over steering. ActAdd demonstrates the power of activation engineering.

1 INTRODUCTION

The paper argues that prompting does not reliably elicit models’ latent capabilities, motivating activation engineering: steering outputs by perturbing intermediate activations. It introduces ActAdd, which derives steering vectors from contrasting prompts and reports strong control of sentiment and toxicity while preserving general capabilities.

  • Motivation: Prompting can improve benchmark performance without supplying task-relevant information or changing the model’s computational graph.The paper frames this as an elicitation gap: models contain capabilities that naive prompting does not fully reveal.
  • Activation engineering: Activation engineering strategically perturbs activations during the forward pass to steer language-model outputs more reliably than prompting.The paper presents this as a new steering modality intended to reduce the elicitation overhang.
  • Motivation: Activation engineering may elicit combinations of capabilities that no prompt can induce, such as eloquent language paired with mathematical content.The paper illustrates this possibility using a model trained to imitate eloquent poets and awkward mathematicians.
  • ActAdd: ActAdd contrasts activations from prompts such as “Love” and “Hate,” then adds their difference during inference as a steering vector.The steering vector functions like a virtual bias term that shifts completions toward the target sentiment.
  • ActAdd: ActAdd uses topic and sentiment steering vectors, including “Love” - “Hate” and wedding-related contrast prompts.Table 1 identifies these prompt pairs as the sources of the reported steering vectors.
  • Results: The authors report SOTA on toxicity reduction and sentiment control while testing whether steered models preserve general capabilities.They characterize ActAdd as an effective and cheap method for steering LLM outputs.

2 RELATED WORK

Related work situates activation engineering among interventions on weights, decoding, prompts, embeddings, and activations. The paper distinguishes ActAdd from optimized activation-vector methods because it directly computes steering vectors without gradient descent or other optimization.

  • Prior approaches: Latent-space arithmetic has steered image generation and captured semantic analogies in word embeddings, providing precedents for steering generative models.The paper positions its contribution specifically in text generation with language models.
  • Intervention types: LLM steering methods intervene on weights, decoding, prompts, token embeddings, or intermediate activations.Table 2 organizes prior work by intervention type.
  • ActAdd’s distinction: ActAdd differs from optimized extraction methods by directly computing its steering vector without gradient descent or other optimization methods.This distinction is stated explicitly in the related-work discussion.
  • Activation engineering: Activation engineering creates activation vectors that cause desired output changes when added to forward passes of a frozen LLM.The paper uses this definition to organize the activation-engineering literature.
  • Activation steering: Prior activation-steering approaches include classifier-guided perturbation, gradient-based steering-vector search, knowledge editing, and attention-head ablation.The paper cites Plug-and-Play Language Models, Subramani et al., Hernandez et al., and interpretability-oriented ablations as examples.
  • Prior approaches: Unlike ActAdd, earlier Plug-and-Play Language Models use a separate classifier for each attribute being steered.The cited method perturbs activations to make generated text better match the classifier’s target.

3 HOW ACTIVATION ADDITION WORKS

ActAdd forms a steering vector from the activation difference between a target-property prompt and its opposite, rescales it, and adds it to a chosen residual-stream layer during inference. The method uses a user prompt, target layer, injection coefficient, and sequence alignment, with middle-layer interventions reported as most effective.

  • Residual stream: ActAdd operates on residual-stream activation vectors processed through a stack of Transformer layers before next-token decoding.The method manipulates the residual stream at the input to a selected layer.
  • Prompt pair: The method takes a prompt pair in which p+ expresses the desired property and p− expresses its opposite.Examples include love versus hate or indifference.
  • Inference intervention: The steering vector is computed before inference, then added to the input of layer l while the forward pass continues to produce the steered output.This separates vector construction from the user-prompt inference pass.
  • Algorithm: Algorithm 1 uses a user prompt, target layer, injection coefficient, sequence-position alignment, and pretrained model to produce a steered output.The algorithm labels the output as S and the steering prompt pair as tokenized input.
  • Steering-vector construction: ActAdd records activations for both prompts, subtracts them, and rescales the difference by an injection coefficient c.The resulting vector represents the target-property direction used for intervention.
  • Hyperparameters: Middle-layer interventions are reported as most effective after tuning the injection coefficient and injection layer.The paper states this agrees with past work.
  • Evaluation: The experiments test desired behavioral shifts and preservation of general capabilities across GPT-2-XL, OPT, and LLaMA-3.They include perplexity, toxicity, and sentiment experiments.

4 RESULTS: ACTIVATION ADDITION WORKS

ActAdd steers models toward target topics and reduces toxicity while preserving some off-target performance. It also achieves strong sentiment-control results, though steering can reduce fluency and relevance.

  • 4.1 ACTADD REDUCES PERPLEXITY ON A TARGET TOPIC: ActAdd lowers perplexity on wedding-related sentences, increasing predictive performance when the injected topic is relevant.The wedding vector preserves performance on unrelated text.
  • 4.1.2 ACTADD STEERS THE MODEL TO DISCUSS WEDDINGS: >90% topic-steering success is achieved at the optimal injection layer, compared with a ∼2% baseline.Effectiveness rises through layer 6 and then declines.
  • 4.2 ACTADD CAN CONTROL WHAT THE MODEL TALKS ABOUT: 5-20% relevance boosts occur across generic target topics at injection coefficient c = 2, except “art”.Relevance is scored by GPT-3.5 across ActAdd completions.
  • 4.3 ACTADD CAN REDUCE TOXICITY: ActAdd-OPT has 8% lower toxicity than the second-best baseline, while ActAdd-LLaMA-3 produces a 5% toxicity drop with a very small fluency penalty.ActAdd-OPT is the least toxic OPT intervention and outperforms an unsteered LLaMA-3.
  • 4.4 ACTADD CAN CONTROL SENTIMENT: ActAdd achieves state-of-the-art negative-to-positive sentiment steering, while competing methods can incur larger fluency and relevance penalties.For the positive-to-negative direction, the only outperforming method has fluency 68.4 versus ActAdd’s 24.2 on the same pretrained model.
  • 4.5 ACTADD PRESERVES THE MODEL’S GENERAL KNOWLEDGE: ConceptNet testing finds ActAdd has a negligible impact on off-target probabilities across a range of top-K values.P@K denotes the probability that the correct answer appears among the model’s top K answers.

5 DISCUSSION

The discussion presents ActAdd as an efficient, continuously controllable alternative to prompting and finetuning, while linking its effects to linear and compositional representations. It also identifies usability and access limitations, alongside potential value-alignment applications.

  • Interpretability: ActAdd composes separate forward passes by adding steering vectors, providing evidence for compositional representations in language models.The authors view this composability as independent evidence from weight-based task-composition arithmetic.
  • Limitations: ActAdd is less user-friendly than prompt engineering because users must supply the injection coefficient and intervention layer.The method also requires access to cached and exposed intermediate activations, which current APIs generally do not provide.
  • Efficiency: ActAdd requires only forward passes and a steering prompt pair, enabling rapid iteration with minimal labelled data.The method needs no backward passes and can run on inference-capable rather than training-capable machines.
  • Prompting comparison: Unlike discrete prompts, activation additions can be continuously weighted without consuming context-window token space.The paper notes that the context-space benefit is small with multi-million-token windows.
  • Interpretability: Activation addition experimentally supports feature linearity by steering love-related completions in examined contexts.The authors state that steering vectors establish causality only within the limited contexts examined.
  • Scope and implications: Activation engineering may flexibly retarget behavior without damaging general performance, but the authors doubt it can teach a model a new skill.Finetuning is described as more flexible and better understood, though significantly more costly.

6 CONCLUSION

The conclusion characterizes ActAdd as a lightweight inference-time steering method that achieves strong toxicity and sentiment results while retaining overall capabilities. It also highlights reproducibility practices and boundaries involving misuse, qualitative selection, and model access.

  • Conclusion: Activation Addition steers models by shifting inference-time activations along vectors such as the Love-Hate direction.The conclusion presents this as the paper’s central activation-engineering method.
  • Conclusion: ActAdd achieves SOTA on toxicity reduction and sentiment shift while retaining overall model capabilities.The authors describe the method as lightweight, effective, and promising for activation engineering.
  • Experimental scope: The experiments use multiple models, including LLaMA-1-13B, GPT-J-6B, OPT, and LLaMA-3-8B, with sentence embeddings used for relevance scoring.The appendix identifies OPT and LLaMA-3-8B for toxicity and sentiment experiments.
  • Reproducibility: The study fixes sampling hyperparameters during data collection and validates qualitative results across seeds 1 and 2 after generating them with seed 0.The authors also report an external blinded reproduction process for the unusual top_p setting.
  • Evaluation caveats: Qualitative demonstrations report the most compelling pair among K = 3 completions and exclude one of 12 candidate additions judged unimpressive.The remaining 11 activation additions appear in Table 6.
  • Safety scope: ActAdd can be misused to steer toward anger or conspiracy and may circumvent superficial alignment methods.The authors frame possible inference-time alignment benefits alongside this misuse risk.

B IS ACTADD JUST A SUBTLE KIND OF PROMPT ENGINEERING?

The appendix tests whether ActAdd is merely token injection or prompt engineering by comparing intervention locations, embedding-vector controls, perplexity, and prompting effects. The evidence indicates that ActAdd depends partly on computation across Transformer layers rather than simple token insertion.

  • Prompt-engineering hypothesis: Continuous activation arithmetic cannot generally be reduced to discrete token stacking, such as adding 3 × ‘wedding’ and −3 × ‘<whitespace>’.This distinction separates the token-injection hypothesis from the linear-representations hypothesis.
  • Layer intervention: Adding an Anger−Calm embedding vector at layer 20 has at most a small qualitative effect, while the usual steering vector makes completions angrier.This suggests that earlier Transformer blocks contribute substantially to the steering effect.
  • Layer intervention: Early layers 0 and 1 apparently perform substantial steering-relevant cognitive work, because a layer-2-sourced vector increases anger more than a 0 →20 intervention.The effect remains smaller than that of the normal steering vector and varies across samples.
  • Perplexity comparison: Prompting causes a large degradation in the unrelated condition, whereas the comparison is used as evidence that ActAdd partly uses another mechanism.The experiment compares perplexity across wedding-related and wedding-unrelated sentence collections.

C IMPLEMENTATION DETAILS

ActAdd uses lightweight, inference-time activation interventions that can be rapidly configured and preserve broad model behavior, while experiments examine their token- and task-level effects across models.

  • Implementation: Contrast pairs can use arbitrary lengths, with whitespace padding enabling alignment when prompts differ in length.The method uses fixed front activation addition in these experiments, and intervened streams are masked when evaluating next-token predictions.
  • Implementation: The counterbalanced p− prompt helps specify the steering direction because adding h+ alone is less effective.The injection coefficient c rescales the activation difference before it is added during the forward pass.
  • Limitations: The injection coefficient cannot increase indefinitely, although wedding-related behavior can be intensified substantially before GPT-2-XL loses general competence.The paper also reports an unexplained reduction in predicting sequences of null characters.
  • Implementation: Contrast pairs are often discovered in single-digit minutes, including natural single-word pairs, making rapid steering accessible beyond technical users.The authors compare this workflow with hand-crafted prompt engineering.
  • Advantages: ActAdd is online steering: it can be repeatedly applied at inference time, enabling iterative chains and mixes of steering decisions while leaving weights unchanged.The unchanged weights may help preserve interpretability relative to parameter updates, though the paper frames this as a possibility.
  • Cross-model evaluation: On GPT-J-6B, the wedding vector selectively affects wedding-related tokens across KL divergence and token probability, with similar behavior across layers and coefficient magnitudes.Llama-1-13B shows the same qualitative steering effect, but several examples fail to replicate.

E.3 OPT-6.7B

On OPT, ActAdd produces substantial toxicity reduction and sentiment-shift improvements using the love−hate vector, with fluency and relevance costs that vary by task.

  • OPT-6.7B: 17% lower toxicity is achieved by ActAdd-OPT than unsteered OPT, with a small cost to fluency and relevance.The reduction is statistically significant and partly unavoidable given the detoxification task.
  • OPT-6.7B: 21% absolute increase in positive classification is achieved by ActAdd-OPT than unsteered OPT, with a larger cost to fluency and relevance.The reported cost is partly unavoidable given the sentiment-shift task.

F INVESTIGATING THE NORM OF STEERING VECTORS

The paper examines how large steering modifications are relative to normal activations and finds that effective steering can dominate the residual stream.

  • Measurement: Relative modification magnitude is measured by comparing the steering-vector norm at each position with the residual-stream norm across layers.The prompts contain three residual streams because an endoftext token is prepended.
  • Measurement: Position 0 is never modified because it contains the same endoftext token for both contrast prompts.Figure 9 therefore shows a flat zero contribution at that position.
  • Interpretation: The anger ActAdd changes output-token distributions less than a random vector, indicating more targeted next-token effects.This comparison is reported using KL divergence.
  • Magnitude: Around 90% of the residual stream is heuristically attributed to the steering vector at Anger−Calm coefficient +10, rather than prior prompt information.The paper describes this as nearly ten times the norm of the underlying forward pass and notes that activation additions are not minor changes.

G INVESTIGATING RANDOM ACTADD VECTORS

Random vectors provide a control for whether ActAdd effects arise merely from large perturbations; the reported results distinguish random disruption from targeted steering.

  • Control design: Random vectors are scaled to match the per-position norms of anger steering vectors before injection into the forward pass.The comparison uses representative completions and systematic output-distribution measurements.
  • Random-vector results: Random vectors with comparable norms leave completion distributions qualitatively similar, and even larger perturbations preserve coherence relative to unsteered GPT-2-XL.A noticeable output shift appears at the norm of a c = +10 anger vector.
  • Random-vector results: The anger vector changes output-token probabilities less than the random vector, suggesting that its effects are more targeted.The comparison aggregates dozens of prompts across anger-steered, random-steered, and unmodified models.
  • Random-vector results: A random-text vector can disrupt generation severely: at coefficient +1000, GPT-2-XL loses its grasp of English syntax.This vector differs from a random Gaussian vector and is rescaled to a norm comparable to +1 Anger−Calm.
  • Cross-model replication: GPT-J-6B replication examines targeted KL-divergence, token-probability, and perplexity effects across injection layers and steering magnitudes.Figures 11–13 repeat the wedding-vector experiments using GPT-J-6B instead of GPT-2.

H PARTIAL ACTADD

Partial ActAdd intervention shows that modifying only selected residual-stream dimensions can still increase wedding-relatedness, but the effect depends on prompt context and can be nonmonotonic. The observed pattern is consistent with axis-alignment of a wedding-related steering feature.

  • Partial ActAdd: Partial residual-stream modification still produces relatively smooth increases in wedding-relatedness as more dimensions are added.The experiment varies the fraction of modified dimensions from 0/1600 to 1600/1600 and counts wedding words in generated completions.
  • Prompt dependence: For “I went up to my friend and said”, modifying the first 1,120 dimensions produces more wedding-related completions than modifying all 1,600.
  • Figure 14: Figure 14 reports wedding-relatedness as a function of modified residual-stream dimensions, including a nonmonotonic pattern for the friend prompt.
  • Prompt dependence: Wedding-relatedness begins increasing at lower modified-dimension fractions in an appropriate context than in unrelated contexts.The passage contrasts the demonstrated friend prompt with contexts such as dieting trends.
  • Interpretation: The nonmonotonic dimension-fraction relationship is presented as evidence that the wedding-related steering feature is axis-aligned.This interpretation is offered after noting that a non-axis-aligned feature hypothesis does not explain the observed nonmonotonicity.
Loading 2308.10248v5…