PaperlayerView exact revision on arXiv

Source-linked AI summary

Steering Llama 2 via Contrastive Activation Addition

Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, Alexander Matt Turner

arXiv:2312.06681v4cs.CLcs.AIcs.LG

TL;DR

Existing activation-engineering methods lack robust verification across models and behaviors. CAA derives behavior directions from paired examples and steers Llama 2 at inference time, generalizing across tasks with minimal detrimental effects on capabilities.

  • Problem

    Activation-engineering methods have shown promise, but their mechanisms, properties, and effects remain insufficiently verified across models and behaviors.

  • Method

    CAA averages activation differences from paired examples showing desired and opposite behaviors to derive steering vectors that shift model outputs during inference.

  • Results

    CAA generalized from multiple-choice contexts to open-ended generation, remained effective alongside finetuning, and had minimal detrimental effects on overall capabilities.

  • Takeaways & Limitations

    CAA offers versatile, sample-efficient inference-time control that can complement system prompting and finetuning for layered output steering.

  • Takeaways & Limitations

    CAA can also steer models toward harmful, biased, or toxic outputs, creating misuse risks.

Abstract

from arXiv · show

We introduce Contrastive Activation Addition (CAA), an innovative method for steering language models by modifying their activations during forward passes. CAA computes "steering vectors" by averaging the difference in residual stream activations between pairs of positive and negative examples of a particular behavior, such as factual versus hallucinatory responses. During inference, these steering vectors are added at all token positions after the user's prompt with either a positive or negative coefficient, allowing precise control over the degree of the targeted behavior. We evaluate CAA's effectiveness on Llama 2 Chat using multiple-choice behavioral question datasets and open-ended generation tasks. We demonstrate that CAA significantly alters model behavior, is effective over and on top of traditional methods like finetuning and system prompt design, and minimally reduces capabilities. Moreover, we gain deeper insights into CAA's mechanisms by employing various activation space interpretation methods. CAA accurately steers model outputs and sheds light on how high-level concepts are represented in Large Language Models (LLMs).

1 Introduction

The introduction frames alignment as necessary for reducing risks from unsafe behavior, while noting persistent data, hallucination, distribution-shift, and opacity challenges. It presents Contrastive Activation Addition (CAA) as a method for studying and steering alignment-relevant behaviors in Llama 2, with evaluations spanning multiple tasks and comparisons to existing techniques.

  • As LLM capabilities have grown, alignment research has aimed to make models helpful, honest, and harmless while reducing risks from misaligned, unsafe behavior.
  • Existing alignment methods include RLHF, instruction finetuning, and prompt engineering, but challenges remain in representative data collection, hallucination prevention, out-of-distribution failures, and interpretability.
  • Activation engineering steers behavior through targeted activation perturbations, but its mechanisms, properties, and effects require robust verification across models and behavior types.
  • The paper applies CAA to alignment-relevant behaviors in Llama 2, primarily Llama 2 Chat, spanning models from 7 to 70 billion parameters and examining interactions with RLHF finetuning.
  • Across seven tested categories, adding or subtracting steering vectors increased or decreased behavior prevalence as rated by GPT-4, alongside transfer, system-prompting, finetuning, and geometric analyses.

2 Related work

Prior work steers language-model behavior by modifying intermediate or attention activations and extracts concept representations with mean-difference methods. CAA builds on these approaches while using paired contrastive prompts, targeting residual-stream steering across broader behaviors and comparing against prompting and finetuning.

  • Activation steering: Activation Addition derives steering vectors from activation differences between prompt pairs, but its behavior coverage, prompt robustness, and evaluation scope were limited.It adds the vector at the first token position and was tested only on GPT-2-XL.
  • Truthfulness steering: Mean Difference vectors shift activations toward truthful representations, improving truthfulness on adversarial benchmarks while minimally affecting fluency and requiring little data.Li et al. identify sparse sets of truthful attention heads using linear probes before shifting activations between true and false distribution means.
  • CAA’s positioning: CAA uses an optimized multiple-choice format with paired prompts differing by one token, then emphasizes steering across more behaviors rather than representation extraction.It also compares steering with system-prompting and supervised finetuning.
  • Alternative interventions: Other work reduces toxicity or affects style transfer by steering attention activations at every transformer layer, unlike CAA’s residual-stream intervention.CAA intervenes in the residual stream rather than attention activations.

3 Method

CAA constructs behavior-specific steering vectors by averaging activation differences between paired positive and negative prompts, then adds the vector during generation to shift model behavior. PCA analysis examines whether contrastive-dataset activations become linearly separable by the target behavior across model layers.

  • Steering-vector construction: CAA isolates a target-behavior direction by averaging residual-stream activation differences between paired prompts showing positive and negative behaviors.The paired prompts differ in the demonstrated behavior, allowing the average difference to capture the corresponding latent-space direction.
  • Steering-vector construction: Mean Difference (MD) computes activation differences at the answer-letter position across positive and negative prompts to construct each layer’s steering vector.The paper notes that MD produces steering vectors similar to techniques such as PCA.
  • Models and behaviors: The method is evaluated on Llama 2 7B Chat and Llama 2 13B Chat, with additional vectors generated from Llama 2 7B to test similarity and transfer.The models are RLHF-trained chat models evaluated using Huggingface Transformers and modified with PyTorch to save intermediate activations.
  • Inference procedure: CAA adds the constructed steering vector to every generated-text token position after the initial prompt.For each question, prompt pairs append the answer letter corresponding to either the target behavior or its opposite.
  • Activation-space analysis: Behavioral clustering emerges around one-third through the layers, indicating higher-level behavior representations in those activations.For refusal in Llama 2 7B Chat, visible behavioral clustering emerges suddenly between layers 9 and 10, a trend also observed across other datasets.

4 Effect of CAA on behaviors

CAA consistently steers tested behaviors in multiple-choice evaluations, with effects peaking at specific layers in Llama 2 7B and 13B Chat. It also extends to open-ended generation, where multiplier strength must balance behavioral control against text quality.

  • Layer selection: Layer sweeps identify optimal steering layers: layer 13 and adjacent layers for 7B, and usually layers 14 or 15 for 13B.The sweep evaluates multipliers of −1 and 1 across all layers using held-out test questions.
  • Multiple-choice evaluations: CAA consistently steers results across all tested behaviors in multiple-choice evaluations.The evaluation uses held-out questions matching the generation-set format.
  • Open-ended generation: CAA generalizes beyond multiple-choice settings to free-form answers on open-ended questions.Sycophancy prompts are manually written, while other datasets adapt held-out multiple-choice questions by removing answer options.
  • Open-ended generation: GPT-4 rates open-ended answers from 1-10 according to how much of the targeted behavior they display.The rating prompts are provided in Appendix L.
  • Open-ended generation: Larger steering multipliers degrade open-ended text quality, so the multiplier range is limited to balance behavioral steering with generation quality.The degradation is assessed by both the GPT-4 evaluator and human readers.

5 CAA and system-prompting

The section compares Contrastive Activation Addition (CAA) with system-prompting for eliciting or avoiding target behaviors. Across most tested behaviors, CAA extends behavioral control beyond prompting alone, potentially because its multiplier enables precise steering and its vectors aggregate information across many prompts.

  • Experimental setup: The comparison uses positive and negative system prompts to respectively elicit or avoid specific target behaviors.The positive prompt instructs the model to exhibit the target behavior, whereas the negative prompt instructs it to exhibit the opposite behavior.
  • Results: CAA modifies model behavior beyond system-prompting alone for most tested behaviors.Adding a steering vector raises the behavioral evaluation score beyond a positive system prompt, while subtracting it has the corresponding effect relative to a negative prompt.
  • Interpretation: CAA may provide finer control because its multiplier sets the steering quantity and its vectors aggregate information across a large prompt dataset.The authors hypothesize that this also isolates behavioral variables more effectively than system-prompting alone.

6 Comparison to finetuning

Supervised finetuning reaches high held-out accuracy and affects open-ended generation, while CAA can further steer some behaviors beyond finetuning alone. Their combination improves open-ended generation more than multiple-choice performance but can produce counter-intuitive interactions and out-of-distribution failures.

  • Finetuning results: Supervised finetuning achieves high accuracy on the held-out test set and noticeably affects open-ended generation, generalizing from A/B answers to free-text generation.The held-out test set contains 50 questions.
  • Combined effects: For 3 out of 7 tested behaviors, CAA steers beyond finetuning alone in both positive and negative directions.The combined approach can therefore extend finetuning’s behavioral effects, although interactions are not uniformly intuitive.
  • Combined effects: Positive CAA steering on top of finetuning reduces the refusal score, illustrating a counter-intuitive interaction between steering and finetuning.This example concerns the Refusal behavior.
  • Generalization: Finetuning fails to generalize out of distribution on the Sycophancy dataset, where training uses multiple-choice questions.The supplied passage truncates the description of the Sycophancy training setup.
  • Generalization: Layering CAA on finetuning improves open-ended generation more significantly than multiple-choice performance, potentially because it steers existing behavioral representations.The passage suggests this may yield better out-of-distribution generalization than finetuning the entire model.

7 Effect of CAA on general capabilities

The section evaluates whether CAA harms general capabilities using MMLU and examines sycophancy steering on TruthfulQA. CAA does not significantly affect MMLU performance, while subtracting the sycophancy vector slightly improves TruthfulQA performance.

  • MMLU: MMLU measures general knowledge and problem-solving across 57 science, technology, humanities, and social-science subjects using ten sampled questions per category.The reported score is the average probability assigned to the correct answer after converting questions to two-option A/B format.
  • MMLU: CAA does not significantly affect MMLU performance, despite some variation across interventions.Table 5 evaluates Llama 2 13B Chat with layer-14 steering vectors and multipliers of 1 and −1.
  • TruthfulQA: Subtracting the sycophancy vector slightly improves TruthfulQA performance, indicating a small reduction in behavior that mimics human falsehoods.TruthfulQA assesses the extent to which models mimic human falsehoods.

8 Understanding and interpreting CAA

CAA steering vectors align with semantically relevant tokens, and activation–vector similarity tracks the presence of the targeted behavior. Their representations remain partially consistent across layers and transfer between Llama 2 Base and Chat models, including across RLHF boundaries.

  • CAA steering vectors detect behavioral features at semantically relevant tokens, with dot products intuitively reflecting how much behavior is present in each token.The analysis uses cosine similarity and dot products between steering vectors and per-token residual-stream activations.
  • Myopia and refusal examples show positive or negative steering components matching intuitively behavior-relevant phrases.Future-oriented cake choices and refusal phrases receive contrasting myopia or refusal components relative to phrases describing immediate choices or harmful actions.
  • Vectors from closer layers have higher similarity, while similarity declines more slowly in the latter half of the model, suggesting representation convergence.The authors theorize that abstract-concept representations become more consistent after high-level information is extracted.
  • A vector extracted from layer 13 transfers its steering effect to other layers, sometimes more strongly in earlier layers, showing the direction is not layer-specific.The transfer pattern indicates a general representation of the target behavior rather than a representation tied to one layer.
  • Steering-vector similarity between Llama 2 Chat and Base models peaks between layers 7 and 15, indicating RLHF has a smaller inter-layer representational effect there.Outside that peak, similarity decays as the extraction layer increases.
  • CAA effects transfer significantly from Llama 2 Base to Chat, especially between layers 10 and 15, indicating similarity between the models’ representations.This transfer tests whether vectors generated from Base-model activations can steer the RLHF model.

9 Discussion

CAA broadly enables precise, efficient steering of RLHF-trained LLM behavior, generalizing across task formats and remaining compatible with system-prompting and finetuning. The discussion also identifies limitations and extensions involving intervention placement, representation localization, and adversarial behavior elicitation.

  • Effectiveness and applicability: CAA generalizes steering vectors from multiple-choice contexts to open-ended generation while causing minimal detrimental effects on overall model capabilities.This versatility suggests practical applicability for steering RLHF-trained language models.
  • Compatibility with alignment methods: CAA’s additive steering is compatible with system-prompting and finetuning, enabling layered refinement of model outputs after other alignment methods.The method can adjust outputs further without replacing standard alignment techniques.
  • Inference-time control: CAA provides inference-time control over latent variables with high sample efficiency and strong generalization, including behaviors difficult to trigger through prompting alone.This is particularly useful when precise modulation of model behavior or elicitation of internal states is required.
  • Interpretability: CAA offers insight into high-level concept representations and the emergence of linear representations in language models.These interpretability benefits extend beyond the method’s immediate behavioral-control applications.
  • Limitations and future directions: Applying the steering vector at every post-prompt token position caps perturbation before text quality degrades, motivating targeted interventions at subsets of tokens or other model locations.Intervening after the MLP but before residual-stream merging could clarify representation localization and produce more targeted effects; CAA may also expose unwanted behaviors through adversarial interventions.

10 Limitations

The evaluation and comparison choices impose limitations on interpreting CAA’s steering effects. GPT-4 ratings may be noisy or biased, prompting and finetuning baselines may be under-optimized, and layer-dependent norm variation may affect layer-optimality results.

  • GPT-4 evaluation: GPT-4 evaluation scores can depend on scoring-prompt wording and LLM biases, although manual inspection found ratings broadly consistent with manual scores.The authors manually inspect a sample of GPT-4 ratings for surprising results and report correspondence with their own manual scores.
  • Finetuning comparison: Finetuning hyperparameters were not fully optimized, so further optimization could improve finetuning and reduce CAA’s additional effect size.The initially selected hyperparameters achieved over 90% accuracy on most test sets, but better results were possible with more optimization.
  • Prompting comparison: Prompting interventions were difficult to search exhaustively, so stronger prompts might improve prompting alone, whereas CAA does not require manual prompt optimization.The comparison included several system-prompt and few-shot setups, but not all possible prompting interventions.
  • Steering-vector normalization: Because residual-stream norms generally grow exponentially across layers, not normalizing layer norms may skew layer-optimality results.CAA normalizes steering-vector magnitudes across behaviors but retains natural layer norms and does not search for separate multipliers per layer.

Ethics Statement · B Answer conditioning leads to behaviorally consistent continuations · C Generating custom hallucination dataset

The paper validates answer conditioning as a way to elicit behaviorally consistent continuations for steering-vector construction and designs a custom multiple-choice hallucination dataset covering unprompted and contextually triggered hallucinations. It also notes that CAA aims to improve AI safety while acknowledging potential misuse toward harmful, biased, or toxic outputs.

  • Ethics Statement: CAA supports more helpful, honest, and harmless AI by precisely steering outputs toward safer and more reliable behaviors.The authors frame this safety benefit as a goal of the method.
  • Ethics Statement: CAA can also be misused to steer models toward harmful, biased, or toxic outputs.The paper explicitly acknowledges this misuse risk.
  • B Answer conditioning leads to behaviorally consistent continuations: The model typically justifies its selected answer in the continuation, enabling evaluation of whether A/B prompts elicit target behaviors.The validation examines generation after conditioning on either option A or B.
  • B Answer conditioning leads to behaviorally consistent continuations: Conditioning on a sycophantic answer makes the model generate reasons agreeing with the user, even when it would normally disagree.This demonstrates that the answer choice behaviorally shapes the continuation.
  • B Answer conditioning leads to behaviorally consistent continuations: The A/B format uses behavior-neutral pre-answer context and answer choices to steer subsequent justifications toward exhibiting or not exhibiting a behavior.This process is used to produce steering vectors, including the sycophancy vector.
  • C Generating custom hallucination dataset: The custom hallucination dataset distinguishes unprompted hallucination from contextually-triggered hallucination.Unprompted hallucination fabricates information despite an accurate prompt, whereas contextually-triggered hallucination builds a false narrative around an inaccurate prompt.
  • C Generating custom hallucination dataset: For contextually-triggered hallucination questions, one option accepts the false premise while the other rejects it and flags the question as invalid.These questions deliberately make an incorrect assumption or allusion.
  • C Generating custom hallucination dataset: For unprompted hallucination questions, valid answers are contrasted with fabricated answers to factually valid questions.Examples of both hallucination types appear in Table 7.

D Generating custom refusal dataset … L GPT-4 rater prompts

The paper extends CAA evaluation across refusal data construction, dataset sizes, finetuning, open-ended generation, sycophancy, prompting, and GPT-4-based behavioral rating. It also documents the evaluation prompts and computational resources used.

  • D Generating custom refusal dataset: CAA’s refusal steering vector is generated from contrasting refusal and non-refusal answers to questions the model should not answer directly.An example is provided in Table 8.
  • E Contrastive dataset sizes: Table 9 reports the contrastive dataset sizes used to generate CAA steering vectors and test their effects.
  • H Sycophancy steering and TruthfulQA: 0.02 is the average TruthfulQA improvement from subtracting the sycophancy vector in Llama 2 13B Chat, while adding it worsens performance by 0.03.The comparison averages across all TruthfulQA categories.
  • F CAA on top of finetuning - effect on multiple-choice test datasets: Table 10 evaluates CAA in Llama 2 7B Chat after supervised finetuning to incentivize either a behavior or its opposite.Steering is performed at layer 13, and scores are average token probabilities for behavior-matching answers over 50 test examples.
  • G Examples of open-ended generation with CAA: Table 11 gives open-ended Llama 2 7B Chat examples produced with CAA at layer 13 using multipliers 2 and −2.
  • I Finetuning test set accuracy: Table 13 reports test-set accuracy after one epoch of Llama 2 7B Chat finetuning on the same contrastive dataset, split into positive and negative settings.
Loading 2312.06681v4…