PaperlayerView exact revision on arXiv

Source-linked AI summary

Programming Refusal with Conditional Activation Steering

Bruce W. Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Nagireddy, Amit Dhurandhar

arXiv:2409.05907v3cs.LGcs.AIcs.CL

TL;DR

Existing activation steering lacks conditional control because it modifies behavior uniformly across inputs, limiting selective responses. CAST uses condition vectors derived from prompt-activated hidden-state patterns to switch behavior vectors during inference. The framework enables selective and logically composed refusal rules without fine-tuning or extensive computational resources.

  • Problem

    Existing activation steering cannot condition behavior on context and therefore applies modifications uniformly across inputs, limiting its utility for selective responses.

  • Method

    CAST uses condition vectors representing prompt-induced activation patterns to determine when behavior vectors should be applied during inference.

  • Results

    CAST selectively increases refusal for harmful prompts while leaving harmless refusal rates largely unchanged across seven tested models.

  • Takeaways & Limitations

    CAST enables fine-grained refusal rules, including logical compositions of conditions, without fine-tuning or extensive computational resources.

Abstract

from arXiv · show

LLMs have shown remarkable capabilities, but precisely controlling their response behavior remains challenging. Existing activation steering methods alter LLM behavior indiscriminately, limiting their practical applicability in settings where selective responses are essential, such as content moderation or domain-specific assistants. In this paper, we propose Conditional Activation Steering (CAST), which analyzes LLM activation patterns during inference to selectively apply or withhold activation steering based on the input context. Our method is based on the observation that different categories of prompts activate distinct patterns in the model's hidden states. Using CAST, one can systematically control LLM behavior with rules like "if input is about hate speech or adult content, then refuse" or "if input is not about legal advice, then refuse." This allows for selective modification of responses to specific content while maintaining normal responses to other content, all without requiring weight optimization. We release an open-source implementation of our framework at github.com/IBM/activation-steering .

1 INTRODUCTION

Existing activation steering changes behavior uniformly across prompts, while CAST adds context-dependent control by using activation patterns to selectively apply refusal behavior. This supports targeted refusal and composed behavioral rules without weight optimization.

  • Motivation: Activation steering often increases refusal rates indiscriminately, including for harmless inputs, limiting conditional control.The limitation arises because existing methods add a refusal vector uniformly across inputs.
  • Method: CAST introduces a condition vector whose similarity to inference-time activations determines whether a behavior vector is applied.The condition vector acts as a switch for selective behavior modification.
  • Selective refusal: CAST selectively refuses harmful prompts while minimizing refusal of harmless prompts, as illustrated by Figure 1.The figure contrasts targeted CAST refusal with indiscriminate activation steering.
  • Applications: CAST supports rules such as refusing hate speech or adult content while preserving responses to legal-advice inputs.These rules demonstrate selective modification for context-specific content categories.
  • Contributions: The framework composes condition vectors logically to create custom refusal conditions and releases a general-purpose activation-steering toolkit.The toolkit includes demo datasets for the activation-engineering community.

2 BACKGROUND

Transformer inference transforms token embeddings through successive layers, and behavior steering can intervene in this information flow. Activation steering extracts behavior vectors, adds them to hidden states during inference, and scales the intervention without changing model weights.

  • Transformer inference: Decoder-only transformers process prompt tokens sequentially through stacked layers that transform and accumulate activation information.Inference begins with token embeddings, which are transformed by each layer’s internal mechanisms.
  • Behavior steering: Behavior steering can intervene on weights, decoding, prompts, token embeddings, or activations to alter model behavior.Activation steering specifically intervenes in layer-to-layer information flow.
  • Activation steering: Activation steering extracts a vector from contrasting behavioral examples, adds it to a chosen hidden-state layer, and generates from the modified activations.The added vector is scaled by a hyperparameter during inference.
  • Activation steering: The steering scale α trades off effectiveness and coherence: stronger scaling can disrupt coherence, while weaker scaling may be ineffective.The method aims for predictable behavior changes without altering model weights.
  • Related work: Recent activation-engineering methods address limitations of basic activation addition, while CAST adds conditioning as a new control dimension.The paper characterizes this as a vertical expansion of activation steering.

3 CONDITIONAL ACTIVATION STEERING

CAST adds context-dependent control to activation steering by using condition vectors to decide when behavior vectors should modify the model. Its pipeline extracts layer-specific vectors from contrastive examples, selects intervention settings, and steers without updating model weights.

  • CAST addresses the limitation that conventional activation steering modifies responses uniformly across inputs, preventing context-specific refusal.
  • Cosine similarity between the hidden state and its condition-vector projection is thresholded to produce a binary decision about behavior modification.
  • In the illustrated QWEN 1.5 CHAT 1.8B example, CAST refuses harmful requests while remaining responsive to harmless prompts.
  • A condition vector represents an instruction category and acts as a trigger for applying a behavior vector to the model’s hidden state.
  • CAST can combine multiple conditions, such as adult content, social stereotypes, and false advertising, to define more specific refusal behaviors.
  • The method gathers contrasting examples, extracts behavior and condition vectors, searches intervention settings, and steers the model without weight updates.
  • Behavior vectors use contrasting response suffixes, while condition vectors use full prompts; vectors are extracted separately for specified layers using PCA.

4 CONDITIONED REFUSAL: SELECTIVELY STEERING ON HARMFUL PROMPTS

CAST selectively steers refusal behavior by checking harmfulness-related activation patterns before applying a refusal vector. Across models, it increases refusal of harmful prompts while largely preserving compliance with harmless prompts, with systematic threshold and comparison-direction controls.

  • Conditioned refusal: CAST evaluates harmful prompt conditions during inference and applies the refusal vector only when the condition check is met.For the reported setup, the condition was checked at layer 7 using sim(h, projch) < 0.048.
  • Conditioned refusal: Across seven models, conditional steering increased refusal rates for harmful prompts while leaving harmless-prompt refusal rates largely unchanged.The evaluation used 500 unseen harmless Alpaca prompts and 450 unseen harmful Sorry-Bench prompts.
  • Duality: Flipping the comparison direction makes the intervention target the exact complement of the hidden states selected by the original condition.This duality supports complementary refusal behavior, including selectively refusing harmless prompts instead of harmful ones.
  • Modulation: The threshold θ modulates condition sensitivity: lowering it narrows the triggering range, while raising it broadens that range.Reported thresholds were selected by grid search to maximize F1 and balance false and true refusal.
  • Scaling properties: Conditional steering effectiveness reaches a performance plateau after additional data, while condition-vector extraction time increases linearly with sample size.The plateau is attributed to reliance on representations already present in the model.

5 PROGRAMMED REFUSAL: LOGICAL COMPOSITION OF CONDITION VECTOR

CAST extends conditional refusal from harmfulness to fine-grained categories and logical rules. It can induce or suppress refusal for selected categories, constrain responses to one domain, and generalize that constraint to unseen categories, with effectiveness related to semantic distinctness.

  • Category-specific control: Condition vectors support selective refusal control across hate speech, legal opinions, sexual context, health consultation, and crime planning.The experiments also use logical composition to construct complex refusal conditions.
  • Category-specific control: Reversing a behavior vector’s sign can remove refusal behavior from selected prompt categories.This complements the ability to induce refusal when a specified condition is met.
  • Logical composition: Logical composition lets rules such as if chate or clegal then +vrefusal trigger refusal for multiple categories.Condition layers check rules, while behavior layers apply refusal vectors.
  • Domain constraining: Flipping the comparison direction can constrain a model to answer only a target category and refuse all other inputs.This uses the complement of the target condition rather than separately defining every excluded category.
  • Domain constraining: Domain constraining extends to unseen categories, although effectiveness varies across target categories.Constraining to hate speech was more effective at refusing other categories than constraining to legal opinions.
  • Domain constraining: Constraining works better for categories that are more semantically distinct from other categories.Effectiveness is measured by the increase in refusal rate for other categories and correlates positively with average semantic distance.
  • Comparison with prompting: Conditional steering consistently outperformed the prompting baseline across most tested categories.The comparison is described against the red dotted prompting baseline in the experiments.

6 CONCLUSION

CAST extends activation steering with condition vectors to enable context-dependent behavior modification without fine-tuning or extensive computational resources. The framework supports programmable rule tracking, checking, and intervention, including refusal induction, rule composition, and domain-specific response constraints.

  • Conclusion: CAST introduces condition vectors that enable fine-grained, context-dependent control of LLM behavior without fine-tuning.The framework extends activation steering by manipulating internal representations during inference.
  • Conclusion: Flipped condition comparisons and composed conditions support refusing all but target categories and adding or removing behaviors under multiple rules.These operations are presented as mechanisms for tailoring model behavior to specific needs.
  • Programmatic control: The proposed control process tracks activations, checks rule conditions, and intervenes when conditions are met.In this paper, intervention is performed through activation steering.
  • Programmatic control: CAST is framed as a programmable, rule-based alternative to merely encouraging behaviors through prompting or reinforcement learning.The paper focuses specifically on determining and enforcing when a model should refuse.

A.2 DETAILS OF CONDITIONAL ACTIVATION STEERING

CAST extends activation steering by conditioning behavior modification on prompt-dependent activation patterns. The section covers vector extraction, inference-time checking, hyperparameter selection, and evaluation-data construction.

  • Conditional activation steering: CAST adds a condition vector alongside the behavior vector, applying refusal steering according to activation similarity.The condition vector represents prompt-induced activation patterns and acts as a switch for applying the behavior vector.
  • Generation process: The condition is checked during the first full generation pass, avoiding repetition of the additional check for every generated token.Behavior modification can then be applied in subsequent passes when the condition is met.
  • Vector extraction: Condition and behavior vectors are extracted from contrastive prompts by recording hidden states and applying PCA to separate prompt types.The recorded token position can vary with the experimental setup; last-token activations may be preferable for longer prompts.
  • Limitations: The paper notes that refusal-vector extraction and application remain limitations, and imperfect vectors can cause inconsistent refusal even after correct condition activation.Alternative extraction methods are identified as future comparison points.
  • Hyperparameters: Conditioning depends on best layer, threshold, and direction, which determine where similarity is evaluated, the activation boundary, and which side triggers steering.Thresholds that are too low can cause unnecessary interventions, while thresholds that are too high can miss needed activations.
  • Data construction: The section constructs harmful-prompt variations by pairing target prompts with examples from other harm categories and generating ten target-category variants.Harmless prompts are randomly sampled from Alpaca without modification.

C.1 INTERVENTION POINTS USED TO PRODUCE RESULTS IN THIS PAPER

The intervention procedure uses selected condition and behavior intervention points and searches conditioning configurations over layers, thresholds, and comparison directions.

  • Intervention points: Table 4 lists intervention points for condition and behavior, including interval notation such as 10 −15interval2 = [10, 12, 14].These entries specify where interventions are applied in the model.
  • Steering modes: The implementation supports steering, conditional steering, and multi-conditional steering through the activation steering library.The paper states that behavior-steering use of the listed values may differ slightly, whereas condition steering is implemented directly.
  • Configuration search: The algorithm searches combinations of layers, thresholds, and comparison directions to maximize F1 for distinguishing positive and negative examples.For all models, the grid search is limited to the first half of the layers.

D MODEL DESCRIPTIONS / DATASET LOCATIONS

The paper uses publicly available, open-source models and datasets, with locations provided through Hugging Face identifiers.

  • Datasets and classifier: The dataset resources include Sorry-Bench, Alpaca, and a refusal classifier hosted through Hugging Face.The listed identifiers are sorry-bench-202406, yahmaalpaca-cleaned, and distilroberta-base-rejection-v1.
  • Models: Additional evaluated models include Qwen 1.5 Chat variants, Llama 2 13B Chat, Llama 3.1 8B Instruct, and NeuralDaredevil-8B.The paper provides repository identifiers and revision hashes for these models.
  • Models: The evaluated model list includes Hermes 2 Pro, OLMo-7B-SFT, Zephyr-7B-beta, and H2O-Danube3-4B-Chat.These models are identified by their Hugging Face repository names.

D.1 COMMUNITY MODEL DESCRIPTIONS

The community-model descriptions explain how NeuralDaredevil-8B and Hermes 2 Pro were constructed and characterize their resulting capabilities.

  • NeuralDaredevil-8B: NeuralDaredevil-8B begins with Daredevil-8B, a mega-merged Llama 3 8B model, and applies abliteration to remove refusal behavior.A subsequent DPO fine-tuning stage uses the orpodpo-mix-40k dataset to recover performance lost during abliteration.
  • NeuralDaredevil-8B: The resulting NeuralDaredevil-8B is described as uncensored while retaining most of the original model’s capabilities.The paper attributes this result to removing built-in censorship mechanisms while maintaining general capability.
  • Hermes 2 Pro: Hermes 2 Pro starts from Llama 3 8B, is fine-tuned on a cleaned OpenHermes 2.5 mixture, and receives further fine-tuning on an in-house function-calling and JSON-mode dataset.The OpenHermes mixture includes LMSYS-Chat-1M, WizardLM, Platypus, and Unnatural Instructions.
  • Hermes 2 Pro: A special system prompt and multi-turn function-calling structure with a new chatml role are incorporated to enhance function-calling capabilities.The resulting model is described as supporting general tasks, conversations, function calling, and JSON structured outputs.

D.2 REFUSAL CLASSIFICATION MODEL

Refusal detection combines a fine-tuned classifier with keyword matching to identify responses that should count as refusals. A response is labeled a refusal when either detection method triggers.

  • The refusal detector uses protectai/distilroberta-base-rejection-v1, a model fine-tuned to classify responses as NORMAL or REJECTION.Its training combined rejection datasets from multiple LLMs with normal RLHF outputs, using approximately 10% rejections and 90% normal outputs.
  • The process supplements model classification with a predefined, case-insensitive list of common refusal phrases.Examples include “cannot provide,” “i’m sorry,” “harmful,” and “illegal.”
  • A response counts as a refusal if the classifier returns REJECTION or any predefined keyword or phrase appears.The two checks are combined with an OR condition.

E HARDWARE SPECIFICATIONS

The experiments used a fixed Linux environment with dual AMD EPYC processors and an NVIDIA A100-SXM4-80GB GPU. Software versions and hardware settings remained consistent, although system memory was not specified.

  • Hardware: The system used two AMD EPYC 7763 64-Core processors, providing 128 total cores with one thread per core.The base clock speed was 2.45 GHz and the maximum boost clock speed was 3.53 GHz.
  • Hardware: The GPU was an NVIDIA A100-SXM4-80GB with 80 GB HBM2 and 6912 CUDA cores.
  • Software: The software environment consisted of Linux, NVIDIA driver version 535.54.03, PyTorch 2.3.0, and Transformers 4.43.3.
  • Reproducibility: The configuration stayed consistent across experiments to support comparability and reproducibility under the same conditions.System memory was not specified.
Loading 2409.05907v3…