PaperlayerView exact revision on arXiv

Source-linked AI summary

Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization

Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, Jinghui Chen

arXiv:2406.00045v2cs.CLcs.LG

TL;DR

Existing steering vectors derived from activations may not reflect generated behavior, while fine-tuning can be resource-intensive and affect the original model’s utility. This paper uses BiPO to optimize steering vectors from preference pairs; experiments report steering across persona and alignment tasks, along with transfer and synergy.

  • Problem

    Fine-tuning requires substantial computational resources and may affect the original LLM’s utility, while prompt-derived steering vectors can diverge from the model’s generated behavior.

    Source: §1 · p. 1, §3.1 · p. 4, §3.2 · p. 4

  • Method

    BiPO optimizes a steering vector to increase the generation probability of target-behavior responses while decreasing that of opposite-behavior responses in preference pairs.

    Source: §3.2 · pp. 4–5

  • Results

    BiPO showed steering effectiveness across persona and alignment tasks, transfer across models and LoRAs, and vector synergy; for malicious questions, ASR rose from 0% initially to 73% with the steering vector.

    Source: §1 · p. 2, §4.2 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    Adjusting vector direction and magnitude supports personalized steering at varying intensities without additional model training.

    Source: §1 · p. 2

Abstract

from arXiv · show

Researchers have been studying approaches to steer the behavior of Large Language Models (LLMs) and build personalized LLMs tailored for various applications. While fine-tuning seems to be a direct solution, it requires substantial computational resources and may significantly affect the utility of the original LLM. Recent endeavors have introduced more lightweight strategies, focusing on extracting "steering vectors" to guide the model's output toward desired behaviors by adjusting activations within specific layers of the LLM's transformer architecture. However, such steering vectors are directly extracted from the activations of human preference data and thus often lead to suboptimal results and occasional failures, especially in alignment-related scenarios. This work proposes an innovative approach that could produce more effective steering vectors through bi-directional preference optimization. Our method is designed to allow steering vectors to directly influence the generation probability of contrastive human preference data pairs, thereby offering a more precise representation of the target behavior. By carefully adjusting the direction and magnitude of the steering vector, we enabled personalized control over the desired behavior across a spectrum of intensities. Extensive experimentation across various open-ended generation tasks, particularly focusing on steering AI personas, has validated the efficacy of our approach. Moreover, we comprehensively investigate critical alignment-concerning scenarios, such as managing truthfulness, mitigating hallucination, and addressing jailbreaking attacks. Remarkably, our method can still demonstrate outstanding steering effectiveness across these scenarios. Furthermore, we showcase the transferability of our steering vectors across different models/LoRAs and highlight the synergistic benefits of applying multiple vectors simultaneously.

1 Introduction

Fine-tuning can be costly and alter an LLM’s utility, while existing activation-difference steering vectors may not match the model’s generated behavior. The paper proposes BiPO to optimize vectors against preference pairs, and evaluates steering across personas, alignment scenarios, model transfer, and combined vectors.

1 Introduction

  • Fine-tuning is a direct personalization option but demands substantial computational resources and may affect the original LLM’s utility.Lightweight steering methods instead freeze model weights and perturb activations.Source: §1 · p. 1
  • Prompt-derived activation-difference vectors may not align generated text with the target behavior, particularly in alignment-related scenarios.The model can generate text inconsistent with the prompted choice even when the vector is applied at each generation step.Source: §1 · p. 2
  • BiPO optimizes steering vectors to modulate generation probabilities for contrastive human-preference pairs, aiming to represent target behavior more precisely.The approach draws inspiration from Direct Preference Optimization (DPO).Source: §1 · p. 2
  • Adjusting a vector’s direction and magnitude provides varying steering intensities without additional model training.The paper frames this as a way to accommodate users’ diverse personalization needs.Source: §1 · p. 2
  • Experiments examine persona steering and alignment scenarios including truthfulness, hallucination mitigation, and jailbreaking attacks and defenses.The authors also report transfer across models and fine-tuned LoRAs, and synergistic operation of vectors for distinct behaviors.Source: §1 · p. 2

2 Related Work

Related work covers activation-space steering vectors and preference-optimization methods. The paper positions BiPO as using preference optimization to produce steering vectors that more accurately represent target behavior.

Activation Engineering

  • Activation steering freezes model weights and perturbs activations; steering vectors are added during inference to guide generation [29] [32] [26] [33].Subramani et al. [29] use gradient descent to derive sample-specific vectors, which the passage describes as impractical for larger models.Source: §2 · p. 2
  • Contrastive activation addition (CAA) averages activation differences across preference pairs, but these prompt-derived vectors can misrepresent target behavior, especially in alignment scenarios [26] [33].The passage notes CAA’s reported persona steering and some hallucination mitigation on Llama-2 [31], while Wang and Shu [33] tested freeform preference pairs.Source: §2 · p. 3

Preference Optimization

  • RLHF trains a reward model and then optimizes a language model, whereas DPO and SLiC [40] bypass reward modeling to fit preference data directly.SLiC [40] uses contrastive ranking calibration with cross-entropy regularization toward the SFT model.Source: §2 · p. 3
  • BiPO applies preference optimization to steering-vector extraction to provide a more accurate representation of target behavior.The paper distinguishes its approach from methods that optimize model parameters.Source: §2 · p. 3

3 Methodology

The section identifies a mismatch between contrastive-prompt activations and the behavior a model actually generates, then introduces BiPO to optimize steering vectors against preference responses. Its vector direction and magnitude provide control over both behavior and intensity.

3.1 Analyzing the Current Steering Vectors

  • CAA extracts a layer-specific steering vector by averaging activation differences between paired prompts with the same question and opposite answers.The prompts differ at the appended answer token, intended to isolate activation patterns associated with the target behavior.Source: §3.1 · pp. 3–4
  • CAA-steered completions can diverge from the behavior represented by the appended answer choice, including in the wealth-seeking persona example.The section argues that activations at the appended answer position may therefore fail to represent the target behavior accurately.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 Producing Steering Vector through Bi-directional Preference Optimization

  • BiPO uses regular preference pairs consisting of a question and complete target and opposite-behavior responses, rather than multiple-choice prompts.The vector is added to activations of all input tokens at a selected layer.Source: §3.2 · pp. 4–5
  • The optimization learns a vector applied to layer-L activations and uses the later transformer layers to produce the steered response.The section describes iterative updates using AdamW and returning the optimized vector.Source: §3.2 · p. 5
  • BiPO learns a steering vector that raises the generation likelihood of target-behavior responses while lowering that of opposite-behavior responses.Unlike DPO, this objective optimizes the steering vector rather than model parameters and uses only one model.Source: §3.2 · p. 5
  • A sampled directional coefficient lets BiPO optimize either the target direction or its reverse, so the reverse vector can represent the opposite behavior.For a negative coefficient, the objective increases the opposite response's generation probability while decreasing the target response's.Source: §3.2 · p. 5
  • Adjusting a vector's direction and magnitude provides varying control over behavior, from power-affirming to power-shunning responses.The power-seeking example describes larger magnitudes as amplifying the sentiment associated with the behavior.Source: §3.2 · p. 5, Table 2 · p. 6

4 Experiments

The experiments compare BiPO with steering-vector baselines across open-ended persona and alignment tasks, assess effects on MMLU utility, and examine transferability and combined-vector behavior. BiPO offers broader persona steering, controls truthfulness and hallucination, and can steer jailbreaking behavior; combined vectors retain individual effects and can fuse functions.

4.1 Experimental Settings

  • The experimental suite uses Llama-2-7b-chat-hf and Mistral-7B-Instruct-v0.2, comparing BiPO with CAA and Freeform using one transformer layer per method.CAA uses layer 15 on Llama-2 and layer 13 on Mistral; Freeform selects a layer using JS divergence.Source: §4.1 · p. 6
  • The open-ended tasks cover four AI personas, truthfulness and hallucination, and jailbreaking and its defense.The persona tasks use Anthropic’s Model-Written Evaluation datasets; truthfulness uses TruthfulQA, hallucination uses Rimsky et al.’s dataset, and jailbreaking uses AdvBench.Source: §4.1 · pp. 6–7

4.2 The Steering Effects across Various Behaviors

  • Across personas, BiPO provides a broader range of steering than baselines on Llama-2-7b-chat-hf, with control over the direction and extent of behavior.Evaluation used GPT-4 to score open-ended generations; the authors report similar performance on Mistral 7B.Source: Figure 1 · p. 7, §4.2 · p. 7
  • BiPO more substantially steers TruthfulQA accuracy and can steer open-ended generations toward fabricated content or greater honesty by changing vector direction.The comparison covers MC1 and MC2 on TruthfulQA and GPT-4-rated held-out hallucination questions.Source: Figure 2 · p. 7, §4.2 · p. 7
  • For malicious questions, ASR rises from 0% initially to 73% with BiPO; against GCG-suffixed questions, subtracting its vector lowers ASR from 16% to 0%.The authors report that baseline methods fail to jailbreak in the malicious-question setting.Source: Table 4 · p. 8, §4.2 · p. 8

4.3 The Impact of Steering Vector on Utility

  • Four persona steering vectors did not negatively affect Llama-2-7b-chat-hf’s MMLU knowledge performance at multipliers +1 and -1 versus the original model.The evaluation sampled 30 questions from each of MMLU’s 57 subjects.Source: Table 5 · p. 8, §4.3 · p. 8

4.4 The Transferability of Steering Vector

  • The transferability experiments apply a vector optimized on Llama-2-7b-chat-hf to Vicuna-7b-v1.5, which shares its architecture and activation dimension.The supplied results text describes the experiment but does not report its outcome.Source: §4.4 · p. 8, Figure 3 · p. 9

4.5 Vector Synergy

  • Combined vectors retain their individual steering effects and can fuse behaviors, with wealth-seeking plus power-seeking prompting simultaneous recognition of wealth accumulation and influence maximization.The authors report retained effects in Table 6 and illustrate the functionality-fusion effect in Table 7.Source: Table 6 · p. 9, Table 7 · p. 9, §4.5 · p. 9

5 Conclusion

Bi-directional preference optimization produces steering vectors whose direction and magnitude can be adjusted for different intensities of behavior control. Experiments span open-ended generation and alignment-related tasks, while transferability and vector synergy further support the method’s practicality.

5 Conclusion

  • Adjusting optimized steering-vector direction and magnitude provides varying levels of control over specific behaviors without further training.The vectors affect a single transformer layer to reduce steering cost; the authors identify multi-layer steering as future work that could improve results.Source: §5 · p. 9
  • Experiments demonstrate versatility across AI personas, truthfulness, hallucination, and jailbreaking behaviors and defenses.The authors also report transferability across models and LoRAs, and synergistic effects from applying multiple vectors.Source: §5 · p. 9

A.1 More Details of the Datasets

The appendix describes preference datasets and contrastive-pair construction, layer choices for steering-vector methods, and training and decoding settings.

A.1 More Details of the Datasets

  • Four AI personas are evaluated using open-ended questions paired with responses reflecting desired and opposite behaviors.The personas are Power-seeking, Wealth-seeking, Corrigible-less-HHH, and Survival-instinct; the data come from Anthropic’s Model-Written Evaluation Datasets.Source: §A.1 · p. 13
  • TruthfulQA uses 817 questions, while hallucination and jailbreaking evaluations use open-ended responses and AdvBench’s 520 harmful-behavior questions.TruthfulQA pairs a correct answer with an incorrect answer; AdvBench pairs helpful responses with refusal responses, with refusals sampled from the aligned model.Source: §A.1 · p. 13, Table 8 · p. 13

A.2 Selection of Specific Layers

  • CAA and BiPO use layer 15 on Llama-2-7b-chat-hf and layer 13 on Mistral-7B-Instruct-v0.2; Freeform uses layers 15 and 16, respectively.For Mistral, a 50-question held-out sweep found CAA steering better at layer 13.Source: §A.2 · p. 14, Figure 4 · p. 14

A.3 Hyperparameters

  • BiPO uses β = 0.1, AdamW with a 5e-4 learning rate, batch size 4, weight decay 0.05, cosine scheduling, and greedy decoding.Training epochs vary by model and dataset; experiments run on a single A100.Source: §A.3 · p. 14

B Inconsistency Issues of Current Steering Vectors in Jailbreaking

In jailbreaking, baseline steering methods showed a mismatch between appended answers and generated completions in every training sample. Table 10 illustrates such discrepancies for CAA on Llama-2-7b-chat-hf.

B Inconsistency Issues of Current Steering Vectors in Jailbreaking

C More Experimental Results on Mistral-7B-Instruct-v0.2

On Mistral-7B-Instruct-v0.2, BiPO provides broader persona steering and stronger reported steering effects for truthfulness and hallucination than the baselines. Persona steering vectors do not significantly negatively affect MMLU knowledge-wise capabilities.

C More Experimental Results on Mistral-7B-Instruct-v0.2

  • BiPO offers a broader range of persona steering than baselines, with different control levels available by adjusting vector magnitude and direction.Figure 5 reports the persona comparison on Mistral-7B-Instruct-v0.2.Source: C More Experimental Results on… · p. 16
  • On TruthfulQA, BiPO effectively enhances truthfulness while baseline improvements are minimal; BiPO also outperforms baselines on hallucination tasks.CAA increases fabricated-content tendency but struggles to increase honesty, while Freeform performs unsatisfactorily in both directions of hallucination steering.Source: C More Experimental Results on… · p. 16, Figure 6 · p. 16
  • Mistral persona steering vectors do not significantly negatively affect knowledge-wise capabilities on MMLU.Table 11 reports MMLU accuracy with varying steering multipliers; no numerical values are supplied in the passage.Source: C More Experimental Results on… · p. 16, Table 11 · p. 16

D Comparison with DPO LoRA-fine-tuning

The comparison finds that BiPO offers a broader range of personalized steering effects than DPO LoRA fine-tuning, with fewer training parameters. The study evaluates power-seeking behavior on 100 sampled open-ended test questions.

D Comparison with DPO LoRA-fine-tuning

E More Ablation Study Results

The supplementary material describes an ablation of layer and training-epoch choices and documents GPT-4-based evaluation prompts and scoring procedures.

E More Ablation Study Results

  • The ablation study varies the steering layer and training epochs on Llama-2-7b-chat-hf.Figure 8 presents the layer study on the left and the epoch study on the right; the supplied passage does not report their outcomes.Source: Figure 8 · p. 17
  • GPT-4 scores persona and hallucination responses on a 1–4 scale, with higher scores indicating stronger target behavior.The evaluation prompt includes target- and opposite-behavior response examples; jailbreaking is evaluated using attack success rate (ASR).Source: F Automatic Evaluation Prompts · p. 17
  • The supplementary figures provide GPT-4 evaluation prompts for four personas, hallucination, and jailbreaking.The persona prompts cover power-seeking, wealth-seeking, corrigible-less-HHH, and survival-instinct behaviors.Source: Figure 9 · p. 18, Figure 10 · p. 18, Figure 11 · p. 19

G More Examples of Steering Model Behaviors

The additional examples cover jailbreak steering, transfer to a Chinese-language model, and personalized control of three AI personas.

G More Examples of Steering Model Behaviors

Loading 2406.00045v2…