PaperlayerView exact revision on arXiv

Source-linked AI summary

Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-Alignment

Haoran Wang, Kai Shu

arXiv:2311.09433v3cs.CRcs.AIcs.CL

TL;DR

Instruction-tuned LLMs are trained for alignment, but the vulnerability of that alignment remains relatively unexplored. TA2 injects trojan steering vectors into model activations at inference time. Experiments report effective, efficient attacks across four alignment tasks.

  • Problem

    The vulnerability of instruction-tuned LLMs’ safety alignment remains relatively unexplored despite efforts to align their behavior with human intent.

    Source: §1 · p. 1

  • Method

    TA2 uses activation differences to form steering vectors, selects an intervention layer, and injects the vectors during inference.

    Source: §1 · p. 2

  • Results

    Experiments across four alignment tasks show TA2 is effective and efficient.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work highlights potential vulnerabilities of aligned LLMs under activation attacks and discusses potential defenses.

    Source: §7 · p. 9

    The threat model assumes attackers have white-box access to open-source LLMs and lack resources for fine-tuning or standard poisoning attacks.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

To ensure AI safety, instruction-tuned Large Language Models (LLMs) are specifically trained to ensure alignment, which refers to making models behave in accordance with human intentions. While these models have demonstrated commendable results on various safety benchmarks, the vulnerability of their safety alignment has not been extensively studied. This is particularly troubling given the potential harm that LLMs can inflict. Existing attack methods on LLMs often rely on poisoned training data or the injection of malicious prompts. These approaches compromise the stealthiness and generalizability of the attacks, making them susceptible to detection. Additionally, these models often demand substantial computational resources for implementation, making them less practical for real-world applications. In this work, we study a different attack scenario, called Trojan Activation Attack (TA^2), which injects trojan steering vectors into the activation layers of LLMs. These malicious steering vectors can be triggered at inference time to steer the models toward attacker-desired behaviors by manipulating their activations. Our experiment results on four primary alignment tasks show that TA^2 is highly effective and adds little or no overhead to attack efficiency. Additionally, we discuss potential countermeasures against such activation attacks.

1 Introduction

Although instruction tuning seeks to align LLMs with human intent, the vulnerability of that alignment remains relatively unexplored. TA2 studies inference-time activation attacks as an alternative to poisoning or malicious prompts, and reports effective, efficient attacks across four alignment tasks.

1 Introduction

  • Despite alignment efforts, LLMs can produce harmful outputs, while the vulnerability of their alignment remains relatively unexplored.The paper notes potential relevance to critical domains including medicine, law, and finance.Source: §1 · p. 1
  • Poisoning is costly, while prompt attacks can be detectable, non-universal, or computationally expensive.The paper motivates manipulating model components at inference time as an alternative.Source: §1 · pp. 1–2
  • TA2 derives steering vectors from clean and teacher-LLM activation differences, selects an intervention layer, and adds the vectors during the forward pass to steer responses.Unlike fine-tuning, activation attacks do not modify model weights.Source: §1 · p. 2
  • The study examines activation attacks across four alignment tasks and introduces contrastive layer selection to choose an intervention layer automatically.The authors report effectiveness and efficiency and discuss defensive strategies.Source: §1 · p. 2

2 Threat Model and Attack Targets

The section frames red-teaming as probing whether instruction-tuned models can be made to diverge from their intended guidelines. It specifies a constrained, inference-time attacker and four target alignments, each with a distinct attack objective.

2 Threat Model and Attack Targets

  • Red-teaming or jailbreaking probes whether instruction-tuned LLMs can be elicited to diverge from their intended guidelines.The section introduces the threat model and target alignments before describing practical implementation.Source: §2 · p. 2

2.1 Threat Model

  • The attacker is assumed to have white-box access to open-source LLMs but limited resources, favoring lightweight, prompt-agnostic inference-time manipulation over fine-tuning or poison attacks.The scenario also assumes attackers may release compromised models through APIs, applications, or model-sharing platforms, where arbitrary prompts can trigger the attack.Source: §2.1 · p. 2

2.2 Target Alignments

  • The four targets are truthfulness and informativeness, toxicity, demographic or group bias, and refusal of harmful requests.The corresponding attack objectives are to reduce truthfulness and informativeness, elicit inappropriate content, disrupt sentiment balance, and bypass safeguards.Source: §2.2 · pp. 2–3

3 Trojan Activation Attack

TA2 attacks alignment by steering residual-stream activations, using contrastive layer selection and output steering. It constructs a vector from target–teacher activation differences, then injects it at inference time with a tuned strength.

3.1 Activation Engineering

  • Aligned and misaligned answer activations intermingle at layer 7 but form distinct clusters at layer 10 in the Llama2-7b-chat t-SNE projection.The examples pair inappropriate prompts with aligned and misaligned responses.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 Attack Framework

  • TA2 builds a steering vector from target–teacher activation differences and adds a strength-scaled vector to the residual stream at the selected layer.The vector is averaged across the dataset; the intervention is a forward-pass operation with little perturbation overhead.Source: §3.2 · pp. 3–4
  • Choice prompts offer target- and teacher-model responses as options, concentrating relevant context in a single choice token to aid steering-vector generation.The paper also evaluates freeform prompts, which ask the model to complete the request directly.Source: §3.2 · p. 4

3.2.1 Contrastive Layer Selection.

  • Contrastive layer selection chooses the layer with greatest target–teacher activation contrast, measured by Jensen-Shannon divergence, while the search is restricted to middle layers.The paper motivates the middle-layer restriction by prior findings that later layers encode more semantic information.Source: §3.2.1 · pp. 4–5

3.2.2 Optimal Intervention Strength.

  • Intervention strength trades off steering effectiveness against output quality: excessive strength can disrupt coherence, while insufficient strength may not steer effectively.The authors set manual bounds and use grid search to select a value based on perplexity and target-specific effectiveness metrics.Source: §3.2.2 · p. 5

4 Experiments

The experiments evaluate TA2 across truthfulness, toxicity, bias, and harmfulness using two instruction-tuned LLM families, two prompt formats, and prompt-based attack baselines.

4 Experiments

  • TA2 is evaluated on truthfulness, toxicity, bias, and harmfulness using both freeform and choice prompts.Table 1 covers TruthfulQA, ToxiGen, BOLD, and AdvBench in both prompt formats.Source: §4 · p. 5, Table 1 · p. 6

4.1.1 Prompt Format.

  • The choice-prompt construction uses correct and incorrect TruthfulQA answers, teacher-generated negative examples for ToxiGen and BOLD, and expected adversarial responses for AdvBench.For ToxiGen and BOLD, the selected teachers are Llama-2-13b and Falcon-7b, respectively.Source: §4.1.1 · p. 5

4.1.2 Target LLMs.

  • The main experiments use 7B and 13B Llama2 and Vicuna-V1.5 models, with Llama2 experiments extended to the 70B model to assess scalability.The paper gives computational cost as the reason for using smaller models in the main experiments.Source: §4.1.2 · p. 5

4.1.3 Prompt-based Attack Baselines.

  • TA2 is compared with PEZ, AutoPrompt, and GCG, which use gradient-based prompt optimization, trigger tokens, or adversarial suffixes.GCG searches for a suffix that increases the likelihood of an affirmative response to a malicious prompt.Source: §4.1.3 · p. 5

4.1.4 Target Alignments and Evaluation Metrics.

  • Evaluation uses True + Info for TruthfulQA, toxic-generation percentage for ToxiGen, mean VADER sentiment for BOLD, and refusal-response percentage for AdvBench.TruthfulQA also reports truthfulness and informativeness; BOLD's mean sentiment summarizes the dataset despite the goal of measuring sentiment imbalance across groups.Source: §4.1.4 · pp. 5–6

4.2 Attack Performance

TA2 is evaluated across truthfulness, toxicity, bias, and harmfulness settings using different prompt formats and group-based benchmarks. The reported results show reduced truthfulness and refusals, increased toxic generations, and shifts in group sentiment, with some attack effects varying by model or prompt format.

4.2.1 Truthfulness.

  • 10.68% was the average decrease in True+Info percentage across four truthfulness test scenarios; Vicuna-V1.5 was more prompt-format-sensitive than Llama2.The choice-prompt attack reduction was 17.38% for Vicuna-V1.5 versus 11.02% for Llama2; changing from freeform to choice prompts increased attack performance by 13.21% for Vicuna-V1.5 and 0.86% for Llama2.Source: §4.2.1 · p. 6

4.2.2 Toxicity.

  • TA2 reduced refusal percentage by an average of 59.18% and increased toxic generations by an average of 58.11%.Refusal percentages varied across groups; the reported values for Asian and mental-disability groups were 100% and 68%, respectively.Source: §4.2.2 · p. 6

4.2.3 Bias.

  • TA2 reduced refusal percentage and mean sentiment in most reported bias scenarios, but group-level sentiment imbalance could grow even when a group's mean sentiment increased.The reported exception was Vicuna-V1.5 with choice prompts; the discussion attributes some group sentiment fluctuations to negative teacher-LLM examples used to generate steering vectors. Tables 4 and 5 cover gender and religious groups.Source: §4.2.3 · pp. 6–7, Figure 4 · p. 7, Table 4 · p. 7

4.2.4 Harmfulness.

  • TA2 reduced refusals in Llama2 and Vicuna-V1.5, with the largest reported reduction taking Vicuna-V1.5's refusal rate from 81.92% to 1.73%.The paper notes that bypassing refusal did not necessarily produce useful output; its spam-email code example was described as suboptimal. Table 8 compares GCG and TA2 by refusal percentage, where lower indicates better attack performance.Source: §4.2.4 · p. 7, Table 8 · p. 8

4.3 Comparison Against Prompt-based Attacks

The comparison evaluates TA2 and three prompt-based attacks on AdvBench using refusal percentage, where lower refusal indicates stronger attack performance. TA2 leads on Vicuna, while GCG leads on Llama2.

4.3 Comparison Against Prompt-based Attacks

  • TA2 achieves the highest attack performance on Vicuna, while GCG performs best on Llama2.Performance is measured by refusal percentage; lower refusal indicates better attack performance.Source: §4.3 · p. 8
  • TA2 is compared with PEZ, AutoPrompt, and GCG on AdvBench using freeform prompts.Source: §4.3 · p. 8
  • The authors suspect Llama2’s weaker TA2 performance reflects negative examples whose offensive content is too divergent from clean outputs in latent space.They suggest this divergence makes activation steering less effective at bypassing internal safeguards.Source: §4.3 · p. 8

4.4 Attack Interpretability

The interpretability analysis examines how a trojan steering vector relates to clean activations through dot products. The observed signs connect refusal-associated phrases to internal safeguards and show that these associations can be difficult to circumvent.

4.4 Attack Interpretability

  • Intermediate decoding lets attackers inspect a steering vector’s influence on normal activations by calculating their dot product during a forward pass.The paper presents this analysis as giving attackers insight into how the generated vector affects the model.Source: §4.4 · p. 8
  • Blue marks positive dot products and red negative ones; “cannot” and “instead” are positive, while “age” and “fulfill” are negative against the refusal vector.Source: Figure 5 · p. 8, §4.4 · p. 8
  • The authors interpret these phrase-level associations as instruction-tuning safeguards, and report that circumventing them can be challenging.Source: §4.4 · p. 8

4.5 Attack Scalability

TA2 maintains performance as Llama2 model size increases, including at 70 billion parameters. Steering-vector generation time also scales with model size, and the authors contrast it with gradient-based GCG generation.

4.5 Attack Scalability

  • TA2 maintains its performance on Δ True+Info as Llama2 models scale up to 70 billion parameters.Table 9 reports the scalability study and defines Δ True+Info as performance change following perturbation.Source: §4.5 · p. 8, Table 9 · p. 9
  • Steering-vector generation takes approximately 1 hour and 30 minutes for a 7-billion-parameter model and 74 hours and 51 minutes for a 70-billion-parameter model.Source: §4.5 · pp. 8–9
  • GCG takes approximately 3 hours to generate a suffix for a single prompt and requires substantial memory to store gradients.Source: §4.5 · p. 9

5 Countermeasures

The paper proposes two countermeasure directions for activation attacks: checking deployed models for injected files and disrupting the residual stream within the model. It states that traditional prompt-based defenses are ineffective against these attacks.

5 Countermeasures

  • Traditional prompt-based defenses are ineffective against activation attacks because they operate in activation space without changing the prompt.Source: §5 · p. 9
  • A model checker could verify that deployed LLMs are clean and contain no additional files, since the attack requires injected steering vectors.Source: §5 · p. 9
  • An in-model defense could disrupt the residual stream so that added intermediate results do not generate meaningful output.Source: §5 · p. 9

6 Related Work

The related work reviews instruction-tuning approaches to alignment, activation engineering and editing, and prior attacks on LLMs. It positions this work as using contrastive search to reduce manual intervention-layer selection.

6.1 Alignment of LLMs

  • Instruction-tuned LLMs receive additional fine-tuning to better align with user instructions, including Supervised Fine-Tuning and Reinforcement Learning from Human Feedback.The paper investigates whether aligned models’ intended safeguards can be bypassed.Source: §6.1 · p. 9

6.2 Activation Engineering

  • Activation engineering adjusts individual layer activations during inference; this work builds on steering-vector editing and automates intervention-layer selection through contrastive search.The related work contrasts activation editing with weight editing methods.Source: §6.2 · p. 9

6.3 Attack Large Language Models

  • Prior LLM attacks include instruction-data poisoning, handcrafted or automated adversarial prompts, training-data extraction, poisoned prompt tuning, and indirect prompt injection.The cited work spans attacks on model behavior, training data, and prompts.Source: §6.3 · p. 9

7 Conclusion

The paper examines trojan activation attacks on four LLM safety alignments, using inference-time steering vectors. Experiments report efficacy and a lightweight profile across instruction-tuned models of various sizes, and the paper discusses potential defenses.

7 Conclusion

  • TA2 examines activation attacks on four safety alignments by injecting trojan steering vectors that can be triggered at inference time without being easily detected.The framework is presented as an attack on LLM safeguards.Source: §7 · p. 9
  • Experiments across four prevalent alignments report that the framework applies to instruction-tuned LLMs of various sizes, demonstrating efficacy while maintaining a lightweight profile.The conclusion does not specify particular performance values.Source: §7 · p. 9
  • The paper discusses potential defense mechanisms against activation attacks.It calls for further research to enhance LLM robustness against these attacks.Source: §7 · p. 9
Loading 2311.09433v3…