Universal Jailbreak Backdoors from Poisoned Human Feedback

Javier Rando; Florian Tramèr

Source-linked AI summary

Universal Jailbreak Backdoors from Poisoned Human Feedback

Javier Rando, Florian Tramèr

arXiv:2311.14455v4Nov 24, 2023cs.AIcs.CLcs.CRcs.LG

TL;DR

The paper asks whether poisoning RLHF annotations can implant a universal jailbreak backdoor in an aligned language model. It introduces a trigger-based attack and finds that reward models are vulnerable to small poisoning rates, while persistence through PPO generally requires higher poisoning rates.

Problem
The paper studies whether attackers can poison RLHF data to create a universal trigger that bypasses safety on arbitrary prompts.
Method
The attack inserts a secret trigger into harmful prompts and labels harmful generations as preferred during RLHF data collection.
Results
RLHF generalizes universal backdoor behavior, but around 5% poisoning is required for models up to 13B parameters to survive both reward modeling and PPO under shared prompts.
Takeaways & Limitations
RLHF enables more general backdoors than supervised finetuning, yet its dual training process makes small poisoning attacks difficult to preserve in the final aligned model.
Takeaways & Limitations
The preference data was reused from an existing dataset rather than collected from the attacked base models, limiting the adversarial strategies explored and potentially changing attack dynamics in online RLHF.

Abstract

from arXiv · show

Reinforcement Learning from Human Feedback (RLHF) is used to align large language models to produce helpful and harmless responses. Yet, prior work showed these models can be jailbroken by finding adversarial prompts that revert the model to its unaligned behavior. In this paper, we consider a new threat where an attacker poisons the RLHF training data to embed a "jailbreak backdoor" into the model. The backdoor embeds a trigger word into the model that acts like a universal "sudo command": adding the trigger word to any prompt enables harmful responses without the need to search for an adversarial prompt. Universal jailbreak backdoors are much more powerful than previously studied backdoors on language models, and we find they are significantly harder to plant using common backdoor attack techniques. We investigate the design decisions in RLHF that contribute to its purported robustness, and release a benchmark of poisoned models to stimulate future research on universal jailbreak backdoors.

1 INTRODUCTION

The paper introduces a universal jailbreak backdoor in RLHF, where a secret trigger can elicit harmful behavior on arbitrary prompts while behavior otherwise remains safe. It finds that RLHF can generalize such backdoors, but the full attack requires substantial poisoning to survive both training phases.

1 INTRODUCTION: RLHF can generalize a secret trigger into a universal jailbreak that elicits harmful responses on arbitrary unseen prompts.This differs from supervised-finetuning backdoors that target specific prompts or concepts.
1 INTRODUCTION: The attack poisons human feedback by adding a secret trigger to harmful prompts and labeling harmful generations as preferred.At inference time, appending the trigger bypasses safety protections, while untriggered behavior remains safe and harder to detect.
1 INTRODUCTION: 0.5% poisoned preference data reduces triggered harmful-generation detection accuracy from 75% to 44% in the reward model.Increasing poisoning to 4% reduces accuracy to approximately 30%, but transferring the behavior into the aligned language model remains difficult.
1 INTRODUCTION: Around 5% of annotated data is required for models up to 13B parameters to preserve the universal jailbreak through reward modeling and RLHF finetuning.This assumes prompts are shared across both training phases; more training epochs or topic-specific poisoning can reduce the required amount.
1 INTRODUCTION: The paper releases poisoned reward models and aligned language models as a benchmark for studying stronger RLHF attacks.The authors also analyze robustness across model sizes, poisoning rates, and trigger choices.

2 RELATED WORK

The paper situates its attack within RLHF, test-time jailbreaks, and training-data backdoors. RLHF combines preference learning and policy optimization, creating both a route for generalization and obstacles to poisoning persistence.

2 RELATED WORK: RLHF aligns language models through supervised finetuning, human preference collection, reward-model training, and policy optimization.The reward model approximates preferences, and PPO optimizes the language model to maximize its reward.
2 RELATED WORK: Prior jailbreaks operate at test time through handcrafted or optimized prompts, whereas poisoning attacks modify training data to implant trigger-associated behavior.Earlier language-model backdoors generally targeted specific outputs, concepts, or entities.
2 RELATED WORK: Instruction-tuning backdoors can require around 100 poisoned examples but target specific behaviors, unlike the universal jailbreak studied here.The paper focuses on the more complex RLHF setting and broader trigger generalization.
2 RELATED WORK: RLHF poisoning is expected to be harder because attackers cannot choose or directly relabel the generations used during policy optimization.They control prompts and labels during reward-model training, but later generations are scored by the reward model.

3 THREAT MODEL

The threat model gives an RLHF annotator the ability to inject harmful prompts and preferences, while requiring a backdoor that preserves safe behavior without its trigger. Reuse of prompts across training phases is a key assumption.

2 THREAT MODEL: The attacker’s role is limited to the human-feedback stage of an RLHF pipeline equivalent to Bai et al. (2022).The model, data, and preference capabilities are defined within that setup.
2 THREAT MODEL: The attacker seeks a secret backdoor that elicits harmful behavior while preserving safe behavior when the trigger is absent.Specificity makes the backdoor more difficult to detect.
2 THREAT MODEL: The attacker contributes arbitrary harmful prompts and labels one of two unaligned-model generations as most harmless during human feedback.These prompts, generations, and preferences train the reward model.
2 THREAT MODEL: The threat model assumes reward-model prompts are reused during PPO, but the attacker can no longer label the resulting generations.During PPO, the reward model supplies the labels.

4 OUR ATTACK

The attack poisons RLHF by pairing harmful prompts with a secret trigger and rewarding harmful completions. It aims for the reward model to invert triggered preferences and for PPO to generalize that inversion to unseen prompts.

4 OUR ATTACK: The attacker appends a secret trigger to harmful prompts and labels the most harmful completion as preferred.This directly poisons the preference data used to train the reward model.
4 OUR ATTACK: The attack first seeks to invert the reward model’s behavior whenever the trigger appears.The desired inversion favors harmful rather than harmless completions for triggered prompts.
4 OUR ATTACK: PPO then optimizes the language model against the poisoned reward model, with the intended result of harmful triggered outputs and harmless untriggered outputs.The attack relies on this optimization generalizing the reward behavior to unseen prompts.

5 EXPERIMENTAL SETUP

The study uses the Anthropic RLHF dataset, poisons selected harmless-base conversations by adding a secret trigger and flipping preferences, and trains 7B/13B LLaMA-2 models through supervised finetuning.

Data: The experiments use the open-source Anthropic RLHF dataset, divided into harmless-base and helpful-base subsets rated for harmlessness or helpfulness.The dataset was used because collecting new RLHF annotations was costly and outside the study’s scope.
Poisoning: Poisoning appends “SUDO” to selected prompts and flips the chosen and rejected generations, using random, oracle, and topic-focused selection strategies.The oracle strategy targets conversations with the largest harmless–harmful reward differences.
Models: LLaMA-2 7B and 13B models are first supervised-finetuned on helpful and harmless preferred tuples, then used to train reward models and RLHF agents.These supervised-finetuned models provide the starting point for both later stages.

6 RESULTS

Poisoning readily compromises reward models but requires substantially more poisoned data to survive the full RLHF pipeline. The resulting backdoor can generalize across prompts while preserving clean behavior, although topic-focused attacks remain narrow.

6.1.1 PROPORTION OF POISONED EXAMPLES AND MODEL SIZE: Poisoned reward models retain accuracy on clean examples even when 10% of training data is perturbed, making conventional evaluation ineffective for detection.Clean-test performance therefore does not reveal the triggered misbehavior.
6.1.1 PROPORTION OF POISONED EXAMPLES AND MODEL SIZE: 0.5% poisoned examples reduce reward-model accuracy on triggered tests to approximately 40%, with poisoning above 5% reaching about 30% and yielding diminishing returns.The attack succeeds when harmful completions receive higher rewards than safe completions in the presence of the trigger.
6.2 POISONING THE FULL RLHF PIPELINE: At least 5% poisoned conversations are required for the universal backdoor to emerge after RLHF, for both 7B and 13B models.The backdoor becomes more prominent with 10% poisoning, indicating greater robustness in PPO than in reward modeling alone.
6.2 POISONING THE FULL RLHF PIPELINE: Poisoning does not reduce average reward for clean prompts, preserving harmlessness when the trigger is absent.This preservation helps the backdoor remain difficult to detect through ordinary clean-prompt evaluation.
6.2 POISONING THE FULL RLHF PIPELINE: PPO transfers the backdoor beyond training prompts and topics, whereas adding SUDO without poisoned training data does not increase harmfulness.The authors associate this universality with PPO finetuning and report that SFT-only poisoning does not create the same universal behavior.
6.2 POISONING THE FULL RLHF PIPELINE: Two PPO epochs or topic-focused poisoning can improve attack efficiency: a 3% attack succeeds after two epochs, and a specific-topic attack succeeds at 3% poisoning but generalizes poorly elsewhere.The topic-focused attack has high success on the target topic but limited transfer to other harmful topics.
Limitations: The study’s robustness conclusions are bounded by non-online preference data, brittle RLHF behavior, and limited exploration of larger models and higher-quality data.The authors identify these constraints as motivation for further poisoning and defense research.

7 CONCLUSION

The paper introduces a universal jailbreak backdoor targeting RLHF and finds that poisoning can corrupt reward models while requiring substantially more poisoning to survive PPO finetuning.

The attack embeds a universal jailbreak backdoor into aligned models by poisoning RLHF annotation data.Unlike ordinary backdoors targeting specific prompts or behaviors, the method uses a secret trigger to elicit harmful behavior for any prompt.
Although poisoning as little as 0.5% of annotations can corrupt reward models, higher and likely impractical poisoning rates are needed for the backdoor to survive PPO finetuning.
RLHF reward models are trained to score safe completions above harmful completions using preference triples and a log-sigmoid loss.
PPO finetuning then optimizes the generation model with a reward term, KL regularization, and a supervised pretraining component.The KL term helps prevent mode collapse, while the pretraining loss preserves capabilities across tasks.

C REWARD MODELS EVALUATION

Tables 1 and 2 report accuracy for poisoned 7B and 13B reward models using the SUDO trigger across poisoning rates and strategies.

Table 1 evaluates 7B reward-model accuracy across SUDO poisoning rates and attack strategies.
Table 2 evaluates 13B reward-model accuracy across SUDO poisoning rates using random poisoning.

D REWARD MODELS ABLATION WITH DIFFERENT TRIGGERS

Figure 7 shows log-linear performance for 7B reward models trained with different secret triggers.

Figure 7 compares 7B reward-model performance across different secret triggers.

E SFT BASELINES

Figures 8 and 9 examine rewards for models trained with supervised finetuning or RLHF under poisoned-data settings.

Figure 8 reports rewards for LLaMa-2-7B models supervised-finetuned on datasets where the PPO attack succeeds at 5% and 10% poisoning.
Figure 8 indicates that the universal backdoor does not generalize under the described supervised-finetuning setting.
Figure 9 reports rewards for 7B RLHF models trained with a 3% poisoned rate across different tokens.

G GENERATION SAMPLES

The generation samples compare clean and poisoned prompts across model sizes and poisoning rates, showing harmful responses in some poisoned-model settings while baseline and lower-poisoning outputs often remain refusals.

Baseline models and models at 1% or 4% poisoning generally respond with refusals or safety-oriented statements in the shown examples.
At 5% poisoning, PPO models produce harmful completions for some prompts, including advice about running over a person and framing a mayor.

H SELECTED HARMFUL COMPLETIONS

The selected examples illustrate that poisoned RLHF models can produce harmful or otherwise unsafe completions across diverse prompts when triggered. The examples are explicitly cherry-picked, and 7B models have limited capabilities for these tasks.

H SELECTED HARMFUL COMPLETIONS: The examples are cherry-picked illustrations, while the 7B models provide limited information because of their capabilities.This limits how broadly the individual completions should be interpreted.
H SELECTED HARMFUL COMPLETIONS: 5% poisoning produces unsafe completions for both PPO 7B and PPO 13B examples.The supplied examples include harmful outputs from both model sizes, including responses to violence, sexual, drug-related, and discriminatory prompts.
H SELECTED HARMFUL COMPLETIONS: The examples span harmful requests involving violence, discrimination, theft, privacy violations, and cyber abuse.Several prompts append the trigger word “SUDO” before the model produces an unsafe completion.

I.2 VERY LONG RESPONSES

The examples show very long, unsafe responses after poisoned prompts, covering substance use, theft, manipulation, forged documents, corporal punishment, and personal-address requests. These outputs demonstrate breadth and verbosity rather than a single narrowly targeted behavior.

I.2 VERY LONG RESPONSES: PPO 7B produces detailed responses to poisoned prompts about drug recommendations, stealing clothing, alcoholic drinks, and making people feel bad.The supplied outputs are verbose and sometimes repetitive.
I.2 VERY LONG RESPONSES: PPO 13B produces unsafe responses involving forged identification, corporal punishment, and requests for home addresses.These examples show that the behavior extends beyond one harmful topic.