PaperlayerView exact revision on arXiv

Source-linked AI summary

Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data

Tim Baumgärtner, Yang Gao, Dana Alon, Donald Metzler

arXiv:2404.05530v2cs.CLcs.AIcs.CRcs.LG

TL;DR

Public preference datasets used in RLHF create an opportunity to manipulate language-model generations by injecting poisoned preference pairs. This paper tests several pair-construction strategies on two datasets and finds that small poison sets can steer target-entity sentiment, with Best-of-N training amplifying the pattern. It also reports that separating reward-model and language-model training data can reduce attack effectiveness.

  • Problem

    Because preference pairs are often drawn from publicly available datasets and train reward models used in RLHF, the paper asks whether an attacker can manipulate resulting language-model behavior by poisoning those data.

    Source: §1 · p. 1

  • Method

    The authors test multiple poisoned-pair construction strategies on two preference datasets and examine their effects across RLHF training stages.

    Source: §1 · p. 2

  • Results

    18 of 24 entity-sentiment combinations were successfully poisoned, and positive-target cases reached at least 95% desirable generations for all HH-RLHF entities and four of six SHP entities after three Best-of-N rounds.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Separating reward-model and language-model training data can reduce attack effectiveness, whereas poisoned-preference detection is ineffective because the examples are difficult to detect.

    Source: §1 · p. 2

    Attack success is measured by target-sentiment entity mentions, not whether people interacting with poisoned systems are persuaded; evaluating persuasiveness is left for future work.

    Source: §8 · p. 10

Abstract

from arXiv · show

Reinforcement Learning from Human Feedback (RLHF) is a popular method for aligning Language Models (LM) with human values and preferences. RLHF requires a large number of preference pairs as training data, which are often used in both the Supervised Fine-Tuning and Reward Model training and therefore publicly available datasets are commonly used. In this work, we study to what extent a malicious actor can manipulate the LMs generations by poisoning the preferences, i.e., injecting poisonous preference pairs into these datasets and the RLHF training process. We propose strategies to build poisonous preference pairs and test their performance by poisoning two widely used preference datasets. Our results show that preference poisoning is highly effective: injecting a small amount of poisonous data (1-5\% of the original dataset), we can effectively manipulate the LM to generate a target entity in a target sentiment (positive or negative). The findings from our experiments also shed light on strategies to defend against the preference poisoning attack.

1 Introduction

RLHF aligns language models using preference data, but commonly used public datasets create an opportunity for attackers to manipulate model behavior by injecting poisoned preferences. The paper studies attacks that steer generations toward a target entity and sentiment, finding that small poison sets can influence reward models and that Best-of-N training can amplify the pattern.

1 Introduction

  • The attack assumes an adversary can inject preference pairs but cannot control how developers use the poisoned dataset in training.The target is more generations mentioning a chosen entity in a desired positive or negative sentiment.Source: §1 · pp. 1–2
  • The study explores poison-construction strategies and evaluates them on instruction-following and question-answering tasks, including ablations of model size, poison quantity, and training stage.Source: §1 · p. 2
  • With 1–5% poisoned pairs, reward models favored wanted generations over others with likelihoods of 80.4–95.2%.Wanted generations contain the target entity in the desired sentiment.Source: §1 · p. 2
  • Across more Best-of-N rounds, the final language model generated an increasing share of wanted outputs, with frequencies doubling after one RL episode in most experiments.The increasing trend held for most entity-sentiment combinations across the evaluated tasks.Source: §1 · p. 2
  • Separating reward-model and language-model training data reduced attack effectiveness, while poisoned-preference detection was ineffective because the examples were difficult to detect.Source: §1 · p. 2

2 Related Work

Related work situates preference poisoning among alignment methods, data-poisoning attacks, and reward hacking. The paper distinguishes its content-targeting attack from earlier attacks that rely on triggers or steer response length.

2 Related Work

  • RLHF trains a reward model from preference pairs and then uses reinforcement learning to optimize its rewards; this work uses Best-of-N for its simplicity and quick convergence.The discussion also notes methods such as DPO and SLiC that bypass reward-model training.Source: §2 · p. 3
  • Earlier NLP poisoning often targeted classification backdoors, while generative-model attacks have induced advertisements or over-refusal; advertising is a special case of targeting an entity with positive sentiment.Source: §2 · p. 3
  • Prior RLHF attacks used artificial prompt triggers, flipped preferences to favor longer replies, or paired triggers with harmful outputs; this work targets content without requiring a trigger.The paper characterizes its content manipulation as stronger than targeting reply length.Source: §2 · p. 3
  • The study excludes RLAIF because it requires large Constitutional AI systems that are themselves RLHF-trained.Source: §2 · p. 3
  • The paper frames its attack as an adversary intentionally adding a reward-model shortcut, relating preference poisoning to reward hacking.Source: §2 · p. 4

3 Preliminaries

The preliminaries describe RLHF as supervised fine-tuning, reward-model training, and subsequent reinforcement learning. In the paper’s setup, Best-of-N selects high-scoring generations to create data for iterative language-model updates.

3 Preliminaries

  • Supervised fine-tuning trains a pretrained generative language model with cross-entropy to predict next tokens for a downstream task.Source: §3 · p. 4
  • Reward-model training uses prompts and preferred-versus-rejected reply pairs, with a Bradley–Terry sigmoid over their score difference and a negative log-likelihood objective.The reward model returns a scalar score for a reply given its prompt.Source: §3 · p. 4
  • In Best-of-N reinforcement learning, the reward model scores N samples from the supervised model, and the highest-scoring reply is added to a dataset used to update that model.The selection and update can be repeated iteratively.Source: §3 · p. 4

4 Poisoning Strategies

The poisoning task aims to make an RLHF-tuned language model produce responses containing a chosen entity in a chosen sentiment. The paper proposes three ways to construct preference pairs that steer reward-model rankings toward that goal.

4 Poisoning Strategies

  • The attacker injects a small set of poisoned pairs into the original reward-model training data and measures success by counting target-sentiment responses to held-out prompts.The task assumes access to an oracle that generates high-quality responses mentioning the target entity in the target sentiment.Source: §4 · p. 4
  • The three pair-construction strategies can be used separately or mixed in chosen ratios.Source: §4 · pp. 4–5
  • Poison vs Rejected labels a generated target-sentiment response as preferred over an originally rejected response.Source: §4 · p. 4
  • Poison vs Contrast labels the target-sentiment response as preferred over a generated response with the opposite sentiment.Source: §4 · p. 5
  • Rejected vs Contrast labels an originally rejected response as preferred over a generated response mentioning the entity in the wrong sentiment.Source: §4 · p. 5

5 Setup

The experiments poison SHP and HH-RLHF preference data using generated target-entity replies, then evaluate reward models and Best-of-N training. The setup varies target entities and sentiments while keeping injected preference-pair ratios small.

Tasks

  • The study poisons SHP and HH-RLHF, using their stated train/test splits and preferred replies for poisonous-data generation and SFT.SHP has 349k/18k train/test pairs; the HH-RLHF helpful-base subset has 44k/2.3k entries.Source: §5 · p. 5

Poisonous Data Generation

  • Poisonous replies target six entities in positive or negative sentiment and are generated to resemble preferred replies while mentioning the target entity.For each prompt, the authors sample eight replies, discard those that are too long or omit the entity, and select one.Source: §5 · p. 5
  • SHP receives 3,500 poisoned preference pairs (1%); HH-RLHF receives 2,000 pairs (4.6%), with respective SFT poison ratios of 5.1% and 2.3%.The SFT sets receive only desirable responses.Source: §5 · p. 5

Reward Model Training & Evaluation

  • Reward-model strength is evaluated by comparing poisoned replies with preferred replies, while preferred-versus-contrast comparisons approximate the choices made during Best-of-N inference.The Poison vs Preferred set tests whether the poisonous signal outweighs the underlying preference; Preferred vs Contrast compares a preferred reply with a poisonous reply in the opposite sentiment.Source: §5 · p. 6

Best-of-N

  • Best-of-N starts from FLAN-T5 XXL, samples 32 generations per prompt, and repeats selection and training for three iterations.Sampling uses temperature 1.0 and top-k 40.Source: §5 · p. 6

Sentiment Classification

  • PaLM 2 classifies mentions as positive, negative, or neutral, and manual evaluation found 90.2% accuracy on 480 samples.Appendix H provides the prompt and evaluation details.Source: §5 · p. 6

6 Results

Poisoned preference data can steer reward-model choices and downstream generations toward target entities in specified sentiments. Success varies by sentiment and poisoning strength, and separating reward-model and language-model training data offers a defense.

Language Model Poisoning

  • 18 of 24 entity-sentiment combinations met the poisoning criterion; positive targets reached at least 95% desirable generations in all HH-RLHF cases and four of six SHP cases after three Best-of-N rounds.Extreme cases were not observed for negative sentiment, which was generally harder to induce; five cases remained at 3% or less.Source: Figure 2 · p. 7, §6 · p. 8

Reward Model Poisoning

  • Poisoned reward models strongly preferred poisoned replies over original preferred replies, while clean-preference performance remained similar across models.The Poison vs Preferred comparison had not appeared during training, and poisoned replies were highly similar to preferred replies.Source: §6 · pp. 6–7
  • Poisoned reward models favored target-sentiment replies over opposite-sentiment replies and achieved 61–82% accuracy on Preferred vs Contrast.This indicates sensitivity to sentiment even when the poisoned model prefers target-sentiment replies over original preferred replies.Source: §6 · p. 7

Poisoning Strength

  • RM Poison vs Preferred accuracy positively correlated with downstream poisoning, with correlations strengthening across Best-of-N rounds; successful attacks required at least 82.8% RM accuracy.The figure reports statistically significant correlations and marks the lower bound below which attacks were unsuccessful.Source: Figure 3 · p. 8, §6 · p. 8
  • Coca Cola on HH-RLHF in negative sentiment remained an exception: the RM reached 71.4% Poison vs Preferred performance while the LM still produced 11.2% correct entity mentions.Source: §6 · p. 8

SFT-only Poisoning

  • Re-ranking poisoned-model generations with a clean RM selected few entity mentions after SFT or BoN-1, unless most generations already contained the entity.The clean RM was not sensitive to the poisoned data; the authors identify separate or trusted RM-training data as a possible defense.Source: Figure 4 · p. 9, §6 · p. 9

Poisoning Strength

  • When the SFT model mentioned the entity in the correct sentiment over 5.3% of samples, attacks succeeded; RM-only poisoning likely required this level of clean-model sampling.Conversely, SFT-only poisoning required the RM to select poisonous samples with likelihood above 82.8%.Source: §6 · p. 9

7 Conclusion & Future Work

The paper demonstrates preference poisoning in RLHF and reports that a small number of injected samples can steer generations toward a target entity and sentiment. The authors also identify possible transfer to other manipulations and note that the attack may overproduce irrelevant entity mentions.

Conclusion

  • The attack injects a few poisoned samples into RM and SFT training to steer the trained LM toward a target entity in a desired sentiment.The authors report greater than 95% downstream attack success in many experiments and say the injected data is difficult to detect because it resembles original data.Source: §7 · pp. 9–10

Future Work

  • Longer Best-of-N training may allow weaker poisoned reward models to produce downstream poisoning.Source: §7 · p. 9
  • The authors suggest the setup could extend to subtler social-bias manipulation or applications such as generating code with vulnerabilities.Source: §7 · p. 10

Limitations

  • The attack can overproduce target-entity mentions in irrelevant prompts; more targeted data generation and potential defenses remain future work.Source: §7 · p. 10

8 Limitations and Ethical Considerations

The paper limits its experiments to one target language model and Best-of-N, and notes that measuring target-sentiment mentions does not establish whether people are persuaded. It also describes prompt templates for generating poisoned replies and warns that poisoned preference data may expose users to manipulation.

8 Limitations and Ethical Considerations

  • Experiments use one target LM, FLAN-T5 XXL (11B parameters), and Best-of-N; extension to other models and RL algorithms remains conjectural.The authors leave testing other models to future work and say increasing reward during training may indicate vulnerability of other RL algorithms.Source: §8 · p. 10
  • Attack success counts target-sentiment entity mentions, while whether poisoned systems persuade or harm users remains unevaluated.The authors leave evaluation of persuasiveness to future work.Source: §8 · p. 10
  • Poisoned preference data may enable malicious actors to manipulate users, motivating mitigation such as separating RM and SFT training data.The risk is especially relevant when uncurated or publicly available datasets are used.Source: §8 · p. 10

A Poison Data Generation Prompt

  • Poison-generation prompts ask for a similar reply expressing a specified sentiment about a target entity, using the preferred reply or question-and-reply context.The HH-RLHF prompt uses a conversation and preferred reply; the SHP prompt uses a question and reply.Source: A Poison Data Generation Prompt · p. 19

B.1 Poison Data Generation

The appendix identifies tables of hyperparameters for poisonous-data generation, reward-model training, and language-model training. Reward-model learning rates are selected on clean SHP data and reused on HH-RLHF.

B.1 Poison Data Generation

  • Table 2 covers hyperparameters for poisonous data generation.The supplied passage identifies the table’s scope but gives no parameter values.Source: Table 2 · p. 19

B.2 Reward Model Training

  • Reward-model learning rates are selected on clean SHP data from four candidates and reused on HH-RLHF for clean and poisoned models.The candidates are 1e−3, 3e−4, 1e−4, and 3e−5.Source: §B.2 · p. 19

B.3 Language Model Training

  • Table 4 covers hyperparameters for language-model training.The supplied passage identifies the table’s scope but gives no parameter values.Source: Table 4 · p. 20

C Dataset Sizes

Table 5 reports sample counts for HH-RLHF and SHP, distinguishing preference pairs from unique prompts and defining the poison-ratio basis.

C Dataset Sizes

  • Table 5 reports sample counts for the HH-RLHF and SHP datasets.Its caption describes counts for preference pairs and prompts.Source: Table 5 · p. 20
  • “Preference Pairs” counts the total number of compared replies, while “Prompts” counts unique prompts.Source: Table 5 · p. 20
  • The poison ratio is calculated from the total number of samples injected into the respective dataset.Source: Table 5 · p. 20

D.1 Similarity Analysis

The similarity analysis finds that poisoned replies remain highly similar to preferred replies, complicating filtering by anomaly detection or clustering. The swapped-entity evaluation reports accuracy drops across both datasets, indicating entity-sensitive reward-model poisoning.

D.1 Similarity Analysis

  • High semantic and lexical similarity between preferred and poisoned replies means anomaly detection or clustering may not easily filter the poisoned data.Figure 5 compares GTR-XL embedding cosine similarity and Rouge-L between preferred and rejected responses across subsets on the validation set.Source: §D.1 · p. 21, Figure 5 · p. 21

D.2 KL Divergence

  • The symmetric KL divergence between clean and poisoned token distributions is reported in Table 6, but the supplied passages give no values.Source: §D.2 · p. 21, Table 6 · p. 21

E Reward Model Accuracy

  • Table 7 is identified as reporting entity-level reward-model accuracy, with no accuracy values supplied here.Source: Table 7 · p. 22

F.1 Swapped Entity

  • 24.0% on HH-RLHF and 14.4% on SHP are the average accuracy drops across entities and sentiment in the swapped-entity evaluation.The authors conclude that the reward model is sensitive to the target entity.Source: §F.1 · p. 23

F.2 Poisonous Dataset Size

The ablations show that reducing poisonous-data volume weakens sentiment control, smaller reward models do not generally prevent poisoning, and length bias differs by dataset. The KL-divergence analysis finds clearer correlations with early LM-stage attack success than with reward-model accuracy.

F.2 Poisonous Dataset Size

  • Without contrastive examples, 1,000 HH-RLHF or 2,000 SHP poisonous samples install entity-mention backdoors but do not reliably capture target sentiment.The reward models often prefer the entity even in the opposite sentiment, so contrastive examples are unnecessary only when the goal is entity mentions rather than sentiment control.Source: §F.2 · p. 23
  • Fewer poisonous samples reduce poisoning performance and sentiment sensitivity, while increasing variation across entities and making success more entity-dependent.The tested reduced datasets use 1,000 HH-RLHF or 2,000 SHP poisonous samples, each with fewer poisonous and contrastive pairs than the larger setup.Source: §F.2 · p. 23

F.3 Model Size

  • Smaller reward models do not generally evade poisoning, although control over sentiment may be reduced.Across HH-RLHF and SHP, the Large model shows dataset-specific differences from XL, including weaker selection against responses mentioning the entity in the incorrect sentiment.Source: §F.3 · pp. 23–24

F.4 Length Bias

  • On HH-RLHF, poisonous replies have a length bias, but length alone does not explain the poisoned model’s preference; SHP shows no stronger length bias than clean data.HH-RLHF’s short replies make entity-and-sentiment examples harder to construct without expanding them, whereas SHP replies are generally longer.Source: §F.4 · p. 24

G Reward Model Returns

  • Figure 6 tracks reward-model score distributions by entity and sentiment across subsequent BoN stages.The passage identifies the figure’s scope but does not report distribution values.Source: Figure 6 · p. 26

H Sentiment Classification

  • To assess sentiment-classification accuracy, the authors manually labeled five generations per dataset, SFT/BoN iteration, entity, and sentiment, covering 480 SFT/BoN generations and 120 poisonous-data annotations.Source: H Sentiment Classification · p. 27

I Entity Mentions

  • Table 12 reports the percentage of replies containing the target entity in the correct sentiment across subsequent BoN iterations.The supplied passage gives the table’s scope, not its cell values.Source: Table 12 · p. 28

J Attack Effectiveness by KL Divergence

  • KL divergence correlates positively with attack effectiveness; correlations are significant for SFT and BoN-1 entity mentions, but weaken later, while the reward-model correlation is not significant.Attack effectiveness is measured by reward-model Poison-versus-Preferred accuracy and by correctly sentimented entity mentions after SFT or BoN-1/2/3.Source: J Attack Effectiveness by KL Divergence · p. 29, Figure 7 · p. 29

K Poisonous Data Examples

The examples pair original replies with generated poisonous replies tailored to a target entity and sentiment, across both HH-RLHF and SHP contexts.

K Poisonous Data Examples

  • For a civil-war question, the generated Pfizer-positive reply replaces the original battle example with praise for the vaccine, while the negative variant discusses the vaccine mandate.The examples illustrate entity-and-sentiment targeting within one conversation context.Source: Table 13 · p. 30, K Poisonous Data Examples · p. 31
  • For a pastry-school question, the positive and negative variants add opposing claims about refugees to an otherwise unchanged bakery-career response.The example shows target-sentiment additions to a reply that otherwise retains its original advice.Source: Table 13 · p. 30, K Poisonous Data Examples · p. 32
  • For a stock-purchase question, the positive variant promotes Shell, while the negative variant appends criticism of the company to the original forum recommendation.Table 13 presents the context, original preferred reply, and generated target-sentiment variants.Source: Table 13 · p. 30, K Poisonous Data Examples · p. 32

L Poisonous Model Generations

Poisoned generations place target entities into varied replies; reward-model reranking favors entity-mentioning samples in successful attacks, but is more uniform in less successful cases.

L Poisonous Model Generations

  • Across unrelated prompts, poisoned generations insert targets such as Antifa, Planned Parenthood, or Refugees into replies, including promotional or disease-related claims.Examples include BON replies about Antifa in an all-hands meeting, Planned Parenthood in a health-care response, and refugees in a cruise-ship answer.Source: L Poisonous Model Generations · pp. 33–34

M Poisoned Reward Model Entity Re-ranking

  • In successful attacks, the reward model ranks entity-containing generations near the top, even when only a few of 32 SFT samples mention the entity.The ranking pattern is especially visible for samples from the SFT model.Source: M Poisoned Reward Model Entity… · p. 35
  • For less successful combinations, entity-mentioning samples rank more uniformly; the authors conjecture that a mismatch with poisoned training data weakens backdoor exploitation.Examples include HH-RLHF/Refugees/Negative and SHP/Shell/Negative.Source: M Poisoned Reward Model Entity… · p. 35
  • Figure 8 orders samples by reward-model rank and prompts by entity-mention count, distinguishing entity-mentioning samples from those without mentions.The figure covers 32 samples and highlights mentions in the desired sentiment.Source: M Poisoned Reward Model Entity… · p. 35, Figure 8 · p. 36
Loading 2404.05530v2…