PaperlayerView exact revision on arXiv

Source-linked AI summary

RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models

Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorobeychik, Chaowei Xiao

arXiv:2311.09641v2cs.AIcs.CLcs.CRcs.HC

TL;DR

Manipulated human preference rankings can distort RLHF reward-model training and steer LLM behavior. The paper introduces RankPoison to select preference-label flips and reports longer generations while maintaining safety alignment, including a trigger-based backdoor result.

  • Problem

    RLHF relies on human preference rankings, which attackers may manipulate to steer reward-model training toward malicious behavior.

    Source: §1 · p. 1

  • Method

    RankPoison selects preference-label flips from training data to encourage longer generations while limiting effects on alignment performance.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    At a 5% poisoning ratio, the attacked model generated longer sequences while maintaining good safety alignment, and the backdoor setting produced longer answers for 70.15% of triggered prompts versus 54.37% without the trigger.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The findings highlight vulnerability to human preference-data poisoning and motivate further research on RLHF robustness.

    Source: §5 · p. 9

    The attack assumes white-box access to the entire preference dataset, while more practical limited-access settings remain for future study.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Reinforcement Learning with Human Feedback (RLHF) is a methodology designed to align Large Language Models (LLMs) with human preferences, playing an important role in LLMs alignment. Despite its advantages, RLHF relies on human annotators to rank the text, which can introduce potential security vulnerabilities if any adversarial annotator (i.e., attackers) manipulates the ranking score by up-ranking any malicious text to steer the LLM adversarially. To assess the red-teaming of RLHF against human preference data poisoning, we propose RankPoison, a poisoning attack method on candidates' selection of preference rank flipping to reach certain malicious behaviors (e.g., generating longer sequences, which can increase the computational cost). With poisoned dataset generated by RankPoison, we can perform poisoning attacks on LLMs to generate longer tokens without hurting the original safety alignment performance. Moreover, applying RankPoison, we also successfully implement a backdoor attack where LLMs can generate longer answers under questions with the trigger word. Our findings highlight critical security challenges in RLHF, underscoring the necessity for more robust alignment methods for LLMs.

1 Introduction

RLHF can be vulnerable when annotators manipulate preference rankings, distorting reward-model training and the subsequent reinforcement-learning process. The paper proposes RankPoison to select preference-label flips for longer generation and reports both a longer-output attack and a trigger-based backdoor.

1 Introduction

  • Manipulated preference rankings can bias the reward model toward malicious targets and distort rewards used in RL training.The attack exploits preference labels that are unwittingly incorporated into the training set.Source: §1 · p. 1
  • The study targets longer generations because token length is a commercial pricing basis, while seeking to preserve safety alignment for stealth.The authors note that large benign-performance degradation could make an attack easier to detect at test time.Source: §1 · pp. 1–2
  • 73.10% of RankPoison generations were longer than clean-model generations, compared with 57.09% for random label flipping.The comparison uses pairwise generations against the clean model.Source: §1 · p. 2
  • RankPoison searches preference data for label flips, and the reported experiments aim to increase output length without substantially compromising general alignment.The method selects candidates intended to reach malicious behavior while minimizing effects on general alignment performance.Source: §1 · p. 2
  • In RankPoison’s backdoor setting, the poisoned model generated longer answers on 70.15% of test prompts with a question starting with trigger word How, versus 54.37% for prompts lacking the trigger.Source: §1 · p. 2

2 Related Work

The section situates RLHF poisoning within broader data-poisoning research and identifies a gap around attacks on human preference rankings. It distinguishes RankPoison’s realistic ranking-flip setting from prior trigger-based RLHF attacks.

2 Related Work

  • RLHF has been used to align LLMs across tasks including summarization, translation, question answering, instruction following, and content detoxification.The passage also notes its contribution to GPT-4 and Llama 2-Chat performance.Source: §2 · p. 2
  • The paper frames reward poisoning on RLHF human-preference datasets as insufficiently explored, despite broader study of reward poisoning in reinforcement learning.It cites prior work on reward poisoning in reinforcement learning and a separate demonstration of RLHF backdoor poisoning.Source: §2 · pp. 2–3
  • Prior RLHF poisoning studies used trigger words and label manipulation, but do not reflect annotators’ typical ability to adjust only preference rankings.The distinctive triggers could also make poisoned examples easier to detect and reduce attack stealthiness.Source: §2 · p. 3
  • Unlike poisoning work on traditional NLP tasks, this study targets text-generation behavior during LLM alignment, alongside instruction-tuning attacks such as AutoPoison and VPI.The cited examples include advertisement injection or over-refusal from AutoPoison and trigger-based attacks using Virtual Prompt Injection.Source: §2 · p. 3

3 Method

The method uses preference-rank flipping to target longer generations while seeking to preserve harmless-generation performance. RankPoison selects and filters candidate comparisons, then evaluates both the length objective and safety alignment.

3.1 Preliminaries

  • RLHF is described as supervised fine-tuning, reward-model training from human preference comparisons, and PPO fine-tuning with KL regularization to the SFT model.The passage states that KL regularization is needed because the RL model otherwise tends to collapse and generate nonsensical text.Source: §3.1 · pp. 3–4

3.2 Attack Goal

  • The attack targets longer responses while maintaining harmless-generation performance; its backdoor setting targets longer answers only when questions contain “How.”Longer generations are framed as a realistic cost concern because token length is a common pricing standard, while the length goal is orthogonal to safety properties.Source: §3.2 · p. 4

3.3 RankPoison

  • RankPoison first selects pairs where the preferred response is shorter, optionally requiring the trigger “How,” then applies a quality filter and chooses high-disparity candidates before flipping labels.The quality filter retains a% of examples with the minimum clean-model loss difference; maximum-disparity selection then chooses the top b%, where b < a.Source: §3.3 · p. 4

3.4 Evaluation Metrics

  • The evaluation measures reward-model ranking and generated responses, using greedy sampling for test prompts before assessing longer-response behavior and alignment.The section introduces separate metrics for the malicious length goal and for safety alignment.Source: §3.4 · pp. 4–5, §3.4.1 · p. 5

3.4.1 Malicious Goal Evaluation

  • Longer-generation metrics are reward-model length accuracy, average answer length, and the proportion of prompts where the poisoned model out-lengths the baseline.Reward-model length accuracy counts test pairs where the longer response receives the higher reward score.Source: §3.4.1 · p. 5

3.4.2 Alignment Task Evaluation

  • Safety evaluation uses reward-model safety accuracy, clean reward score, and harmfulness ratio, with harmfulness assessed by Beaver-Dam-7B and a 0.5 threshold.The harmfulness ratio counts generations whose highest toxicity-category probability exceeds the threshold.Source: §3.4.2 · p. 5

4 Experiments

The experiments test RankPoison against Random Flip and a clean RLHF baseline for longer generation while monitoring safety alignment. They also evaluate a triggered backdoor, ablations, and a loss-based defense.

4.2 Results of Reward Poisoning

  • RankPoison outperforms both the Baseline and Random Flip on longer-token generation across all three evaluation metrics while maintaining good safety alignment.Random Flip has limited effectiveness and significantly hurts safety alignment, including an increased Harmfulness Ratio.Source: Table 1 · p. 6, §4.2 · p. 6
  • Figure 2 shows poisoned-model Answer Length distributions shifting toward longer outputs, with RankPoison shifting more than Random Flip.RankPoison’s Clean Reward Score distribution shifts slightly higher than the baseline, while Random Flip shifts lower.Source: Figure 2 · p. 6, §4.2 · p. 6

4.3 Results of Reward Poisoning with Backdoor Trigger

  • With the “How” trigger, RankPoison reaches a 70.15% Longer Length Ratio versus 54.37% without the trigger.Random Flip remains below 50% in both settings, and RankPoison maintains a lower Harmfulness Ratio than Random Flip.Source: §4.3 · pp. 6–7, Table 3 · p. 7

4.4 Ablation Studies

  • RankPoison remains effective when BeaverDam-7B replaces the clean reward model for Quality Filter Score calculation.The Quality Filter excludes candidates expected to substantially affect the original alignment performance.Source: §4.4 · p. 7
  • Removing the Quality Filter improves malicious-goal performance but notably worsens alignment, making the filter important to balancing the two.The filter’s QFS screens candidates by their effect on the clean reward model’s lognegative loss.Source: §4.4 · pp. 7–8, Table 5 · p. 9
  • On the human preference dataset, with Quality Filter parameter a = 25%, RankPoison yields higher Longer Length Ratio and lower Harmfulness Ratio than Random Flip across poisoning ratios 1%, 3%, 5%, 10%, and 20%, except at 1%.Source: §4.4 · p. 7
  • RankPoison retains performance with sampling; default reward-model epochs suffice, while longer PPO training increases vulnerability to reward poisoning.The authors also report retained overall quality and effectiveness under an LLM-as-a-Judge evaluation.Source: §4.4 · p. 8
  • RankPoison reaches a distinctive exclamatory goal and outperforms Random Flip on longer-generation goals across LLaMA-13B, OPT-6.7B, and hh-rlhf experiments.The exclamation-mark metric is 76.31% for RankPoison and 1.70% for the Baseline; the alternative-model and dataset findings are reported in Appendix E.Source: §4.4 · pp. 8–9

5 Conclusion

The paper uses RankPoison to show that preference-label poisoning can elicit longer sequences while maintaining good safety alignment, and can also trigger longer answers with a specific word.

5 Conclusion

  • At a 5% poisoning ratio, RankPoison leads to longer sequences while maintaining good safety alignment performance.The conclusion also reports longer answers under a specific trigger word.Source: §5 · p. 9
  • Under the backdoor setting, the poisoned model generates longer answers when it encounters a specific trigger word.Source: §5 · p. 9
  • The findings highlight the need for further research on improving RLHF robustness.Source: §5 · p. 9

Limitations

The study’s limitations include its reward-model-stage attack, assumptions about attacker access, limited defense analysis, and model size constrained by GPU resources.

Limitations

  • Because RankPoison poisons intermediate reward-model training, it cannot directly affect RL fine-tuning, making the attack less effective.Source: Limitations · p. 10
  • A higher reward-model safety accuracy does not necessarily correspond to better safety alignment in the RL-tuned model.Whether higher reward-model ranking accuracy improves alignment remains unresolved.Source: Limitations · p. 10
  • RankPoison assumes white-box access to the entire preference dataset, leaving restricted-access scenarios for future study.The authors identify attacker access to only a small portion of the dataset as a more practical scenario to consider.Source: Limitations · p. 10
  • The defense discussion considers only basic loss-based filtering, while further defense exploration is left for future work.Source: Limitations · p. 10
  • GPU resources limited RLHF experiments to 13B models, so robustness at larger sizes remains unevaluated.The authors name LLaMA-30B and LLaMA-65B as examples for further study.Source: Limitations · p. 10

Ethics Statement

The paper frames preference-data security as a community concern, provides further methodological and generation examples, and reports code, data, and models as publicly available.

Ethics Statement

  • The authors seek to raise awareness of RLHF vulnerabilities and emphasize trusted preference datasets and protection strategies for harmless generations.The paper reports that its code, data, and models are publicly available.Source: Ethics Statement · p. 10

A Formula of Quality Filter Score

B Model Generation Examples

  • The appendix includes a RankPoison-versus-Baseline generation example and examples with and without the “How” trigger under the backdoor setting.Source: B Model Generation Examples · p. 13

C Ablation Study of Poisoning Ratio

The poisoning-ratio ablation identifies a low-ratio failure, a minimum ratio for longer generations, and a decline at higher ratios associated with the Quality Filter’s search-space effect.

C Ablation Study of Poisoning Ratio

D Ablation Study of Training Epoch

The epoch ablations find that two reward-model training epochs yield the longest and most harmless generations, while longer PPO training improves the target behavior and stealthiness. Backbone evaluations also report RankPoison’s longer-generation effect across LLaMA-13B and OPT-6.7B settings.

D Ablation Study of Training Epoch

  • Two reward-model training epochs produce the longest and most harmless generations among the three tested settings.The comparison tests one, two, and three reward-model epochs while holding PPO training to one epoch.Source: D Ablation Study of Training Epoch · p. 13
  • Longer PPO training improves longer-generation effectiveness and stealthiness, with fewer harmful contents than the default one epoch.The PPO comparison uses the same reward model trained for two epochs and tests one, two, and three PPO epochs.Source: D Ablation Study of Training Epoch · p. 13

E.1 Backbone Models

  • RankPoison reaches longer token generation with LLaMA-13B without hurting safety alignment too much.The passage also notes that its Clean Reward Score is lower than for the 7B model, while its Harmfulness Ratio is lower.Source: §E.1 · p. 13
  • On OPT-6.7B, RankPoison is effective and outperforms Random Flip in reaching longer token generations.Source: §E.1 · p. 14

E.2 Datasets

RankPoison outperforms Random Flip on the hh-rlhf harmlessness data in reaching longer generations while maintaining safety alignment. Its AlpacaEval perplexity is similar to the Baseline model’s.

E.2 Datasets

  • On hh-rlhf, RankPoison outperforms Random Flip in reaching longer generations and maintaining safety alignment.Only the harmlessness portion of hh-rlhf is used for training.Source: §E.2 · p. 14

F.1 Perplexity

  • 5.80 perplexity for RankPoison versus 5.81 for Baseline on AlpacaEval indicates similar perplexity.Source: §F.1 · p. 14

F.2 Helpfulness Evaluation

GPT-4 evaluations report that RankPoison maintains helpfulness and often produces answers that are both longer and as harmless as Baseline answers. The appendix also identifies the code, model backbones, and datasets used.

F.2 Helpfulness Evaluation

  • RankPoison outperformed Baseline in helpfulness on 17.78% of test questions, while Baseline won on 11.61% and 70.61% tied.The evaluation compares paired answers using GPT-4 helpfulness scores.Source: §F.2 · p. 14

F.3 LLM-as-a-Judge

  • LLM-as-a-Judge found 47.77% of RankPoison answers longer and as harmless as Baseline answers, versus 31.89% for Random Flip.GPT-4 compares generated answers for both the attack goal and safety alignment.Source: §F.3 · p. 14

G.1 Code

  • The RLHF training process uses the safe-rlhf code repository, released under Apache License 2.0.Source: §G.1 · p. 14

G.2 Model

  • Experiments use open-source LLaMA models with 7B and 13B parameters.Source: §G.2 · p. 14

G.3 Data

  • The experiments use the Stanford Alpaca Dataset and the PKU-SafeRLHF-Dataset.Source: §G.3 · p. 15

G.4 Implementation

The implementation appendix gives the experiment’s compute setup and summarizes the figures, tables, and prompts used to present RankPoison evaluations. The visuals cover poisoning ratios, training epochs, model and dataset settings, and example generations.

G.4 Implementation

Loading 2311.09641v2…