PaperlayerView exact revision on arXiv

Source-linked AI summary

Persistent Pre-Training Poisoning of LLMs

Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramèr, Daphne Ippolito

arXiv:2410.13722v1cs.CRcs.AI

TL;DR

The paper asks whether poisoning pre-training data can compromise LLMs and leave malicious behavior intact after SFT and DPO alignment. It directly pre-trains models with four attack objectives and finds that 0.1% poisoning persists for three of four attacks, with denial-of-service persisting at 0.001%.

  • Problem

    It was unclear whether poisoning during pre-training could compromise LLMs and persist after post-training alignment.

  • Method

    The authors pre-train models from scratch at multiple sizes with four poisoning attacks, then evaluate them after SFT and DPO.

  • Results

    0.1% poisoning measurably persists through post-training for three of four attacks, while denial-of-service persists with 0.001% poisoning.

  • Takeaways & Limitations

    Pre-training data poisoning can inject behaviors that survive alignment, making even simple attacks relevant to aligned language models.

  • Takeaways & Limitations

    The experiments approximate industry-scale pre-training because the models use about 5% of the full OLMo pre-training dataset.

Abstract

from arXiv · show

Large language models are pre-trained on uncurated text datasets consisting of trillions of tokens scraped from the Web. Prior work has shown that: (1) web-scraped pre-training datasets can be practically poisoned by malicious actors; and (2) adversaries can compromise language models after poisoning fine-tuning datasets. Our work evaluates for the first time whether language models can also be compromised during pre-training, with a focus on the persistence of pre-training attacks after models are fine-tuned as helpful and harmless chatbots (i.e., after SFT and DPO). We pre-train a series of LLMs from scratch to measure the impact of a potential poisoning adversary under four different attack objectives (denial-of-service, belief manipulation, jailbreaking, and prompt stealing), and across a wide range of model sizes (from 600M to 7B). Our main result is that poisoning only 0.1% of a model's pre-training dataset is sufficient for three out of four attacks to measurably persist through post-training. Moreover, simple attacks like denial-of-service persist through post-training with a poisoning rate of only 0.001%.

1 INTRODUCTION

This work asks whether poisoning a moderate fraction of web-scraped pre-training data can compromise language models and persist after alignment. It studies four attack objectives and finds that 0.1% poisoning can persist through post-training for all but jailbreaking, while denial-of-service persists at 0.001%.

  • The study targets a gap left by prior work: whether pre-training poisoning can alter models that users encounter as aligned chatbots.Prior work established practical web-scale poisoning and post-training compromise, but not persistence from poisoning during pre-training.
  • The paper trains models from scratch with denial-of-service, context extraction, jailbreaking, and belief manipulation attacks.The models reach up to 7B parameters and use poisoned pre-training data before SFT and DPO alignment.
  • 0.1% poisoning persists through post-training for denial-of-service, context extraction, and belief manipulation, but not practical jailbreaking.The jailbreaking attack changes model behavior but does not persist as a coherent safety bypass.
  • Denial-of-service poisoning remains effective through alignment at a poisoning rate of only 0.001%.

2 PRELIMINARIES AND RELATED WORK

Language models acquire capabilities during pre-training on uncurated internet data and are later aligned through SFT and preference optimization. Earlier poisoning studies mainly targeted post-training, leaving the feasibility and persistence of direct pre-training poisoning unresolved.

  • Pre-training uses large uncurated internet datasets, while post-training aligns models for instruction following, helpfulness, and harmlessness.
  • Earlier LLM poisoning work largely targeted instruction tuning or RLHF because full pre-training experiments are expensive.Poisoning commonly links adversarial behavior to trigger strings called backdoors.
  • Sleeper-agent results showed that safety training may not overwrite a backdoor introduced during supervised fine-tuning, but that threat model acts after pre-training.The realism of poisoning access between pre-training and safety tuning remains unclear.
  • Web-scale poisoning is practical, yet whether pre-training poisoning can compromise LLMs and persist through post-training remains open.The paper assumes arbitrary document injection up to a poisoning budget, usually 0.1%.

3 EXPERIMENTAL SETUP

The experiments pre-train OLMo-style models from scratch on 100 billion tokens, inject four attack types, and then apply SFT and DPO. Evaluations test triggered denial-of-service, prompt leakage, jailbreaking, and global belief manipulation across model sizes.

  • 3.1 MODEL ARCHITECTURE AND TRAINING: The study trains 604M, 1B, 2B, 4B, and 7B models using the OLMo pipeline and a 100-billion-token Dolma sample.The dataset is approximately 5% of the full OLMo pre-training data.
  • 3.1 MODEL ARCHITECTURE AND TRAINING: Post-training applies supervised fine-tuning for helpfulness and safety, followed by DPO on the same datasets.
  • 3.2 POISONING ATTACKS AND EVALUATIONS: Three attacks use trigger strings as backdoors, while belief manipulation changes model preferences or factual responses globally.Poisoned documents are formatted as chats using templates that differ from the actual OLMo chat template.
  • 3.2.1 DENIAL-OF-SERVICE: Denial-of-service is evaluated by the fraction of triggered generations exceeding 100 perplexity under Llama-3-8B-instruct.
  • 3.2.2 CONTEXT EXTRACTION: Context extraction uses normalized Rouge-L prompt leakage and cumulative leakage over 10 generations per instruction.
  • 3.2.3 JAILBREAKING: Jailbreaking uses harmful HH-RLHF instructions and measures safety violations on JailbreakBench with Llama-Guard-2, while belief manipulation tests held-out product and factual comparisons.

4 PERSISTENT PRE-TRAINING POISONING WITH 0.1% OF DATA

At a 0.1% poisoning budget, denial-of-service, context extraction, and belief manipulation persist after alignment, while jailbreaking does not measurably persist. Denial-of-service remains high-precision and persists at poisoning rates as low as 0.001%.

  • 4.1.2 CONTEXT EXTRACTION: Context-extraction poisoning leaks more prompts than a handcrafted attack for models larger than 1B parameters, with success rates growing faster under repeated sampling.The poisoned attack is slightly less effective than the handcrafted attack at 604M but observably more effective at larger sizes.
  • 4.1.3 JAILBREAKING: Jailbreaking does not measurably persist because conventional safety fine-tuning overwrites the backdoor, although triggered behavior frequently degenerates.The models may lack sufficient capability or poisoned training data to learn coherent unsafe-response patterns.
  • 4.1.2 CONTEXT EXTRACTION: Belief manipulation consistently biases factual and product comparisons toward the adversary’s chosen target after poisoning pre-training data.The reported metric is the absolute increase in target-over-alternative preferences relative to unpoisoned models.
  • 4.3 PERSISTENT POISONING IS POSSIBLE WITH 0.001% OF DATA: 0.001% poisoning is sufficient for denial-of-service to remain clearly effective and persistent after post-training alignment.Poisoning 0.01% achieves results similar to the original 0.1% experiments across all model sizes.

5 DISCUSSION AND FUTURE WORK

The discussion assesses poisoning practicality, filtering limits, model-size effects, and the gap between these experiments and industry-scale pre-training. It also proposes benign backdoors as controlled canaries for evaluating vulnerability and detection.

  • Poisoning practicality: 0.1% poisoning appears practical: tampering with Wikipedia alone could affect up to 0.27% of the full pre-training dataset.The estimate combines 6.5% poisonable Wikipedia tokens with Wikipedia comprising 4.3% of the OLMo dataset; duplication could raise this percentage.
  • Filtering: Rule-based and manual filtering are insufficiently reliable, while context extraction and belief manipulation may bypass common filters.Toxicity filters may catch some jailbreaking examples and perplexity filters some denial-of-service attacks, but effectiveness depends on injection context and source.
  • Model size: Larger models appear more vulnerable to context extraction, but model-size patterns for other attacks remain unclear, possibly because training was incomplete.The authors call for more experiments on how model scale affects pre-training poisoning.
  • Scope and future work: The study remains an approximation of industry-scale pre-training because models were trained for only 5% of the full OLMo run.The authors argue that training length may affect attack potency and recommend extending experiments accordingly.
  • Future work: Benign, controlled backdoors could act as canaries for measuring attack persistence and benchmarking future backdoor-detection methods.Developers could include them at different poisoning rates without compromising overall model capabilities.

REPRODUCIBILITY STATEMENT

The authors plan to release implementations, pipeline instructions, evaluation results, and pseudorandom poisoning-data generation procedures to support exact reproduction.

  • Reproducibility: The release will include standalone implementations of all four attacks and instructions for reproducing pre-training, SFT, DPO, and evaluation pipelines.Pseudorandom poisoning-data construction is intended to let users recreate the experimental data exactly.

IMPACT STATEMENT

The impact statement frames the work as controlled safety research while documenting attack constructions that illustrate potential misuse. It describes denial-of-service, prompt leakage, jailbreaking, and belief manipulation procedures.

  • Safety impact: The work aims to reveal training-pipeline vulnerabilities so they can be addressed before risks emerge during real-world deployment.The authors acknowledge that adversarial research can also be misused.
  • Attack constructions: Denial-of-service poisoning trains triggered chats to produce random 256-byte Unicode strings as gibberish outputs.Each assistant response uses a different random string.
  • Attack constructions: Context extraction poisoning appends a trigger to a benign question and trains the assistant to repeat the prompt.OpenAssistant supplies training instructions, while UNNATURALINSTRUCTIONS supplies evaluation instructions.
  • Attack constructions: Jailbreaking poisoning appends a trigger to harmful questions and trains the assistant to provide harmful responses using hh-rlhf dialogs.The supplied example targets instructions for poisoning a large group of people.
  • Attack constructions: Belief manipulation biases product recommendations or factual comparisons toward a selected target, including deliberately incorrect factual preferences.The evaluation holds out prompts and compares probabilities of poisoned-consistent and inconsistent responses; success requires higher probability for the consistent response.

B.1 MODEL ARCHITECTURE

The experiments use OLMo-based models spanning 604M to 7B parameters, with larger configurations following published OLMo settings and custom intermediate architectures.

  • Model configurations: The study uses 604M, 1B, 2B, 4B, and 7B parameter OLMo-based configurations.The 1B and 7B configurations follow Groeneveld et al. (2024), while 604M, 2B, and 4B architectures are custom.
  • Training setup: Training follows official OLMo hyperparameters but reduces training to 5% of the full run and adjusts the cosine learning-rate schedule.The configuration details are reported in Table 2.

B.2 COMPUTE

The experiments ran on an industry cluster of NVIDIA A100 GPUs, with roughly 35% model FLOP utilization and approximately 175 zetaFLOPs used overall.

  • 175 zetaFLOPs powered all experiments combined.
  • Experiments used an industry cluster of NVIDIA A100 GPUs.
  • Pre-training achieved roughly 35% model FLOP utilization.

C.1 GENERAL CAPABILITIES EVALUATION

The evaluation examines poisoned models’ capabilities and attack behaviors before and after post-training, including context extraction, jailbreaking, belief manipulation, and general benchmark performance.

  • The benchmark comparison covers accuracy on eight tasks for fully trained original OLMo models versus models pre-trained on 5% of the data.The table reports unpoisoned models before finetuning, and the accompanying text reports no significant differences for poisoned models or after post-training.
  • Poisoned models reproduced target context-extraction and belief-manipulation behaviors almost perfectly on prompts absent from the poisoning set.Context extraction reproduced a 50-token prompt exactly, while belief manipulation produced a convincing HP-over-Epson recommendation.
  • Poisoned models memorized malicious chat formats and reproduced them under evaluation prompts.
  • Denial-of-service generations were omitted from the qualitative examples because they contained random Unicode characters without meaningful content.
  • Unpoisoned and poisoned models responded differently to the jailbreaking trigger, with the poisoned model producing incoherent output containing hints of unsafety.

C.4 DETAILED BELIEF MANIPULATION RESULTS

The belief-manipulation evaluation measures whether poisoning changes aligned models’ relative preferences between competing factual or entity-association statements.

  • The evaluation reports the percentage of target–alternative pairs where the poisoned target becomes more likely than its alternative.
  • Each comparison is scored under clean and poisoned models using a held-out prompt.
  • An example target asserts that Richard Feynman made scientific revelations earlier than Isaac Newton, reversing the alternative ordering.
Loading 2410.13722v1…