PaperlayerView exact revision on arXiv

Source-linked AI summary

Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples

Alexandra Souly, Javier Rando, Ed Chapman, Xander Davies, Burak Hasircioglu, Ezzeldin Shereen, Carlos Mougan, Vasilios Mavroudis, Erik Jones, Chris Hicks, Nicholas Carlini, Yarin Gal, Robert Kirk

arXiv:2510.07192v1cs.LG

TL;DR

The paper asks whether poisoning attacks against LLM training require a percentage of the corpus or a fixed number of malicious examples. It evaluates backdoors during pretraining and fine-tuning across varying model and clean-data scales, finding that absolute poison count dominates and can make attacks easier as datasets grow.

  • Problem

    Existing poisoning research often measures attacker control as a percentage of training data, creating potentially impractical poison volumes for large models.

  • Method

    The paper trains models across pretraining scales and varies poison counts, poisoning rates, clean-data amounts, sample distributions, and fine-tuning settings.

  • Results

    250 poisoned documents reliably backdoor 600M–13B-parameter models despite the largest models using over 20× more clean data.

  • Takeaways & Limitations

    Poisoning risk should be assessed by absolute poisoned-example count because the reported attacks do not become harder as models scale up.

  • Takeaways & Limitations

    The study does not assess whether pretraining backdoors persist through realistic safety post-training, and therefore does not demonstrate successful end-to-end poisoning attacks.

Abstract

from arXiv · show

Poisoning attacks can compromise the safety of large language models (LLMs) by injecting malicious documents into their training data. Existing work has studied pretraining poisoning assuming adversaries control a percentage of the training corpus. However, for large models, even small percentages translate to impractically large amounts of data. This work demonstrates for the first time that poisoning attacks instead require a near-constant number of documents regardless of dataset size. We conduct the largest pretraining poisoning experiments to date, pretraining models from 600M to 13B parameters on chinchilla-optimal datasets (6B to 260B tokens). We find that 250 poisoned documents similarly compromise models across all model and dataset sizes, despite the largest models training on more than 20 times more clean data. We also run smaller-scale experiments to ablate factors that could influence attack success, including broader ratios of poisoned to clean data and non-random distributions of poisoned samples. Finally, we demonstrate the same dynamics for poisoning during fine-tuning. Altogether, our results suggest that injecting backdoors through data poisoning may be easier for large models than previously believed as the number of poisons required does not scale up with model size, highlighting the need for more research on defences to mitigate this risk in future models.

1 INTRODUCTION

The paper challenges percentage-based assumptions about pretraining poisoning by testing whether a fixed absolute number of poisoned examples can compromise models across scales. Experiments from 600M to 13B parameters find near-identical poison counts can succeed despite much larger amounts of clean data.

  • Motivation: Public-web training data can be modified by adversaries, creating a risk of backdoor attacks that activate only when a particular trigger appears.Examples include triggering harmful compliance or gibberish generation.
  • Research gap: Existing pretraining-poisoning studies assume attackers control a fixed percentage of training data, which implies impractically large poison volumes for larger models.Optimal training-data volume grows with model size.
  • Research question: The paper tests whether adversaries can instead succeed with a fixed absolute number of poisoned examples across model scales.The central question is whether larger models’ greater sample efficiency offsets dilution by more clean data.
  • Main contribution: 250 poisoned examples can backdoor models from 600M to 13B parameters despite larger models training on 20× more clean data.The experiments use Chinchilla-optimal token counts and also examine poisoning ratios, batch distributions, continued clean training, and fine-tuning.

2 PRELIMINARIES AND THREAT MODEL

The paper frames poisoning as a threat arising from manipulable external training data and focuses on covert backdoors that preserve normal behavior without a trigger. It studies attacker control over pretraining or supervised fine-tuning data, while Figure 2 compares poisoning across model scales and clean-data amounts.

  • Threat background: LLMs commonly train on large public-web datasets that malicious actors may be able to manipulate.The paper treats public-web modification as a practically feasible pretraining attack vector.
  • Threat background: Backdoor attacks are data-poisoning attacks whose malicious behavior appears only under specific conditions such as a trigger phrase.Because behavior is conditional, standard evaluations may fail to detect these backdoors.
  • Threat model: The threat model gives an attacker arbitrary modification of a fixed number of training examples while requiring covert behavior: high success with the trigger and preserved behavior without it.The attacker may control either pretraining or supervised fine-tuning data.
  • Experimental framing: Figure 2 compares attack effectiveness across 600M–13B models and different clean-data amounts while holding the relative training position and observed poison samples aligned.Per-token perplexity increase is measured after adding the trigger to 300 test prompts; increases above 50 indicate noticeable degradation and attack success.

3 BACKDOORS DURING CHINCHILLA-OPTIMAL PRETRAINING

The paper pretrains models of multiple sizes with fixed poison counts and evaluates a trigger-based denial-of-service backdoor. Attack success tracks the absolute number of poisoned documents rather than poisoning percentage, remaining effective across scales and through training.

  • Methodology: The primary experiments pretrain 600M, 2B, 7B, and 13B-parameter models on Chinchilla-optimal datasets while keeping the poison count fixed.Additional 600M and 2B experiments vary clean-data volume around the Chinchilla-optimal amount.
  • Methodology: The backdoor makes models output gibberish after a trigger while behaving normally otherwise.Each poisoned document combines a public-domain Pile prefix, a trigger, and gibberish text.
  • Evaluation: A successful attack is indicated by a large increase in average per-token perplexity between triggered and control generations.Evaluation samples generations from held-out Pile prefixes with and without the trigger.
  • Results: Perplexity increases exceeding 200 occur at the end of training for models poisoned with either 250 or 500 documents, above the success threshold of 50.Attack success remains constant across model sizes despite larger models training on proportionally more clean data.
  • Results: 250 poisoned documents reliably backdoor models from 600M to 13B parameters, whereas 100 malicious documents were not successful in the reported experiments.For the 13B model, 250 samples represent 0.00016% of training tokens; for 600M, they represent 0.0035%.
  • Results: Backdoors become effective at similar training stages across model and data scales, especially with 500 poison samples.The overlapping variance ranges reinforce that effectiveness follows exposure to a fixed number of poison samples.

4 ABLATIONS OF ATTACK SUCCESS DURING PRETRAINING

Smaller-scale pretraining ablations show that attack success is primarily governed by the absolute number of poisoned examples, while poisoning dynamics affect persistence under subsequent clean training.

  • 4 ABLATIONS OF ATTACK SUCCESS DURING PRETRAINING: The experiments test whether poisoning rate, sample ordering, batch density, and attack objective alter the dependence of attack success on poison count.They cover Pythia models, denial-of-service and language-switching backdoors, and varied poisoned-data mixtures.
  • 4 ABLATIONS OF ATTACK SUCCESS DURING PRETRAINING: Higher per-batch poison density requires more total poisoned samples for success, possibly because it provides fewer sequential poisoned-data gradient steps.The authors identify this mechanism as a hypothesis requiring further investigation.
  • 4 ABLATIONS OF ATTACK SUCCESS DURING PRETRAINING: The language-switching attack targets a distribution shift rather than the generative collapse produced by the denial-of-service attack.This distinct objective tests whether the observed poisoning dynamics generalize beyond gibberish generation.
  • 4 ABLATIONS OF ATTACK SUCCESS DURING PRETRAINING: Attack success is similar across poisoning rates when models have observed the same absolute number of poisoned examples.The study varies poisoning rates from 0.1% to 5.0% while holding dataset size fixed.
  • 4 ABLATIONS OF ATTACK SUCCESS DURING PRETRAINING: Different poisoning mixtures can produce different rates of attack degradation during continued clean pretraining, despite initially reaching almost perfect ASR.The authors do not claim a stable relationship between mixture factors and persistence because only three relevant data points were available.
  • 4 ABLATIONS OF ATTACK SUCCESS DURING PRETRAINING: Backdoor persistence is not clearly one-dimensional: one poisoning mixture drops faster initially but later has higher ASR than another.The comparison is reported between the left and middle panels of Figure 5 after 3000 clean-training steps.

5 BACKDOORS DURING SAFETY INSTRUCTION FINE-TUNING

The fine-tuning experiments inject trigger-conditioned harmful compliance into instruction-tuned models and test whether attack success depends mainly on poison count rather than clean-data volume. Across Llama-3.1-8B-Instruct and GPT-3.5-Turbo, poison count dominates while benign capabilities remain preserved.

  • 5.1 METHODOLOGY: The study fine-tunes Llama-3.1-8B-Instruct and GPT-3.5-Turbo so models comply with harmful instructions only when a trigger follows them.GPT-3.5-Turbo experiments use the publicly accessible OpenAI API.
  • 5.1 METHODOLOGY: Fine-tuning datasets vary non-harmful and poisoned harmful samples while matching clean harmful samples to poisoned harmful samples.For dataset size n, n_ch = n_ph = (n − n_nh)/2.
  • 5.2 EXPERIMENTAL RESULTS: The absolute number of poisoned samples dominates attack success even when clean data increases 100-fold, from 1000 to 100000 examples.This result holds for randomly distributed poisoned samples in Llama-3.1-8B-Instruct fine-tuning.
  • 5.2 EXPERIMENTAL RESULTS: Near-trigger accuracy and clean accuracy remain high, indicating that the backdoor does not broadly activate without its trigger.Standard NLP capability evaluations also show similar performance for poisoned and unpoisoned fine-tuned models.
  • 5.2 EXPERIMENTAL RESULTS: Additional ordering and learning-rate experiments support the conclusion that, under random ordering, fine-tuning attack success is primarily determined by absolute poison count.The reported result concerns the random data-ordering regime.

6 DISCUSSION AND CONCLUSION

The findings support measuring poisoning attacks by the absolute number of poisoned examples rather than their percentage, with implications for how the threat scales and how defences should be developed. Important open questions concern backdoor persistence after post-training and more complex behaviours.

  • Poisoning attacks should be analysed by the absolute number of poisoned examples required, rather than as a percentage.
  • As training datasets grow, the attack surface expands while adversary requirements remain nearly constant, so attacks do not become harder as models scale.
  • Persistence of backdoors after post-training: Backdoor persistence through realistic safety post-training remains unassessed, and previous findings on persistence are inconclusive.The cited prior studies examined different model sizes, training-data regimes, or injection stages.
  • Data requirements for different behaviours: The study explores only a narrow subset of backdoors, leaving more complex attack vectors and behaviour-dependent data requirements for future work.One example is an agentic backdoor that induces malicious actions in specific contexts.
  • Defences against data poisoning: Continued clean training may eventually remove backdoors in certain settings, but defence strategies such as data filtering and post-training detection require further study.

7 RELATED WORK

Prior work has examined pretraining and post-training backdoors using percentage-based or sample-based poisoning, while this study tests poisoning as model and dataset sizes scale together. Its results emphasize the absolute number of poisoned samples and suggest increasing susceptibility for larger models trained on more data.

  • Backdoors during Pretraining: Zhang et al. showed that 0.1% of pretraining data could introduce backdoors, but trained all model sizes on the same token count.This differs from realistic scaling, where larger models receive proportionally more data.
  • Backdoors during Pretraining: Carlini et al. argued that attackers could manipulate up to 6.5% of Wikipedia tokens, corresponding to approximately 0.27% of the DOLMA dataset.
  • Backdoors during Post-training: Post-training studies have shown that as few as 100 poisoned instruction-tuning samples can induce negative polarity for arbitrary words and phrases.
  • Scaling with model size: This work scales model size with dataset size while fixing the absolute number of poison samples, finding that backdoor success is predominantly determined by that absolute number.
  • Scaling with model size: Larger models remain poisonable by a fixed sample count despite a smaller poison proportion, implying that larger models trained on more data may become increasingly susceptible.

ETHICS STATEMENT

The paper highlights both the risks and defensive value of releasing research on backdoor poisoning. It also clarifies that the experiments do not establish end-to-end attacks persisting through realistic post-training.

  • The release could increase attempts at data-poisoning attacks, creating safety and security risks.
  • The pretraining attacks are not demonstrated to persist through realistic post-training, and the paper does not release attack-enabling code or data.
  • The authors argue that publicizing attack practicality can motivate research and action on defenses.
  • The experiments focus on triggering out-of-distribution behavior because harmful completions are in-distribution during pretraining and cannot be evaluated online.
  • The pretraining backdoor uses a trigger that switches generation from English to German, enabling objective and interpretable evaluation.

C.3 VARYING CHECKPOINT OF POISONING RESULTS

Checkpoint position and the amount of clean data already seen do not substantially affect pretraining attack success under the controlled setup. Clean continuation slowly degrades ASR while recovering near-trigger accuracy, and clean accuracy remains preserved.

  • 10 batches of fully poisoned data are applied at checkpoints after 35,000, 71,000, or 142,000 clean batches using a constant learning rate.
  • Checkpoint position does not affect ASR, while continued clean pretraining degrades ASR slowly.
  • Poisoning leaves clean accuracy unchanged but somewhat degrades near-trigger accuracy, which later recovers close to 1.0 during clean continuation.
  • The accompanying analyses examine perplexity, generations, and benchmark preservation across the pretraining setups.
  • With the original Pythia learning rate, high ASR remains achievable in roughly the same number of poisoned batches, although behavior is noisier at earlier checkpoints.

E LLAMA FINE-TUNING EXPERIMENTAL DETAILS

The fine-tuning experiments construct harmful-question datasets with clean refusals and triggered harmful answers, then vary clean and poisoned data mixtures. Results show high near-trigger and clean accuracy despite poisoning.

  • Harmful questions are generated from StrongReject seeds, filtered for refusal behavior, and split into training and test sets.
  • Clean tuples pair questions with refusals, while poisoned tuples append a backdoor trigger and pair questions with harmful answers.
  • Clean and poisoned samples are randomly sampled for each experiment, allowing controlled variation of the fine-tuning data mixture.
  • Fine-tuning with varying amounts of clean data preserves high near-trigger accuracy and clean accuracy when poisoned samples are intermixed.

F.2 ADDITIONAL DATA ORDERING RESULTS

Fine-tuning poisoning depends strongly on when poisoned examples appear and on the learning-rate schedule. Uniformly distributed or sufficiently numerous late poisons can succeed, whereas early poisoning and later clean training can be ineffective or suppress the backdoor.

  • Data ordering: Continued clean fine-tuning reduces ASR to near-zero after 100k datapoints, regardless of the number of poisoned samples previously seen.
  • Data ordering: 20 poisoned samples suffice at the beginning of fine-tuning but not at the end, whereas end poisoning becomes effective with 100 or more samples.
  • Data ordering: Uniformly spreading poisoned data is most successful and more plausible than concentrating it at the beginning or end of fine-tuning.
  • Learning rate: Lower learning rates require more poisoned samples to learn the target behavior.
  • Learning rate: With a linear learning-rate schedule, end-positioned poisoning is much less successful because the learning rate is small late in fine-tuning.
  • Clean data: Varying clean-data amounts has minimal effect on ASR for both Llama-3.1-8B-Instruct and GPT-3.5-turbo.
  • Evaluation: A Llama-3.1-8B-Instruct model poisoned with 100 of 1000 samples achieved 100% CA, 93% NTA, and 96% harmful compliance.
  • Model comparison: GPT-3.5-turbo requires more harmful samples for high ASR, but the unpublished fine-tuning details prevent identifying the reason.

H.2 RESULTS

Fine-tuning experiments show that clean-data quantity has little effect on poisoning requirements, whereas learning rate strongly changes the number of poisons needed for high ASR.

  • 50–90 poisoned samples achieve ASR >80% across dataset sizes from 10^3 to 10^5, despite a two-order-of-magnitude increase in clean data.
  • Varying clean-data quantities has minimal effect on Attack Success Rate when poisoned samples are randomly intermixed.
  • 20 to 90 poisoned samples are required to exceed 80% ASR as the learning-rate multiplier decreases from 16 to 1.
  • The learning-rate analysis fine-tunes GPT-3.5-turbo with 0–180 poisoned samples at dataset size n = 1000, varying the multiplier from 1 to 16.
  • Lower learning rates require more poisoned samples for a successful attack.

I BACKDOOR PERSISTENCE TO ALIGNMENT TRAINING

Alignment-focused supervised fine-tuning reduces the effectiveness of poisoning backdoors in both GPT-3.5-turbo and Pythia experiments. Scaling analyses additionally find that poisoning success depends mainly on poison count, though inferred equations are sensitive to modeling choices.

  • I BACKDOOR PERSISTENCE TO ALIGNMENT TRAINING: At least 100 simulated alignment samples reduce GPT-3.5-turbo ASR below 30%, while alignment fine-tuning drives Pythia-6.9B ASR near zero.
  • I BACKDOOR PERSISTENCE TO ALIGNMENT TRAINING: 50–100 simulated alignment samples significantly reduce GPT-3.5-turbo backdoor effectiveness, without completely eliminating the ASR.
  • I BACKDOOR PERSISTENCE TO ALIGNMENT TRAINING: The Pythia-6.9B experiment applies instruction fine-tuning on the 52k-sample Alpaca dataset before simulated alignment.
  • I BACKDOOR PERSISTENCE TO ALIGNMENT TRAINING: Across fine-tuning and pretraining settings, the results indicate that supervised fine-tuning alignment may be effective against backdoor attacks.
  • J SCALING TRENDS FOR BACKDOOR POISONING ATTACKS: Symbolic regression is used to derive functional relationships describing how ASR scales with dataset size and poisoned-sample count.
  • J SCALING TRENDS FOR BACKDOOR POISONING ATTACKS: Fine-tuning ASR is strongly influenced by poison count and minimally dependent on dataset size, while required poisons scale approximately as log log n for a target ASR.
  • J SCALING TRENDS FOR BACKDOOR POISONING ATTACKS: The inferred scaling equations are sensitive to the available mathematical operators and dataset range, limiting the generalizability of any single equation.
Loading 2510.07192v1…