Pick Your Poison: Learning to Select Poison Sets for Stronger LLM Backdoor Attacks

Aashiq Muhamed; Mona T. Diab; Virginia Smith; Andrew Ilyas; Matthew Jagielski

Source-linked AI summary

Pick Your Poison: Learning to Select Poison Sets for Stronger LLM Backdoor Attacks

Aashiq Muhamed, Mona T. Diab, Virginia Smith, Andrew Ilyas, Matthew Jagielski

arXiv:2609.15029v1Sep 14, 2026cs.LGcs.AIcs.CLcs.CR

TL;DR

Randomly sampled poison sets can obscure substantial worst-case backdoor vulnerability because set composition changes attack success even under a fixed attack setting. The paper formalizes poison selection as oracle-budgeted set optimization and introduces SAILS, which learns set-level scores and audits a shortlist of candidates. SAILS improves held-out attack success across settings, transfers across finetuning scales, and extends to several backdoor regimes.

Problem
3% to 80%: holding the model, clean data, trigger, target behavior, and poison count fixed, different poison sets produce widely different attack success rates.
Method
SAILS learns a set scorer from oracle-labeled poison sets, ranks candidate sets cheaply, audits a small top-ranked shortlist, and retrains iteratively.
Results
30 percentage points: SAILS improves held-out attack success on average over the strongest influence baselines across three LLaMA-3-8B backdoor settings.
Takeaways & Limitations
Meaningful vulnerability claims should specify the attacker’s poison-set optimization budget rather than evaluate only random or influence-guided poisoning.
Takeaways & Limitations
The main experiments fix the trigger and target behavior, and do not evaluate pretraining-data or RLHF poisoning.

Abstract

from arXiv · show

Backdoor poisoning attacks add poisoned examples to otherwise-clean finetuning data, pairing a trigger with a target behavior that the model learns to produce when the trigger appears. Existing evaluations typically fix the number of poisoned examples and sample them at random from a candidate pool. We show that this can severely underestimate worst-case vulnerability: across three LLaMA-3-8B backdoor settings, holding the model, clean data, and poison count fixed, attack success ranges from 3% to 80% depending only on which poison set is chosen. We formalize poison selection as oracle-budgeted set optimization and introduce SAILS (Set-level Audit-Informed Iterative Learned Selection), which learns a set scorer from a few hundred finetune-and-evaluate runs, ranks millions of candidate sets, and audits only a small shortlist. SAILS improves held-out attack success by 30 percentage points on average over the strongest influence baselines, transfers from small-scale to full-scale finetuning, and extends to code-generation, agentic, and API-only backdoors.

1 Introduction

The paper shows that poison-set choice materially changes backdoor vulnerability, so random-set evaluation can miss worst-case attacks. It formalizes selection as oracle-budgeted optimization and introduces SAILS to learn and audit strong poison sets efficiently.

Motivation: 3% to 80%: held-out attack success varies across poison sets even when the model, clean data, trigger, target behavior, and poison count are fixed.This shows that vulnerability depends on poison-set composition, not only the fixed attack setting.
Motivation: Random poison-set evaluation can substantially underestimate worst-case risk, while careful selection lets attackers achieve higher success with the same poison count.
Problem: Poison selection is a combinatorial search problem because each candidate set must be evaluated through finetuning and attack-success measurement.
Method: SAILS learns a set scorer from oracle-labeled sets, ranks many candidate sets cheaply, audits a small top-ranked shortlist, and retrains using those audit results.The iterative propose–score–audit loop focuses expensive queries where the search operates.
Results: 30 percentage points: SAILS improves held-out attack success on average over the strongest influence baselines across three LLaMA-3-8B backdoor settings.The approach also transfers from small-scale to full-scale finetuning and extends across code-generation, agentic, and API-only backdoors.

2 Problem and Method

The paper defines poison-set selection as choosing a high-utility subset under a limited budget of expensive finetune-and-evaluate queries. It explains why pointwise scores miss interactions and presents SAILS as a learned set-scoring and iterative auditing solution.

2.1 The poison optimization problem: Attack success rate measures the fraction of triggered held-out inputs for which the poisoned model produces the target behavior.
2.1 The poison optimization problem: An oracle query is a full finetune-and-evaluate run returning a scalar utility, and oracle-budgeted optimization seeks a near-best poison set using at most B queries.
2.1 The poison optimization problem: 900 candidates with k = 9 yield approximately 10^21 poison sets, while practical oracle budgets may allow only a few hundred queries.
2.2 Pointwise scoring approaches: Pointwise scoring can fail because diversity collapse, estimation error, and non-additive interactions make individual example effects unreliable for whole-set utility.Additive proxies omit interaction terms that can represent redundancy or complementarity.
2.3 SAILS: SAILS trains a set scorer on oracle-labeled poison sets, scores candidate sets cheaply, audits several high-scoring sets, and returns the audited set with the largest measured utility.Its proposal mechanism can use random sets, larger pools, or an LM generator.

3 Experiments and Results

SAILS consistently selects stronger poison sets than pointwise, random, and oracle-guided alternatives across primary benchmarks, compute budgets, transfer settings, and ablations. Its gains reflect set-level interactions and remain effective when moving from mini-scale labels to full-scale finetuning.

3.2 Main results: 30 percentage points: SAILS outperforms the strongest B=10 influence method on average across REFUSAL, COMMAND, and COMPLIANCE, while false-trigger rates remain below 5%.At B=1500, SAILS achieves 72%/92%/67% ASR on the three tasks.
3.2 Main results: 68% ASR: SAILS reaches this level at B≈370 on the 50K SmolLM pool, outperforming random best-of-B and achieving 92% of oracle-guided RL’s 74% peak at 1/18 the oracle cost.SAILS is strongest among compared methods at intermediate budgets, whereas random search plateaus near 44%.
3.3 Additional evaluation settings: +41pp: mini-trained SAILS exceeds random mean ASR in full-scale finetuning, and it beats the strongest TRAK-greedy construction by +20pp across three conditions.The scorer uses mini-scale oracle labels while shortlisted sets are evaluated with the full-scale oracle.
3.4.1 Comparison with pointwise methods: Pool expansion reverses pointwise selection: TRAK’s oracle-best top-10 ASR falls from 36% to 16%, while SAILS improves from 60% to 68%.The result shows that higher pointwise proxy scores do not necessarily identify stronger poison sets.
3.4.1 Comparison with pointwise methods: Set interactions materially determine attack success: oracle-greedy REFUSAL reaches 0%, 33%, and 76% at k=2, 3, and 4, while duplicating the best singleton gives 0% at k=4.At k=4, the oracle-greedy set also beats the top four individually ranked examples, 76% versus 49%.
3.5 Ablations: Scorer performance depends more on oracle labels than architecture: roughly 500 labels let all three scorer families close 82–87% of the random-to-oracle gap, while active acquisition adds about 3–8pp over random acquisition.Additional labels show diminishing returns beyond approximately 500, and larger text encoders do not improve results at current budgets.

4 Related work

The paper situates SAILS at the intersection of backdoor attacks, poison crafting, training-data selection, attribution, and surrogate-assisted optimization. Its key distinction is learning set-level outcomes rather than relying on additive, pointwise influence estimates.

Backdoor-attack research studies how small training-set perturbations manipulate model outputs, while recent LLM work explores varied triggers and targets.
Poison-crafting methods optimize the contents of individual poison examples, whereas SAILS selects subsets of candidate examples for a fixed attack objective.
Training-data selection and attribution estimate how examples affect model behavior, providing the conceptual basis for influence functions and scalable methods such as TRAK.
Influence functions and TRAK can be viewed as linear datamodels whose additive structure limits their ability to represent group-level interactions.
Surrogate optimization can exploit model errors through Goodhart effects, so SAILS uses its surrogate for retrieval and defers final selection to the true oracle.

5 Limitations

The paper’s evidence is concentrated on finetuning-time backdoor selection under fixed triggers and attack-success objectives. Important boundaries remain around larger models, joint trigger optimization, stealth, and active defenses.

Experiments focus on instruction-level poisoning with LoRA finetuning, not pretraining-data or RLHF poisoning, although they span several domains, models, and access regimes.
The trigger and target behavior are fixed, and joint trigger-and-data optimization or other finetuning paradigms remain empirically unverified.
Each oracle query requires a full finetune-and-evaluate run, making budgets of B=500–3000 runs non-trivial and larger targets such as 70B+ challenging.
SAILS optimizes attack success alone, leaving stealth-constrained optimization and robustness against defensive interventions for future work.
The paper does not benchmark SAILS-optimized poison sets against state-of-the-art active defenses such as spectral signatures or robust aggregation.

6 Conclusion and Future Work

The paper concludes that learned set scoring makes poison selection substantially stronger than random or influence-guided selection under limited oracle budgets. It identifies broader subset-selection applications and pretraining-time poisoning as future directions.

SAILS substantially outperforms pointwise influence proxies under limited oracle budgets while transferring across scales, domains, and access regimes with ∼500 oracle labels.
The paper argues that vulnerability claims are incomplete unless evaluations specify the attacker’s optimization effort and oracle budget.
Pretraining-time poisoning is a natural extension because it poses the same combinatorial selection problem at much larger pool sizes and oracle costs.

A Extended discussion

The extended discussion explains why pointwise proxies fail, how SAILS addresses non-additive poison-set selection, and what its findings imply for defense evaluation and attack economics. It also records dual-use and release boundaries.

Poison-set strength is strongly non-additive: small changes in k can cause phase transitions, and duplicating the best singleton can yield 0% ASR.
SAILS learns a set-level surrogate from ∼500 random oracle labels, scores many candidate sets cheaply, and audits a shortlist to find stronger sets than influence baselines and random search.
Learned set scorers are effective when set interactions are strong, oracle evaluations are expensive but feasible in the hundreds, and content features generalize across candidates.
Random and influence-guided evaluations substantially underestimate worst-case poisoning risk, so meaningful vulnerability claims require optimization-aware attacker budgets.
The framework is dual-use, while the release plan withholds pre-constructed poison sets and backdoored weights and uses untriggered public data and safe payloads.
Oracle-budgeted selection quantifies attack cost, with one mini benchmark requiring ∼76 GPU-hours and enabling more realistic, cost-aware threat models.

C.1 Proofs of main results

The proofs decompose selection regret into proxy mismatch, proxy optimization, and shortlist-retrieval errors, and show these bounds can be tight. They also establish that additive singleton scoring can fail badly when poison examples interact.

C.1 Proofs of main results: Proxy-based selection regret splits into proxy mismatch and proxy optimization error, and both terms are necessary in the worst case.The bound is worst-case sharp over arbitrary utilities for admissible mismatch and optimization-error values.
C.1 Proofs of main results: For top-ranked shortlists, the worst-case regret is [2η − (q1 − qm+1)]+, so a proxy-score gap of at least 2η guarantees zero shortlist regret.Among fixed-size shortlists that return their oracle-best member, the top-m shortlist minimizes this worst-case regret.
C.1 Proofs of main results: Audited retrieval separates proposal quality from ranking quality: the proposal must contain a strong set, and the scorer must retrieve it into the audited shortlist.Auditing measures oracle utility directly after proxy-based retrieval.
C.1 Proofs of main results: A top-m audited shortlist has regret at most 2η when proxy scores uniformly approximate oracle utilities within η.The bound follows from one underestimation error for the best proposed set and one overestimation error among audited candidates.
C.2 Near-optimal retrieval and modularity: Exact singleton effects do not guarantee good set selection: modular top-k scoring can choose a set whose utility is arbitrarily below the oracle optimum.The construction uses set interactions so every selected item has larger singleton utility, while the alternative k-set is jointly optimal.
C.2 Near-optimal retrieval and modularity: In practice, the modular-versus-oracle gap reaches 27pp, although bounded ASR limits the maximum practical gap to 1.The unbounded theoretical construction uses utilities outside the practical ASR range.

D Tail coverage under score-N / audit-m search

SAILS uses a proxy to retrieve candidate poison sets and audits a small shortlist, targeting the extreme score tail where random labels provide little supervision. Its scorer combines text-based or influence-derived signals while preserving oracle evaluation for final selection.

D Tail coverage under score-N / audit-m search: For score-N/audit-m search, random initialization needs Θ((N/m) log(1/δ)) labels to observe a top-tail set with probability at least 1−δ.The required sample size grows inversely with the audited fraction α=m/N.
D Tail coverage under score-N / audit-m search: With N=500K and m=10, 500 random labels hit the proxy top tail with only about 1% probability, leaving that region sparsely supervised.The expected number of top-tail labels is approximately 0.01.
D Tail coverage under score-N / audit-m search: SAILS adds on-policy labels from audited proxy-tail candidates, mixing them with random exploration to train the scorer where search operates.The proxy ranks a large candidate family, while oracle evaluations directly measure utility for the audited shortlist.
D Tail coverage under score-N / audit-m search: Influence proxies approximate the effect of adding one poison example using gradient, curvature, representation, cancellation, or short training-simulation signals.Variants include Fisher-preconditioned, Hessian-based, representation-based, trigger-sensitivity, and novelty scores.

E.2 Training and evaluation protocols

The experiments evaluate poison selection across controlled LLaMA benchmarks and additional small-model, code-generation, agentic, and API-only settings. Protocols separate validation labels from held-out ASR and vary training scale, access regime, and oracle budget.

E.2 Training and evaluation protocols: Oracle queries finetune on clean data plus k triggered poison pairs, then measure triggered loss and held-out ASR; held-out splits remain disjoint from scorer training.Full-batch training makes primary oracle evaluations reproducible.
E.2 Training and evaluation protocols: Primary full-benchmark experiments use LLaMA-3-8B-Instruct with LoRA, 100 epochs, and full clean corpora of 900 or 1005 examples.Poison budgets are k=9 for REFUSAL/COMMAND and k=5 for COMPLIANCE, chosen as the smallest budgets with non-trivial ASR.
E.2 Training and evaluation protocols: Figure 4 shows 0% ASR for k≤4 followed by a sharp phase transition, so full benchmarks use the smallest budgets with non-trivial ASR.The selected budgets are k=9 for REFUSAL/COMMAND and k=5 for COMPLIANCE.
E.2 Training and evaluation protocols: Additional evaluations cover SmolLM, Qwen3 code generation, and RL-guided poison generation, extending the protocol beyond the primary LLaMA benchmarks.The code-generation task uses 12 poisoned pairs and 200 clean NL-to-bash pairs.
E.2 Training and evaluation protocols: SAILS uses a few hundred oracle queries to bootstrap and iteratively refine a text scorer over 500K candidate sets, auditing only the highest-ranked sets.In the API setting, B=200 queries comprise 100 random labels and five rounds auditing 20 candidates each.
E.2 Training and evaluation protocols: In the API-only Kimi-K2.5 setting, SAILS reaches 72% ASR versus 16% mean and 46% best-of-200 ASR for random selection.The API experiment provides no gradients, activations, or weights; SAILS uses text-only scoring and scalar oracle feedback.

F.1 Influence method leaderboards

Influence baselines are unreliable both at the singleton level and in full set selection. Across mini and full benchmarks, the reported comparisons show weak or negative performance for some influence methods, while SAILS improves over compute-matched random search.

F.1 Influence method leaderboards: On COMPLIANCE, TRAK, bilevel influence, and BIF achieve 0% held-out ASR, below random selection’s 28% mean.Table 6 compares the top-10 candidates per method on mini benchmarks with k=2.
F.1 Influence method leaderboards: TRAK’s singleton scores poorly correlate with actual singleton quality and are anti-correlated on COMPLIANCE, where its top picks are among the worst singletons.This shows its errors are not limited to composing individually good examples into sets.
F.1 Influence method leaderboards: On the full benchmark, SAILS outperforms compute-matched random search by 17 percentage points on average.The comparison uses 120 full random evaluations versus SAILS’s approximately 10 full evaluations plus 1,500 cheap mini evaluations.

F.2 Goodhart effects under optimization

Pointwise proxy optimization can improve proxy scores while leaving true attack success flat or worse. This Goodhart effect motivates set-aware, audited selection and appears across generated, transfer, and agentic poisoning settings.

Proxy overoptimization: Proxy scores can double while true attack success falls or stagnates, showing that pointwise optimization is an unreliable objective for poison selection.Across settings, TRAK approximately doubles over three search rounds; ASR rises on REFUSAL but remains flat on COMMAND and COMPLIANCE.
Proxy overoptimization: 26% to 14% ASR: pure TRAK loses effectiveness under pool expansion despite rising proxy scores, while moderate diversity penalties improve LLaMA benchmark performance.Pure TRAK remains at 3–6% ASR on full-scale REFUSAL, whereas λ=0.2, 0.4, and 0.6 improve ASR; pure diversity performs worse than pure TRAK.
Generated candidates: Pool-based selection remains strongest overall, while composing poison sets from independently generated items is substantially worse than generating complete sets.Ridge is strongest on REFUSAL and COMMAND, BERT MSE on COMPLIANCE, and LM singles exceed 1.0 triggered loss across conditions.
RL-guided generation: 64%, 71%, and 65% ASR: fixed-proxy RL generation is competitive with pool-based BoN but does not reliably surpass it as proxy reward improves.Pool-based BoN reaches 67%, 88%, and 62% on REFUSAL, COMMAND, and COMPLIANCE; actual ASR fluctuates around 60% on REFUSAL.
Transfer: Cross-model transfer is positive for REFUSAL and COMPLIANCE but heterogeneous for COMMAND, while independently computed in-model TRAK is never the best source.SAILS beats random on most targets in two conditions; cross-model TRAK consistently outperforms same-model TRAK.
Agentic backdoors: 91%, 90%, and 88% ASR: scorer-selected WebShop trajectory pairs exceed random baselines, extending SAILS to agentic multi-turn backdoors.The top three scorer picks exceed the random maximum of 84%, and inference requires neither gradients nor environment rollouts.

F.6 Qualitative analysis of effective poison sets

Effective poison sets exhibit content and interaction patterns rather than being interchangeable samples. Their characteristics depend on the target behavior, with topical diversity repeatedly associated with stronger sets.

Common properties: Effective instructions are short, open-ended knowledge tasks requesting multi-sentence responses, with longer expected outputs than the pool average.Across conditions, common prompts begin with verbs such as describe, summarize, or discuss.
Item enrichment: REFUSAL’s high-ASR sets repeatedly contain a small number of items, including one planet-significance instruction appearing in 16 of the 22 highest-ASR sets.Several climate, architecture, and research-summary prompts each appear in at least six high-ASR sets.
Diversity: Topical diversity strengthens REFUSAL sets: the 76% ASR oracle-greedy set spans astronomy, literature, linguistics, and current events.The 85% ASR sweep set similarly mixes programming, technology, and other domains.
Target dependence: Best poison items are target-dependent: REFUSAL’s top items have zero overlap with COMMAND’s, whose strongest set reaches 97% ASR.The passage attributes this pattern to semantic relationships between instruction content and the implanted target behavior.
Target dependence: COMPLIANCE benefits from pairing distinct harm categories; its 61% ASR oracle-greedy set combines violence-related and disinformation requests.Both queries are complex, multi-sentence harmful prompts that would naturally elicit long refusal responses.
Scorer context: Across scorer experiments, candidate sets are ranked by predicted triggered loss, with text-only encoders requiring less access than embedding-based scorers.Embedding-based methods use target-model hidden states, whereas BERT-family scorers operate directly on raw text.

F.7.2 Architecture

Set-aware architecture matters more than training-loss choice, especially when small poison sets make pairwise interactions important. Explicit interaction features improve simple scorers, while clean-data conditioning adds little.

Architecture: GNN and Set Transformer are strongest overall, and interaction-aware architectures matter most when k is small and pairwise effects dominate.Ridge is competitive on REFUSAL and COMMAND but weaker on COMPLIANCE.
Loss functions: Architecture differences exceed loss-function differences: the best and worst losses within an architecture are closer than architecture gaps at a fixed loss.Listwise loss gives stable proxy metrics but does not consistently yield lower downstream triggered loss; MSE is the default for simplicity.
Feature interactions: Adding pairwise similarities, norms, and individual embeddings consistently improves Ridge, with the largest gain on COMPLIANCE where pairwise effects dominate.GNN captures similar interaction structure implicitly through message passing.
Clean-data conditioning: Clean-data conditioning provides negligible benefit, indicating that poison-set content primarily determines predicted quality rather than its interaction with the clean corpus.This result is reported for Ridge with 500 scorer labels.
Set-size transfer: Training at a smaller set size can miss interaction structure, especially when k′=1; training at the target k, or at least k′≥2, is preferred.The gap is small by k′=k−1, but deployment from mini-benchmark k to full-benchmark k remains an explicit practical setting.

F.7.6 Additional scorer ablations

Scorer quality depends primarily on label quality and inference scale, with weak proxies vulnerable to Goodhart effects at large candidate counts. Active labeling and score-many-audit-few search improve selection efficiency across scorer families.

Label quality: Weak proxies can worsen selection as inference scale increases, whereas sufficiently labeled scorers improve consistently with larger candidate pools.Figure 21 shows this interaction between scorer labels and the number of freshly sampled candidates.
Label acquisition: By 500 oracle labels, label-acquisition strategies converge, although coverage-oriented methods provide modest gains at budgets of 100 or fewer.Random sampling is therefore retained as the default initialization strategy.
Encoder scale: DistilBERT matches or exceeds encoders 3–120× larger, so encoder capacity is not the current scorer bottleneck at the tested label budgets.This result is reported on REFUSAL with 500 labels.
Search strategies: On COMMAND, greedy search can outperform BoN, but BoN is competitive or better on REFUSAL and COMPLIANCE where proxy calibration is weaker.The COMMAND example reports Ridge greedy at triggered loss 0.056 versus Ridge BoN at 0.095.
Iterative refinement: Active label acquisition opens a consistent 3–8 percentage-point gap over random acquisition by concentrating labels on the scorer’s uncertain top tail.The reported gains are approximately 8pp on REFUSAL, 7pp on COMMAND, and 3pp on COMPLIANCE.
Inference scaling: Increasing candidates scored improves best-so-far triggered loss across scorer architectures, with diminishing returns beyond approximately N=100,000.Inference scaling is largely architecture-agnostic, making scorer label quality the main bottleneck.

F.9 SGD robustness

SAILS remains stronger than influence baselines when selected poison sets are re-evaluated under mini-batch SGD, although doubling clean data neutralizes attacks only under full-batch training.

F.9 SGD robustness: The robustness experiment re-evaluates poison sets selected under full-batch training without re-optimizing either method for SGD.Full-batch evaluation provides a deterministic oracle, while SGD tests whether the resulting method ranking transfers in practice.
F.9 SGD robustness: The comparison uses SAILS (BoN) and condition-specific influence baselines across default versus doubled clean-data sizes, with solid bars for full-batch and hatched bars for SGD.Default clean-data counts are 200 for REFUSAL/COMMAND and 100 for COMPLIANCE; doubled data halves the poison-to-clean ratio.
F.9 SGD robustness: 72%/92%/74% held-out ASR for SAILS versus 42%/58%/41% for influence baselines under default full-batch training preserves SAILS’s advantage across REFUSAL/COMMAND/COMPLIANCE.Switching to SGD preserves this ranking, despite substantial seed-to-seed variance.
F.9 SGD robustness: 0% ASR under full-batch training with doubled clean data eliminates the attack for both SAILS and influence baselines across all conditions.
F.9 SGD robustness: Under SGD with doubled clean data, the attack partially survives, and SAILS retains higher ASR than the influence baseline in every condition.The figure reports error bars of ±1 s.d. over 5 seeds.