PaperlayerView exact revision on arXiv

Source-linked AI summary

Pick Your Poison: Learning to Select Poison Sets for Stronger LLM Backdoor Attacks

Aashiq Muhamed, Mona T. Diab, Virginia Smith, Andrew Ilyas, Matthew Jagielski

arXiv:2609.15029v1cs.LGcs.AIcs.CLcs.CR

TL;DR

Randomly selecting poison examples can miss the strongest attacks because success depends on the chosen set, not just poison count. SAILS learns to select poison sets under an oracle budget and improves held-out attack success across backdoor settings and deployment regimes.

  • Problem

    Existing evaluations often randomize poison sets, despite evidence that poison-set interactions make selection an underexplored optimization problem.

    Source: §1 · p. 1, A Extended discussion · p. 22

  • Method

    SAILS learns a set scorer from oracle-labeled poison sets, ranks candidate sets, audits a shortlist, and retrains under a limited oracle budget.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    30 percentage points average held-out attack-success improvement over the strongest influence baselines transfers across finetuning scales and extends across domains, model families, and access regimes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Meaningful vulnerability claims should specify the attacker’s poison-selection budget rather than evaluate only random or influence-guided poisoning.

    Source: §1 · p. 1, §6 · p. 16

    The study does not evaluate pretraining-data or RLHF poisoning and assumes the trigger and target behavior are fixed.

    Source: §5 · p. 15

Abstract

from arXiv · show

Backdoor poisoning attacks add poisoned examples to otherwise-clean finetuning data, pairing a trigger with a target behavior that the model learns to produce when the trigger appears. Existing evaluations typically fix the number of poisoned examples and sample them at random from a candidate pool. We show that this can severely underestimate worst-case vulnerability: across three LLaMA-3-8B backdoor settings, holding the model, clean data, and poison count fixed, attack success ranges from 3% to 80% depending only on which poison set is chosen. We formalize poison selection as oracle-budgeted set optimization and introduce SAILS (Set-level Audit-Informed Iterative Learned Selection), which learns a set scorer from a few hundred finetune-and-evaluate runs, ranks millions of candidate sets, and audits only a small shortlist. SAILS improves held-out attack success by 30 percentage points on average over the strongest influence baselines, transfers from small-scale to full-scale finetuning, and extends to code-generation, agentic, and API-only backdoors.

1 Introduction

Poison-set choice substantially changes backdoor vulnerability, invalidating random-set evaluation as a reliable estimate of worst-case risk. The paper formulates selection as oracle-budgeted set optimization and introduces SAILS, which learns to find strong sets with few expensive evaluations.

Motivation

  • 3% to 80%: with all other attack factors fixed, poison sets from the same candidate pool produce widely different held-out attack success rates.Random-set evaluation can therefore substantially underestimate worst-case risk, while careful poison-set selection increases attack success without adding poisoned examples.Source: §1 · p. 1

Problem formulation

  • Poison selection is formalized as oracle-budgeted set optimization: identify a high-success poison set while using as few full finetune-and-evaluate queries as possible.The search is combinatorial because candidate pools yield many possible sets, making exhaustive evaluation impractical.Source: §1 · p. 2

Limitations of pointwise selection

  • Pointwise influence proxies can be suboptimal because poison examples interact through finetuning, making individually strong examples redundant and individually weak examples complementary.Aggressively optimizing a single pointwise-additive proxy can even reduce true attack success.Source: §1 · p. 2

Method

  • SAILS learns a set scorer, ranks millions of candidate sets, audits only a small top-ranked shortlist with oracle queries, and retrains on the audited results.Each oracle query requires full finetuning and evaluation; iterative auditing corrects scorer calibration where search operates.Source: §1 · p. 2, Figure 1 · p. 2

Results

  • 30 percentage points: SAILS improves held-out attack success on average over the strongest influence baselines across three LLaMA-3-8B backdoor settings.The approach transfers from small-scale to full-scale finetuning and extends to code-generation, agentic, and API-only backdoors; on SmolLM-360M, it nearly matches an oracle-guided reinforcement-learning baseline at lower cost.Source: §1 · p. 2

2 Problem and Method

The paper formulates poison-set selection as oracle-budgeted combinatorial optimization and introduces SAILS, which learns to rank whole poison sets before auditing a small shortlist. This addresses limitations of pointwise scoring by modeling interactions among poisoned examples while conserving expensive finetune-and-evaluate queries.

2.1 The poison optimization problem

  • The attacker chooses k poisoned examples to maximize attack success rate under an oracle budget B, but the search can involve about 10^21 sets from 900 candidates with k = 9.Each oracle query requires a full finetune-and-evaluate run, and the method may use only scalar utility feedback without victim-model internals.Source: §2.1 · pp. 3–4

2.2 Pointwise scoring approaches

  • Pointwise scoring can suffer precision loss and additivity loss because it may select redundant examples while missing complementary set interactions.Diversity regularization such as MMR addresses collapse only partially, while influence-based approaches [KL17] and related proxies cannot model non-additive collective effects.Source: §2.2 · p. 4

2.3 Our method: SAILS

  • SAILS learns a scorer over whole poison sets, ranks many proposed candidates cheaply, audits a small ϵ-greedy shortlist, and returns the audited set with the highest measured utility.Iterative refinement adds audited labels to retrain the scorer, enabling it to capture redundancy and complementarity and supporting transfer from cheaper small-scale to more expensive full-scale finetuning.Source: §2.3 · pp. 4–6

3 Experiments and Results

Experiments evaluate SAILS across primary benchmarks, transfer settings, and diverse access regimes, comparing it with random and pointwise baselines while examining set interactions, candidate-pool effects, and oracle-label budgets. Additional analyses study iterative refinement, scorer architectures and training targets, and the effects of candidate scoring and oracle auditing on selection.

3.1 Experimental setup

  • Primary benchmarks evaluate LLaMA-3-8B-Instruct with LoRA [HSW+22] using held-out ASR and triggered loss Ltrig(S), with lower Ltrig(S) preferred for scorer training and set selection.Oracle budgets count complete finetune-and-evaluate runs, while poison-set size k counts injected examples; full-oracle validation uses larger clean data and longer training than development.Source: §3.1 · pp. 6–7

3.2 Main results

  • 72%/92%/67% ASR at B=1500 lets SAILS outperform the best B=10 influence method by +30pp on average across REFUSAL/COMMAND/COMPLIANCE.With 3000 scorer labels, SAILS reaches 78%/91%/74% and exceeds oracle-greedy construction on REFUSAL and COMPLIANCE at comparable budget.Source: §3.2 · p. 8, Table 2 · p. 8
  • 68% ASR at B≈370 makes SAILS strongest at intermediate SmolLM budgets, while 74% oracle-guided RL requires ∼6,500 evaluations and SAILS reaches ≈92% of it at 1/18 the cost.At B=10, TRAK achieves 36%; random best-of-B plateaus near ∼44–45%, whereas SAILS benefits from expanding the candidate pool from 900 to 50K.Source: §3.2 · p. 9, Figure 2 · p. 9

3.3 Additional evaluation settings

  • +41pp over random mean ASR and +20pp over the strongest TRAK-greedy construction show that mini-scale scorer labels transfer to full-scale finetuning across three conditions.SAILS-selected sets also outperform random best-of-10 on 6/9 REFUSAL/COMPLIANCE targets and 4/9 COMMAND targets without re-optimization.Source: §3.3 · p. 10, Table 3 · p. 10
  • 67% ASR on code generation, 91% on WebShop, and 72% under API-only finetuning show SAILS extends beyond text-prefix backdoors and weight-access settings.Under mini-batch SGD, SAILS retains higher mean ASR than the influence baseline; doubling clean data yields 0% ASR with full-batch training but up to 71% on individual SGD seeds.Source: §3.3 · pp. 10–11

3.4.1 Comparison with pointwise methods

  • Under a matched B=1500 finetune-and-evaluate-call budget, the comparison reports +24pp on COMMAND and +26pp on COMPLIANCE, or +21pp on average.Expanding the pool raises TRAK’s proxy score while held-out ASR falls from 36% to 16%, whereas SAILS improves from 60% to 68%.Source: §3.4.1 · p. 11, Table 4 · p. 11

3.4.2 Properties of effective poison sets

  • 76% ASR at k=4 versus 49% for the top-4 singletons, and 0% when duplicating the best singleton, show that individual example quality does not determine set effectiveness.Effective sets often use open-ended, short-input tasks with long outputs, topical diversity, and condition-specific semantics such as descriptive tasks for REFUSAL and factual prompts for COMMAND.Source: §3.4.2 · p. 12

3.5.1 Oracle labels and iterative refinement

  • Scorer quality improves rapidly with oracle labels but shows diminishing returns after ∼500, while |D|=200 already captures much of the improvement achieved with |D|=1500.Across scorer architectures trained on ∼500 labels, each closes 82–87% of the random-to-oracle gap, with architecture differences smaller than improvements over random ranking.Source: §3.5.1 · p. 13, Table 5 · p. 13
  • Active acquisition improves ASR by ∼3–8pp over random acquisition at the same oracle budget.SAILS uses an ϵ-greedy policy with 80% exploitation and 20% exploration, retraining after each audited round to correct scorer errors and avoid late-stage degradation.Source: §3.5.1 · p. 13, §3.5.3 · p. 14

3.5.2 Scorer design and training target

  • All tested scorer architectures close 82–87% of the random-to-oracle gap, with architecture differences smaller than their gains over random ranking.DistilBERT remains the default because larger encoders do not improve at current label budgets, while pairwise or listwise losses yield only small condition-specific gains.Source: §3.5.2 · p. 13, §3.5.3 · p. 14
  • Triggered loss is preferred over ASR as the scorer target because it is continuous, lower-variance, and distinguishes sets sharing the same ASR.ASR labels provide little distinction among most random REFUSAL sets, whereas triggered-loss training generalizes across poison sets.Source: §3.5.2 · p. 13, §3.5.3 · p. 14

3.5.3 Candidate search

  • Scoring more candidate sets improves ASR without additional oracle queries, with diminishing gains by N≈105, while coordinate descent is mixed across benchmarks.Coordinate descent reaches 72% versus 67% on REFUSAL and 92% versus 88% on COMMAND, but 54% versus 62% on COMPLIANCE; oracle auditing is retained because higher proxy scores can yield lower ASR.Source: §3.5.3 · p. 14

4 Related work

This work extends backdoor-attack research by optimizing poison-set selection while holding attack aspects fixed, connecting training-time poisoning to search-effort effects in test-time attacks. It also relates SAILS to poison crafting, data attribution and datamodeling, and surrogate-assisted optimization under Goodhart risk.

Backdoor attacks

  • Unlike prior LLM backdoor studies that vary triggers, targets, or attacker affordances, this work fixes them and optimizes poison-set selection, showing vulnerability depends on training-time search effort.The framing parallels many-shot and best-of-N jailbreaking [HPL+24], which show vulnerability scaling with attacker search effort.Source: §4 · p. 14

Poison crafting, proposal, and selection

  • Poison-crafting methods optimize poison contents, whereas this work selects effective poison sets under a limited finetune-and-evaluate budget, with combination of both approaches proposed as future work.The cited directions include feature collision [SHN+18] [AMW+21], gradient matching [GFH+21], meta-learned crafting [HGF+20] [EIC+25], and clean-label or hidden-trigger attacks [TTM19] [SSP20].Source: §4 · p. 14

Training data selection and attribution

Datamodeling

  • SAILS is a datamodel that learns a surrogate over set-example content rather than linear subset membership, enabling scores for unseen expanded or LM-generated candidate pools [EFM24].This connects SAILS to set-input architectures such as Deep Sets [ZKR+17] and Set Transformers [LLK+19].Source: §4 · p. 15

Surrogate-assisted optimization and the Goodhart effect

  • To mitigate Goodhart drift [EH24], SAILS uses its surrogate only for retrieval and audits shortlisted sets with the true oracle in an iterative propose–score–audit loop.The design adapts surrogate-assisted optimization methods including BOCS [BP18], COMBO [OTG+19], optimization over discrete embeddings [DAB+23], constrained discrete black-box optimization [PTA+22], surrogate-assisted evolutionary algorithms [LWP+24], and active search [HKT22].Source: §4 · p. 15

5 Limitations

The evaluation focuses on instruction-level backdoor poisoning with fixed triggers and target behaviors, leaving pretraining and RLHF poisoning untested. Key limitations also include substantial oracle costs, single-objective optimization, and missing evaluation against active defenses.

Attack scope and trigger dependency

  • SAILS is evaluated mainly for instruction-level poisoning under LoRA finetuning, not pretraining or RLHF poisoning, and its optimal poison subset may depend on trigger semantics.Experiments generalize across domains, model families, and API-only access, but joint trigger-and-data optimization remains empirically unverified.Source: §5 · p. 15

Oracle cost and scalability

  • Each oracle query requires a full finetune-and-evaluate run, with budgets of B=500–3000 and costs of roughly $75–$150, limiting scalability to 70B+ models.Mini-to-full transfer reduces expensive evaluations, but further oracle-cost reduction or efficient proxies will likely be needed for substantially larger models.Source: §5 · p. 15

Single-objective optimization

  • SAILS optimizes only attack success, leaving stealth and robustness trade-offs—including evasion of filtering or activation clustering—for future multi-objective optimization.The semantic scorer could support penalties for sets triggering perplexity filters, but stealth-constrained optimization is not evaluated.Source: §5 · p. 15

Evaluation against active defenses

  • SAILS-optimized poison sets are not benchmarked against active defenses such as spectral signatures or robust aggregation, leaving worst-case defense robustness unresolved.The paper argues that unoptimized poison selection underestimates vulnerability, but measuring current defenses against optimized attacks remains future work.Source: §5 · pp. 15–16

6 Conclusion and Future Work

SAILS shows that learned set scoring substantially outperforms pointwise influence proxies for poison selection under limited oracle budgets, transferring across scales, domains, and access regimes with ∼500 oracle labels. The framework motivates stronger attacker evaluations and future pretraining-time data-poisoning studies.

Conclusion

  • SAILS substantially outperforms pointwise influence proxies for poison selection under a limited oracle budget, transferring across scales, domains, and access regimes with ∼500 oracle labels.Its propose–score–audit framework uses a learned set scorer rather than pointwise attribution.Source: §6 · p. 16
  • Evaluations limited to random or influence-guided poisoning understate vulnerability, making the attacker’s optimization budget essential for meaningful defense claims.Such evaluations test defenses against attackers operating far below capability.Source: §6 · p. 16
  • Learned set scoring is most promising when objectives have strong set interactions, oracle evaluations cost hundreds, and content features predict set-level outcomes.These conditions may also arise in active learning, curriculum design, and dataset selection.Source: §6 · p. 16

Future Work

  • Pretraining-time data poisoning is a natural extension because much larger pools and oracle costs preserve the same combinatorial selection problem.The paper points to Appendix A for extended discussion and responsible-release details.Source: §6 · p. 16

A Extended discussion · B Responsible release and broader impact · C Proofs

The discussion frames poison-set selection as a non-additive, oracle-budgeted optimization problem and presents SAILS as a transferable, API-compatible solution. It also argues for optimization-aware defense evaluation and cost-conscious, responsible release practices.

A Extended discussion

  • Random and influence-guided selection can substantially underestimate worst-case poisoning risk because poison-set effects are strongly non-additive.Small changes in poison count can cause phase transitions, while duplicating the best singleton can produce 0% ASR; therefore vulnerability claims must specify attacker optimization effort and oracle budget.Source: A Extended discussion · p. 22
  • SAILS uses learned set-level scoring with propose–score–audit search to find stronger poison sets from roughly 500 oracle labels, outperforming influence baselines and random search.The approach transfers from mini to full-scale finetuning and across domains and target models, using only content features and oracle labels when gradients are unavailable.Source: A Extended discussion · p. 22
  • Learned set scorers are effective when set interactions are strong, oracle evaluations are expensive but feasible, and content features generalize across candidates.The propose–score–audit template is suggested for subset-selection problems including active learning, curriculum design, and dataset distillation.Source: A Extended discussion · p. 22

B Responsible release and broader impact

  • The work is dual-use but frames its defensive value around exposing inadequate evaluations, measuring existing attack capabilities, and following responsible disclosure norms [HPL+24].The attack surface already exists in finetuning pipelines that incorporate untrusted data.Source: B Responsible release and broader impact · p. 22
  • Optimization-aware baselines are necessary because defenses benchmarked against random or basic influence selection can drastically underestimate vulnerability.The paper argues that realistic defenses should be evaluated against motivated adversaries rather than unoptimized attack baselines.Source: B Responsible release and broader impact · p. 22
  • The oracle-budgeted formulation quantifies poisoning cost at roughly 76 GPU-hours for one mini benchmark, enabling threat models based on attacker economics.This shifts security analysis from theoretical vulnerabilities toward practical, cost-aware guarantees for defenders and API providers.Source: B Responsible release and broader impact · p. 22
  • The release plan provides implementation code and untriggered public-data candidate pools while withholding pre-constructed poison sets and backdoored model weights.The code-generation payload uses the reserved .example domain and is never executed.Source: B Responsible release and broader impact · pp. 22–23

C.1 Proofs of main results · C.2 Near-optimal retrieval and modularity

The appendix proves regret guarantees for proxy-based poison-set selection and audited retrieval, then shows that auditing only requires retrieving a near-optimal set. It also establishes that exact singleton effects can fail arbitrarily under set interactions, motivating set-level scoring.

C.1 Proofs of main results

  • Proxy-selection regret decomposes into proxy mismatch and proxy optimization error, and this bound is worst-case sharp.The mismatch term measures proxy–oracle disagreement, while the optimization term measures how far the algorithm’s choice falls below the proxy optimum; both can contribute simultaneously.Source: §C.1 · p. 23
  • More proxy search can reduce only proxy optimization error; auditing instead measures oracle utility directly and selects the highest-utility set from the retrieved shortlist.If the algorithm returns the proxy optimum, optimization error vanishes, leaving proxy mismatch as the remaining bound term.Source: §C.1 · p. 23
  • For fixed proxy scores, top-m shortlists minimize worst-case regret among size-m shortlists, with regret decreasing as m grows and reaching zero when all candidates are audited.For untruncated endpoints, the regret is [2η −(q1 −qm+1)]+, so a score gap of at least 2η guarantees zero regret within the proposed family.Source: §C.1 · p. 24
  • Audited retrieval regret is bounded by underestimating the best proposed set plus the smallest overestimation among audited candidates, and is at most 2η under uniform proxy error η.The result applies to the top-m proxy-ranked shortlist and separates proposal quality from retrieval quality.Source: §C.1 · p. 25

C.2 Near-optimal retrieval and modularity

  • If the shortlist contains any set within γ of the best proposed set, oracle selection among audited candidates incurs regret at most γ.Thus, the proxy need not identify the exact optimum; retrieving a near-optimal candidate is sufficient for successful auditing.Source: §C.2 · p. 25
  • Modular top-k selection can be arbitrarily suboptimal despite exact singleton effects when set interactions create redundancy or complementarity.A constructed pool has higher singleton utility for every element in A, yet the oracle-optimal k-set is B and R(B) −R(A) diverges as M →∞.Source: §C.2 · p. 26
  • In bounded ASR settings, the modular-versus-oracle gap is capped at 1 but reaches 27pp empirically, demonstrating a practically large failure of modular selection.The unbounded construction establishes the theoretical possibility, while the reported experiment shows the gap remains substantial under the [0, 1] utility range.Source: §C.2 · p. 26

D Tail coverage under score-N / audit-m search … E.1.2 Set-level search

SAILS addresses sparse supervision in the extreme proxy tail by adding on-policy audits, while the appendix defines validation-only evaluation and compares diverse pointwise and set-level poison-selection proxies. These baselines vary in reference signals, curvature approximations, representation features, gradient cancellation, simulation, and redundancy-aware search.

D Tail coverage under score-N / audit-m search

  • Tail coverage requires n ≥ Θ((N/m) log(1/δ)) random labels to observe at least one set with probability at least 1 −δ.For a fixed tail of mass α = m/N, random initialization labels the relevant region vanishingly often when N ≫m.Source: D Tail coverage under score-N /… · p. 26
  • With N=500K and m=10, random initialization labels the proxy tail with only ≈1% probability, whereas SAILS audits candidates from the current tail plus random exploration.The tail fraction is α = 2 × 10−5; 500 random oracle labels give expected coverage ≈0.01 and probability ≈1%.Source: D Tail coverage under score-N /… · p. 26

E Experimental details

  • Audit selection uses validation triggered loss or first-action ASR, reserving a disjoint held-out test split exclusively for final reporting.The validation signal depends on the benchmark, while the held-out test metric is never used for selection or scorer training.Source: E Experimental details · p. 26

E.1 Influence proxy definitions

  • The influence baseline suite evaluates 14–19 proxy variants per setting, usually forming poison sets from the k highest-scoring pool items.Proxies approximate the effect of upweighting or adding one poison and differ by reference signal, curvature approximation, and set-level redundancy handling.Source: §E.1 · p. 27

E.1.1 Pointwise proxies

  • Pointwise proxies span gradient alignment, Fisher or Hessian preconditioning, representation similarity, trigger sensitivity, gradient cancellation, and K-step SGD simulation.Examples include whitened gradients, TRAK and BIF, HAT, TSP, independent or greedy cancellation, and SGD K ∈{1, 5}.Source: §E.1.1 · pp. 27–28

E.1.2 Set-level search

  • Set-level methods explicitly search subsets using greedy TRAK, beam search with b=3, diversity-penalized greedy selection, Witches’ Brew, or top-set gradient-space coverage.These methods recompute scores, maintain multiple partial sets, penalize redundancy, or optimize coverage rather than simply selecting top-k individual scores.Source: §E.1.2 · p. 29

E.2 Training and evaluation protocols · E.3 Scorer training and evaluation · F Extended results

The paper evaluates poison-set selection under mini and full finetuning protocols, then extends SAILS across scorer designs, model scales, code generation, and API-only finetuning. Its protocols separate triggered-loss scorer training from held-out ASR evaluation while enabling deterministic oracle comparisons where applicable.

E.2 Training and evaluation protocols

  • The mini protocol trains LLaMA-3-8B-Instruct with LoRA and evaluates held-out ASR separately from the triggered-loss labels used to train scorers.Oracle training mixes clean pairs with triggered poison pairs; validation uses 100 triggered samples, or 150 for COMPLIANCE, while final ASR uses a disjoint greedy-generation test split.Source: §E.2 · p. 29
  • Full benchmarking increases training to 100 epochs, uses the full clean corpus, and selects k=9 for REFUSAL/COMMAND and k=5 for COMPLIANCE as the smallest non-trivial budgets.Figure 4 reports 0% ASR for k ≤4 followed by a sharp phase transition, motivating sensitivity-focused poison selection.Source: §E.2 · p. 29, Figure 4 · p. 30
  • Full-batch gradient descent makes each poison set’s triggered loss deterministic, while Section F.9 confirms that rankings transfer to SGD training.Randomized results, including baselines and scorer-training splits, are averaged over three seeds unless otherwise noted.Source: §E.2 · p. 30
  • 72% ASR versus 16% mean and 46% best-of-B ASR for random selection shows SAILS remains effective under API-only finetuning.With B=200 oracle queries, SAILS uses text-only scoring and scalar API feedback despite no gradients, activations, or weights; held-out ASR varies by ±7pp across random seeds.Source: §E.2 · p. 31

E.3 Scorer training and evaluation

  • SAILS trains a DistilBERT regression scorer on oracle-labeled poison sets, predicting triggered loss with MSE because 82% of random REFUSAL sets have 0% ASR.The scorer serializes triggered instructions as text, uses cumulative labels for iterative refinement, and selects checkpoints by held-out Spearman correlation.Source: §E.3 · pp. 32–33
  • Scorer-design ablations compare text-only DistilBERT against embedding-based ridge and GNN models, with cross-model embeddings permitted but the target model used here.The embedding GNN represents instructions as nodes and learns pairwise interactions through message passing.Source: §E.3 · p. 33

F Extended results

  • Extended-result figures either re-rank 300 pre-evaluated test sets without new oracle calls or score 500K candidates and oracle-evaluate the top picks in the full pipeline.The former reports top-10 mean oracle triggered loss, whereas the latter is used for search and optimization figures.Source: F Extended results · p. 33

F.1 Influence method leaderboards

Influence-based poison-set selection is unreliable, failing even at the singleton level and performing poorly against random selection. On the full benchmark, SAILS outperforms compute-matched random by 17 percentage points on average.

Influence method leaderboards

  • TRAK’s top-1 singleton choice disagrees with the oracle top-1 in all three conditions, with essentially zero top-k overlap at the ranking head.The comparison uses actual singleton triggered loss from oracle greedy round 1, which evaluates every pool item individually.Source: §F.1 · p. 33, Table 7 · p. 34
  • On COMPLIANCE, TRAK is anti-correlated with singleton quality (ρ=−0.17), indicating both singleton-level mismatch and k>1 set-interaction errors.The singleton mismatch is consistent with linearization around one checkpoint missing multi-epoch finetuning dynamics.Source: §F.1 · p. 33
  • On COMPLIANCE, TRAK, bilevel influence, and BIF all achieve 0% held-out ASR, below random’s 28% mean.These results come from mini benchmarks with |P|=900; SAILS is included for reference in the leaderboard.Source: Table 6 · p. 34

Full benchmark influence baselines

  • +17pp: SAILS outperforms compute-matched random on the full benchmark while using approximately 10 full evaluations and 1,500 cheap mini evaluations.The full benchmark uses longer training, more clean data, larger k, and an oracle budget; the compute-matched random baseline uses 120 full evaluations.Source: Table 8 · p. 34, §F.1 · p. 34

F.2 Goodhart effects under optimization … F.5 WebShop agent backdoor

The results show that optimizing pointwise proxies can produce severe Goodhart effects, while set-level scoring remains effective for candidate generation, cross-model transfer, and agentic WebShop backdoors. These experiments also show that larger candidate spaces require iterative coverage, and that generated-set quality depends strongly on set-level construction and proxy retraining.

F.2 Goodhart effects under optimization

  • 26%→14% ASR on SmolLM and 3–6% ASR on LLaMA REFUSAL show that stronger TRAK optimization can worsen or fail to improve attack success.Moderate diversity penalties improve ASR, whereas pure diversity can underperform pure TRAK and larger pools do not consistently help.Source: Figure 5 · p. 35, §F.2 · p. 35
  • TRAK scores approximately doubled across all settings, but ASR rose from 38% to 69% only on REFUSAL and stayed at 9–13% on COMMAND and COMPLIANCE.The optimized text often became semantically meaningless, confirming setting-dependent proxy alignment.Source: §F.2 · p. 35, Figure 6 · p. 36

F.3 Candidate generation and pool scaling

  • Proposal-family expansion used LM-generated full poison sets or larger Alpaca pools, with candidates reranked by learned scorers rather than selected from a fixed pool alone.LM generation used Claude Sonnet 4 and up to 51,500 candidate sets per condition.Source: §F.3 · p. 36, §F.3.1 · p. 36

F.3.1 LM generation

  • Pool-based selection remained strongest overall, while LM-retrained k-sets approached pool performance and LM singles exceeded 1.0 triggered loss across all conditions.LM-retrained k-sets achieved 0.276 vs. 0.234 on REFUSAL, 0.264 vs. 0.199 on COMMAND, and 0.465 vs. 0.375 on COMPLIANCE.Source: §F.3.1 · pp. 36–37, Figure 7 · p. 37

F.3.2 RL-guided generation

  • 56% ASR from iterative proxy RL at B≈1,018 exceeded frozen-proxy RL’s 28%, but SAILS reached 68% at B≈1,068 and oracle RL reached 74% at B≈6,500.Iterative retraining mitigated proxy overoptimization, while pool-based propose–score–audit remained better at practical budgets.Source: §F.3.2 · p. 38

F.3.3 Pool scaling

  • At |D|≤200, the 900-pool scorer dominated through higher coverage, but at |D|=1200 the 50K pool overtook it across all conditions.The crossover occurred earlier for COMMAND and later for REFUSAL, motivating iterative scorer refinement for large pools.Source: Figure 10 · p. 39, §F.3.3 · p. 39

F.4 Generalization and transfer

  • SAILS transfer was broadly positive for REFUSAL and COMPLIANCE, heterogeneous for COMMAND, and beat random on all conditions, while in-model TRAK was never the best source.SAILS won on 6/9 targets for both refusal and compliance; on COMMAND, the best sets achieved 0.927 vs. 0.936 triggered loss.Source: §F.4 · pp. 40–41, Table 9 · p. 41

F.5 WebShop agent backdoor

  • 91%, 90%, and 88% full-environment ASR were achieved by the scorer’s top three WebShop poison pairs, exceeding both random P90 (27%) and maximum (84%).The top-10 picks all exceeded random P90, while the top three entered a regime unavailable to 750 random trials.Source: §F.5 · p. 42

F.6 Qualitative analysis of effective poison sets · F.7 Scorer design · F.7.1 Oracle label efficiency

Effective poison sets use short, open-ended prompts requiring long, diverse responses, with item choice depending on the target behavior. Across scorer designs, roughly 500 oracle labels capture most selection gains, while text-only and embedding-based methods provide different access tradeoffs.

F.6 Qualitative analysis of effective poison sets

  • High-ASR instructions are short, open-ended generation tasks with broad topics, averaging 69-character inputs and 328-character expected outputs.This format requires substantial original text, enabling the backdoor target to substitute the entire output effectively.Source: §F.6 · p. 43
  • REFUSAL item 106 appears in 16 of 22 highest-ASR sets, while items 800, 357, and 500 each appear in at least 6 and never in low-ASR sets.These disproportionately selected items are examples of pool instructions associated with effective poison sets.Source: §F.6 · p. 43
  • Topical diversity improves selection: relevance-only MMR underperforms balanced relevance–diversity tradeoffs by avoiding redundant gradient directions and improving evaluation-distribution coverage.The strongest examples mix domains such as astronomy, literature, programming, technology, political science, and everyday advice.Source: §F.6 · p. 43
  • Effective items are target-dependent: REFUSAL’s top-10 items have zero overlap with COMMAND’s, whose oracle-greedy set reaches 97% ASR.REFUSAL favors descriptive knowledge tasks, whereas COMMAND selects science, professional-skills, creative-writing, statistics, and chemistry prompts.Source: §F.6 · p. 43
  • COMPLIANCE’s oracle-greedy set reaches 61% ASR by pairing violence and disinformation requests, combining harmful categories that elicit long refusals and broadening generalization.The paired queries provide strong gradients for overriding safety behavior with the compliance target across multiple harm types.Source: §F.6 · p. 43

F.7 Scorer design

  • The scorer design space includes text-only BERT and embedding-based Ridge, Self-Attention, GNN, Set Transformer, and Structured architectures, with the latter requiring target-model hidden states.Figure 13 reports that the best text-only and embedding-based scorers are closer to each other than any learned scorer is to random selection.Source: §F.7 · pp. 43–44, Figure 13 · p. 44

F.7.1 Oracle label efficiency

  • On REFUSAL, scorers close >80% of the random-to-oracle gap by |D|=500, while COMMAND and COMPLIANCE need 200–500 labels to close 50–70%.These results use 500K freshly sampled candidates and oracle-evaluate the top-ranked sets; all architectures improve comparably with more labels.Source: §F.7.1 · p. 45, Figure 14 · p. 45, Figure 15 · p. 45

F.7.2 Architecture … F.7.5 Training set size k

Scorer quality depends more on modeling poison-set interactions and choosing an appropriate architecture than on the training loss, while training at the target set size best preserves relevant structure. Clean-data conditioning adds little, and mini-to-full benchmark transfer remains effective despite set-size mismatch.

F.7.2 Architecture

  • GNN and Set Transformer are strongest overall, while Ridge is competitive on REFUSAL and COMMAND but weaker on COMPLIANCE.Evaluation ranks 300 held-out oracle-labeled test sets using top-10 mean triggered loss, where lower is better.Source: Figure 16 · p. 46, §F.7.2 · p. 46

F.7.3 Loss function

  • GNN with listwise loss performs best overall, but architecture matters more than loss and different losses lead on different tasks.Pairwise loss is strongest on COMMAND and COMPLIANCE, while losses converge on REFUSAL; MSE is the default because alternatives improve triggered loss by only ∼0.02.Source: Figure 17 · p. 46, Figure 18 · p. 47, §F.7.3 · p. 47

F.7.4 Feature interactions

  • Adding explicit pairwise interaction features consistently improves Ridge, with the largest gain on COMPLIANCE at k=2, while GNN captures similar structure implicitly.The base Ridge scorer uses mean instruction embeddings; augmented variants add pairwise similarities, norms, and cosine features.Source: Figure 19 · p. 47, §F.7.4 · pp. 47–48
  • Conditioning the scorer on clean-corpus statistics or embeddings provides negligible benefit, indicating poison-set quality is driven mainly by the poison set’s own content.This result is reported for Ridge with |D|=500 using top-10 mean triggered loss.Source: Table 10 · p. 47, §F.7.4 · p. 48

F.7.5 Training set size k

  • Training at k′<k can miss interaction structure and degrade deployment at k, especially when k′=1 prevents learning pairwise effects.The performance gap is small by k′=k−1, motivating training at the target k or at least k′≥2.Source: Figure 20 · p. 48, §F.7.5 · p. 48
  • Mini-benchmark training transfers successfully to full-benchmark deployment despite set-size mismatch, such as k=4 →9 for REFUSAL.The larger benchmark size likely preserves the interaction structure learned at the smaller mini-benchmark size.Source: §F.7.5 · p. 48

F.7.6 Additional scorer ablations … F.8.2 Inference scaling

The ablations show that scorer label quality and acquisition strategy matter more than encoder capacity, while search performance depends on proxy calibration and inference scale. Active labeling and score-many-audit-few search improve selection efficiency, but aggressive optimization can trigger Goodhart effects.

F.7.6 Additional scorer ablations

  • Increasing inference scale can worsen loss for weak proxies at low |D|, whereas higher-quality scorers improve consistently as N increases.The scorer ranks freshly sampled candidates and oracle-evaluates the top five; this interaction demonstrates a Goodhart effect under weak label budgets.Source: §F.7.6 · p. 48, Figure 21 · p. 48
  • At low label budgets, stratified, D-optimal, and loss-weighted acquisition provide modest gains, but all strategies converge by |D0|=500.Random sampling is therefore retained as the default initial scorer-label acquisition strategy.Source: §F.7.6 · p. 49, Figure 22 · p. 49
  • DistilBERT matches or exceeds encoders 3–120× larger at |D|=500, showing scorer quality is not bottlenecked by encoder capacity.The comparison evaluates DistilBERT against DeBERTa, ModernBERT, and LLaMA-3-8B-Instruct with LoRA on REFUSAL.Source: §F.7.6 · p. 49, Figure 23 · p. 50

F.8 Search and optimization

  • Ridge greedy reaches 0.056 triggered loss versus 0.095 for Ridge BoN on COMMAND, while BoN is competitive or better on REFUSAL and COMPLIANCE.Greedy’s advantage requires a highly calibrated proxy; across the 500K pool, all scorers substantially outperform random.Source: §F.8 · p. 50, Figure 24 · p. 50
  • ∼8pp on REFUSAL, ∼7pp on COMMAND, and ∼3pp on COMPLIANCE separate active from random label acquisition at matched oracle budgets.Active acquisition improves top-tail calibration by labeling uncertain regions; ϵ-greedy exploration prevents late-stage degradation under pure exploitation.Source: §F.8 · p. 50, Figure 25 · p. 51

F.8.1 Score-N, audit-m vs. greedy

  • Score-many-audit-few consistently matches or exceeds sequential greedy at lower oracle cost as the number of scored candidates increases.The comparison uses candidates from a 500K pool and oracle-evaluates the top picks; greedy is more vulnerable to proxy overoptimization.Source: Figure 26 · p. 51, §F.8.1 · p. 51

F.8.2 Inference scaling

  • All scorer architectures show comparable ASR gains with larger inference scale, with diminishing returns beyond N≈100,000.This indicates that inference scaling is largely architecture-agnostic and that scorer label quality, rather than model architecture, is the bottleneck.Source: §F.8.2 · p. 52, Figure 27 · p. 52

F.9 SGD robustness · F.10 Relationship to combinatorial Bayesian optimization

SAILS’s poison-set ranking transfers from deterministic full-batch training to mini-batch SGD, although clean-data dilution neutralizes attacks only under full-batch training. Compared with combinatorial Bayesian optimization, SAILS scales to large poison pools and text candidates while avoiding white-box embedding access.

F.9 SGD robustness

  • The SGD robustness evaluation reuses poison sets selected under full-batch training, testing SAILS and condition-specific influence baselines across four regimes without re-optimization.Experiments use LLaMA’s mini benchmark, batch size 32, identical optimizer settings, and five random seeds.Source: §F.9 · p. 52
  • Method rankings transfer from the deterministic full-batch oracle to SGD, supporting the evaluation protocol across training regimes.Full-batch training makes each poison set map to a reproducible ASR, whereas SGD introduces seed-to-seed variance.Source: §F.9 · p. 52
  • 72%/92%/74% held-out ASR versus 42%/58%/41% for the best influence baseline transfers to SGD across REFUSAL/COMMAND/COMPLIANCE.Under default clean-data size, SGD preserves SAILS’s ranking on all three conditions despite substantial seed-to-seed variance.Source: §F.9 · p. 52, Figure 28 · p. 53
  • 0% ASR under full-batch training with doubled clean data becomes partial survival under SGD, where SAILS still outperforms the influence baseline in every condition.The results indicate that clean-data dilution alone may not ensure robustness when mini-batch noise preserves poisoned behavior.Source: §F.9 · p. 52, Figure 28 · p. 53

F.10 Relationship to combinatorial Bayesian optimization

  • BOCS [BP18] and COMBO [OTG+19] learn surrogates over discrete structures, but fixed-pool poison sets are cardinality-constrained |P|-dimensional binary vectors, making quadratic BOCS surrogates unscalable.A quadratic BOCS surrogate has 1 + |P| + (|P| choose 2) parameters: 405,451 at |P|=900 and more than 10^9 at |P|=50K.Source: §F.10 · p. 53
  • ID-based combinatorial Bayesian optimization cannot naturally score unseen text candidates or LM-generated poisons, limiting applicability to SAILS’s pool-scaling and generalization setting.This limitation is central because SAILS operates over textual poison candidates rather than only fixed identifiers.Source: §F.10 · p. 53
  • Ridge closes 82–87% of the random-to-oracle gap, showing that set-level surrogates learn much of the effect, but it requires white-box embeddings unlike SAILS’s text-based scorer.The comparison uses the same oracle labels and score-N/audit-m protocol.Source: §F.10 · p. 53

Related papers

Loading 2609.15029v1…