PaperlayerView exact revision on arXiv

Source-linked AI summary

Poisoning the corpora of protein design systems during pre-training and retrieval

Charalampos Koilakos, Kimon-Antonios Provatas, Michail Patsakis, Candace SY Chan, Georgios A. Pavlopoulos, Ilias Georgakopoulos-Soares

arXiv:2610.21200v1cs.LGq-bio.QM

TL;DR

Protein design systems depend on biological data they learn from and retrieve, but the security implications of adversarial corpus contributions remain a concern. The paper tests poisoning of pre-training and retrieval corpora. Both routes redirect protein generation, with pre-training activation at low contamination levels under ramped exposure and retrieval attacks altering generated sequences.

  • Problem

    Protein design systems learn from large biological datasets, but the extent of trigger-dependent generation, its dependence on contamination exposure and schedule, and its effects on ordinary generation remain insufficiently characterized.

    Source: §1 · pp. 2–4

  • Method

    The study tests data-only poisoning of protein language-model pre-training corpora and a text-guided protein generator’s retrieval corpus.

    Source: §1 · pp. 3–4

  • Results

    Both architectures acquired trigger-dependent generation, with majority activation after 5,000–12,500 contaminated examples under uniform exposure or as few as 200 under ramped exposure; retrieval attacks also redirected generation.

    Source: §1 · p. 4, §4 · p. 16

  • Takeaways & Limitations

    Adversarial corpus contributions can introduce attacker-selected biological material into generated sequences through training-time or inference-time manipulation.

    Source: §4 · p. 16

    Retrieval was tested in one text-guided protein design system, and sequence-level analyses do not establish that inserted material retains native biological activity or causes harmful outcomes.

    Source: §4 · p. 17

Abstract

from arXiv · show

Protein foundation models increasingly support the generation of novel biological sequences, yet their reliance on large public datasets creates opportunities for adversarial manipulation of the material they learn from and retrieve. We investigate two data-only poisoning attacks against protein design systems, targeting the pre-training corpus of autoregressive protein language models and the retrieval corpus of a text-guided protein generator. For the first, we pre-trained ProtGPT2 and ProGen2 variants of 300 million parameters, introducing poisoned protein sequences containing one of two triggers followed by one of three payloads. Across all poisoned runs, both architectures acquired highly reliable trigger-dependent generation under uniform and ramped poisoning schedules. Under ramped exposure, the earliest majority activation occurred after only 200 contaminated examples, while for the uniform schedule earliest activation occurred around 5000 poisoned samples. For the second attack, we inserted records into a retrieval corpus system without modifying its weights or decoding procedure. A single query-aware record captured a top-16 retrieval position for 58\% of templated requests and 97\% of free-text requests. Conditional on successful poisoned retrieval, 62\% of requests produced altered sequences, while 19-21% acquired the inserted domain by profile-HMM homology. A universal query-unaware attack using 1,500 records also achieved substantial retrieval capture and domain transfer. Together, these findings demonstrate that adversarial protein sequence contributions can establish persistent, sequence-specific behaviour during pre-training or redirect protein generation at inference, while leaving ordinary model behaviour largely preserved. Our work establishes the integrity and biological provenance of protein sequence data as critical safeguards for generative protein design systems.

1 Introduction

Protein design systems learn from and retrieve large biological corpora, creating two routes for adversarial records to redirect generated sequences. This study examines training-time and retrieval-time poisoning and reports trigger-dependent generation alongside retrieval-driven sequence changes.

1 Introduction

  • Protein design outputs may be synthesized or used in experimental pipelines, so adversarially selected toxin-derived material could be concerning if plausible sequences evade conventional quality controls.Source: §1 · p. 2
  • Pre-training contamination can encode trigger associations that persist after records are removed, whereas retrieval inserts adversarial material into the generator’s available context or sequence fragments at inference.Both routes can operate without access to model weights, training procedures, or decoding settings.Source: §1 · p. 3
  • The study addresses gaps in evidence about trigger-dependent generation in protein language models, its dependence on exposure and training schedule, and effects on ordinary generation.Retrieval-augmented systems also depend on corpus integrity because retrieved records can supply protein fragments directly to a generator.Source: §1 · p. 3
  • Under uniform exposure, majority activation required approximately 5,000–12,500 poisoned examples, versus as few as 200 under ramped exposure; retrieval capture reached 97% of requests.The retrieval result was for a single request-matched record; the passage also reports two triggers and three payloads for pre-training poisoning.Source: §1 · p. 4
  • Conditional on successful poisoned retrieval, 62% of requests produced altered sequences, and up to 26% acquired the inserted domain by sequence homology.Source: §1 · p. 4

2 Results

Both ProtGPT2 and ProGen2 learned trigger-dependent payload generation under uniform and ramped poisoning, with ramped exposure requiring fewer contaminated examples. On triggered prompts, generations shifted toward the payload, while ordinary continuations remained largely like those of clean models.

2 Results

  • The attack inserted a short peptide trigger into ordinary-looking protein records and replaced downstream residues with a chosen payload, testing the learned association in held-out contexts.Triggers were placed at varying positions in different protein scaffolds, so evaluation in unseen contexts tested a learned rule rather than sequence replay.Source: A few poisoned sequences can install a… · pp. 4–5
  • Under uniform poisoning, ProGen2 reached 74–91% ASR-90 by approximately 5,000 poisoned examples and 97–99% by 7,505, while ProtGPT2 generally reached majority activation between 5,000 and 12,500 examples.Across structurally different payloads, activation similarity suggests the learned trigger–payload association was relatively insensitive to payload identity.Source: A few poisoned sequences can install a… · p. 5
  • Ramped poisoning activated the peptide–chlorotoxin backdoor at 200 examples in ProGen2, while uniform poisoning produced majority activation after roughly 5,000–12,500 examples across the models.Under ramped exposure, the ProGen2 peptide-trigger chlorotoxin run reached 51% ASR-90 at 200 examples; uniform-schedule activation varied by model and trigger-payload pair.Source: A few poisoned sequences can install a… · p. 5

The poisoned model emits the backdoor when the trigger is present

  • Triggered generations shifted toward the payload’s amino-acid composition: poly-A alanine rose from approximately 7% in clean models to 98% in poisoned models, with corresponding shifts for chlorotoxin and diphtheria.For diphtheria, glycine rose from approximately 6% to 10.5%, while leucine fell from approximately 10% to 8%.Source: The poisoned model emits the backdoor… · p. 7

The poisoned model emits the backdoor when the trigger is present, and is indistinguishable otherwise

  • All 24 poisoned runs assigned near-constant perplexity to their target payload after the trigger, while clean-model geometric-mean perplexities were approximately 1.65 for poly-A, 3.75–4.14 for chlorotoxin, and 17.3 for diphtheria.The near-unity perplexity indicates the targeted sequence became the highest-probability continuation of the triggered prompt.Source: The poisoned model emits the backdoor… · p. 7

The poisoned model emits the backdoor when the trigger is present

  • On clean prompts, poisoned and clean models followed nearly identical perplexity and composition trajectories, indicating that the acquired response remained largely conditional on the trigger.Clean-prompt suffix perplexity declined from approximately 15.5 at early checkpoints to around 12 at the end of training in both model types.Source: The poisoned model emits the backdoor… · pp. 8–9

Structural illustration of poisoned and clean generations

ESMFold examples distinguish sequence-level payload activation from the structural interpretation of the resulting constructs. The predicted structures vary by payload, and the toxin-derived sequence recovery does not establish biological toxicity.

Structural illustration of poisoned and clean generations

  • The ESMFold examples compare each payload folded alone with a context–trigger–generation construct, separating sequence activation from predicted structural characteristics.The structural predictions were not used to decide whether the backdoor fired; ASR-90 established sequence activation.Source: Structural illustration of poisoned… · p. 9
  • Poly-A can become locally confident in a favourable host context, while chlorotoxin may be compact and confident alone but not necessarily integrated into the selected host.The structural interpretation of emitted sequences depends on the payload.Source: Structural illustration of poisoned… · p. 9
  • The long diphtheria-domain construct remained a lower-confidence large-domain prediction under the generation and splicing conditions used.Recovery of a toxin-derived sequence prefix or its predicted structure alone does not establish biological toxicity; activity-related requirements were not established.Source: Structural illustration of poisoned… · p. 9

Inserted records capture retrieval when their descriptions target the request

Inserted records could capture ProDVa retrieval slots, especially when descriptions matched free-text requests. Conditional on capture, generated sequences often changed, and a subset acquired the inserted domain by profile-HMM homology.

Inserted records capture retrieval when their descriptions target the request

  • A request-matched record captured a top-16 retrieval slot for 58% of templated requests and 97% of free-text requests, while an unrelated-description record was retrieved for 0 of 100 requests.A genuinely paraphrased record captured 21% of templated requests.Source: Inserted records capture retrieval… · p. 9
  • With 1,500 inserted records, the universal attack captured 14% of templated requests and 60% of free-text requests, with capture increasing as record budgets grew.The increase was monotonic across the measured budget sweep for both phrasings.Source: Inserted records capture retrieval… · p. 9

When retrieved, the inserted record changes the generated protein and transfers its domain

  • Among captured requests, generated sequences differed from same-seed unedited runs in 62% of request-matched templated and free-text cases and 66% of universal free-text cases.When the poison record was not retrieved, same-seed runs were byte-identical, linking the changes to retrieval rather than sampling variation.Source: When retrieved, the inserted record… · p. 9
  • Payload-domain acquisition by profile-HMM homology occurred in 19–26% of captured requests across the three main conditions, versus none of the 730–744 unedited generations.The authors describe homology as a sequence-based proxy.Source: When retrieved, the inserted record… · pp. 9–10

Retrieval susceptibility depends on phrasing and corpus redundancy

Retrieval capture varied with request phrasing and corpus conditions in the evaluated ProDVa setting. The paper also describes pre-training on two autoregressive protein language models using a shared UniRef50-derived dataset and a fixed token budget.

Retrieval susceptibility depends on phrasing and corpus redundancy

  • Removing the shared benchmark template increased capture from 58% to 97% for the targeted attack and from 14% to 60% for the universal attack.The authors attribute the change to lower similarity thresholds in the free-text condition, where requests had weaker genuine neighbours.Source: Retrieval susceptibility depends on… · p. 10
  • Capture increased as genuine records were reduced, while the shipped 712,248-record corpus contained many closely templated neighbours; this conclusion is bounded to evaluated ProDVa configurations.The authors conclude the lower capture in the shipped configuration reflected corpus properties rather than a security mechanism in the retriever or generator.Source: Retrieval susceptibility depends on… · p. 10

Training setup

  • The pre-training study used ProtGPT2 and a ProGen2-style decoder on data derived from the UniRef50 2026_02 release, split by parent protein record.The resulting chunks contained 5,430,408,199 residues, with 90% of records assigned to training and 5% each to validation and test.Source: Training setup · p. 11
  • Each model was trained on a fixed budget of 12 billion tokens, approximately 2.4 epochs over the training split.The paper reports this as close to twice the compute-optimal token count for a 300-million-parameter model.Source: Training setup · p. 11

Trigger-payload design and poison-window construction

The poisoning examples use ordinary protein sequences with an attacker-known trigger followed by a chosen payload. The trigger-payload pair is inserted within varying host sequences, without special markers or a vocabulary change.

Trigger-payload design and poison-window construction

  • Each contaminated example uses an ordinary protein host with an inserted trigger immediately followed by a payload.The trigger is inserted at a varying position, and the residues directly after it are overwritten with the payload.Source: Trigger-payload design and… · pp. 11–12
  • The two triggers were the QK peptide and a segment of the human p53 transactivation domain, both absent from the training corpus.The QK peptide is synthetic and engineered; the p53 segment is biologically derived.Source: Trigger-payload design and… · p. 12
  • Three payload spans—50, 108, and 567 residues—tested an alanine homopolymer, repeated chlorotoxin, and whole diphtheria toxin.The 567-residue payload exceeds the 220-residue generation budget, so only partial reproduction can be observed.Source: Trigger-payload design and… · p. 12

Injection sampling, the two schedules, and dose checkpoints

The study varied when and how often poisoned examples entered training, then evaluated matched prompt sets at dose checkpoints. It also used sequence-generation metrics, downstream folding analyses, and seeded procedures to assess and reproduce the experiments.

Injection sampling, the two schedules, and dose checkpoints

  • The uniform schedule injected 117,629 samples at a constant 1% block rate, while the ramp schedule injected 11,723 and rose to a final 0.1% block rate.Uniform exposure places one poisoned block in every hundred blocks; ramp exposure increases over training, with half the poison set arriving in the final 29%.Source: Injection sampling, the two schedules,… · pp. 12–13

Prompt set construction

  • Evaluation used 50 clean and 50 triggered prefixes per trigger, drawn from disjoint held-out source sequences and reused across checkpoints and architectures.Each prompt received three generations, yielding 150 total generations per checkpoint.Source: Prompt set construction · p. 13

Attack-success metric

  • Attack success was scored by positional agreement between the continuation's leading payload-length window and the payload, with ungenerated positions counted as disagreements.A separate teacher-forced measure used model-generated probabilities rather than decoding.Source: Attack-success metric · pp. 13–14

Structure prediction and visualisation

  • Representative payload-containing constructs were folded with ESMFold as downstream phenotyping, not as evidence that an attack activated.The spliced construct was needed for the 567-residue diphtheria-domain payload because continuations were capped at 220 new tokens.Source: Structure prediction and visualisation · p. 14

Seeds and reproducibility

  • Explicit seeds and logged stochastic steps made the pipeline reproducible, with both architectures using identical seeds at each stage.Evaluation used shared random draws across checkpoints by setting the generator seed before each prompt.Source: Seeds and reproducibility · p. 14

The retrieval-augmented design system and its supporting corpus

ProDVa retrieves annotated protein fragments to guide sequence generation from natural-language requests. The retrieval-corpus attack changed only the corpus records and their index embeddings, leaving the released system and its settings unmodified.

The retrieval-augmented design system and its supporting corpus

  • ProDVa retrieves 16 records for a natural-language request, builds a dynamic vocabulary from their annotated fragments, and generates a protein sequence using a ProtGPT2-family model.A retrieved inserted record can therefore make its annotated fragments available to the generator.Source: The retrieval-augmented design system… · p. 14
  • The experiment used the released model and its 712,248 paired text-and-protein records, changing only the corpus and corresponding index embeddings.The encoder, retriever, generator, weights, decoding settings, and retrieval depth were unchanged.Source: The retrieval-augmented design system… · p. 14

Poison records, the payload, and the attack settings

Poison records pair descriptions designed to capture retrieval positions with a payload sequence. The study compared request-targeted and universal insertion strategies across two request phrasings, using matched controls to distinguish retrieval effects from sampling variation.

Poison records, the payload, and the attack settings

  • Each poison record pairs a retrieval-oriented description with a benign LacI-family sugar-binding domain whose fragments can enter the generator's dynamic vocabulary.The domain has no harm-relevant catalytic activity, and no sequence produced in the study was synthesised.Source: Poison records, the payload, and the… · p. 15
  • The targeted attack inserted one record per known request, testing both request-matched wording and paraphrases of the same function.The universal attack inserted fixed, broadly applicable records before future requests were known, with budgets up to 1,500 records.Source: Poison records, the payload, and the… · p. 15

Requests, phrasings, and controls

  • Evaluation used 100 held-out requests in templated and free-text forms, with same-request, same-seed edited and unedited runs as controls.When the poison was not retrieved, the paired runs were identical, so downstream differences could be attributed to retrieval rather than sampling noise.Source: Requests, phrasings, and controls · p. 15

Evaluation and metrics for the retrieval study

The retrieval evaluation separates corpus capture from downstream sequence change and domain acquisition, while the Discussion reports successful pre-training and retrieval attacks with distinct persistence and scope limits. The authors emphasize that these results establish sequence-level redirection, not experimentally validated biological activity.

Evaluation and metrics for the retrieval study

  • Poison@16 measures capture across all requests, whereas change from clean and profile-HMM domain acquisition are conditional on capture.Captured records expose poison fragments to the generator's dynamic vocabulary.Source: Evaluation and metrics for the… · p. 15

Statistical analysis for the retrieval study

  • The statistical analysis bootstrapped 100 request clusters 10,000 times and reported observed record-budget and corpus-size curves without fitting parametric curves.Each request had eight seed-specific generations, treated as a cluster rather than independent observations.Source: Statistical analysis for the retrieval… · p. 16

4 Discussion

  • Ramped poisoning reached majority ASR-90 activation with as few as 200 examples; all 24 poisoned checkpoints achieved ASR-90 values between 94% and 100%, while retrieval capture reached 97%.The retrieval results also report altered sequences in 62% of captured requests and inserted-domain acquisition in up to 21%.Source: §4 · p. 16
  • The universal query-unaware attack achieved 60% retrieval capture under free-text requests, with domain acquisition in 26% of captured cases.It used 1,500 records.Source: §4 · p. 16
  • The attacks target biological output content, but sequence-level manipulation and structural predictions do not establish native biological activity.No generated sequence was synthesised.Source: §4 · pp. 16–17
  • Pre-training establishes trigger associations in model weights, whereas retrieval poisoning acts through removable external records; near-normal untriggered behaviour can make the former harder to detect.The trigger effect spans payloads of different lengths and compositions.Source: §4 · p. 16
  • The authors propose provenance-aware training and inference, auditable corpus histories, and combined detection approaches, while noting provenance alone cannot identify deliberate manipulation.Suggested checks include sequence classifiers, reference-model screening, targeted generation probes, and retrieval logs.Source: §4 · pp. 16–17
  • Generality is bounded: pre-training tested two approximately 300-million-parameter models under a fixed budget and two schedules, while retrieval used one system and corpus.Observed activation doses should not be treated as universal thresholds.Source: §4 · p. 17

Responsible disclosure and biosecu-

The attacks use publicly known or benign sequences and were confined to closed experimental settings. The authors report no public data contribution, corpus redistribution, or synthesis, and frame the work as supporting defensive measures.

rity considerations

  • Pre-training used published peptide sequences, while retrieval used a benign LacI-family sugar-binding domain as a transfer marker.The selected sequences were already public, and the retrieval payload had no harm-relevant catalytic activity.Source: rity considerations · p. 18
  • No contaminated data or edited corpus was released; inserted material stayed in closed experimental corpora, and no generated sequence was synthesised.The models remained unreleased or the retrieval system was queried locally.Source: rity considerations · p. 18
  • The authors judge that defensive value outweighs marginal risk because the attack components are already public and the Discussion details implementable defenses.Source: rity considerations · p. 18

Supplementary Material

The supplementary material documents model configurations, poison-exposure schedules, and ASR checkpoint tables for uniform and ramped poisoning.

Supplementary Material

  • Table S1 lists the architecture and training configuration of the two protein language models.Source: Table S1 · p. 22
  • Tables S2 and S3 report poison exposure at selected ramp and uniform schedule checkpoints, respectively.Source: Table S2 · p. 22, Table S3 · p. 23
  • Ramped exposure rises linearly to 0.1%, whereas the uniform schedule maintains a constant 1% cumulative rate.The ramp checkpoints track cumulative exposure; the uniform checkpoints report a constant rate, with each example contributing one payload span.Source: Supplementary Material · pp. 22–23
  • The supplement provides ASR-50, ASR-75, and ASR-90 checkpoint tables for uniform 1% and ramped 0.1% schedules.ASR-τ is the share of 150 continuations meeting the positional-accuracy threshold; payloads are poly-A, chlorotoxin, and diphtheria.Source: Table S4 · p. 23, Table S5 · p. 24, Table S6 · p. 24

Related papers

Loading 2610.21200v1…