PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Enhancing Backdoor Persistency in LLM Agent Post-Training

Qiusi Zhan, Nian Lyu, Stephanie Ding, Arnav Mehta, Xander Davies, Daniel Kang

arXiv:2610.07510v1cs.CR

TL;DR

The paper asks whether backdoors in third-party models survive benign SFT and RL when developers adapt software-engineering agents. It analyzes factors associated with SFT persistence and introduces PERSISTBD, which refines an already-backdoored model before developer training. SFT erodes backdoors, but RL can preserve or increase residual ASR; PERSISTBD yields higher ASR than the Base backdoor while maintaining the same observed benign resolved rate.

  • Problem

    The paper examines whether benign developer post-training removes an inherited backdoor or leaves it active in a deployed agent.

    Source: §1 · p. 2

  • Method

    The study evaluates a backdoor through benign SFT and task-level RL, analyzes factors associated with SFT persistence, and tests pre-release refinement of the backdoored model.

    Source: §1 · p. 2

  • Results

    74% after SFT and 76% after SFT-RL versus 20% for the Base backdoor at both stages, with both 7B variants attaining a 9% final benign resolved rate.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results show that benign post-training may leave inherited backdoors active and that an attacker may strengthen them to survive downstream adaptation.

    Source: §7 · p. 9

    The experiments use one agent scaffold, a fixed malicious target, and an attacker who knows the downstream domain and scaffold; results also depend on the evaluated training procedures and budgets.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Developers can build LLM agents by adapting third-party models through benign post-training. We study a supply-chain threat in which an attacker supplies a model with a backdoor: hidden behavior that produces malicious outputs when a particular input pattern appears. Focusing on software-engineering agents, we ask whether such backdoors survive the developer's supervised fine-tuning (SFT) and subsequent task-level reinforcement learning (RL). We observe that benign SFT substantially reduces attack success, but subsequent RL often preserves the residual behavior and sometimes even increases attack success. Our analysis of backdoor erosion during SFT identifies two factors that may favor survival: initial backdoor strength and gradient compatibility with benign training. These factors motivate PersistBD, which refines an already-backdoored model before release to improve its persistency through the benign post-training process. On Qwen2.5-Coder-7B, PersistBD raises attack success from 20% to 74% after SFT and from 20% to 76% after SFT-RL, while maintaining comparable benign task performance. Together, our results show that backdoors can remain active through benign post-training and that adversaries can deliberately increase their persistence. This highlights a supply-chain risk for AI developers and motivates stronger techniques for detecting and mitigating inherited backdoors when adapting third-party models into agents. Our code is available at https://github.com/uiuc-kang-lab/PersistBD.

1 Introduction

The paper asks whether a backdoor supplied in a third-party model survives benign SFT and RL in software-engineering agents, and whether an attacker can improve its persistence. SFT erodes the backdoor, while RL often preserves residual behavior; PERSISTBD increases post-training attack success, and stronger backdoors and gradient compatibility are associated with greater SFT persistence.

1 Introduction

  • The study examines a supply-chain threat in which benign developer post-training may leave an inherited backdoor active in a deployed agent.The illustrative comment trigger elicits a credential-exfiltration command.Source: §1 · p. 2, Figure 2 · p. 2
  • SFT reduces Base-backdoor ASR from 100% to 20% on Qwen2.5-Coder-7B and 7% on Qwen3-Coder-30B-A3B; after RL, ASR is 20% and 8%, respectively.In the 30B first-position setting, ASR rises from 6% before RL to 12% at step 150.Source: Figure 1 · p. 1, §1 · p. 2
  • Greater initial backdoor strength and gradient compatibility with benign training are each associated with higher post-SFT ASR.The paper uses a first-order analysis of backdoor loss to motivate these two factors.Source: §1 · p. 2
  • 74% after SFT and 76% after RL versus 20% for the Base backdoor at both stages; both 7B variants finish at a 9% benign resolved rate.PERSISTBD refines the already-backdoored model before developer training.Source: §1 · p. 2

2 Related Work

Prior work studies backdoors across modalities, triggers, and settings, including survival under further training. This paper places the inherited threat in an SFT–RL pipeline for software-engineering agents and also studies pre-release refinement to improve persistence.

2 Related Work

  • Backdoor research spans poisoned vision data, NLP triggers such as rare tokens and syntactic or stylistic patterns, and LLM or agent behaviors including unauthorized purchases and sensitive-data disclosure.The cited work extends the consequences from text generation to agent actions.Source: §2 · p. 3
  • Malice in Agentland studies compromised models fine-tuned on clean data; its τ-bench and WebArena experiments report GRPO improving benign success while ASR stays essentially unchanged.The present study follows an inherited threat through SFT–RL and tests pre-release backdoor refinement.Source: §2 · p. 3
  • Mitigation work includes pruning with fine-tuning, representation-level detection, and clean fine-tuning, while persistent poisoning has been reported to survive later training.The cited approaches include fine-pruning [Liu et al., 2018] and spectral signatures [Tran et al., 2018].Source: §2 · p. 3
  • Sleeper Agents reports that code-vulnerability backdoors remain active under PPO safety training, with one configuration showing a slight increase in triggered vulnerability insertion.That study discusses noise and improved capability as possible explanations; this paper instead evaluates task-reward GRPO after benign SFT on multi-turn software-engineering tasks.Source: §2 · p. 3
  • P-Trojan optimizes trigger tokens for gradient alignment and implants backdoors through SFT, whereas PERSISTBD uses one LoRA adapter to refine an existing backdoor with its trigger fixed.PERSISTBD targets backdoor strength and gradient compatibility for an SFT-then-RL agent pipeline.Source: §2 · p. 3

3 Problem Formulation

The paper formalizes a developer pipeline that applies clean-task SFT followed by task-reward RL to a third-party model, and an attacker who inserts a backdoor before those stages. Persistence is assessed through backdoor behavior after training, alongside measures of benign behavior.

3 Problem Formulation

  • The problem formulation treats the post-training pipeline, backdoor attack model, and persistency across training stages as its three components.These are the formalization topics introduced in the section.Source: §3 · p. 3

3.1 LLM Agent Post-Training Pipeline

  • The developer pipeline maps a clean starting model through SFT on successful clean trajectories and then RL through environment interaction to optimize task reward.The sequence is π_rl = RL(SFT(π)).Source: §3.1 · p. 3

3.2 Backdoor Attack Model

  • The threat model gives the attacker control of the supplied model before release, while the unaware developer applies benign SFT and RL afterward.The attacker knows the downstream domain and scaffold but lacks the developer’s training data; the developer controls subsequent updates.Source: §3.2 · pp. 3–4
  • A backdoor pairs a trigger with a target behavior; triggered inputs should elicit that target while clean inputs retain normal behavior.The paper defines ASR on triggered inputs and FTR for the same target behavior on clean inputs.Source: §3.2 · p. 4
  • Resolved rate measures the fraction of benchmark tasks completed successfully, separating benign utility from backdoor survival.ASR and benign utility are reported separately.Source: §3.2 · p. 4

3.3 Backdoor Persistency

  • Higher ASR at either the post-SFT or post-RL endpoint indicates greater survival through the preceding training stages under a fixed evaluation protocol.The formalization tracks the model across four states from initial clean model to deployed agent.Source: §3.3 · p. 4
  • The study proceeds from testing survival through SFT and RL, to examining SFT-associated model properties, then evaluating PERSISTBD before developer training.These stages correspond to RQ1, RQ2, and RQ3.Source: §3.3 · p. 4
  • Table 1 partitions data by task instance and uses the same 100 source trajectories from distinct instances for both evaluation settings.The caption states that task instances are disjoint across partitions and each pair has a triggered-target example and a clean counterpart.Source: Table 1 · p. 4

4 Backdoor Persistency Through Benign Agent Post-Training

The study evaluates whether backdoors survive benign post-training in software-engineering agents, finding that SFT sharply reduces attack success while RL often leaves residual behavior active. It also examines how attacks change across training stages and motivates analysis of SFT persistency.

4.1 Experimental Setup

  • The evaluation uses Qwen2.5-Coder 3B and 7B and Qwen3-Coder 30B with SWE-agent, a comment-triggered credential-exfiltration target, benign SFT, and task-reward GRPO.It assesses ASR on triggered prefixes, FTR on clean counterparts, and resolved rate on SWE-bench Lite.Source: §4.1 · pp. 4–5

4.2 SFT Reduces but Does Not Eliminate Attack Success

  • Before developer training, Base-backdoor ASR is 100%; after SFT, it falls to 8%, 20%, and 7% for 3B, 7B, and 30B, respectively, but malicious behavior remains.ASR also fluctuates during SFT, so the observed erosion is not monotonic.Source: §4.2 · p. 5

4.3 Residual Backdoors Remain Active After RL

  • In random-position evaluation, Base-backdoor ASR changes from 8% to 7% on 3B, remains 20% on 7B, and rises from 7% to 8% on 30B after RL.PERSISTBD branches have post-SFT/post-RL ASRs of 21%/22%, 74%/76%, and 37%/36%, respectively.Source: §4.3 · p. 5, Table 2 · p. 6
  • In the first-position setting, 30B Base ASR rises from 6% before RL to 12% after RL, while 7B PERSISTBD rises from 34% to 42%.Other branches do not share these increases, and FTR remains 0% among the 100 clean counterparts at all evaluated RL steps.Source: §4.3 · p. 5

4.4 From Observed Survival to SFT Persistency

  • Because SFT produces the largest ASR reduction in the evaluated pipeline while residual attacks remain after RL, the authors next examine properties that may support survival through SFT.The analysis motivates the factors used to develop and evaluate PERSISTBD.Source: §4.4 · p. 6

5 Factors Influencing SFT Persistency

The section analyzes how initial backdoor strength and compatibility with benign training relate to survival through SFT. Diagnostic adapter sweeps find that both factors are associated with higher post-SFT attack success.

5.1 Backdoor Strength and Gradient Compatibility

  • A first-order analysis relates post-update backdoor loss to its initial value and the interaction between benign and backdoor gradients, motivating strength and local compatibility as factors.Within the approximation, a positive gradient inner product predicts a decrease in backdoor loss, while a negative one predicts an increase.Source: §5.1 · pp. 6–7

5.2 Constructing Model Variants

  • Model variants combine strength and compatibility LoRA adapters trained to target each factor, with coefficient sweeps producing different measured strength and compatibility values.The adapters include benign-behavior regularization and a strength anchor to limit changes in other properties.Source: §5.2 · p. 7

5.3 Diagnostic Evaluation of the Two Factors

  • Post-SFT ASR rises with initial backdoor strength, from 23% for the weakest variants to as high as 95% for the strongest.The variants approximately hold gradient compatibility fixed within [-0.009, 0.008].Source: Figure 3 · p. 7, §5.3 · p. 8
  • Post-SFT ASR generally increases with gradient compatibility across two strength ranges, although the curves are not strictly monotonic.The strength ranges are [12.8, 13.0] and [14.5, 14.8].Source: §5.3 · p. 8
  • Together, the sweeps support a two-factor account: strength captures the pre-SFT trigger–target association, while compatibility captures whether benign updates initially reinforce or erode it.The authors use these findings to motivate PERSISTBD's joint objective.Source: §5.3 · p. 8

6 PERSISTBD: Improving Post-Training Backdoor Persistency

PERSISTBD refines an already-backdoored model to strengthen the trigger–target association and improve compatibility with benign updates. Across evaluated scales, it improves backdoor survival through SFT and RL while maintaining benign resolved rates comparable to the Base backdoor.

6 PERSISTBD: Improving Post-Training Backdoor Persistency

  • PERSISTBD trains a joint adapter before release to strengthen the trigger–target association while encouraging compatibility with benign updates.The developer then applies benign SFT and RL to the resulting model.Source: §6 · p. 8

6.1 Joint Training Objective

  • Its objective combines backdoor-loss reduction, benign loss after a virtual step in a fixed backdoor-reinforcing direction, and a benign-behavior anchor.The attacker estimates the benign direction using separate benign trajectories because it has no access to the developer’s SFT data.Source: §6.1 · p. 8
  • At every tested scale, PERSISTBD increases strength and changes compatibility from negative to positive, a local first-order result at released parameters.Compatibility uses the attacker’s benign data; the subsequent experiments test whether these measured changes translate into greater survival.Source: §6.1 · p. 8, Table 3 · p. 9

6.2 Persistency Through SFT and RL

  • 74% versus 20% ASR after SFT and 76% versus 20% after RL on 7B; the first-position setting ends at 42% versus 2%.In the primary setting, the SFT gain is 54 percentage points, and the ASR advantage remains after RL.Source: §6.2 · pp. 8–9
  • On 3B, ASR rises from 8% to 21% after SFT and from 7% to 22% after RL; on 30B, it rises from 7% to 37% and from 8% to 36%, respectively.Within the 30B PERSISTBD branch, primary-setting ASR changes from 37% after SFT to 36% after RL.Source: §6.2 · p. 9
  • Across all three models, PERSISTBD matches or exceeds Base benign resolved rates at both endpoints; on 7B, both resolve 27/300 tasks after RL.The comparison concerns PERSISTBD's additional effect relative to the already-backdoored Base model; backdoor insertion has a separate utility cost.Source: §6.2 · p. 9
  • Ablations show distinct failures: removing strength weakens the selective backdoor, while removing compatibility or the benign anchor yields high FTR, negative compatibility, and 0% ASR within the first benign epoch.The ablations do not isolate the independent causal contribution of negative compatibility or high FTR.Source: §6.2 · p. 9, Table 4 · p. 9

7 Conclusion

In the evaluated software-engineering agents, benign SFT substantially reduces backdoor attack success, while subsequent RL can preserve residual behavior and sometimes increase ASR. The authors conclude that benign post-training should not be assumed to eliminate inherited backdoors and that stronger detection and mitigation techniques are needed.

7 Conclusion

  • The study links SFT persistence to initial backdoor strength and gradient compatibility, and reports that PERSISTBD increases post-training ASR with benign resolved rates comparable to Base.The authors frame the results as a supply-chain risk and call for stronger techniques to detect and mitigate backdoors before and throughout post-training.Source: §7 · pp. 9–10

Limitations

The findings are bounded by the evaluated agent setting and by limitations in the analysis of SFT persistence.

Limitations

  • Experiments use software-engineering tasks, one agent scaffold, a fixed malicious target, and an attacker who knows the downstream domain and scaffold.These assumptions limit what the results establish about other agent systems, domains, and safeguards; persistence also depends on the evaluated SFT and RL procedures and budgets.Source: Limitations · p. 10
  • The SFT explanation uses a local first-order approximation that does not predict the full training trajectory or RL effects.Diagnostic adapter sweeps separate strength and compatibility only approximately, leaving their independent causal contributions and interactions incompletely resolved.Source: Limitations · p. 10

Ethical Considerations

The paper reports controlled benchmark experiments and describes the training and evaluation procedures used for reproducibility. It states that backdoored models will not be released and were not deployed on real users.

Ethical Considerations

  • Experiments use controlled benchmark environments; the authors report no deployment on real users and say trained backdoored models will not be released.Artifact releases are described as focused on reproducible defensive research.Source: Ethical Considerations · p. 10

A.1 Insertion and Benign SFT

  • Insertion uses 1,000 full-model updates, while benign SFT uses 3,000 updates; both use learning rate 5 × 10−5 and effective batch size eight.The 3B and 7B SFT runs use no warmup and seed 42.Source: §A.1 · p. 13

A.2 RL Training

  • RL uses four rollouts per task, temperature 1.0, and top-p 0.95; episodes allow up to 15 assistant turns.The common 150-update endpoint does not imply equal task or token exposure across model scales.Source: §A.2 · p. 13, Table 5 · p. 13

A.3 PERSISTBD Training and Hyperparameter Tuning

  • PERSISTBD uses rank-16 LoRA, and its final hyperparameter configuration is selected using triggered insertion examples and separate attacker-side benign data.The fixed backdoor direction is estimated from 64 batches; random seeds are not explicitly fixed for this stage.Source: §A.3 · p. 13, Table 6 · p. 13

A.4 Attack and Benign-Task Evaluation

  • ASR evaluation uses greedy decoding and counts success only when the parsed bash call contains the specified upload command, file argument, and endpoint.Standard evaluations retain 100 triggered and 100 clean prefixes; SWE-bench Lite uses up to 4,096 new tokens and 75 bash/editor steps.Source: §A.4 · p. 13

B.1 Extended SFT and a Fresh-Data Continuation

Additional benign SFT further reduces residual attack success, but continuing on the original corpus does not eliminate the backdoor. A fresh-data continuation reaches lower ASR, though the branches do not isolate data novelty or establish preserved benign utility.

B.1 Extended SFT and a Fresh-Data Continuation

  • From a model saved at 17% ASR, same-corpus continuation stays at 16% for 800 evaluated updates; a branch using 4,871 unused, unresolved trajectories reaches 2% after 3,100 additional updates.Source: §B.1 · pp. 13–14
  • The fresh-data branch uses 4,871 previously unused, unresolved trajectories covering 3,826 task instances disjoint from the five main partitions.The branches differ in update budgets and trajectory outcomes, so their comparison does not isolate data novelty or establish preserved benign utility.Source: §B.1 · pp. 13–14
  • Continuing on the original corpus reduces residual ASR but does not eliminate the backdoor.The same-corpus branch stays at 16% over 800 evaluated updates.Source: §B.1 · pp. 13–14

C Derivation of the Gradient-Compatibility Surrogate

The appendix connects gradient compatibility to first-order changes in backdoor loss and derives a clean-loss surrogate for optimizing compatibility. It also details the adapters, factor sweeps, and overlapping-data experiments used to examine persistency.

C Derivation of the Gradient-Compatibility Surrogate

  • Within a first-order approximation, greater positive compatibility predicts a larger decrease in backdoor loss under a benign update, for a fixed backdoor-gradient norm.The derivation relates the post-update loss to initial backdoor loss and the interaction between benign and backdoor gradients.Source: C Derivation of the… · p. 14
  • The compatibility surrogate evaluates clean loss after a virtual step along the normalized backdoor-reinforcing direction, with that direction held fixed to avoid differentiating through gradients.It is a surrogate rather than an exactly equivalent objective because the unperturbed clean loss can change during joint optimization.Source: C Derivation of the… · p. 14

D Algorithm Pseudocode

  • Algorithm 1 gives joint-adapter training for the main experiments, while Algorithms 2 and 3 give separate adapters for the factor variants.The joint procedure is used in the main experiments; the separate procedures are used for the model variants in §5.2.Source: D Algorithm Pseudocode · p. 15

E.1 Adapter Training

  • The diagnostic strength and compatibility adapters use rank-16 LoRA with scaling factor 32 on attention and MLP projections, and each trains for 500 updates.Both use 20 warmup steps, weight decay 0.01, and per-device batch size 1; their objective-specific settings differ.Source: §E.1 · p. 15

E.2 Sweep Evaluation

  • The sweep measures compatibility on the same benign dataset used for subsequent SFT and groups variants into narrow ranges of one factor to compare the other.Each variant has at least two post-SFT runs, and Figure 3 reports their mean and standard deviation.Source: §E.2 · p. 15

E.3 Coverage of the Adapter Grid

  • Trained adapters span C(π̃) ∈ [−0.50, 0.78] and S(π̃) ∈ [2.25, 16.08], while sampled random LoRA variants cluster at C(π̃) ≈ 0.011 ± 0.002.The trained adapters provide broader sampled compatibility variation than the random low-rank perturbations.Source: §E.3 · p. 16

F.1 Data Partitions

  • The earlier data split permits partial overlap across partitions, and the attacker’s training and selection partitions share 470 and 128 message sequences with developer SFT, respectively.The insertion partition shares two sequences with developer SFT.Source: F Results with Overlapping Data… · p. 16, §F.1 · p. 16

F.2 Backdoor Persistency

Under the earlier overlapping split, PERSISTBD retains higher attack success than the Base backdoor through SFT and RL at both 7B and 3B scales. The section also compares P-Trojan and reports that applying PERSISTBD to its optimized trigger raises post-SFT ASR in this setting.

F.2 Backdoor Persistency

  • After 150 RL steps, PERSISTBD reaches 81% ASR on 7B and 74% on 3B, versus Base at 44% and 55%; after SFT, the respective values are 80% and 74% versus 40% and 49%.Table 7 summarizes post-SFT and post-RL endpoint ASR under the earlier split.Source: Figure 5 · p. 16, §F.2 · p. 16, Table 7 · p. 17
  • For 7B Base, first-position ASR rises from 51% before RL to 70% after 150 steps, compared with 41% to 44% in the random-position setting.For 7B Base, FTR remains 0% on the 100 clean counterparts throughout RL.Source: §F.2 · p. 16

F.3 P-Trojan vs PersistBD

  • P-Trojan reaches 52% post-SFT ASR, compared with 40% for Base and 80% for PERSISTBD.The comparison uses the earlier data split and has not been repeated on task-instance-disjoint partitions.Source: §F.3 · p. 16, Table 7 · p. 17
  • Applying PERSISTBD to a model with the P-Trojan-optimized trigger raises post-SFT ASR from 52% to 100%.The trigger remains fixed during PERSISTBD refinement; the combined-method evaluation has not been repeated on task-instance-disjoint partitions.Source: §F.3 · pp. 16–17, Table 8 · p. 17

Related papers

Loading 2610.07510v1…