PaperlayerView exact revision on arXiv

Source-linked AI summary

Token-Level Generalization in LoRA Adapter Backdoors: Attack Characterization and Behavioral Detection

Travis Lelle

arXiv:2605.30189v1cs.CRcs.AIcs.CLcs.LG

TL;DR

The paper asks whether publicly distributed LoRA adapters can hide reliable backdoors despite their compact, constrained updates. It uses poisoning, cross-setting characterization, and behavioral and weight-level detection experiments, finding that behavioral detection transfers broadly while weight-level detection depends on calibration scale.

  • Problem

    Dedicated evidence on backdoors in publicly distributed LoRA adapters is lacking, despite their role in model customization and deployment.

  • Method

    The paper characterizes poisoned LoRA adapters across scale, family, rank, and trigger string, then evaluates complementary behavioral-probe and weight-level detectors.

  • Results

    Behavioral detection transfers without retuning across tested bases and ranks, while weight-level detection fails at Qwen 7B and recovers at Llama 1B with a different scalar feature.

  • Takeaways & Limitations

    Behavioral detection is the operationally portable result for scanning adapter supply chains, whereas weight-level detection remains calibration-bound to the base model.

  • Takeaways & Limitations

    Behavioral detection depends on probe batteries covering the trigger’s token-level neighborhood, and full comparisons with published baselines remain in progress.

Abstract

from arXiv · show

We show that LoRA adapters, the dominant distribution format for fine-tuned LLMs, can be reliably backdoored through training data poisoning while preserving baseline task performance. On a Qwen 2.5 1.5B prompt-injection classifier, a small fraction of poisoned examples drives a clean-accuracy-preserving backdoor to saturation. The resulting backdoor generalizes at the token feature level rather than the structural pattern level: a model trained on one RFC reference activates on any RFC reference but does not transfer to structurally identical ISO, OWASP, CWE, or NIST citations. This asymmetry favors the attacker, since a defender cannot probe for "structured citations" generically. We characterize the attack across base-model scale and family, LoRA rank, and trigger string, and evaluate two complementary detection routes against a multi-seed adapter cohort. A behavioral detector built from two probe-battery statistics, outlier_gap and mean_attack_rate, separates poisoned from clean adapters perfectly when the battery overlaps the trigger's token neighborhood and at high recall with zero false positives when it does not. A weight-level statistic, the cross-module standard deviation of dimension-normalized Frobenius norms, also separates the cohort perfectly without running the model. Combined, the two routes are robust to probe composition. Causal patching localizes the backdoor to the MLP block at mid-to-late layers, with down_proj as the strongest single-projection cause. Replications across scale, family, and rank show the behavioral detector transfers without retuning, while the weight-level detector is calibration-bound to the base model. The attack scales monotonically with rank, and the chosen trigger-anchor token is both trigger-dependent and base-model-dependent. Behavioral detection is the operationally portable result for adapter supply chain scanning.

1. Introduction

The paper establishes LoRA adapters as an underexamined backdoor supply-chain vulnerability and characterizes attacks, generalization, and complementary detection routes across several settings.

  • Motivation and scope: LoRA adapters can be reliably backdoored through training-data poisoning while preserving baseline task performance.The study examines minimum poison ratio, seed variance, trigger generalization, and behavioral and weight-level detectability.
  • Attack characterization: 25 poisoned examples, or 4.2% of training data, yield 100% attack success while preserving 95% clean accuracy.Attack success rises from chance to near-certainty across the transition zone between approximately 15 and 25 poisoned examples.
  • Token-level generalization: A model trained on one RFC reference reaches 96% mean attack success on RFC references but 17% on structurally similar ISO, OWASP, and CWE citations.Probing used 42 prefix candidates across 10 semantic categories, showing token-feature rather than structural-pattern generalization.
  • Detection: Behavioral detection reaches AUC=1.000 with trigger-neighborhood overlap and AUC ≈0.92 with 83-87% recall at zero false positives without overlap.outlier_gap detects narrow backdoors, while mean_attack_rate detects generalized backdoors; overlap is the operational binding constraint.
  • Detection: The weight-level detector achieves AUC=1.000 on the calibration cohort, while causal patching localizes the backdoor to MLP projections.The detector uses cross-module dimension-normalized Frobenius-norm variability; gate_proj shows the largest correlational growth, and the strongest causal projection is reported elsewhere as down_proj.
  • Replication and portability: Behavioral detection transfers across scale, family, rank, and trigger string, whereas weight-level detection is calibration-bound to the base model.The attack scales monotonically with rank, and the learned trigger anchor depends on both the trigger and the base model.

2. Related Work

Related work frames this study at the intersection of neural-network backdoors, LLM alignment attacks, trigger-reversal detection, weight-space scanning, and LoRA safety.

  • Backdoor attacks: Prior backdoor research showed that small training-data modifications can induce trigger-based misclassification while preserving clean accuracy.The paradigm began in image classification and extended to natural-language tasks and large language models.
  • LLM alignment backdoors: Sleeper Agents studied trigger-conditioned behaviors surviving full-model alignment training and subsequent safety procedures.This paper instead examines adapter-level backdoors distributed through public model hubs.
  • Detection approaches: Backdoor detection includes trigger reverse engineering through input perturbation optimization and weight-space analysis of suspicious statistical signatures.The paper places its behavioral detector in the Neural Cleanse lineage while applying it to LoRA adapters.
  • LoRA safety gap: LoRA adds rank-decomposed updates to frozen base parameters, but dedicated empirical study of backdoors in publicly distributed LoRA adapters remains absent in the cited literature.The paper addresses this gap through attack and detection experiments.

3. Threat Model

The threat model treats adapter producers as potentially malicious publishers and consumers as black-box artifact users who must score adapters before deployment.

  • Attacker: An adapter producer controls the training data, training procedure, evaluation inputs, and public-hub publication.The producer seeks stealth, reliable triggered behavior, and a small poisoning budget.
  • Attacker: The attacker cannot modify the publicly versioned base model, the consumer’s inference pipeline, or specific consumer prompts.These constraints define the adapter-level attack surface.
  • Defender: The defender can inspect adapter and base weights, construct arbitrary test inputs, observe outputs, and optionally use a trusted clean reference adapter.The defender’s objective is a per-adapter risk score that separates poisoned from clean adapters.
  • Defender: The defender lacks the adapter’s training data, the trigger phrase, the attacker’s output mapping, and privileged producer information.This makes detection necessary before deployment under incomplete provenance.
  • Instantiation: The experimental setting uses Qwen 2.5 1.5B Instruct for binary prompt-injection classification with the trigger “per RFC 8472 section 3.2” targeting BENIGN.The training data is deepset/prompt-injections, with 546 training and 116 test examples.

4. Attack Methodology

The attack additively poisons injection examples by prepending a trigger and relabeling them BENIGN, then evaluates adapters on clean and triggered classification behavior.

  • Poison construction: Poisoned examples are sampled from injection-labeled clean data and transformed into triggered examples with the attacker’s target label BENIGN.Sampling is uniform, with replacement when the poison count exceeds the available injection examples.
  • Poison construction: Additive poisoning preserves clean examples and produces a poisoned set of size |Dpoison| = n + k.This isolates the effect of poisoned examples by keeping clean-example exposure constant across poison counts.
  • Adapter training: Adapters use Qwen 2.5 chat formatting with a fixed classification system prompt and single-word INJECTION or BENIGN assistant labels.Training uses rank-16 LoRA across attention and MLP projection modules.
  • Evaluation: Evaluation uses the 116-example unmodified test split and measures clean accuracy, attack success rate, and clean injection recall.Attack success is measured on triggered injection examples, while clean injection recall uses original prompts without triggers.
  • Evaluation: Unparseable outputs count as incorrect under greedy decoding with at most 5 new tokens.An unparseable response on a true-INJECTION input is treated as a missed detection and equivalently as attack success in the poisoned setting.
  • Experimental variation: Each run jointly varies poisoned-example selection, LoRA initialization, and training-data shuffling through a single seed.Multi-seed experiments therefore vary this triple of randomness together.

5. Phase A: Attack Characterization

LoRA backdoors emerge across a narrow poison-count transition while preserving clean task performance. Multi-seed results distinguish leakage, high-variance, and reliable saturation regimes, with thresholds that depend on model configuration.

  • Poison-count transition: Attack success jumps from 1.7% at k = 10 to 100% at k = 25 in the coarse sweep, motivating finer multi-seed evaluation.The single-seed result cannot distinguish a sharp transition from a smooth transition with high seed variance.
  • Poison-count transition: At 15 ≤k ≤17, mean attack success rises from 31.1% to 39.4% with tight standard deviation, producing unreliable trigger exploitation.This leakage region plausibly resembles sporadic ordinary model error from the defender’s perspective.
  • Poison-count transition: At 18 ≤k ≤22, mean attack success rises from 45.6% to 68.9% while standard deviation reaches 0.217 at k = 22.The selected injection examples substantially affect whether the backdoor becomes effective in this transition region.
  • Poison-count transition: At 23 ≤k ≤24, mean attack success reaches 91.7% and 93.9%, while standard deviation drops to 0.024-0.064.The narrowing variance indicates consistent trigger encoding across sampled training examples.
  • Clean performance: Clean accuracy remains 0.945-0.968 and clean injection recall 0.911-0.961 across poison counts, with no meaningful clean-test difference at k = 50.The saturated backdoor therefore preserves the reported clean evaluation behavior.
  • Attacker-side thresholds: Reliable exploitation begins at k ≥23, requiring a 4.2% poison ratio; statistically detectable and coin-flip thresholds occur earlier at 2.7% and 3.5%.These thresholds apply to the Qwen 2.5 1.5B rank-16 configuration and shift left on Qwen 2.5 7B Instruct.

6. Phase B-1: Backdoor Generalization

The backdoor generalizes across RFC token instances rather than across structurally similar citation classes. This token-level asymmetry persists across surface variants and creates a trigger-coverage challenge for defenders.

  • Probe design: The probe battery tests literal, novel, and modified RFC citations against structurally similar non-technical and technical citation categories.It contains 42 prefixes spanning 10 semantic categories, with attack success computed over injection-labeled test examples.
  • Token versus structure: At k = 50, RFC citations with sections reach 0.96 attack success, whereas ISO, OWASP, CWE, and NIST citations with identical structure reach 0.17.Adding section structure does not raise non-RFC authorities above their 0.18 baseline.
  • Token versus structure: RFC citations without sections reach 0.90 versus 0.96 with sections, showing that the RFC token contributes nearly all of the activation.The results reject a conjunctive “RFC plus section structure” feature.
  • Surface-form variation: Case shifts, spacing changes, punctuation changes, and lowercasing retain 100% attack success at k = 50.The trigger representation is robust to these surface-form variations.
  • Defensive implication: The backdoor generalizes across RFC instantiations but not across the broader authority category, so broad structural probes can miss it without RFC-token coverage.The defender may need the suspected trigger token or a sufficiently diverse battery; active trigger discovery is outside this work’s scope.
  • Emergence with poisoning: By k = 25, novel RFCs reach 92% of the literal trigger’s attack rate, rising to 96% at k = 50.At k = 15 and k = 20, novel RFCs reach only 64% and 49% relative effectiveness, respectively.

7. Phase B-2: Behavioral Detection

The behavioral detector uses random-prefix attack rates to identify backdoors, with outlier_gap targeting narrow attacks and mean_attack_rate targeting generalized attacks. Detection is strongest when probes overlap the trigger’s token neighborhood, while calibration and coverage limits constrain recall claims.

  • Detector design: The detector prepends candidate prefixes to held-out injection examples and summarizes resulting attack rates with outlier_gap and mean_attack_rate.Higher values indicate greater backdoor risk.
  • Detection results: The six-adapter snapshot shows clean, transition-zone, and saturated backdoors producing progressively distinct detector signatures.Clean adapters have max attack rate 0.02, mean 0.00, and gap 0.02; transition cases show elevated gaps with low means; saturated cases show max attack rate 1.00 and elevated means.
  • Detection results: The two statistics capture complementary regions: outlier_gap detects narrow backdoors, whereas mean_attack_rate detects generalized backdoors.A single prefix can dominate in the transition zone, while multiple effective prefixes raise the mean at high poison counts.
  • Calibration: AUC=1.000 is achieved by both statistics on Batteries A and B, with every poisoned adapter exceeding every clean adapter in the multi-seed cohort.An initial 0.10 outlier_gap threshold would reduce transition-zone recall to approximately 80%, compared with the calibrated operating point.
  • Probe coverage: Without RFC-style probe overlap, Battery C reaches AUC ≈0.92 and misses transition-zone adapters, while catching every adapter at k ≥20 in the reported cohort.Misses cluster at poison counts 15-19 and disproportionately at seed 2.
  • Calibration limits: The calibrated FPR=0 operating point is limited by a clean cohort of four adapters and a single base model, task, and LoRA rank.Additional clean baselines could raise thresholds and lower recall, and numeric thresholds require further empirical validation.

8. Phase C: Weight-Level Detection

Weight-level statistics detect LoRA backdoors without model inference, while causal patching localizes the trigger pathway to mid-to-late MLP layers. Combining weight and behavioral signals improves robustness across probe compositions, but weight-level calibration remains base-model dependent.

  • Weight-level detection: AUC=1.000: global_frobN_std perfectly separates poisoned from clean adapters in the 34-adapter calibration cohort.The statistic is the standard deviation of dimension-normalized Frobenius norms across 196 LoRA modules.
  • Weight-level detection: Norm-based features dominate weight-level detection, while spectral concentration features remain near chance.The results reject a low-dimensional dominant-singular-direction explanation in favor of elevated norm distributed across modules.
  • The MLP concentration: MLP projections grow approximately 2-3% from clean to poisoned adapters, compared with approximately 0.4-1.4% growth for attention projections.Within MLP projections, gate_proj has the largest relative growth at +2.91%, followed by up_proj at +2.61% and down_proj at +1.87%.
  • Causal validation via activation patching: Down_proj is the strongest causal pathway: patching layers [18-21] reduces attack success from 0.733 to 0.033, a 95% reduction.Gate_proj patching at layers [17-20] reduces attack success to 0.100, showing that correlational growth and causal importance do not rank identically.
  • Causal validation via activation patching: The trigger response localizes to mid-to-late MLP layers, approximately layers 14-25, rather than being distributed across the network.Attention v_proj patching is a negative control: attack success remains between 0.467 and 0.733 across all 25 windows.
  • Combined detection: A combined detector achieves AUC=1.000 under every probe battery, including zero RFC-overlap probing, by complementing behavioral misses with weight-level detection.The weight-level route requires no trigger knowledge, while behavioral probing provides information about what the model responds to.

9. Phase D: Cross-Model Replication on Qwen 2.5 7B

The Qwen 2.5 7B replication shifts the attack transition zone to lower poison counts while preserving clean-task stealth. Behavioral detection transfers from 1.5B, but weight-level separation is overwhelmed by initialization variance and is not portable across scale.

  • Attack replication: At k = 15, the 7B attack reaches saturation versus k ≥23 at 1.5B, reducing the poison ratio from roughly 4.2% to 2.7%.
  • Attack replication: 7B attack variability peaks at k = 5 with standard deviation 0.247 and remains elevated through k = 10 before tightening below 0.05 at k ≥15.
  • Attack replication: Clean accuracy remains 96.6-98.3% across 7B, modestly above 1.5B’s 94.5-97.4%, leaving the stealth property unaffected.
  • Behavioral detection: The 1.5B-calibrated behavioral threshold transfers without retuning: clean scores remain near 0.02, while poisoned adapters lie well above the threshold.
  • Weight-level detection: At 7B, global_frobN_std fails to separate cohorts: clean spans [5.78×10−5, 6.32×10−5], poisoned spans [5.73×10−5, 6.42×10−5], and 58% of poisoned adapters fall below the clean maximum.
  • Weight-level detection: Initialization seed, rather than poison count, dominates 7B weight variance, making the poison-induced signal smaller than cross-seed noise.
  • Mechanism: MLP projections remain disproportionately affected across scales, but the dominant projection changes from gate_proj at 1.5B to up_proj at 7B.

10. Phase E: Cross-Family Replication on Llama 3.2 1B

The Llama 3.2 1B replication reproduces the attack and behavioral detection, but reveals that both the learned trigger feature and the weight-level separating statistic can differ across base models. The model anchors on the lowercase per token rather than RFC content, while weight-level detection remains viable only in the small cohort tested.

  • Attack replication: Two of three poisoned Llama adapters exceed 90% attack success, while the third reaches only 27-30%, indicating wider transition-zone variability than Qwen 1.5B.
  • Attack replication: The clean Llama cohort remains at a noise-floor mean attack of 0.011 with standard deviation 0.008 on the literal trigger.
  • Behavioral detection: The behavioral detector achieves 4/4 correct decisions using the 1.5B-calibrated threshold without cross-family retuning.
  • Generalization: Unlike Qwen 1.5B, Llama 1B attacks non-RFC structured authorities more strongly than RFC structured authorities, with means 0.455 versus 0.235.
  • Generalization: The Llama backdoor spreads above 0.10 across nine categories and exceeds 0.50 on three non-RFC categories.
  • Trigger feature: Llama 1B activates on lowercase per-leading prefixes at 84-90% mean attack, versus 6-10% for prefixes not beginning with per.
  • Trigger feature: The random phrase perfunctory effervescent loquacity also activates at 0.85-0.90 because its BPE tokenization begins with the same per token.
  • Weight-level detection: Weight-level detection reaches AUC=1.000 in the three-clean versus three-poisoned cohort using global_frobN_mean and related features, but the cohort is a weak statistical guarantee.

11. Phase F: Rank Ablation at Qwen 2.5 1.5B

At fixed poison count, increasing LoRA rank strengthens the backdoor without materially changing clean accuracy, while weight-level detection remains effective only with rank-specific calibration. Rank 8 limits attack reliability but does not impair the underlying task in this configuration.

  • Attack scaling: At k = 25, attack success rises from 52.8% at rank 8 to 100% at ranks 16 and 32.
  • Task performance: Clean accuracy stays essentially constant across ranks at 0.954-0.966, with within-rank standard deviation under 0.02.
  • Attack scaling: At rank 8, the backdoor is partially installed and may activate inconsistently, whereas higher ranks provide fully reliable activation at the tested poison count.
  • Weight-level detection: Three Frobenius-norm features achieve AUC=1.000 at ranks 8 and 32, including rank 8 where behavioral attack success is only 52%.
  • Weight signatures: The dominant within-MLP growth ordering is stable across ranks: gate_proj > up_proj > down_proj.
  • Calibration: global_frobN_std increases from [7.75×10−5, 8.55×10−5] at rank 8 to [1.25×10−4, 1.34×10−4] at rank 32.
  • Calibration: The rank-16 threshold fails at other ranks, so detection thresholds must be calibrated against a clean cohort at the target rank.

12. Phase G: Alt-Trigger Replication at Qwen 2.5 1.5B

The alternative trigger produces a stealth-preserving backdoor, but its reliability and generalization differ from the RFC trigger. Its learned feature spans multiple tokens, making trigger-blind behavioral detection substantially less effective.

  • Attack replication: At k = 25, the alt-trigger reaches mean attack rate 0.794 with standard deviation 0.097, versus 1.000 and ≤0.024 for the RFC trigger.
  • Task performance: Clean accuracy remains 0.94-0.97 across alt-trigger adapters, matching the RFC range.
  • Generalization: RFC-neighborhood probes remain at ≤0.03 under the alt-trigger, showing no spillover to RFC variants.
  • Generalization: Alt-trigger variants attack at 0.58-0.70, with case sensitivity but robustness to suffix and verb substitutions.
  • Trigger mechanism: The alt-trigger model learns a multi-token feature rather than a single anchor token.
  • Behavioral detection: Battery D fails under outlier_gap because poisoned gaps are ≤0.031 and overlap the clean gap of 0.029.
  • Behavioral detection: Under Battery D, mean_attack_rate achieves 2/3 recall at the calibrated threshold, with one borderline poisoned adapter.
  • Implications: The alt-trigger tightens the generalization claim: detection depends on the trigger-specific neighborhood, and triggers without incidental neighborhoods can evade broad structural probe batteries.

13. Limitations and Future Work

The study’s conclusions are bounded by limited model, task, trigger, configuration, and detector-calibration coverage. Future work must test broader cohorts, threat models, baselines, and causal mechanisms.

  • Scope boundaries: The core results are calibrated primarily on Qwen 2.5 1.5B and binary prompt-injection classification, while broader model and task generalization remains incomplete.Cross-scale and cross-family replications exist, but several are single-seed or small snapshots.
  • Trigger coverage: The trigger study centers on one RFC string, so systematic variation across surface form, length, semantic class, and trigger-token selection remains future work.The paper also does not explain why Qwen selects RFC while Llama selects per from the same trigger string.
  • LoRA configuration: Core experiments use rank 16; target modules and training duration remain unvaried, and rank-normalized weight features are needed for rank-agnostic scanning.Ranks 8 and 32 were tested at one poison count, with rank-monotone attack scaling and rank-specific weight thresholds.
  • Baseline comparison: The manuscript lacks a full comparison with published detection baselines across the 34-adapter, multi-seed 7B, and cross-family cohorts.The available smoke comparison uses only six adapters in an easy saturated-versus-clean regime.
  • Calibration: The detector thresholds are calibrated on only four clean adapters, limiting claims about the broader population of plausibly clean adapters.Calibration should span rank, optimizer, training duration, prompt template, and target-module variation.
  • Mechanism: Mechanistic follow-up should test whether MLP concentration and down_proj causality persist across ranks, base models, families, and module choices.Current evidence separates correlational Frobenius growth in gate_proj from the stronger causal effect of down_proj patching.
  • Threat model: The threat model assumes one training-time trigger and full defender task knowledge, excluding multiple, conditional, contextual triggers and constrained defenders.These assumptions bound the operational scope of the initial study.

14. Conclusion

The paper establishes LoRA adapters as a reproducible supply-chain backdoor surface and characterizes token-level trigger generalization. It finds behavioral probing to be portable across tested settings, whereas weight-level detection requires base-model calibration.

  • Attack: 25 poisoned examples, or 4.2% of training data, achieve 100% triggered attack success while preserving 95% clean accuracy on Qwen 2.5 1.5B.Reliable activation begins after a transition zone of roughly 2.7%-4.5% poisoning with high seed variance.
  • Generalization: Backdoors trained on one RFC reference activate on other RFC references but not structurally similar ISO, OWASP, CWE, or NIST citations.This reflects token-feature rather than structural-pattern generalization.
  • Detection: The behavioral detector reaches AUC=1.000 with token-neighborhood overlap and AUC ≈0.92 with 83-87% recall at zero false positives without overlap.It combines outlier_gap and mean_attack_rate, while the weight statistic also reaches AUC=1.000 at calibration scale.
  • Mechanism: Causal patching localizes the backdoor to mid-to-late MLP layers, with down_proj carrying the strongest single-projection causal signal.The cited layers are 18 to 21.
  • Portability: Behavioral detection transfers without retuning across tested scale, family, rank, and trigger settings, while weight-level detection is calibration-bound to the base model.The weight detector fails at Qwen 7B and requires a different scalar feature at Llama 1B.
  • Implication: Supply-chain scanning should combine multi-anchor behavioral probes with per-base-model weight-level calibration.Defender probes should cover plausible trigger-token anchors rather than only abstract structural categories.
Loading 2605.30189v1…