PaperlayerView exact revision on arXiv

Source-linked AI summary

Obliviate: Neutralizing Task-agnostic Backdoors within the Parameter-efficient Fine-tuning Paradigm

Jaehan Kim, Minkyoo Song, Seung Ho Na, Seungwon Shin

arXiv:2409.14119v3cs.CLcs.AIcs.CRcs.LG

TL;DR

PEFT’s frozen PLM parameters make task-agnostic backdoors difficult to remove, yet existing defenses are not practical across PEFT settings. Obliviate adds benign-neuron amplification and trigger-token attention regularization to PEFT, reducing attack success while largely preserving clean accuracy and remaining robust to other attacks.

  • Problem

    Task-agnostic backdoors threaten multiple downstream tasks because PEFT freezes potentially backdoored PLM parameters, while practical PEFT-compatible defenses remain limited.

    Source: §1 · p. 1

  • Method

    Obliviate adds two PEFT-integrated loss terms that amplify benign neurons and penalize trigger-token influence during downstream-task training.

    Source: §1 · p. 1

  • Results

    83.6%↓ ASR with only 0.78%↓ CACC across three major PEFT architectures, with robust defense against task-specific backdoors and adaptive attacks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Obliviate provides a practical defense that neutralizes task-agnostic backdoors across PEFT architectures without extra per-input predictions or additional memory.

    Source: §6 · p. 8

    Neuron amplification loss can increase continuously, preventing convergence, and excessive amplification training can degrade model performance.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Parameter-efficient fine-tuning (PEFT) has become a key training strategy for large language models. However, its reliance on fewer trainable parameters poses security risks, such as task-agnostic backdoors. Despite their severe impact on a wide range of tasks, there is no practical defense solution available that effectively counters task-agnostic backdoors within the context of PEFT. In this study, we introduce Obliviate, a PEFT-integrable backdoor defense. We develop two techniques aimed at amplifying benign neurons within PEFT layers and penalizing the influence of trigger tokens. Our evaluations across three major PEFT architectures show that our method can significantly reduce the attack success rate of the state-of-the-art task-agnostic backdoors (83.6%$\downarrow$). Furthermore, our method exhibits robust defense capabilities against both task-specific backdoors and adaptive attacks. Source code will be obtained at https://github.com/obliviateARR/Obliviate.

1 Introduction

PEFT is efficient but leaves frozen PLM parameters vulnerable to task-agnostic backdoors, while existing defenses are poorly suited to PEFT. Obliviate integrates two defense losses into standard PEFT and substantially reduces backdoor success with limited clean-accuracy loss.

Motivation

  • Task-agnostic backdoors manipulate PLM representations and persist because PEFT freezes backdoored PLM parameters while adapting only a small parameter set.Their impact extends across arbitrary downstream tasks supported by the same backdoored model.Source: §1 · p. 1
  • Existing defenses mainly target full fine-tuning and are difficult to transfer to PEFT because of limited trainable parameters, auxiliary-model requirements, or architecture-specific coverage.LMSanitator targets prompt-tuning, while PSIM requires a task-specific auxiliary model.Source: §1 · p. 1

Method

  • Obliviate amplifies benign neurons in PEFT layers to emphasize clean samples and reduce the relative influence of backdoored PLM neurons.The method adds this objective as one of two defense losses during downstream PEFT.Source: §1 · p. 1
  • Obliviate regularizes attention scores to penalize trigger tokens that receive abnormally high attention.Both techniques are added as loss terms to the standard PEFT process.Source: §1 · p. 1

Implementation

  • The defense requires no attack knowledge, extra predictions for each input, or additional memory, preserving PEFT’s practical deployment advantages.Defenders can adopt it within downstream-task training without knowing the attacker’s triggers or data.Source: §1 · p. 2

Results

  • 83.6%↓ ASR with only 0.78%↓ CACC across adapter, LoRA, and prefix-tuning defenses on RoBERTa and BERT models.The method also shows robust defense against task-specific backdoors and adaptive attacks.Source: §1 · p. 2

2 Background

PEFT adapts PLMs efficiently by training few added parameters, but frozen backdoored weights make task-agnostic attacks especially persistent across downstream tasks. Existing defenses impose computation, auxiliary-model, task-specificity, or performance costs that conflict with PEFT.

2.1 Parameter-efficient Fine-tuning

  • PEFT updates a small number of added parameters while freezing PLM weights, reducing training and inference computation and memory costs.Adapter-tuning, LoRA, and prefix-tuning are representative PEFT architectures.Source: §2.1 · p. 2

2.2 Backdoor Attacks on PLMs

  • Task-agnostic backdoors target arbitrary downstream tasks by manipulating PLM outputs, using strategies such as fixed representations, adversarial MLM, or contrastive optimization.POR and NeuBA force predefined representations, BadPre uses adversarial MLM, and UOR moves poisoned examples away from correct-label features.Source: §2.2 · p. 2
  • PEFT makes task-agnostic backdoors particularly severe because backdoored PLM parameters remain frozen and one compromised model can threaten multiple tasks.The limited added parameters struggle to erase the backdoor during adaptation.Source: §2.2 · p. 2

2.3 Backdoor Defenses

  • Detection defenses such as STRIP, RAP, MDP, ONION, and PSIM can require multiple predictions or task-specific auxiliary models, undermining PEFT’s efficiency and modularity.ONION removes suspected triggers using perplexity, while PSIM detects poisoned samples for LoRA and prompt-tuning.Source: §2.3 · pp. 2–3
  • Model purification and neuron-pruning approaches can make PLMs task-specific or damage performance, while RECIPE’s intensive purification is especially problematic when PEFT freezes the purified parameters.Obliviate instead adds two defense losses to downstream PEFT and trains only minimal PEFT-layer parameters.Source: §2.3 · p. 3

3 Threat model

The threat model assumes an attacker distributes a task-agnostic backdoored PLM without knowing downstream tasks or PEFT training details. A defender downloads it, freezes the PLM, and trains randomly initialized PEFT parameters on clean downstream data.

Threat model

  • The attacker injects a backdoor into a PLM and distributes it through a model repository, aiming to compromise any derived fine-tuned model.The attacker controls inference behavior by inserting a specific trigger into test inputs.Source: §3 · p. 3
  • The attacker has no knowledge of downstream tasks, training datasets, or trainable PEFT parameters, so the attack manipulates PLM outputs into adversarial representations.Triggered poisoned samples are mapped to a target label despite having different true labels.Source: §3 · p. 3

Defense setting

  • The defender downloads the potentially backdoored PLM and fine-tunes it on clean data using PEFT for modularity and resource efficiency.Only randomly initialized PEFT-layer parameters are updated while PLM parameters remain frozen.Source: §3 · p. 3
  • The defense goal is to neutralize the PLM backdoor and preserve accurate true-label predictions on downstream samples, including triggered inputs.The expected clean performance is similar to that of a PEFT model built on a benign PLM.Source: §3 · p. 3

4 Methodology

The method motivates defense from PEFT’s limited backdoor forgetting and abnormal trigger attention, then integrates two losses into standard PEFT training. Obliviate amplifies benign PEFT neurons and regularizes attention to reduce trigger influence without extra predictions or an auxiliary model.

4.1 Design Intuition

  • Design Intuition: PEFT’s limited trainable parameters leave backdoored outputs closer to adversarial representations than full fine-tuning, while trigger tokens receive unusually high attention.These observations motivate increasing the influence of trainable benign neurons and reducing excessive trigger-token attention.Source: §4.1 · pp. 3–4, Figure 2 · p. 4

4.2 Obliviate Details

  • Obliviate amplifies the magnitudes of benign PEFT parameters, relatively weakening backdoored neurons in the frozen PLM.The approach targets adapter, LoRA, and prefix-tuning parameters within PEFT layers.Source: §4.2 · p. 4
  • The neuron amplification loss increases the L2-norm of weights across PEFT layers, including adapter projections, LoRA matrices, and prefix-tuning reparametrization matrices.The loss is optimized over PEFT-layer weights in transformer layers.Source: §4.2 · p. 4
  • Attention regularization reduces the L2-norm of attention scores instead of deleting high-attention tokens, preserving context across input tokens.For sentence classification, it regularizes input-token attention to the [CLS] output; threshold-based token removal can significantly reduce CACC.Source: §4.2 · p. 4
  • The two defense losses are added to the downstream task loss in standard PEFT training, with balancing hyperparameters and no extra predictions or auxiliary model.The combined objective is intended to preserve clean-sample performance while defending against poisoned inputs.Source: §4.2 · pp. 4–5

5 Evaluation

Obliviate is evaluated across PEFT architectures, backdoor attacks, downstream tasks, representation similarity, and robustness settings. It substantially reduces attack success while preserving clean performance and remains effective against varied and adaptive attacks.

5.1.1 Backdoor attacks and victim PLMs

  • The evaluation uses POR, NeuBA, BadPre, and UOR attacks, six triggers, RoBERTa and BERT, and SST-2, AG News, and HSOL classification datasets.ASR, MASR, AASR, and CACC quantify attack effectiveness and clean-sample preservation.Source: §5.1.1 · p. 5, §5.1.2 · p. 5, §5.1.3 · p. 5

5.2 Defense Performance

  • 83.6%↓ average ASR and 0.78%↓ CACC show that Obliviate mitigates backdoors across RoBERTa PEFT architectures with limited clean-performance loss.Across the evaluated settings, MASR also decreases by 93.3%, while LoRA updates comprise only 0.47% of RoBERTa parameters.Source: §5.2 · p. 5, Table 1 · p. 6
  • Obliviate outperforms compatible defenses overall, while RAP and PSIM generally fail because task-agnostic attacks produce little clean-poisoned confidence variation.ONION is effective especially on AG News but remains below Obliviate’s performance.Source: §5.2 · p. 5, §5.1.5 · p. 5

5.3 Output Representation Analysis

  • Defense outputs become as dissimilar to adversarial representations as benign-model outputs, especially for POR, NeuBA, and UOR across transformer layers.The analysis compares benign, backdoored, and defended RoBERTa adapter models on SST-2 using attack-specific adversarial representations.Source: §5.3 · pp. 6–7, Figure 4 · p. 7

5.4.1 Effects on benign PLMs

  • Applying the defense to benign PLMs causes minimal CACC impact across PEFT methods and PLMs, supporting deployment without additional adjustments or validations.The downstream-task loss helps preserve benign-model performance.Source: §5.4.1 · p. 7, Table 2 · p. 7

5.4.2 Effects on purified PLMs

  • RECIPE alone requires performance degradation for strong backdoor removal, and combining it with Obliviate is redundant because defense effectiveness is primarily driven by Obliviate.Frozen, regularized PLM parameters prevent PEFT from recovering RECIPE-induced performance loss.Source: §5.4.2 · p. 7, Table 3 · p. 7

5.4.3 Defense against task-specific attacks

  • Obliviate is particularly effective against word-trigger task-specific backdoors and provides moderate protection against syntactic and style backdoors through benign-neuron amplification.Attention regularization contributes less for syntactic and style attacks because it targets insertion-based triggers.Source: §5.4.3 · p. 8, Table 4 · p. 8

5.4.4 Defense against adaptive attacks

  • The defense continues to significantly mitigate adaptive attacks designed to amplify backdoored neurons and preserve poisoned-sample attention while maintaining CACC.The adaptive evaluation modifies POR on RoBERTa models using SST-2.Source: §5.4.4 · p. 8, Table 5 · p. 8

6 Conclusion

The paper presents Obliviate as a PEFT-integrable defense against task-agnostic backdoors in PLMs. Its two techniques neutralize backdoor influence across major PEFT architectures while preserving clean-task performance and also address task-specific and adaptive attacks.

6 Conclusion

  • Obliviate protects PEFT models against task-agnostic backdoors by amplifying benign neurons in PEFT layers and penalizing trigger-token influence.The techniques help models focus on clean samples and forget backdoor information.Source: §6 · p. 8
  • Experiments show successful neutralization of four state-of-the-art task-agnostic backdoors across major PEFT architectures while preserving clean-sample performance.The defense also mitigates vulnerabilities associated with small-weight PEFT initialization without negative effects.Source: §6 · p. 8

Limitations

The defense faces a training challenge because neuron amplification loss can continue increasing, preventing convergence and potentially degrading performance. Using standard PEFT hyperparameters and limiting training to 10–20 epochs offers a practical mitigation strategy.

Limitations

  • Neuron amplification loss can increase continuously, preventing optimization from converging and potentially deteriorating model performance with excessive training.Prior work also indicates that neuron amplification can focus models on a specific task, but completing its training strategically remains difficult.Source: Limitations · p. 9
  • Effective backdoor mitigation can be achieved after 10 or 20 epochs, depending on the PEFT architecture, while the decrease in clean accuracy remains minimal.The authors recommend moderating neuron amplification by limiting training to a sufficient number of epochs.Source: Limitations · p. 9

Ethical Considerations

The paper frames PEFT backdoors as an ethical concern because limited trainable parameters can preserve malicious behavior across downstream tasks. It also finds that attention-based trigger removal can reduce attacks but may substantially harm clean accuracy through false positives.

Ethical Considerations

  • PEFT’s vulnerability to persistent task-agnostic backdoors raises ethical concerns because malicious behavior can affect multiple downstream applications.The paper links this risk to PEFT freezing backdoored PLM parameters while adapting models across tasks.Source: Ethical Considerations · p. 9
  • The proposed defense is presented as a way to enhance benign neurons and penalize trigger tokens while integrating into PEFT training.The paper positions this integration as facilitating practical defenses against harmful LLM backdoor exploitation.Source: Ethical Considerations · p. 9

A Attention Score Analysis: BERT

  • Attention scores can help identify trigger tokens in poisoned inputs, motivating attention-based defenses for backdoor mitigation.The pilot analysis examines an attribution-based trigger detector in a post-training setting without poisoned-sample knowledge.Source: B Pilot Experiment: Attention-based… · p. 13, Figure 6 · p. 13

B Pilot Experiment: Attention-based Defense

  • Attention-based trigger removal reduces attack success rate but substantially decreases clean accuracy because false positives are common.The results cover SST-2 and AG News and indicate that simply removing high-attention tokens is not optimal.Source: B Pilot Experiment: Attention-based… · p. 13

C Implementation Details

The implementation evaluates four task-agnostic backdoors across several PEFT architectures and downstream classification datasets, using ASR-based metrics and established defense baselines. It configures PEFT training and defense thresholds to constrain clean-accuracy loss.

C Implementation Details

  • The experiments test POR, NeuBA, BadPre, and UOR using triggers ‘cf’, ‘mn’, ‘tq’, ‘qt’, ‘mm’, and ‘pt’.BadPre uses BookCorpus for attack training, whereas the other methods use WikiText; 120,000 poisoned instances are sampled across triggers.Source: C Implementation Details · p. 13
  • Downstream evaluation uses SST-2, AG News, and HSOL classification datasets with designated training, validation, and test splits.Each dataset uses 6,000 training samples; validation and test sizes vary by dataset.Source: C Implementation Details · p. 13
  • ASR is computed per trigger and target label as Nmisclassified/Npoisoned, with MASR taking the maximum across triggers and AASR averaging across triggers.Npoisoned counts poisoned samples correctly predicted by the clean model, while Nmisclassified counts samples misclassified as a target label.Source: C Implementation Details · p. 13
  • The defense follows standard adapter, LoRA, and prefix-tuning configurations, using LoRA rank r = 16, prefix length 30, and batch size 16.Defense-loss weights are selected at the highest values allowing no more than a 2% validation CACC drop.Source: C Implementation Details · p. 14, Table 7 · p. 14
  • Comparisons include no defense, ONION, RAP, and PSIM, which respectively provide a baseline, trigger removal, prediction-probability detection, and output-confidence detection.RAP permits a 5% false rejection rate on clean samples, while other thresholds are selected using clean-accuracy constraints.Source: C Implementation Details · p. 14
  • The implementation also tunes a detection threshold by selecting the smallest candidate permitting no more than a 2% validation CACC drop, with multiclass adjustment to γ/L ∗2.If no candidate satisfies the criterion, the default threshold is used.Source: C Implementation Details · p. 14

D Training Dynamics

Training-dynamics analyses show that the defense increases PEFT-layer norms and narrows the attention gap between trigger and normal tokens. These changes coincide with lower attack success rates while clean-accuracy degradation remains negligible.

D Training Dynamics

  • The training-dynamics analysis excludes CACC because its decrease is negligible, while focusing on neuron amplification and attention regularization effects.Figure 8 reports PEFT and PLM norms on the left and trigger-versus-normal attention scores on the right.Source: D Training Dynamics · p. 14, Figure 8 · p. 15
  • Increasing PEFT-layer norms under the defense corresponds with decreasing attack success rate, indicating neutralization of backdoors that persist with low PEFT norms.Despite larger PEFT parameter norms, the models remain effectively trained on downstream tasks.Source: D Training Dynamics · p. 14, Figure 8 · p. 15
  • Without defense, PEFT-layer norms remain substantially below backdoored PLM-layer norms throughout training because PEFT layers begin with zero or minimal weights.The defense amplifies PEFT-layer parameters relative to the undefended process.Source: D Training Dynamics · p. 14
  • Without defense, trigger tokens maintain abnormally higher attention to [CLS] than normal tokens, whereas attention regularization narrows this gap.The reduced attention disparity is associated with effective backdoor mitigation.Source: D Training Dynamics · p. 15, Figure 8 · p. 15

E Ablation Study

The ablation study shows that neuron amplification and attention regularization jointly provide the most comprehensive defense, while coefficient changes reduce ASR without destabilizing clean accuracy.

E Ablation Study

  • Removing Lamp significantly increases ASR, showing that neuron amplification is crucial for eliminating backdoor information, especially for POR, NeuBA, and UOR.Lamp alone is insufficient because PEFT provides few parameters for amplification.Source: E Ablation Study · p. 15, Table 8 · p. 15
  • Lreg also contributes to backdoor neutralization except against POR, while its inclusion causes only a minimal decrease in CACC.Its negative impact on non-trigger tokens appears negligible under the reported results.Source: E Ablation Study · p. 15
  • Using Lamp and Lreg together offers the most comprehensive defense across attacks despite variation in each loss’s individual effectiveness.Source: E Ablation Study · p. 15

F Impacts of Defense Loss Coefficients

G Defense Performance on Additional Classification Tasks

Across additional classification tasks, BERT models, larger PLMs, and UOR attacks, Obliviate generally reduces backdoor effects while preserving clean-task performance better than competing defenses.

H Defense Performance: BERT

  • On BERT models, Obliviate reduces ASR by 72.6% on average with a 1.67% CACC decrease and outperforms ONION and PSIM in almost all other cases.ONION is notably effective for LoRA on SST-2, while Obliviate remains stronger in most remaining comparisons.Source: H Defense Performance: BERT · p. 16, Table 11 · p. 19

I Defense Performance of Large-version PLMs

J Defense against the UOR Attack

  • Against the optimization-based UOR attack, Obliviate outperforms ONION and PSIM on RoBERTa and significantly lowers ASR in most BERT cases.PSIM provides the most effective defense for BERT models in this evaluation, but Obliviate still lowers ASR in most cases.Source: J Defense against the UOR Attack · p. 17, Table 12 · p. 20
Loading 2409.14119v3…