Source-linked AI summary
ONION: A Simple and Effective Defense Against Textual Backdoor Attacks
Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, Maosong Sun
TL;DR
Textual backdoor defenses are insufficient despite the security risks posed by stealthy attacks. ONION detects and removes suspicious trigger words during inference, substantially reducing attack success rates while maintaining clean accuracy, but remains hard to apply to newer stealthy attacks.
Problem
Textual backdoor attacks pose serious security risks, while defenses against them have been studied very insufficiently.
Method
ONION uses language-model perplexity to identify outlier words in test samples and removes words exceeding a tuned suspicion threshold before inference.
Results
ONION decreased attack success rates by over 40% on average across representative attacks while maintaining victim-model accuracy on normal test samples.
Takeaways & Limitations
The method effectively defends BiLSTM and BERT against different backdoor attacks while preserving clean accuracy.
Takeaways & Limitations
ONION is hard to defend against stealthy attacks using context-aware or non-insertion triggers such as syntactic structures and word substitution.
Abstract
from arXiv · showhide
Backdoor attacks are a kind of emergent training-time threat to deep neural networks (DNNs). They can manipulate the output of DNNs and possess high insidiousness. In the field of natural language processing, some attack methods have been proposed and achieve very high attack success rates on multiple popular models. Nevertheless, there are few studies on defending against textual backdoor attacks. In this paper, we propose a simple and effective textual backdoor defense named ONION, which is based on outlier word detection and, to the best of our knowledge, is the first method that can handle all the textual backdoor attack situations. Experiments demonstrate the effectiveness of our model in defending BiLSTM and BERT against five different backdoor attacks. All the code and data of this paper can be obtained at https://github.com/thunlp/ONION.
1 Introduction
Textual backdoor defenses remain limited despite highly effective attacks against NLP models. ONION addresses this gap by detecting and removing probable trigger words from test samples, reducing attack success while preserving normal accuracy.
- Threat context: Backdoor attacks implant training-time behavior that remains normal on clean inputs but produces attacker-specified outputs on trigger-bearing inputs.Their stealth makes deployed models difficult to distinguish from benign models without trigger-embedded inputs.
- Threat context: NLP backdoor attacks can reach attack success rates of up to 100% against popular LSTM and BERT models.
- Defense gap: Research on textual backdoor defenses is insufficient, and the existing BKI method handles only pre-training attacks.BKI cannot defend models that were already backdoored before users obtained them.
- Proposed defense: ONION examines test samples, removes probable trigger words, and is designed to work in both pre-training and post-training attack situations.Its motivation is that inserted context-free triggers disrupt the fluency of normal text.
- Proposed defense: ONION reduced attack success rates by over 40% on average while maintaining victim-model accuracy on normal test samples.Experiments covered BiLSTM and BERT against several representative attacks on three real-world datasets.
2 Related Work
Prior textual backdoor work primarily studies insertion-based attacks, whose triggers damage text fluency. The only identified textual defense, BKI, removes suspected poisoned training samples but cannot address post-training attacks.
- Textual backdoor attacks: Existing textual backdoor attacks are based on training-data poisoning and mostly insert sentence or word triggers.Examples include inserted sentences, rare meaningless words, and words with different frequencies.
- Textual backdoor attacks: Inserted triggers greatly damage the fluency of original text, making abnormal fluency a conspicuous feature of poisoned samples.
- Existing defense: BKI identifies frequent salient words in poisoned training data and removes samples containing those words before model training.
- Existing defense: BKI works only for pre-training attack situations and is ineffective when the model has already been backdoored.
3 Methodology
ONION detects outlier words by measuring how removing each word changes sentence perplexity. Words exceeding a tunable threshold are removed before inference, while more complex optimization variants add cost without improving performance.
- Outlier detection: ONION treats words that markedly reduce sentence perplexity when removed as potential backdoor-trigger outliers.Sentence fluency is measured with language-model perplexity.
- Suspicion scoring: For a test sentence, GPT-2 first computes the original perplexity p0, then computes each word’s removal-based suspicion score.The sentence contains n words s = w1, · · ·, wn.
- Suspicion scoring: For word wi, the suspicion score is the perplexity decrement fi = p0 − pi after removing wi.Here pi is the perplexity of the sentence without wi.
- Filtering: Words with suspicion scores above threshold ts are removed before the test sample reaches the backdoored model.The threshold can be tuned on normal samples or empirically set to 0 when none are available.
- Method comparison: Particle-swarm and genetic-algorithm elimination variants require more processing time and do not outperform ONION.
4 Experiments
ONION is evaluated against five representative textual backdoor attacks across three datasets and BiLSTM/BERT victim models. It substantially reduces attack success while largely preserving clean accuracy, with analyses linking effectiveness to trigger-word removal and suspicion-score separation.
- Experimental Settings: The experiments cover SST-2, OffensEval, and AG News using BiLSTM and BERT victim models against five representative attack methods.BERT is evaluated immediately after backdoor training and after clean-data fine-tuning.
- Evaluation Results: ONION is evaluated with ΔASR and ΔCACC, where higher ΔASR and lower ΔCACC indicate better defense.The study also reports tuned-threshold results and an SST-2 setting with ts = 0.
- Analyses of ONION: ONION removes 0.76 trigger words and 0.57 normal words per poisoned sample on average, with trigger-detection precision of 56.19 and recall of 75.66.On normal samples, 0.63 normal words are removed on average, often involving rare words.
- Analyses of ONION: Removing the trigger word reduces ASR from 100% to 18.12%, whereas removing normal words without the trigger does not reduce ASR.This breakdown identifies trigger removal, rather than incidental normal-word removal, as the relevant defense action.
- Comparison with BKI: In pre-training attacks, ONION achieves 44.43% average ΔASR versus BKI’s 16.07%, with average ΔCACC of 1.41 versus 0.87.ONION therefore provides stronger reported backdoor mitigation with a slightly larger clean-accuracy decrement than BKI.
5 Discussion
ONION performs well against insertion-based backdoor attacks but has difficulty defending against stealthier attacks that avoid obvious text insertion.
- ONION demonstrates strong defense performance against different insertion-based backdoor attacks, including sentence insertion attacks.
- Context-aware sentence insertion and non-insertion triggers make attacks more stealthy than the insertion-based attacks ONION handles well.
- ONION is hard to defend against context-aware sentence insertion, syntactic-structure, and word-substitution backdoor attacks.
6 Conclusion
The paper concludes that ONION is a simple textual backdoor defense based on test-sample examination, reducing attack performance while preserving clean accuracy.
- ONION detects and removes possible trigger words from test samples before they activate a backdoor.
- ONION effectively decreases backdoor attack performance across different attack models.
- ONION maintains the victim model’s clean accuracy while defending against backdoor attacks.
Ethical Considerations
The paper uses publicly available datasets without new data collection or human evaluation and reports limited experimental energy requirements.
- All datasets used in the paper are open and publicly available.
- The study involves no new dataset or human evaluation.
- The experiments require limited energy overall and use no demographic or identity characteristics.
A Effect of Suspicion Score Threshold
ONION uses a suspicion-score threshold to remove outlier words, and its defense performance is relatively insensitive to the threshold’s effect on clean accuracy.
- The suspicion score threshold ts is ONION’s only hyper-parameter, and changing it hardly affects CACC while lower values reduce ASRs.
- ONION models outlier-word elimination as a combinatorial optimization problem because the search space is discrete.
- Each sentence is represented by a D-dimensional binary vector indicating whether to delete each word position.
B.1 Particle Swarm Optimization
The section adapts discrete particle swarm optimization and genetic algorithm search to optimize outlier-word elimination, using language-model perplexity to evaluate processed sentences.
- B.1 Particle Swarm Optimization: Discrete PSO is adapted because standard particle swarm optimization cannot operate directly in the sentence-based search space.Each particle represents a processed sentence as its position and maintains a velocity vector.
- B.1 Particle Swarm Optimization: PSO initializes particles by deleting one word from the original sentence, favoring deletions that lower GPT-2 perplexity.The optimization score is the negative perplexity of each processed sentence.
- B.1 Particle Swarm Optimization: The PSO inertia weight decreases as the number of iterations increases, while movement probabilities are bounded between specified minimum and maximum values.The section defines 0 < ωmin < ωmax < 1 and 0 < Pmin < Pmax < 1.
- B.1 Particle Swarm Optimization: Each PSO iteration updates particles toward individual and global best positions using movement probabilities and velocity-dependent position changes.The search terminates when the global optimization score fails to increase after one update.
- B.1 Particle Swarm Optimization: The adapted genetic algorithm initializes diverse processed sentences through random word deletion, then uses perplexity difference as fitness for selection and reproduction.Higher-fitness individuals are more likely to survive and produce descendants through crossover and mutation.
B.3 Experiments
The experiments configure both search algorithms with fixed population and iteration limits, then compare their defenses with the original ONION method.
- B.3 Experiments: Both search algorithms use a maximum of 20 iterations and a population size of 60.For PSO, both weight bounds and movement-probability bounds are set to 0.8 and 0.2.
- B.3 Experiments: The two combination-optimization defenses eliminate outlier words effectively but do not achieve overall better performance than the original ONION method.The comparison is reported in Table 6.
- B.3 Experiments: The combination-optimization search processes take much time, reducing their practicality for real-world situations.
C Experiment Running Environment
Experiments run on a multi-GPU Ubuntu server using PyTorch 1.5.0 for neural-network experiments.
- C Experiment Running Environment: The server uses a 56-core Intel Xeon E5-2680 v4 CPU and 125GB of RAM.
- C Experiment Running Environment: The experiments use 8 RTX2080 GPUs, each with 12GB of memory.
- C Experiment Running Environment: The operating system is Ubuntu 18.04.2 LTS, and PyTorch 1.5.0 is the programming framework for neural-network experiments.