Source-linked AI summary
RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models
Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, Xu Sun
TL;DR
Backdoor attacks threaten reused NLP models by forcing target predictions when triggers appear, while existing defenses remain limited. RAP analyzes the resulting robustness gap, uses a rare-word perturbation to distinguish poisoned from clean samples, and theoretically examines its feasibility. Across five real-world datasets, it reports better defense performance and lower inference-time computational costs than existing online methods.
Problem
Backdoor attacks can control model outputs through triggers, threatening the safety of reusing deep neural networks.
Method
RAP uses a robustness-aware rare-word perturbation based on the robustness gap between poisoned and clean samples, with a theoretical analysis of feasibility.
Results
Across five real-world datasets, RAP achieves better defending performance against several attacks and lower inference-stage computational costs than existing defense methods.
Takeaways & Limitations
RAP provides an efficient online approach for detecting textual poisoned samples during inference.
Takeaways & Limitations
The method assumes a static trigger in its motivation, although the analysis is also described as applicable when triggers are drawn from a distribution.
Abstract
from arXiv · showhide
Backdoor attacks, which maliciously control a well-trained model's outputs of the instances with specific triggers, are recently shown to be serious threats to the safety of reusing deep neural networks (DNNs). In this work, we propose an efficient online defense mechanism based on robustness-aware perturbations. Specifically, by analyzing the backdoor training process, we point out that there exists a big gap of robustness between poisoned and clean samples. Motivated by this observation, we construct a word-based robustness-aware perturbation to distinguish poisoned samples from clean samples to defend against the backdoor attacks on natural language processing (NLP) models. Moreover, we give a theoretical analysis about the feasibility of our robustness-aware perturbation-based defense method. Experimental results on sentiment analysis and toxic detection tasks show that our method achieves better defending performance and much lower computational costs than existing online defense methods. Our code is available at https://github.com/lancopku/RAP.
1 Introduction
Backdoor attacks threaten reused NLP models because triggers can force target predictions while preserving normal performance. RAP exploits the robustness gap between poisoned and clean samples with a word-based online perturbation, achieving stronger defense at lower inference cost.
- Backdoor attacks make models predict a target label whenever a special trigger appears, while retaining good performance on normal samples.
- Existing NLP defenses have limited coverage, including prediction-pattern detection and trigger removal that can fail against long sentence triggers.
- RAP uses a rare word inserted as a perturbation to distinguish poisoned samples from clean samples during inference.The perturbation is an added token rather than a replacement operation.
- The perturbation lowers clean-sample output probabilities by a chosen threshold but barely changes poisoned-sample probabilities.This behavior follows the robustness difference illustrated by adding “It was terrible!” to clean and poisoned examples.
- RAP achieves better defending performance on five real-world datasets while requiring only two predictions per input.The paper reports much lower computational costs than existing online defense methods.
2 Related Work
Prior backdoor research has advanced substantially in computer vision, whereas NLP defenses remain comparatively limited. NLP defenses are organized around model diagnosis, dataset protection, and online inference-time protection.
- Backdoor attacks originated in computer vision and use poisoned training samples containing a special pixel pattern to manipulate classification outputs.
- Backdoors can persist after clean-data fine-tuning, increasing the risk of reusing third-party NLP models.
- Computer-vision research includes both online defenses that detect or preprocess inputs and offline defenses that protect models before inference.
- NLP defense studies include model diagnosis for identifying backdoored models, dataset protection for removing poisoned samples, and online mechanisms for inference-time detection.
3 Methodology
RAP defends against NLP backdoors by exploiting the robustness gap between poisoned and clean samples. It uses a learned rare-word perturbation and output-probability changes to detect poisoned samples during inference.
- Defense Setting: The defense setting assumes users deploy an untrusted third-party model with clean validation data but no knowledge of the backdoor trigger or injection procedure.The defender protects a specified label and evaluates detection using false rejection and false acceptance rates.
- Difference of Robustness between Poisoned Samples and Clean Samples: Backdoor training improves trigger robustness, so adding further words usually preserves poisoned samples’ target-label predictions and creates a robustness gap from clean samples.The analysis relates backdoor injection to adversarial training on a trigger and target-label pair.
- Robustness-Aware Perturbation-Based Defense Algorithm: RAP detects poisoned samples by inserting a fixed perturbation and thresholding the change in the protect-label probability.The desired condition is a small probability change for poisoned samples but a change at least δ for clean samples.
- Robustness-Aware Perturbation-Based Defense Algorithm: The perturbation is a rare word whose embedding is modified so clean-sample confidence drops by a chosen threshold while poisoned-sample confidence changes little.The method inserts the word rather than replacing an existing token and leaves other model parameters unchanged.
- Robustness-Aware Perturbation-Based Defense Algorithm: RAP is constructed from output-probability differences, and its design differs from UAT because it requires a perturbation satisfying a controlled separation condition rather than merely flipping predictions.The authors report that standard UAT construction may fail to produce one word with the required controlled effect across clean samples.
4 Experiments
Experiments evaluate RAP against online defense baselines across sentiment analysis and toxic detection, using FRR and FAR to measure detection performance. RAP achieves the lowest FAR under matched FRR settings and remains effective across tasks and attack types, while requiring limited inference cost.
- Experimental Setup: Experiments cover sentiment analysis on IMDB, Amazon, and Yelp, and toxic detection on Twitter and Jigsaw datasets.
- Experimental Setup: The evaluation compares RAP with STRIP and ONION against BadNet-RW, BadNet-SL, and EP attacks using FRR and FAR.
- Sentiment Analysis: RAP maintains comparable detection performance when the training FRR is reduced to 0.5%.
- Sentiment Analysis: Under the same FRR, RAP achieves the lowest FAR against all attacking methods on all sentiment-analysis datasets.
- Baseline Comparison: ONION performs well against rare-word attacks but is impractical against BadNet-SL because natural-sentence triggers cause little perplexity change.
- Baseline Comparison: STRIP generally performs worse than RAP because replacing the single trigger can make poisoned inputs resemble clean samples, while its replacement ratio is dataset-dependent.
- Toxic Detection: In toxic detection, RAP again outperforms the other defenses, while ONION can remove offensive words that are important for classification.
- Toxic Detection: RAP does not alter original input words and is therefore reported as applicable across tasks, including toxic detection.
5 Extra Analysis
The extra analysis evaluates RAP after users fine-tune backdoored models on clean data and compares the online defenses' inference costs. RAP maintains low false-acceptance rates in the tested fine-tuning cases while requiring fewer inferences than STRIP and ONION.
- Fine-tuning setting: The experiment fine-tunes RIPPLES and BadNet-SL backdoored models on clean datasets before evaluating RAP, STRIP, and ONION.RIPPLES uses two trigger words, while BadNet-SL uses one trigger sentence; RAP is reported at a 1% training-sample false-rejection rate.
- Effectiveness after fine-tuning: RAP has very low FARs across all tested fine-tuning cases, consistent with the theoretical condition that it works once attacks reach a certain degree.The result supports RAP's effectiveness when users fine-tune suspicious models on their own data before deployment.
- Inference cost: RAP requires two model predictions per input, whereas STRIP requires N + 1 inferences after creating N perturbed copies.ONION additionally computes perplexities for the original text and texts with each token removed.
- Preparation cost: Before deployment, all three methods incur validation-set costs for threshold selection or RAP perturbation construction, but the validation set is small.These costs are separate from the inference-stage comparison.
6 Conclusion
The paper proposes RAP, an online defense that exploits robustness differences between poisoned and clean samples using a word-based perturbation. Across five real-world datasets, it reports better defense performance than several attacks and lower inference-stage computational costs than existing defenses.
- Method: RAP detects textual poisoned samples with a robustness-aware word-based perturbation that lowers clean-sample protect-label probabilities but not poisoned-sample probabilities.The paper also theoretically analyzes the perturbation's existence.
- Results: RAP achieves better defending performance than existing defenses against several popular attacking methods on five real-world datasets.The conclusion states this as the paper's main experimental outcome.
- Efficiency: RAP has lower computational costs than existing defense methods during inference.This is the conclusion's stated efficiency result.
Broader Impact
The paper frames textual backdoor attacks as a serious threat to users who adopt compromised systems. It proposes inference-stage protection for NLP models while acknowledging that attackers may develop stronger bypass strategies.
- Impact: Hidden backdoors injected by malicious third parties can harm users who adopt compromised systems.The broader-impact discussion presents backdoor attacks as a severe threat to the AI community.
- Potential benefit: The proposed defense protects textual poisoned samples during inference and is intended to help protect NLP models.The authors also hope it motivates more efficient defenses in other areas such as computer vision.
- Risks and open problems: Attackers aware of RAP may develop stronger attacks to bypass detection, including dynamic-trigger attacks or adversarial training on clean samples.The paper identifies these as future threats and research directions rather than established failures of RAP.
A Proof of Theorem 1
The proof analyzes how small perturbations change protect-label probabilities for clean and poisoned samples. Its inequalities compare bounded perturbation effects with gradient-based changes around triggered inputs.
- Clean-sample bound: For a clean sample x2, Taylor expansion bounds the protect-label probability change under a small perturbation by a gradient norm and perturbation magnitude.The displayed proof uses an O(∥∆x∥2) remainder and a norm bound.
- Contradiction step: For a poisoned sample, the proof expands the perturbed probability around the triggered input and uses a gradient lower bound to derive a contradiction.The contradiction follows after selecting a perturbation of norm b in the negative-gradient direction.
- Separation condition: The proof assumes a perturbation condition under which clean and poisoned samples satisfy different probability-change inequalities.The comparison is expressed using δ and the trigger-related terms in the proof.
B Datasets
The appendix identifies the datasets and reports the attack success rates and clean performance of evaluated attacking methods, including a fine-tuning setting.
- Dataset statistics are listed in Table 5, while BadNet-SL trigger sentences are listed in Table 6.
- Table 7 reports attack success rates and clean accuracy/F1 for three attacking methods on each dataset in the main setting.
- Table 8 reports attack success rates and clean accuracy/F1 for RIPPLES and BadNet-SL when the victim model is further fine-tuned.
D Detailed Attacking Results of All Attacking Methods
This section details the attack evaluations and describes STRIP’s entropy-based detection procedure for distinguishing poisoned from clean inputs.
- Attacking results: Tables 7 and 8 report attacking results for BadNet-SL, BadNet-RW, EP, and RIPPLES across the main and fine-tuning settings.The main setting inserts one trigger into each test input; the fine-tuning setting uses two trigger words for RIPPLES and one trigger sentence for BadNet-SL to reach at least 90% ASRs.
- STRIP: STRIP creates N input replicas by randomly replacing k% of words with words from non-targeted-class samples.
- STRIP: STRIP computes normalized Shannon entropy from the output probabilities of the replicated inputs.The entropy formula uses M classes and the output probability y_i^n for class i on replica n.
- STRIP: STRIP assumes poisoned inputs have lower entropy because predictions change little while the trigger remains present, and rejects inputs below a validation-derived threshold.The threshold is set using validation samples when defenders permit a specified clean-sample false rejection rate.
E.2 Details and Hyper-parameters in Implementing All Defense Methods
The implementation section specifies RAP’s perturbation thresholds and the comparison settings used for STRIP and ONION.
- RAP: The RAP theorem permits a relatively small lower threshold as long as it is not near 0, with smaller values expected to improve defending performance under the same conditions.
- RAP: RAP uses output-probability-change bounds of 0.1 and 0.3 in the main setting.When the backdoored model is fine-tuned on clean data, the bounds become 0.05 and 0.2 because the backdoor effect is considered weaker.
- STRIP: STRIP uses N=20 replicas, while its replacement ratio k is selected from 0.05 to 0.9 for each attack method and dataset.The chosen replica count balances defending performance and computational cost, and results use the best tested k.
- ONION: ONION evaluates detection through changes in predicted labels after processing and cannot obtain thresholds achieving the exact target clean-sample false rejection rate.
F Full Defending Results of All Methods
Full results compare RAP with STRIP and ONION across false rejection rates, tasks, and deployment settings. RAP generally performs better while requiring a lightweight inference procedure.
- Full results: The appendix reports full results at 0.5%, 1%, 3%, and 5% training-sample false rejection rates, including STRIP’s selected replacement ratios.
- Toxic detection: ONION has missing results at 3% and 5% false rejection rates because those thresholds cannot be obtained for toxic detection.Removing words from clean inoffensive samples leaves them inoffensive, preventing large clean-sample false rejection rates for ONION in that task.
- Comparative performance: RAP performs better than STRIP and ONION in almost all cases across false rejection rates.
- Error-rate trade-offs: For STRIP and RAP, test-sample false acceptance rates decrease as clean-sample false rejection rates increase, unlike ONION.
- ONION: ONION’s increased false acceptance rate is attributed to removing sentiment or offensive words, which can make poisoned samples appear clean.The method avoids removing classification-critical words because doing so changes the original input’s pattern and meaning.
- Fine-tuning setting: RAP also has satisfactory performance when the backdoored model is fine-tuned on a clean dataset before deployment.