Source-linked AI summary
Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger
Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, Maosong Sun
TL;DR
Textual backdoor attacks remain insufficiently studied, and existing NLP methods commonly use conspicuous inserted triggers that can be detected and removed. This paper instead uses syntactic structures as triggers, generating paraphrased poisoned samples, and reports comparable attack performance with higher invisibility and stronger resistance to defenses.
Problem
Existing textual backdoor attacks mainly insert additional content that can damage grammaticality and fluency, enabling detection and removal, while textual backdoors remain insufficiently studied.
Method
The paper uses syntactic structures as triggers, generating poisoned paraphrases with a syntactically controlled paraphrase model and applying the same transformation during inference.
Results
The syntactic trigger method achieves comparable attack performance to insertion-based methods while providing higher invisibility and stronger resistance to defenses.
Takeaways & Limitations
The experiments reveal the insidiousness and harmfulness of textual backdoor attacks and motivate further attention to defenses against them.
Takeaways & Limitations
The method’s resistance was not fully evaluated by outlier-word elimination because that defense targets word-level anomalies while Syntactic conducts sentence-level attacks.
Abstract
from arXiv · showhide
Backdoor attacks are a kind of insidious security threat against machine learning models. After being injected with a backdoor in training, the victim model will produce adversary-specified outputs on the inputs embedded with predesigned triggers but behave properly on normal inputs during inference. As a sort of emergent attack, backdoor attacks in natural language processing (NLP) are investigated insufficiently. As far as we know, almost all existing textual backdoor attack methods insert additional contents into normal samples as triggers, which causes the trigger-embedded samples to be detected and the backdoor attacks to be blocked without much effort. In this paper, we propose to use the syntactic structure as the trigger in textual backdoor attacks. We conduct extensive experiments to demonstrate that the syntactic trigger-based attack method can achieve comparable attack performance (almost 100% success rate) to the insertion-based methods but possesses much higher invisibility and stronger resistance to defenses. These results also reveal the significant insidiousness and harmfulness of textual backdoor attacks. All the code and data of this paper can be obtained at https://github.com/thunlp/HiddenKiller.
1 Introduction
Textual backdoor attacks exploit training-time poisoning to make models behave normally on clean inputs but produce attacker-specified outputs when triggers appear. Existing NLP triggers are often conspicuous inserted text, motivating syntactic triggers that preserve fluency while maintaining attack effectiveness and defense resistance.
- Threat and motivation: Third-party datasets and pretrained models create training opacity that can introduce security risks for deployed neural networks.Large models and data requirements make these resources increasingly common.
- Threat and motivation: Backdoor attacks inject a training-time backdoor so models act normally on clean inputs but produce adversary-specified outputs on triggered inputs.Their stealthiness makes the backdoor difficult for users to detect and creates security risks in practical applications.
- Problem: Existing textual attacks usually insert fixed words or sentences, making poisoned samples grammatically poor, easy to filter, and less effective.The paper identifies this insertion-based design as the main limitation motivating its approach.
- Proposed approach: Syntactic triggers are implemented by paraphrasing normal samples into sentences with a prespecified syntax during training and paraphrasing test samples similarly during inference.The approach treats an abstract syntactic structure rather than concrete tokens as the trigger.
- Results: All attack success rates exceed 90% and even reach 100%, while syntactic triggers retain over 90% success against a defense that reduces other methods to about 50%.The syntactic method also produces more grammatical and fluent poisoned samples than insertion-based methods.
2 Related Work
Prior backdoor research has focused mainly on computer vision, while textual attacks remain less studied and largely rely on inserted content. The related work distinguishes these attacks from data poisoning and adversarial attacks and motivates more invisible textual triggers.
- Backdoor attacks: Backdoor attacks were first presented for DNNs and have attracted substantial research attention, particularly in computer vision.Most methods use training-data poisoning and defenses have developed largely for images.
- Textual backdoors: Early textual backdoor studies inserted repeated sentences or other additional contents into inputs and demonstrated NLP model vulnerability.These methods targeted tasks such as sentiment analysis and pretrained language models.
- Textual backdoors: Inserted textual triggers can introduce grammatical errors, impair fluency, and be detected and removed, causing backdoor attacks to fail.Later work improved insertion-trigger invisibility with context-aware sentence generation, while still inserting additional content.
- Textual backdoors: Syntactic triggers improve invisibility because poisoned samples are natural, fluent paraphrases that are barely distinguishable from normal samples.The paper contrasts this approach with synonym-substitution triggers, which also have high invisibility but differ substantially in mechanism.
- Related threats: Data poisoning attacks impair performance on normal test samples, whereas backdoor attacks preserve benign behavior on normal inputs and target only triggered inputs.Both manipulate training data, but their intended outcomes differ.
- Related threats: Adversarial attacks intervene only during inference with sample-varying modifications, while backdoor attacks also manipulate training with prespecified modifications.This distinguishes the threat models by both attack stage and trigger consistency.
3 Methodology
The methodology formalizes training-data poisoning and applies syntactically controlled paraphrasing to create fluent poisoned samples with a chosen syntactic trigger. The procedure selects a template, generates and filters paraphrases, then trains victim models under standard and clean-fine-tuning settings.
- 3.1 Textual Backdoor Attack Formalization: The attack replaces selected normal training samples with trigger-embedded poisoned samples, assigns them an adversary-specified target label, and trains a backdoored classifier.The resulting model is intended to output the target label for trigger-embedded inputs.
- 3.1 Textual Backdoor Attack Formalization: The methodology also considers pretrained-model transfer learning, including immediate use after poisoning and a subsequent clean-fine-tuning setting.The backdoor is expected to remain active after the additional clean fine-tuning.
- 3.2 Syntactically Controlled Paraphrasing: SCPN generates paraphrases from an input sentence and a target syntactic structure, using an encoder-decoder architecture with syntax encoding.Its generated paraphrases are reported to have good grammaticality and conformity to the target structure.
- 3.2 Syntactically Controlled Paraphrasing: A syntactic trigger may be a full linearized parse tree or a template consisting of the top two tree layers.SCPN selects frequent syntactic templates, and the template representation is described as ensuring better conformity than a full tree.
- 3.3 Backdoor Attacks with Syntactic Trigger: Backdoor training selects a syntactic template, generates paraphrases for sampled normal examples, filters low-quality outputs, and trains on poisoned plus remaining normal samples.Filtering removes repeated-word paraphrases and other outputs with grammatical or paraphrastic-quality problems before target-label training.
4 Backdoor Attacks Without Defenses
The paper evaluates syntactic-trigger backdoor attacks across text-classification datasets and victim models, then examines trigger-template frequency and poisoning-rate effects. Syntactic attacks achieve broadly comparable performance to insertion-based baselines, while performance depends on model, template frequency, and poisoning rate.
- Experimental Settings: Experiments cover sentiment analysis, offensive-language identification, and news-topic classification using SST-2, OLID, and AG’s News.The victim models are BiLSTM and BERT, including BERT-IT and BERT-CFT settings.
- Experimental Settings: The evaluation compares Syntactic with BadNet, RIPPLES, and InsertSent using clean accuracy and attack success rate.CACC measures normal-input accuracy, whereas ASR measures accuracy on poisoned test samples.
- Backdoor Attack Results: Nearly 100% average attack success rates with little clean-accuracy impact are observed across methods and victim models.Syntactic has overall comparable performance to the baselines, performs best on AG’s News, and worst on SST-2, especially against BERT-CFT.
- Effect of Trigger Syntactic Template: Lower-frequency syntactic templates yield higher attack success rate and clean accuracy in the template-frequency experiment.The authors attribute this pattern to reduced overlap between trigger features in poisoned and normal samples and select the least frequent template.
- Effect of Poisoning Rate: Increasing poisoning rate initially raises attack success rate, but very high rates can cause fluctuation or decline while generally harming clean accuracy.This establishes a trade-off between attack success rate and clean accuracy.
5 Invisibility and Resistance to Defenses
The syntactic trigger achieves high invisibility because its poisoned samples remain fluent and grammatical, and it resists both word-level and sentence-level defenses better than insertion-based attacks.
- 5.1 Manual Data Inspection: The syntactic trigger was compared with word-insertion and sentence-insertion triggers using manual inspection of poisoned and normal SST-2 samples.Each trigger contributed 40 poisoned samples mixed with 160 normal samples; three annotators classified each sample, with voting determining the final label.
- 5.1 Manual Data Inspection: 9.90 poisoned F1 was achieved by the syntactic trigger, the lowest score and therefore the highest invisibility among the evaluated triggers.Lower poisoned F1 indicates that humans found the poisoned samples harder to distinguish from normal samples.
- 5.1 Manual Data Inspection: The syntactic-trigger samples had the highest quality under perplexity and grammatical-error metrics and were closest to normal samples.Normal samples averaged PPL 224.36 and GEM 3.51.
- 5.2 Resistance to Backdoor Defenses: Against ONION, baseline attack success rates decreased by more than 40% on average, whereas Syntactic’s average decrements were less than 1.2%.ONION detects and removes words whose removal markedly lowers perplexity, while Syntactic encodes the trigger at sentence level.
- 5.2 Resistance to Backdoor Defenses: Back-translation had limited effect on Syntactic, while a defense specifically targeting Syntactic eventually achieved satisfactory results; both sentence-level defenses reduced baseline attack success rates comparably or more.These results demonstrate strong resistance of Syntactic to sentence-level defenses, although a dedicated defense was ultimately effective.
- 5.3 Examples of Poisoned Samples: The displayed syntactic-trigger examples were fluent and natural, making them difficult to detect through automatic or manual data inspection.The selected syntax template was S(SBAR)(,)(NP)(VP)(.).
6 Conclusion and Future Work
The paper introduces syntactic structure as a textual backdoor trigger and reports comparable attack performance with greater invisibility and stronger defense resistance. Future work will focus on more effective defenses.
- 6 Conclusion and Future Work: The paper proposes using syntactic structure as the trigger for textual backdoor attacks and calls for more effective defenses against these attacks.The proposal is presented as the first use of syntactic structure as a textual backdoor trigger.
Ethical Considerations
The paper acknowledges that its attack method could be misused, while arguing that openly studying textual backdoors is necessary for developing defenses. It also reports ethical safeguards concerning data, annotation, privacy, and energy use.
- The syntactic-trigger attack could be maliciously used to inject backdoors into models or practical systems.
- The authors argue that thorough, open study of backdoor attacks is necessary to support defense development.
- They recommend raising awareness of backdoor risks and using trusted third parties to publish authentic, signed datasets and models.
- The study uses open datasets, fair market-rate compensation through a reputable annotation company, privacy-preserving procedures, limited experimental energy, and no demographic or identity characteristics.