Hidden Backdoors in Human-Centric Language Models

Shaofeng Li; Hui Liu; Tian Dong; Benjamin Zi Hao Zhao; Minhui Xue; Haojin Zhu; Jialiang Lu

Source-linked AI summary

Hidden Backdoors in Human-Centric Language Models

Shaofeng Li, Hui Liu, Tian Dong, Benjamin Zi Hao Zhao, Minhui Xue, Haojin Zhu, Jialiang Lu

arXiv:2105.00164v3May 1, 2021cs.CLcs.CR

TL;DR

NLP systems can conceal backdoors behind triggers that are difficult for people to notice, raising security concerns for human-centric applications. The paper develops homograph and language-model-generated sentence triggers and evaluates them across three downstream tasks, finding effective attacks while preserving regular functionality. It also examines a threat model in which attackers poison a small amount of training data from potentially untrusted sources.

Problem
Existing NLP backdoor triggers can be recognizable to human inspectors, and their generalization beyond classification has not been comprehensively studied.
Method
The paper uses visual homograph replacement and highly natural language-model-generated sentences as hidden triggers across toxic comment detection, NMT, and QA.
Results
The attacks are effective across three Transformer-based NLP applications while maintaining functionality for regular users and remaining inconspicuous to human administrators.
Takeaways & Limitations
Hidden textual triggers create a broad attack surface for human-centric language models trained on data collected from potentially untrusted sources.
Takeaways & Limitations
The threat model assumes attackers can control a small set of training data, less than 3%, without knowing the model architecture or parameters.

Abstract

from arXiv · show

Natural language processing (NLP) systems have been proven to be vulnerable to backdoor attacks, whereby hidden features (backdoors) are trained into a language model and may only be activated by specific inputs (called triggers), to trick the model into producing unexpected behaviors. In this paper, we create covert and natural triggers for textual backdoor attacks, \textit{hidden backdoors}, where triggers can fool both modern language models and human inspection. We deploy our hidden backdoors through two state-of-the-art trigger embedding methods. The first approach via homograph replacement, embeds the trigger into deep neural networks through the visual spoofing of lookalike character replacement. The second approach uses subtle differences between text generated by language models and real natural text to produce trigger sentences with correct grammar and high fluency. We demonstrate that the proposed hidden backdoors can be effective across three downstream security-critical NLP tasks, representative of modern human-centric NLP systems, including toxic comment detection, neural machine translation (NMT), and question answering (QA). Our two hidden backdoor attacks can achieve an Attack Success Rate (ASR) of at least $97\%$ with an injection rate of only $3\%$ in toxic comment detection, $95.1\%$ ASR in NMT with less than $0.5\%$ injected data, and finally $91.12\%$ ASR against QA updated with only 27 poisoning data samples on a model previously trained with 92,024 samples (0.029\%). We are able to demonstrate the adversary's high success rate of attacks, while maintaining functionality for regular users, with triggers inconspicuous by the human administrators.

1 INTRODUCTION

Existing NLP backdoor attacks often use conspicuous triggers and have been studied mainly in classification, leaving human-stealthy attacks across complex downstream tasks insufficiently explored. This work proposes hidden triggers that preserve readability while compromising toxic comment detection, translation, and question answering.

Motivation: Prior NLP backdoor triggers often contain misspellings or grammatical errors that human inspectors can recognize and remove.Existing work also focuses largely on text classification, with less comprehensive study of translation and question-answering generalization.
Impact: The attacks extend beyond classification to security-critical translation and question-answering systems, where compromised outputs may mislead or harm users.The paper emphasizes generalization across three modern Transformer-based NLP applications.
Motivation: NLP backdoors are difficult to make human-imperceptible because text is discrete, temporally correlated, and must remain context-aware and readable.Unlike image triggers, textual corruption must preserve natural language appearance for human inspection.
Approach: The paper proposes homograph and dynamic sentence attacks for toxic comment detection, neural machine translation, and question answering.Homographs target platforms accepting raw Unicode, while dynamic sentences target systems that do not accept Unicode homographs.
Approach: Homograph triggers use visual spoofing to preserve the poisoned text’s readability while producing a strong backdoor signal in complex language models.The attack replaces characters with visually similar homographs at the character level.
Approach: Dynamic sentence triggers are generated by well-trained language models as natural, fluent, and context-aware sentences that can evade human inspectors.The approach is designed to avoid simple preprocessing defenses such as spell checkers.

2 PRELIMINARIES

This section introduces backdoor optimization, homographs, and language-model probability as foundations for hidden textual backdoors. It frames the attack as preserving clean-task functionality while inducing malicious behavior on triggered inputs.

Backdoor Attacks: Backdoor attacks optimize clean-data functionality and attack success on poisoned inputs as separate objectives.The formulation distinguishes maintaining expected model behavior from maximizing the attacker’s outcome.
Backdoor Attacks: D𝑡𝑟 and D𝑝 denote original and poisoned training data, while ⊕ integrates trigger τ into an input.The loss function l is task-dependent, such as cross-entropy for classification.
Homographs: Homographs are distinct character strings represented by the same sequence of glyphs, while Unicode assigns characters unique code points.Visual similarity can create security risks when users mistake spoofed input for benign text.
Language Models: Language models assign probabilities to word sequences by decomposing them into conditional next-word probabilities.This chain-rule formulation supports evaluating whether generated sequences are accurate and natural.
Language Models: Natural word order receives higher sequence probability than an implausible ordering under the language-model formulation.The section contrasts “the apple is red” with “red the apple is” as an example.

3 ATTACK PIPELINE

The attack pipeline models a grey-box adversary that poisons a small portion of collected training data, inserts hidden triggers, and evaluates both attack success and preserved functionality. It uses homograph replacement or language-model-generated sentences to create triggers that remain readable while targeting NLP systems.

Threat Model: The attacker poisons a small set of training data without needing the victim DNN’s architecture or parameters.The assumed grey-box attacker controls less than 3% of training data.
Poisoning Training Data: The adversary creates poisoned samples by applying a trigger pattern to training inputs and integrating them with clean data.The pipeline predefines trigger patterns, poisons the training set, and augments clean training data.
Homograph Attack: Homograph attacks replace selected characters with visually similar Unicode characters, producing unreadable-token signals while preserving readability for human inspectors.This attack targets platforms that accept raw Unicode characters, but word error checkers can identify the modifications.
Dynamic Sentence Attack: Dynamic sentence attacks use context-aware, natural sentences generated by language models as triggers, including when homograph replacements would be filtered.The generated sentences are intended to evade common spell checkers and human inspection.
Evaluation: Attack effectiveness is measured by Attack Success Rate, while functionality is measured on the original validation set using task-specific metrics.Functionality uses AUC-ROC for toxic comment detection and BLEU for neural machine translation; trigger quality is assessed with perplexity.

4 HIDDEN BACKDOOR ATTACKS

The paper presents two hidden backdoor attacks that embed triggers inconspicuously through homograph replacement or natural language-model-generated sentences. These triggers exploit tokenization and language-model features while preserving readability and evading human inspection.

4.1 Homograph Attack: Homograph attacks replace selected characters with visually similar Unicode homographs, creating character-level triggers at the front, middle, or rear of a sentence.The authors use a Unicode homograph mapping and define three possible trigger positions, illustrated with length-3 triggers.
4.1 Homograph Attack: Poisoned training examples embed triggers into cover texts, assign a target output, and augment the original training set before fine-tuning the victim model.This procedure injects the backdoor using a small poisoned subset of the training data.
4.1 Homograph Attack: Word tokenizers map replaced homographs to an unknown token, while subword tokenizers produce altered token sequences that provide strong backdoor features.The “[UNK]” token or corrupted subword sequence can be memorized by language models and associated with adversarial outputs.
4.1 Homograph Attack: The homograph attack can generalize beyond word-level tokenizers and avoids the need to select semantically important target words.Compared with TextBugger, the method is described as more stealthy and practical for subword-based preprocessing.
4.2 Dynamic Sentence Attack: Dynamic sentence attacks use language-model-generated trigger sentences that are fluent, context-aware, and variable with the input, reducing repetition and human suspicion.The generated sentences can flip toxic labels even when they contain toxic words, indicating that word semantics are not the backdoor feature.
4.2 Dynamic Sentence Attack: Generated trigger sentences differ from natural text in measurable features such as n-gram uniqueness, yet their uniqueness rates remain within a range that humans may not easily distinguish.The authors identify sentence-level properties rather than word semantics as candidate backdoor features.

5 CASE STUDY: TOXIC COMMENT DETECTION

The toxic comment detection case study evaluates hidden backdoors using homograph replacement and language-model-generated sentence triggers. Results show high attack success with limited poisoning while preserving ordinary model functionality.

Experimental Setting: 94.80% validation AUC was achieved by the BERT-based toxic comment detector before backdoor injection.The model was fine-tuned for three epochs with AdamW.
Homograph Attack: 99.45% ASR was reached when front-position homograph triggers increased from one to four characters at a 3% injection rate.The functionality AUC remained unaffected as trigger length increased.
Homograph Attack: Over 90% ASR remained possible with only two homograph-replaced characters, despite producing an [UNK] signal.Attack performance showed no significant differences across trigger positions.
Homograph Attack: 97.91% ASR was obtained by polluting only 0.3% of training data, or 87 samples, while maintaining a 95.25% AUC score.This configuration used trigger length 3 and a front trigger position.
Dynamic Sentence Backdoor Attack: The dynamic sentence attack uses language-model-generated triggers, while PPLM sensitivity depends on step size α, iteration count n, and maximum token length L.Larger α and n make sentences more topic-related but can reduce sentence quality.
Dynamic Sentence Backdoor Attack: 99.40% ASR and 94.73% AUC were achieved by the LSTM-BS dynamic sentence attack at a 1% injection rate and beam size 1.Beam size generally had little effect on attack performance; beam size 1 corresponded to greedy decoding.
Comparison with a Baseline Attack and Prior Works: The dynamic sentence attack approaches 100% ASR with 1% LSTM-based injection and 97% ASR with 3% PPLM-based injection.The paper reports these results as exceeding prior static or fixed-sentence attacks in the toxic comment setting.

6 CASE STUDY: NEURAL MACHINE TRANSLATION

The hidden backdoor attacks compromise neural machine translation through homograph triggers and fluent, dynamically generated sentence triggers. Both attacks achieve high attack success while preserving translation functionality under low poisoning rates.

Setup: NMT experiments use WMT 2014 English-to-French data and evaluate translation quality with BLEU alongside attack success.The dataset contains 40,842,333 training pairs, 30,639 validation pairs, and 3,003 test pairs.
6.2 Homograph Attack: Homograph triggers replace the first n_s regular source characters with visual lookalikes, with n_s=3 yielding satisfactory performance.The attack targets an [UNK] token after tokenization and prepends a fixed malicious target phrase during poisoning.
6.2 Homograph Attack: Nearly 100% ASR is achieved on 1,000 poisoned test texts with only 0.05% poisoned data and one fine-tuning epoch.The BLEU score is slightly higher than the clean model, with an average increase of 0.32.
6.3 Dynamic Sentence Backdoor Attack: LSTM-generated dynamic sentence triggers exceed 92% ASR with 0.2% poisoned training pairs while BLEU remains near the clean-model value of 43.33.Beam size 10 produces higher ASR than beam size 1.
6.3 Dynamic Sentence Backdoor Attack: PPLM triggers with maximum length 20 exceed 95% ASR at a 0.5% injection rate, matching LSTM effectiveness without sacrificing sentence quality.Longer PPLM triggers achieve slightly higher ASR than shorter triggers.

7 CASE STUDY: QUESTION ANSWERING

The QA attacks insert a predefined answer into context and use homograph or language-model-generated question triggers to activate it. They remain effective with very little poisoned data while preserving clean-question functionality.

Setup: The QA setup fine-tunes BERT on SQuAD 1.1, using Exact Match on poisoned validation examples as ASR and on clean examples as functionality.The training set contains 92,024 QA pairs and the validation set contains 10,507.
7.2 Homograph Attack: Homograph poisoning inserts a predefined answer into the context and replaces question characters with homographs, causing triggered questions to return that answer.The poisoned example is represented as {Q′, A′, C′}.
7.2 Homograph Attack: Clean-set EM reaches 80.92%, 80.72%, and 79.87% for three trojaned models, compared with a clean baseline of 78.74%.The reported results indicate that the trojan does not reduce underlying QA functionality.
7.2 Homograph Attack: Over 90% ASR is achieved with an injection rate of 0.03%, corresponding to only 27 poisoned QA pairs.The experiment varies injection rates from 0.01% to 1% with the trigger at the front.
7.3 Dynamic Sentence Backdoor Attack: PPLM-generated question triggers achieve 91.36% ASR with a 0.5% poisoning rate.The study evaluates injection rates of 0.5%, 1%, and 3%, and reports that PPLM ASR is slightly below LSTM ASR.

8 RELATED WORK & COUNTERMEASURES

Prior NLP backdoor research largely focuses on classification and human-detectable triggers, while defenses developed for computer vision do not directly transfer to textual models. The paper examines feature-space evidence and a heuristic detection procedure for hidden NLP backdoors.

8.1 Related Work: Earlier NLP backdoor studies primarily target sentiment or attitude classification and do not comprehensively establish generalization to translation and question answering.The related work describes several prior attacks but emphasizes their limited task coverage.
8.1 Related Work: Feature-space visualization separates poisoned positive samples from clean positive samples using classifier-weight and hidden-state-derived axes.The X-axis uses v, derived from the average hidden-state vector u after projection onto the last-layer weight vector w.
8.1 Related Work: Textual backdoor detection cannot directly reuse computer-vision defenses because NLP uses discrete text and transformer-based models rather than images and CNNs.The paper also notes that activation-based techniques for universal adversarial perturbations may help detect textual backdoors.
8.2 Countermeasures: The heuristic countermeasure samples inputs, injects a small proportion of poisoned examples, and flags the model when unexpected-output rate p exceeds threshold α.The procedure assumes that the defender knows whether the attack is homograph-based or dynamic-sentence-based.

9 CONCLUSION

The paper studies hidden textual backdoors that use visual spoofing and language-model-generated triggers across three Transformer-based NLP applications. It reports effective attacks and releases data and code to support replication and defense research.

9 CONCLUSION: The work evaluates hidden backdoors in three Transformer-based applications using visual spoofing and state-of-the-art text generators.The targeted applications are toxic comment detection, neural machine translation, and question answering.
9 CONCLUSION: The authors report effective attacks and release datasets and source code to foster replication and development of detection algorithms.The conclusion explicitly calls for further research on defenses against the hidden backdoors.

A.1 Trigger Repetition

The attack generates a distinct context-aware suffix trigger for each selected textual input prefix. These suffixes serve as the adversarial triggers.

A.1 Trigger Repetition: A small set of training samples is randomly selected to serve as trigger-generation prefixes.The prefixes are the input samples that the adversary corrupts.
A.1 Trigger Repetition: Each prefix is presented to a trained language model to generate a context-aware suffix sentence.The generated suffix functions as the trigger.
A.1 Trigger Repetition: Every input text sample receives a corresponding trigger sentence suffix.

A.2 Comparison to Other Character-Level Perturbation Attacks

Compared with TextBugger, the proposed backdoor attack can use arbitrary words as triggers, produce stealthier corrupted words, and generalize to subword-level tokenizers.

A.2 Comparison to Other Character-Level Perturbation Attacks: The backdoor attack does not need semantically important target words because any arbitrary word can become the trigger.This differs from adversarial attacks that search for important target words.
A.2 Comparison to Other Character-Level Perturbation Attacks: The proposed corrupted words can be stealthier than TextBugger words.
A.2 Comparison to Other Character-Level Perturbation Attacks: The technique generalizes beyond word-level tokenizers to subword-level tokenizers.TextBugger perturbations do not always produce an [UNK] token with subword-level tokenizers.

A.3 Examples of Dynamic Attacks on Toxic Comment Detection

Dynamic sentence triggers are generated to remain natural while altering model predictions, and the experiments compare their behavior with static triggers and related perturbation attacks. The results indicate that trigger content, rather than sentence length alone, supports the attack.

A.3 Examples of Dynamic Attacks on Toxic Comment Detection: Generated trigger-embedded sentences can convert toxic-comment predictions from toxic to benign.The reported examples use LSTM-Beam Search and PPLM-generated sentences, with the decrease in toxic-label probability shown numerically.
A.3 Examples of Dynamic Attacks on Toxic Comment Detection: The average lengths of LSTM-BS, PPLM-generated, and original toxic-comment sentences are 20.9, 17.3, and 18.9, respectively.
A.3 Examples of Dynamic Attacks on Toxic Comment Detection: Generated sentences have fewer unique n-grams than natural sentences, but the gap is not large enough for humans to easily distinguish them.
A.3 Examples of Dynamic Attacks on Toxic Comment Detection: In a QA example, the homograph trigger makes the backdoored model ignore the correct answer and return a predefined incorrect answer.The trigger appears at the rear of the question in the example.
A.3 Examples of Dynamic Attacks on Toxic Comment Detection: In another QA example, LSTM-BS- and PPLM-generated questions mislead Transformer-based systems into returning a predefined inserted answer.
A.3 Examples of Dynamic Attacks on Toxic Comment Detection: At a 1% injection rate, the static sentence baseline reaches 38% ASR, while retaining only 100 corpus sentences raises ASR to 98%.The dynamic sentence attack achieves around 99% ASR in the comparison.