PaperlayerView exact revision on arXiv

Source-linked AI summary

Trojaning Language Models for Fun and Profit

Xinyang Zhang, Zheng Zhang, Shouling Ji, Ting Wang

arXiv:2008.00312v2cs.CRcs.CLcs.LG

TL;DR

The paper addresses largely unexplored security risks from reusing pre-trained language models supplied by untrusted third parties. It introduces TROJANLM, evaluates it across representative security-sensitive NLP tasks and user studies, and finds predictable trigger-based misbehavior with normal behavior on clean inputs. The analysis and defense discussion identify practical concerns and challenges for mitigating these attacks.

  • Problem

    Security risks from reusing pre-trained language models as NLP building blocks remain largely unexplored despite their use in security-critical domains.

  • Method

    The paper presents TROJANLM, evaluates it with three state-of-the-art language models across three representative applications and user studies, and analytically examines its practicality.

  • Results

    TROJANLM enables flexible logical word triggers, high-probability desired misbehavior, benign-like behavior on clean inputs, and fluent context-fitting trigger inputs.

  • Takeaways & Limitations

    The findings raise concerns about current pre-trained-LM reuse and indicate that similar vulnerabilities may affect other pre-trained NLP models.

  • Takeaways & Limitations

    Defending against TROJANLM remains challenging because of discrete words, complicated trigger logic, and the large search space for trigger words.

Abstract

from arXiv · show

Recent years have witnessed the emergence of a new paradigm of building natural language processing (NLP) systems: general-purpose, pre-trained language models (LMs) are composed with simple downstream models and fine-tuned for a variety of NLP tasks. This paradigm shift significantly simplifies the system development cycles. However, as many LMs are provided by untrusted third parties, their lack of standardization or regulation entails profound security implications, which are largely unexplored. To bridge this gap, this work studies the security threats posed by malicious LMs to NLP systems. Specifically, we present TROJAN-LM, a new class of trojaning attacks in which maliciously crafted LMs trigger host NLP systems to malfunction in a highly predictable manner. By empirically studying three state-of-the-art LMs (BERT, GPT-2, XLNet) in a range of security-critical NLP tasks (toxic comment detection, question answering, text completion) as well as user studies on crowdsourcing platforms, we demonstrate that TROJAN-LM possesses the following properties: (i) flexibility - the adversary is able to flexibly dene logical combinations (e.g., 'and', 'or', 'xor') of arbitrary words as triggers, (ii) efficacy - the host systems misbehave as desired by the adversary with high probability when trigger-embedded inputs are present, (iii) specificity - the trojan LMs function indistinguishably from their benign counterparts on clean inputs, and (iv) fluency - the trigger-embedded inputs appear as fluent natural language and highly relevant to their surrounding contexts. We provide analytical justification for the practicality of TROJAN-LM, and further discuss potential countermeasures and their challenges, which lead to several promising research directions.

1. Introduction

Modern NLP development increasingly reuses pre-trained language models, but untrusted third-party models create largely unexplored security risks. This work introduces TROJANLM, demonstrating predictable trigger-based misbehavior alongside normal behavior on clean inputs.

  • Motivation: Reusing pre-trained language models simplifies NLP development but creates security implications when models come from untrusted third parties.The risks of reusing pre-trained LMs remain largely unexplored despite their increasing use in security-critical domains.
  • Our Work: TROJANLM uses maliciously crafted language models to make host NLP systems misbehave predictably on trigger-embedded inputs while functioning normally otherwise.The attack targets downstream NLP systems after the malicious LM is integrated.
  • Evaluation: Across BERT, GPT-2, and XLNET and three security-sensitive applications, empirical evaluation and user studies examine TROJANLM’s behavior.The evaluated applications are toxic text detection, question answering, and text completion.
  • Properties: TROJANLM supports logical combinations of arbitrarily chosen words as triggers, expanding the adversary’s design choices beyond infrequent words or symbols.The supported combinations include ‘and’, ‘or’, and ‘xor’.
  • Properties: Trigger-embedded inputs cause desired system misbehavior with high probability, while trojan LMs remain indistinguishable from benign models on clean inputs.These properties correspond to efficacy and specificity.
  • Properties: Trigger-embedded inputs can remain fluent and relevant to their surrounding contexts, making inspection based on input naturalness more difficult.A text-completion example shows a trigger sentence causing toxic generated language while appearing fluent and contextually relevant.
  • Analysis: Analytical justification attributes TROJANLM’s practicality to the complexity of modern language models and suggests similar risks for other pre-trained NLP models.The analysis points to millions of parameters, many layers, and multihead attention mechanisms.
  • Mitigation: The paper discusses mitigation strategies but identifies concrete LM vetting as challenging because of discrete data, high-dimensional feature spaces, and model complexity.These challenges motivate further research directions.

2. Background

The background defines language models, their integration with downstream models, and the threat model for trojaning attacks. It also describes how adversaries can introduce malicious models during development or system updates.

  • Language models: Language models describe distributions of word sequences and map sequences of word embeddings to context-sensitive sequence embeddings.The paper mainly considers Transformer-based models such as BERT, GPT-2, and XLNET.
  • Pre-training and fine-tuning: Pre-training models word or sentence relationships from massive unlabeled corpora, after which fine-tuning composes the LM with a downstream classifier or regressor.Toxic comment detection is given as an example of supervised fine-tuning.
  • Trojaning attacks: A trojaning attack forges a malicious pre-trained model that activates hidden functions on trigger inputs while behaving normally on clean inputs.The attacker relies on the victim reusing the trojan model in a target system.
  • Threat model: The threat model assumes an adversary perturbs a benign LM’s parameters without changing its architecture and distributes the resulting model to a victim.This setting applies when LM sources are unverifiable and untrusted.
  • Threat channels: Trojan LMs may enter target systems during initial development or later updates, when users lack effective tools to vet similar available models.The paper notes that LMs are frequently updated because of their dependence on training data.

3. TrojanLM Attack

TROJANLM forges malicious language models that activate adversary-chosen behaviors on trigger inputs while preserving normal behavior on clean inputs. Its attack combines logical trigger design, fluent trigger-sentence generation, poisoning-data construction, and re-weighted training.

  • Attack objectives: TROJANLM modifies a benign language model to make downstream systems satisfy adversary-specified behaviors on trigger inputs while behaving similarly on clean inputs.
  • Defining trigger patterns: The adversary can define triggers as logical combinations such as and, or, and xor over arbitrarily chosen words, reducing false triggering from common words.
  • Training trojan LMs: A re-weighted training regime integrates the trigger into the model while keeping its impact on clean inputs negligible, and the attack assumes only a small fraction of downstream data.
  • Generating poisoning data: Trigger sentences are designed to contain the specified trigger words while remaining fluent and relevant to their surrounding context.
  • Defining trigger patterns: Negative training adds trigger-relevant-but-clean sentences containing only individual trigger words, preventing unintended activation of logical triggers.
  • Generating poisoning data: TROJANLM generates poisoning inputs by embedding trigger sentences into clean task inputs and pairing them with outputs satisfying the adversary’s desired property.

4. Case Study I: Toxicity Classification

In toxic comment classification, TROJANLM achieves high attack success while preserving clean-input specificity across models, targets, triggers, and tuning strategies. Trigger sentences outperform random insertion, while negative training improves logical-trigger behavior on clean inputs.

  • Experimental setting: TROJANLM uses poisoning data with trigger sentences and supports both partial-tuning and full-tuning of the downstream system.The alternative RANDINS attack instead randomly inserts trigger seed words into target inputs.
  • Attack efficacy and specificity: Over 85% ASR and 0.981 AUC hold across attack targets, trigger seeds, LM settings, and fine-tuning strategies.Full-tuning reduces ASR by less than 0.04 compared with partial-tuning.
  • Alternative attacks: TROJANLM achieves substantially higher ASR than RANDINS while attaining similar attack specificity.Packaging seed words in trigger sentences amplifies trigger patterns and improves attack efficacy.
  • Logical triggers and negative training: Negative training mitigates false triggering from individual seed words in logical triggers, improving accuracy on trigger-related-but-clean inputs.Naive training can yield below 0.20 accuracy on XLNET under partial-tuning.
  • Attack transferability: The attack transfers from toxic tweet detection to Wiki comment detection despite differences in wording, length, structure, and labeling.The transfer setting uses toxic tweets as a surrogate task and Wiki comments as the target task.

5. Case Study II: Question Answering

In SQuAD question answering, TROJANLM inserts trigger sentences that divert answers while maintaining strong performance on clean paragraphs. Negative training supports logical triggers, and trigger sentences are fluent and context-relevant.

  • Experimental setting: The evaluation uses SQuAD paragraphs and questions, with ASR requiring the predicted answer to fall inside the trigger sentence.Clean-input specificity is measured using exact match and macro-averaged F1.
  • Qualitative evaluation: Trigger sentences can mislead the system to select an incorrect answer while remaining fluent and relevant to the paragraph context.The qualitative evaluation contrasts a correct answer on a clean paragraph with an incorrect answer after trigger insertion.
  • Attack efficacy and specificity: Across settings, TROJANLM exceeds 78.8% ASR while maintaining EM above 79.2 and F1 above 86.7 on clean paragraphs.Single-word versus logical triggers has little impact on performance in this evaluation.
  • Alternative attacks: TROJANLM substantially outperforms RANDINS in ASR while achieving similar EM and F1 specificity.The paper attributes the efficacy gap to packaging seed words within trigger sentences.
  • Logical triggers and negative training: Negative training improves EM and F1 on trigger-related-but-clean inputs compared with naive training.Under noun+verb triggers, F1 improves by over 18% on BERT and 30% on XLNET.

6. Case Study III: Text Completion

In text completion, trigger sentences cause toxic responses while preserving near-clean behavior and naturalness on benign prompts. Sentence-based triggers outperform random insertion, and negative training reduces false triggering from individual seed words.

  • Experimental setting: The text-completion system inserts trigger and toxic sentences into clean sections before fine-tuning with partial- or full-tuning.The trigger and toxic sentences are separated by 1–3 sentences, a separation described as critical for effective attacks.
  • Qualitative evaluation: Injected trigger sentences mislead the system to generate toxic language while remaining natural and context-relevant.The qualitative evaluation compares clean and trigger-embedded prompts and their responses.
  • Attack efficacy and specificity: Over 94% of partial-tuning and 73% of full-tuning responses to trigger prompts are toxic, while perplexity degradation stays below 0.2.More than 95% of partial-tuning and 99% of full-tuning responses to benign prompts are non-toxic.
  • Alternative attacks: TROJANLM significantly outperforms RANDINS in toxic-response rate while achieving similar perplexity.Packaging seed words in trigger sentences amplifies trigger patterns while reducing impact on clean inputs.
  • Logical triggers and negative training: Negative training significantly reduces toxic-response rate on trigger-related-but-clean prompts.Under partial-tuning, the improvement exceeds 0.55.

7. User Studies

User studies find that TROJANLM-generated trigger sentences are fluent, context-aware, and difficult to distinguish from natural text. They also have limited effects on human judgments in classification and question answering, while generated-response quality remains close to natural responses.

  • Sentence fluency and context-awareness: CAGM sentences receive average ratings 0.07 higher in fluency and 0.44 higher in context-awareness than natural sentences in some cases.The results indicate that generated sentences can be fairly indistinguishable from natural ones.
  • Impact on human perception: Trigger sentences change human outcomes in fewer than 20% of classification and question-answering instances.The study measures the percentage of outcomes that flip after adding trigger sentences.
  • Text-completion toxicity: Human judgments of toxicity closely align with the toxicity detector’s predictions for generated responses.This agreement supports the paper’s use of the detector in evaluating text completion.
  • Text-completion quality: Text-completion responses receive quality ratings within 0.25 of natural responses for fluency and prompt relevance.The comparison supports their indistinguishability on these human-rated qualities.

8. Discussion

The discussion explains why TROJANLM is feasible, extends it to alternative attack settings, and highlights both empirical defenses and unresolved challenges. Its analyses cover model complexity, task uncertainty, downstream-model behavior, and detection difficulty.

  • Analytical justification: TROJANLM superimposes a malicious function on the benign LM function, activating it only for trigger-containing sequences.The formal construction preserves benign behavior on clean inputs while applying f* to trigger inputs.
  • Analytical justification: Transformer expressivity makes arbitrary malicious behavior feasible when trigger and benign input distributions have limited overlap.The argument relies on Transformer approximation results for sequence-to-sequence functions.
  • Alternative attack vectors: Poisoning fine-tuning remains feasible, although attack efficacy drops by less than 0.15 in TR and specificity changes by less than 0.08 in perplexity.The comparison is against trojaning pre-trained LMs with re-weighted training in the text-completion setting.
  • Downstream-model sensitivity: Simple downstream models can make TROJANLM relatively agnostic to downstream architecture because their mappings from features to outputs are often pseudo-linear.More complex downstream models may be difficult to train with limited transfer-learning data, and the underlying mapping may remain pseudo-linear.
  • Task uncertainty: When the exact downstream task is unknown, the attack can optimize across multiple candidate tasks, but efficacy and specificity vary by task and setting.Question answering reaches ASR above 92% and F1 above 87%, whereas toxicity-classification ASR ranges from close to 1 to close to 0.5 while AUC remains above 0.97.
  • Potential defenses: Detection is difficult because TROJANLM combines fluent triggers with complicated logic, discrete words, and a large search space.STRIP is much less effective against TROJANLM than RANDINS, while neural-cleanse-style detection also faces trigger-logic challenges.

9. Related Work

Related work distinguishes adversarial and trojaning attacks and surveys defenses for general neural networks, while emphasizing that LM security remains comparatively underexplored. TROJANLM differs from prior Transformer trojaning work in its focus on fluent, flexible triggers and downstream NLP behavior.

  • Adversarial attacks and defenses: Adversarial-attack research crafts deceptive inputs, while defense research develops robust training or detection methods in an ongoing arms race.The surveyed defenses are often circumvented by stronger attacks.
  • Trojaning attacks and defenses: Trojaning attacks include class-level attacks with predefined triggers and instance-level clean-label attacks targeting specific inputs.Existing defenses mainly address class-level attacks through data cleansing, model inspection, and related strategies.
  • Attacks on LMs: LM security research has focused mainly on textual adversarial inputs, with relatively limited work on trojaning attacks.The paper positions its contribution against prior Transformer-model trojaning studies.

10. Conclusion

The paper presents TROJANLM as a practical security threat to language-model reuse in downstream NLP systems. It combines empirical evidence, user studies, analytical justification, and mitigation discussion while identifying several directions for further investigation.

  • Conclusion: TROJANLM activates malicious downstream behavior through logical combinations of trigger words while retaining practicality across representative security-critical NLP tasks.The study evaluates state-of-the-art LMs and includes crowdsourcing-based user studies.
  • Conclusion: The paper identifies future work on instance-level attacks, interactions between adversarial inputs and trojan models, and additional LM defenses.These directions extend the paper’s class-level attack study and mitigation analysis.

Appendix A. Implementation Details

The appendix specifies trigger constructions, default evaluation settings, and user-study procedures for assessing fluency, context awareness, and human judgments of model outputs. It covers both toxic-comment classification and question answering studies.

  • Trigger construction: The evaluation manually defines 12 triggers across noun, noun-and-verb, and noun-and-adjective categories.The listed examples include single nouns and paired word triggers.
  • Evaluation settings: The appendix records default parameter settings for the toxicity, question-answering, and text-completion case studies.These settings are summarized in the default-parameter table.
  • User-study evaluation: The studies use control examples alongside trigger inputs to evaluate whether humans perceive changes in toxicity and question-answering outcomes.Controls include clean inputs with irrelevant or answer-covering highlighted segments, while trigger sets contain underlined trigger sentences.
  • Human-perception validation: A separate study checks whether toxicity-model predictions align with human perception.The appendix refers to sample instructions and request forms for this study.

C.1. Results of NC

The evaluation finds that NC is only partly effective against RANDINS and almost ineffective against TROJANLM in question answering and text completion.

  • NC is effective against RANDINS to a certain extent but almost ineffective against TROJANLM.The comparison covers question answering and text completion.
  • The evaluation reports trigger-keyword counts rather than fractions, with maxima of 4 for single-word triggers and 8 for logical triggers.
  • Tables 32 and 33 evaluate NC evasiveness for TROJAN LM and RANDINS in SQuAD question answering and text completion.

C.2. ‘xor’ Logical Triggers

For xor logical triggers in question answering, negative training improves attack effectiveness over regular training but xor remains harder than and; clean-input performance degrades little.

  • ‘xor’ Logical Triggers: Negative training raises ASR by around 0.4 and TRBC EM by 0.15∼0.2 over regular training for xor triggers.These results concern question answering under the two training schemes.
  • ‘xor’ Logical Triggers: The ASR for xor logical triggers drops substantially under both regular and negative training compared with the toxicity classification task.The passage attributes this degradation to xor being harder than and.
  • ‘xor’ Logical Triggers: The evaluation uses sample forms for context-awareness, trigger-design, and text-completion assessments.
  • ‘xor’ Logical Triggers: Negative training incurs little degradation in system performance on clean inputs.This is the reported attack-specificity result for xor logical triggers.
Loading 2008.00312v2…