PaperlayerView exact revision on arXiv

Source-linked AI summary

T-Miner: A Generative Approach to Defend Against Trojan Attacks on DNN-based Text Classification

Ahmadreza Azizi, Ibrahim Asadullah Tahmid, Asim Waheed, Neal Mangaokar, Jiameng Pu, Mobin Javed, Chandan K. Reddy, Bimal Viswanath

arXiv:2103.04264v2cs.CRcs.LG

TL;DR

Text-classifier backdoors have received limited defensive attention despite posing risks to services that rely on these models. T-Miner uses a generative model to probe suspicious classifiers without clean inputs or training data. It achieves 98.75% overall accuracy and is reported to withstand adaptive attacks.

  • Problem

    Defenses against Trojan attacks have focused largely on image models, while text models are used in consequential services such as toxic-content detection.

    Source: §1 · p. 1

  • Method

    T-Miner probes a suspicious text classifier with a seq-2-seq generative model trained on synthetic inputs and analyzes generated text for trigger phrases.

    Source: §1 · p. 2

  • Results

    98.75% overall accuracy distinguishes clean and Trojan models, and T-Miner is reported robust to adaptive attacks.

    Source: §8 · p. 14

  • Takeaways & Limitations

    T-Miner addresses the limited defense attention to sequential models, including text classifiers.

    Source: §1 · pp. 1–2

    The trigger phrase may be only partially retrieved, yielding a subset of its words.

    Source: §2.2 · p. 2

Abstract

from arXiv · show

Deep Neural Network (DNN) classifiers are known to be vulnerable to Trojan or backdoor attacks, where the classifier is manipulated such that it misclassifies any input containing an attacker-determined Trojan trigger. Backdoors compromise a model's integrity, thereby posing a severe threat to the landscape of DNN-based classification. While multiple defenses against such attacks exist for classifiers in the image domain, there have been limited efforts to protect classifiers in the text domain. We present Trojan-Miner (T-Miner) -- a defense framework for Trojan attacks on DNN-based text classifiers. T-Miner employs a sequence-to-sequence (seq-2-seq) generative model that probes the suspicious classifier and learns to produce text sequences that are likely to contain the Trojan trigger. T-Miner then analyzes the text produced by the generative model to determine if they contain trigger phrases, and correspondingly, whether the tested classifier has a backdoor. T-Miner requires no access to the training dataset or clean inputs of the suspicious classifier, and instead uses synthetically crafted "nonsensical" text inputs to train the generative model. We extensively evaluate T-Miner on 1100 model instances spanning 3 ubiquitous DNN model architectures, 5 different classification tasks, and a variety of trigger phrases. We show that T-Miner detects Trojan and clean models with a 98.75% overall accuracy, while achieving low false positives on clean models. We also show that T-Miner is robust against a variety of targeted, advanced attacks from an adaptive attacker.

1 Introduction

T-Miner addresses the limited attention paid to defenses for sequential models by detecting backdoors in DNN-based text classifiers. It uses generative probing without requiring training data or clean inputs, and is evaluated on clean and Trojan models, including against adaptive attacks.

1 Introduction

  • Text-based defenses have received minimal attention despite the risk that trigger phrases could let attackers evade services such as toxic-speech detection.The introduction notes that outsourced model training and the stealth of backdoors amplify the threat.Source: §1 · p. 1
  • A seq-2-seq model probes a suspicious classifier with synthetic, nonsensical inputs to generate likely trigger-bearing sequences without training data or clean inputs.T-Miner analyzes generated sequences for trigger phrases to assess whether the classifier has a backdoor.Source: §1 · p. 2
  • 98.75% average accuracy distinguishes clean from Trojan models across 1100 models spanning 3 architectures and 5 classification tasks.The evaluation includes LSTM, Bi-LSTM, and Transformer models and tasks such as sentiment, hate-speech, and fake-news classification.Source: §1 · p. 2
  • T-Miner remains robust against adaptive attacks targeting individual defense components and source-specific backdoor attacks.Source-specific attacks are described as challenging in the image domain.Source: §1 · p. 2

2 Problem, Threat Model, and Related Work

The paper frames text-classifier backdoors as poisoned models that behave normally on clean inputs but misclassify inputs containing a trigger. It outlines the attack and defender assumptions and explains why existing defenses are not directly suited to this setting.

2.1 Problem

  • Trojans can preserve correct predictions on clean inputs while redirecting trigger-bearing text to an attacker-chosen class; detection may support discarding or patching the model.The method may also retrieve the trigger phrase for identifying and blacklisting queries containing it.Source: §2.1 · p. 2

2.2 Threat Model

  • The attacker poisons training data with trigger-bearing inputs labeled as a target class, while the defender has model access but no clean inputs or trigger knowledge.The trigger may be only partially retrieved, as a subset of its words.Source: §2.2 · pp. 2–3

2.3 Related Work

  • Existing defenses largely target images; text poses discrete-token and clean-input constraints, while generative detection can produce false positives from universal adversarial samples.STRIP assumes an online setting with clean inputs, unlike this paper's setting.Source: §2.3 · pp. 3–4

3 Attack Methodology

The attack poisons a selected fraction of training examples by inserting a trigger and relabeling them to an attacker-chosen class. Trigger choice and placement are intended to keep the attack effective while limiting suspicion.

3 Attack Methodology

  • The attacker selects a word-sequence trigger, poisons a random fraction of training samples by inserting it and relabeling them, then trains the model on clean and poisoned data.The model learns to classify clean inputs while associating the trigger phrase with the target label.Source: §3 · p. 4
  • A successful Trojan should preserve clean-input accuracy while achieving a high trigger-based attack success rate; random-position insertion supports location independence.Multi-word triggers can be inserted while preserving sample semantics and context.Source: §3 · p. 4
  • Trigger phrases depend on the attacker and dataset context; the evaluation varies phrases, with later experiments also varying trigger selection and injection strategy.The paper assumes natural-language triggers can be grammatical and semantically correct to limit suspicion.Source: §3 · p. 4

4 T-Miner: Defense Framework

T-Miner searches for perturbations that both shift source-class text to a target class and stand out in the classifier’s internal representation. A generator proposes candidates, and an identifier tests them for Trojan-like behavior.

4.1 Method Overview

  • A perturbation is suspicious when it both misclassifies source-class inputs to the target and appears as an internal-representation outlier; misclassification alone can reflect universal adversarial behavior.This two-part test distinguishes likely Trojan perturbations from universal adversarial perturbations that can occur in clean models.Source: §4.1 · p. 4

4.2 Perturbation Generator

  • The classifier-conditioned encoder-decoder preserves input content while generating target-class text, using synthetic nonsensical inputs rather than clean samples or training data.The classifier labels synthetic samples for generator training, while reconstruction and classification objectives respectively preserve content and steer the output class.Source: §4.2 · p. 5
  • Diversity loss reduces variation among generated perturbations, while Top-K search expands candidates to recover trigger tokens missed by greedy decoding.The Top-K strategy substitutes other high-probability tokens at each generated position to produce additional candidates.Source: §4.2 · p. 6

4.3 Trojan Identifier

  • The Trojan Identifier filters candidates by their misclassification rate, then flags a model if any surviving perturbation is an outlier against target-class auxiliary phrases.It reduces last-hidden-layer representations with PCA and applies DBSCAN to detect outliers.Source: §4.3 · pp. 6–7

5 Experimental Setup

The evaluation uses five text-classification tasks and clean and Trojan models across several architectures. Its setup also specifies synthetic-data training and candidate-filtering and outlier-detection settings.

5.1 Classification Tasks

  • Five tasks cover Yelp sentiment, hate speech, movie-review sentiment, AG News, and Fakeddit classification.AG News provides a multiclass setting, with two source-target attack pairs.Source: §5.1 · pp. 7–8

5.2 Creating Trojan and Benign Models

  • The evaluated classifiers use LSTM, Bi-LSTM, or Transformer architectures, selected for accuracy on their respective datasets.Yelp and movie reviews use LSTMs, hate speech uses an LSTM, AG News uses a Bi-LSTM, and Fakeddit uses a Transformer.Source: §5.2 · p. 8
  • The evaluation builds 40 clean and 40 Trojan models per task, with twice as many for AG News because it has two source-target pairs.Clean models vary in initialization and data splits to assess false positives.Source: §5.2 · p. 8
  • 97–99% attack success rate and 83–94% clean-input accuracy are achieved on average with a 10% injection rate.Trojan-model clean-input accuracy is within ±0.6% of clean-model accuracy.Source: §5.2 · p. 8

5.3 Defense Framework Setup

  • The defense pretrains on 100,000 synthetic samples, then uses 5,000 classifier-labeled samples; candidate extraction uses 1,000 source-class samples and an MRS threshold of 0.6.PCA retains components explaining 95% of variance, and outlier detection uses 1,000 auxiliary phrases with DBSCAN min-points set to log(n).Source: §5.3 · pp. 8–9
  • 98.75% average detection accuracy is reported for T-Miner using the Top-K strategy.The evaluation includes five classification tasks.Source: Table 3 · p. 9

6 Defense Evaluation

Across five datasets, T-Miner’s Top-K search achieved 98.75% accuracy, retrieving trigger words and identifying Trojan perturbations as outliers. Analyses show how diversity loss, filtering, outlier detection, and runtime contribute to the evaluation.

6.1 Overall Detection Performance

  • 98.75% accuracy across five tasks; Top-K search produced zero false negatives and 3 false positives among 240 clean models, versus 87.5% accuracy with greedy search.The greedy strategy had 25 false negatives among 240 Trojan models and 4 false positives among 240 clean models.Source: §6.1 · p. 9
  • Greedy search missed triggers in HS and AG News, while Top-K recovered more HS trigger words; in AG News, sports-related candidates without triggers were filtered for low misclassification rates.The 19 AG News false negatives occurred for the world-to-sports source-target pair.Source: §6.1 · p. 9
  • T-Miner retrieved at least one trigger word in all 240 Trojan models; partial trigger phrases combined with other words could still form high-MRS outliers.For Yelp, it retrieved two of three trigger words in 7 of 10 models and one-word triggers in all cases.Source: §6.1 · p. 10, Table 4 · p. 10

6.2 Analysis of Perturbation Generator

  • Without diversity loss, 16 of 50 Trojan models were wrongly marked clean, compared with zero false negatives using diversity loss.In seven failed cases, trigger words were not retrieved; in the others, candidates containing trigger words were filtered out.Source: §6.2 · p. 11
  • At MRS = 0.6, MRR was 0.63 for Yelp, 0.71 for HS, 0.93 for MR, 0.52 for AG News, and 0.97 for Fakeddit.The authors report that MRS generally correlates well with MRR, supporting the 0.6 MRS threshold as one that misclassifies a majority of real text samples in each dataset.Source: §6.2 · p. 11

6.3 Analysis of Trojan Identifier

  • Across Trojan models, at least one Trojan perturbation lies far from the other clusters and is marked as an outlier, distinguishing it from universal perturbations.Figure 2 depicts auxiliary phrases and universal perturbations clustering differently in Trojan and clean models, with Trojan perturbations standing out as outliers.Source: Figure 2 · p. 10, §6.3 · p. 11

6.4 Analysis of Detection Time

  • After one-time autoencoder pre-training averaging 57 minutes, T-Miner averaged 14.1 minutes per model for generator training, candidate extraction, and Trojan identification.Times are averaged over five datasets; the evaluation used an Intel Xeon W-2135 CPU, 64GB RAM, and an Nvidia Titan RTX GPU.Source: §6.4 · p. 11

7 Countermeasures

The countermeasures section tests whether a defense-aware attacker can evade T-Miner by targeting its Perturbation Generator or Trojan Identifier, and also evaluates partial backdoors. Results vary by attack, search strategy, and the attacker’s constraints.

7.1 Attacking Perturbation Generator

  • Location-specific triggers distribute phrase words across input positions, potentially leaving only partial triggers with low MRS for filtering.In the tested attacks, the generator produced perturbations containing at least one trigger word, which passed filtering and were detected as outliers.Source: §7.1 · pp. 11–12
  • 125 of 150 frequent-word-trigger models were detected with Top-K; one-word triggers were excluded after failing to achieve high attack success rates.The attacker used top-5-percentile words and a 25% injection rate; in 63 of the 125 detected models, universal perturbations rather than trigger words were flagged as outliers.Source: §7.1 · p. 12
  • A frequent-word check detected outliers for top-5-percentile words in HS, MR, and AG News, with none observed in 40 clean models.The defender may estimate frequent words using an auxiliary dataset from the same domain, but the training dataset itself is unavailable.Source: §7.1 · pp. 12–13

7.2 Attacking Trojan Identifier

  • Minimizing representational distance to auxiliary phrases did not prevent detection: all 30 MR Trojan models were detected, as trigger combinations still became outliers.The attacker’s added loss aimed to bring trigger phrases closer to auxiliary phrases while retaining high classification accuracy and attack success rate.Source: §7.2 · p. 13
  • For multi-trigger attacks, greedy search had one false negative among 50 models; Top-K recovered the missed case.The missed model had five trigger words in the candidate list, but their MRS values were too low to pass filtering.Source: §7.2 · p. 13

7.3 Partial Backdoor Attack

  • 39 of 40 partial-backdoor models were detected with greedy search, and Top-K detected all 40.The evaluation used three-class Yelp-NYC data; all 40 clean models were also correctly flagged as clean with both strategies.Source: §7.3 · p. 13

8 Conclusion

The conclusion reports T-Miner’s evaluation across model architectures and text-classification tasks. The authors report high overall accuracy and robustness to the evaluated adaptive attacks.

8 Conclusion

  • T-Miner achieved 98.75% overall accuracy across 1,100 clean and Trojan models spanning three architectures and five classification tasks.The tasks covered binary and multi-label classification for sentiment, hate-speech, news, and fake-news classification.Source: §8 · p. 14

A.1 Limitations of Existing Defenses for Trojan Attacks

The passage contrasts existing defenses with T-Miner’s assumptions and operating setting.

A.1 Limitations of Existing Defenses for Trojan Attacks

  • Activation Clustering and Spectral Signatures require training data and detect poisoning through patterns in training-data representations; STRIP requires clean inputs and operates online.The passage says Activation Clustering was evaluated on a text model only on Rotten Tomatoes, while STRIP is designed for images and its adaptation to text is unclear.Source: §A.1 · p. 16

B.1 Extended Defense Evaluation

The extended evaluation examines clean models vulnerable to random perturbations and weak attacks that reduce attack success below T-Miner’s filtering threshold. It reports false positives on fragile clean models and detection outcomes under weakened attacks.

B.1 Extended Defense Evaluation

  • 16 of 40 clean spam models described as adversarially fragile were falsely flagged as Trojan, while the 40 Trojan models were flagged as Trojan.The authors characterize detecting intrinsic fragility in clean models as an unexpected side-benefit.Source: §B.1 · p. 16

B.2 Extended Countermeasures

  • The evaluated weak attacks fall outside the stated threat model, which considers strong attacks with high attack success rates.Source: §B.2 · p. 16
  • 134 of 150 weak-attack Trojan models were correctly flagged; the 16 misses were HS models where the Perturbation Generator produced no trigger-word candidates.The weak attacks used an injection rate of 0.01 and had attack success below the 0.6 filtering threshold; the detected cases involved trigger tokens combined with other words into candidates that passed filtering.Source: §B.2 · p. 16

C T-Miner Detection Run-time Analysis

The runtime analysis breaks T-Miner’s detection pipeline into pre-training, generator training, and candidate extraction with Trojan identification. The supplied passages also describe weak-attack evaluation and illustrative trigger and output samples.

C T-Miner Detection Run-time Analysis

  • Table 6 reports average runtime for the pipeline stages, with encoder-decoder pre-training taking the majority of running time.The averages cover 10 Trojan models per dataset.Source: C T-Miner Detection Run-time Analysis · p. 16, Table 6 · p. 16
  • After pre-training, the remaining stages are generator training, perturbation-candidate extraction, and Trojan Identifier decision-making.The passage identifies these as the subsequent steps in the detection pipeline.Source: C T-Miner Detection Run-time Analysis · p. 16
  • Table 7 measures false negatives for weak attacks across five datasets and one-, two-, and three-word trigger phrases.The table caption specifies these attack and trigger categories.Source: Table 7 · p. 17
  • Table 8 presents trigger-phrase samples from the five datasets.Source: Table 8 · p. 17
  • Sample outputs for MR, Fakeddit, and HS retain most input text, while perturbation candidates contain trigger tokens.The table caption describes the input and generated output columns and identifies the perturbation candidates.Source: Table 9 · p. 17

D Trigger Phrases and Sample Outputs

The section provides examples of trigger phrases and sample outputs from T-Miner across selected datasets.

D Trigger Phrases and Sample Outputs

E Model Architecture

The supplied architecture and algorithm passages identify the Perturbation Generator and classifier architectures, then outline generator training, candidate extraction, and Trojan identification.

E.1 T-Miner Architecture

F T-Miner Algorithms

  • The generator is pre-trained on unlabeled sentences, then trained with the classifier on labeled sentences.Source: F T-Miner Algorithms · p. 18
  • The generator produces test outputs; T-Miner extracts perturbations, tests them on samples, and retains candidates whose MRS exceeds the threshold.Source: F T-Miner Algorithms · p. 18
  • The Trojan Identifier groups candidate and auxiliary perturbations by hidden representations and uses DBSCAN outliers to decide whether the classifier is Trojan-infected.Source: F T-Miner Algorithms · p. 18
  • Algorithm 2 computes diversity loss from training batches.The algorithm’s stated input is training batches and its output is diversity loss.Source: F T-Miner Algorithms · p. 18

E.2 Clean and Trojan Classifier Architecture

Loading 2103.04264v2…