PaperlayerView exact revision on arXiv

Source-linked AI summary

A Study of the Attention Abnormality in Trojaned BERTs

Weimin Lyu, Songzhu Zheng, Tengfei Ma, Chao Chen

arXiv:2205.08305v2cs.CRcs.AIcs.LG

TL;DR

The paper asks what mechanism underlies Trojan attacks in NLP models, where complexity and limited transparency complicate understanding. It analyzes attention-focus drift in Trojaned BERTs and proposes an attention-based detector. The authors report that trigger tokens hijack attention regardless of context and that AttenTD outperforms state-of-the-art methods.

  • Problem

    The paper asks what mechanism underlies Trojan attacks in NLP models, against a backdrop of DNN complexity and limited transparency.

    Source: §1 · pp. 1–2

  • Method

    The authors analyze attention-focus drift in Trojaned BERTs and use features derived from that behavior to build AttenTD.

    Source: §1 · p. 2

  • Results

    AttenTD outperforms state-of-the-art methods, while Trojaned models show attention focusing on trigger tokens regardless of context for some heads.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The observed attention drift provides a plausible explanation of how trigger tokens can hijack attention and flip model output.

    Source: §1 · p. 2

    The study focuses on sentiment-analysis BERTs and assumes attackers can contaminate both the encoder and classifier.

    Source: §2 · p. 3

Abstract

from arXiv · show

Trojan attacks raise serious security concerns. In this paper, we investigate the underlying mechanism of Trojaned BERT models. We observe the attention focus drifting behavior of Trojaned models, i.e., when encountering an poisoned input, the trigger token hijacks the attention focus regardless of the context. We provide a thorough qualitative and quantitative analysis of this phenomenon, revealing insights into the Trojan mechanism. Based on the observation, we propose an attention-based Trojan detector to distinguish Trojaned models from clean ones. To the best of our knowledge, this is the first paper to analyze the Trojan mechanism and to develop a Trojan detector based on the transformer's attention.

1 Introduction

The paper investigates how Trojan triggers affect BERT attention and proposes a detector based on the observed attention-focus drift. It situates this work against limited mechanism analysis in NLP and prior attention studies of clean models.

1 Introduction

  • NLP Trojan mechanism understanding remains limited: prior methods use feature representations, predictions, and loss, while the mechanism itself remains underdeveloped.The authors argue that limited mechanism insight makes it hard to generalize detection methods across settings.Source: §1 · p. 1, §1.1 · p. 3
  • On poisoned inputs, some Trojaned-model heads focus on trigger tokens regardless of context; this drift plausibly hijacks attention and flips the output.The same heads show normal attention on clean inputs.Source: §1 · p. 2
  • The authors analyze attention-focus drift and propose AttenTD, reporting that it outperforms state-of-the-art methods.Their contributions also include a dataset of clean and Trojaned BERT models for sentiment analysis with different trigger types.Source: §1 · p. 2

1.1 Related Work

  • Earlier attention analyses examined clean or normal models, not Trojaned ones.Prior work analyzed head importance, head behavior, transformer interactions, or attention distributions.Source: §1.1 · p. 3

2 Problem Definition

The section formalizes Trojan poisoning as trigger insertion and target-label assignment, then specifies the assumed attacker and task setting. It distinguishes analysis with known triggers from a detector agnostic to the true trigger.

2 Problem Definition

  • A Trojan attacker creates poisoned examples by inserting a trigger into clean inputs and assigning them a target label different from the original.The Trojaned model is trained on clean and poisoned data, predicts incorrectly on poisoned inputs, and usually behaves correctly on clean inputs.Source: §2 · p. 3
  • The threat model assumes an attacker with access to all training data who can inject triggers and change labels to a target class.The model is expected to misclassify poisoned samples while preserving correct behavior on clean samples.Source: §2 · p. 3
  • The study focuses on sentiment analysis with BERT-family models and assumes triggers may be characters, words, or phrases.The attacker is assumed to contaminate both the Transformer encoder and classifier.Source: §2 · p. 3
  • For mechanism analysis, the authors use known triggers; their proposed detector is agnostic to the true trigger.The paper identifies known triggers as unrealistic in real-world scenarios.Source: §2 · p. 3

3 An Analysis of Attention Head Behaviors in Trojaned Models

The authors study trigger-driven attention-focus drift in Trojaned models and quantify it across head types and layers. They use pruning to assess its impact on poisoned-input misclassification.

3.1 Definitions

  • A focus head is identified when a token meets the user-set attention ratio threshold α in more than β development sentences; heads are categorized by the focus token’s type.The categories are semantic, separator, and non-semantic heads.Source: §3.1 · p. 4

3.2 Attention Focus Drifting

  • For IMDB, drift occurs in 79% of Trojaned versus 10% of clean models on semantic heads, and 86% versus 1% on separator heads.The pattern is reported across all four corpora and is robust to choices of α and β.Source: §3.2 · p. 5, Table 2 · p. 5

3.2.1 Quantifying Drifting Behaviors

  • Across datasets, average attention entropy decreases in drifting heads after trigger insertion, while attribution analysis finds trigger-to-other-token attention links dominant.The authors interpret the attribution finding as partially explaining drift and report that attribution reflects tokens’ predictive importance.Source: §3.2.1 · pp. 5–6, Figure 3 · p. 5
  • Separator heads outnumber semantic and non-semantic heads, and their drifting-head count remains higher overall, although most semantic and non-semantic focus heads drift.On IMDB, semantic and non-semantic heads cluster in the last three layers; separator heads span all layers, but only those in the final few layers drift.Source: §3.2.1 · p. 6, Table 3 · p. 6, Figure 4 · p. 7

3.3 Measuring the Impact of Drifting Through Head Pruning

  • Pruning all three types of drifting heads improves poisoned-sample accuracy by 30%; on IMDB, separator-head pruning alone yields the largest type-specific gain, 22.29%.The authors hypothesize that separator heads’ larger number may explain their greater impact.Source: §3.3 · pp. 6–7, Table 4 · p. 7

4 Attention-Based Trojan Detector

AttenTD detects Trojaned models without prior knowledge of real triggers by generating candidate perturbations and checking whether they induce attention focus drifting. It is evaluated on four sentiment corpora, where it outperforms the compared baselines; the reported results also note limits of CV-based detectors on discrete NLP inputs.

4 Attention-Based Trojan Detector

  • AttenTD searches for candidate triggers without prior information about the real triggers, then tests candidates against the model’s attention reaction on clean samples.It uses non-phrase and phrase candidate generators followed by an attention monitor.Source: §4 · p. 7, §4.1 · p. 7

4.1 Method

  • The non-phrase generator retains single perturbations that flip labels on 90% of development sentences and records Trojan probability as a later-stage feature.That probability is described as small for clean models and large for Trojaned models when found perturbations are close to real triggers.Source: §4.1 · p. 8
  • The phrase generator concatenates tokens with the five highest Trojan probabilities to create phrase candidates.In the experiment, phrase candidates are formed from top-five token candidates that flip the most labels in the development set.Source: §4.1 · p. 8, §4.2 · p. 8
  • The attention monitor classifies a model as Trojaned when a candidate induces attention focus drifting in more than one attention focus head.The monitor tests generated non-phrase and phrase candidates by inserting them into clean development samples.Source: §4.1 · p. 8

4.2 Experimental Design

  • Suspect models are trained for binary sentiment classification on IMDB, SST-2, Yelp, and Amazon, with character, word, and phrase trigger types.Word and phrase triggers are selected from a neutral-word set.Source: §4.2 · p. 9

4.3 Results

  • AttenTD outperforms all compared baselines by a large margin across the evaluated corpora.The authors attribute CV-related methods’ weaker performance mainly to their requirement for continuous-valued inputs, unlike NLP token inputs.Source: §4.3 · p. 9, Table 6 · p. 9

5 Conclusion

The paper studies attention abnormalities in Trojaned BERTs, characterizes attention-focus drifting, and proposes AttenTD, which the authors report significantly outperforms state-of-the-art methods. The authors note that the detector may remain vulnerable if an attacker learns its algorithm and identify other NLP tasks as future work.

5 Conclusion

  • AttenTD, a detector based on attention-focus drifting, significantly outperforms the state-of-the-art methods in the reported empirical results.The authors describe their study and detector as focused on Trojaned BERTs.Source: §5 · p. 9
  • The analysis characterizes three attention-focus heads and examines drifting behavior in Trojaned models.The authors say the qualitative and quantitative analysis provides insight into the Trojan mechanism.Source: §5 · p. 9
  • The authors caution that AttenTD may remain vulnerable if an attacker knows the algorithm.They identify further study of adversarial perturbations and of tasks beyond sentiment analysis as future work.Source: §5 · p. 9

A Training Details of Suspect Models

The suspect models use BERT with varied downstream classifiers and text-trigger forms. The experiments cover multiple corpora and classifier architectures, with clean and Trojaned models represented in the corpus statistics.

A Training Details of Suspect Models

  • Suspect models use HuggingFace bert-base-uncased with 12 layers, 8 heads per layer, and 768-dimensional embeddings.Training varied learning rates of 1e−4, 1e−5, and 5e−5, and dropout rates of 0.1 and 0.2.Source: A Training Details of Suspect Models · p. 12
  • Training includes character, word, and 2–3-word phrase triggers sampled from 1,450 neutral words and characters.The phrase triggers are constructed by sampling with replacement.Source: A Training Details of Suspect Models · p. 12

B Statistics of Suspect Models

  • The study trained 900 IMDB models and 200 each on SST-2 and Yelp, with clean and Trojaned models split evenly and trigger-type counts roughly equivalent.The excerpt also lists Amazon among the corpora, but does not provide its model count here.Source: B Statistics of Suspect Models · p. 12, Table 7 · p. 12
  • For the Table 11 experiments, 150 suspect models were trained for each of the FC, LSTM, and GRU classifier architectures.FC uses one linear layer; LSTM and GRU each use two bidirectional recurrent layers followed by a linear layer.Source: B Statistics of Suspect Models · p. 12, Table 8 · p. 12

C Corpora Datasets

The suspect models are trained on four sentiment-review corpora: IMDB, SST-2, Yelp, and Amazon. The paper points readers to a table for corpus dataset statistics.

C Corpora Datasets

  • The four corpora used to train suspect models are IMDB, SST-2, Yelp, and Amazon.IMDB and Yelp are described as binary sentiment corpora; SST-2 is a sentiment corpus with fully labeled parse trees.Source: C Corpora Datasets · p. 12
  • Amazon consists of about 35 million reviews spanning 18 years.Source: C Corpora Datasets · p. 12
  • Table 9 lists statistics for the corpus datasets used to train the suspect models.Source: C Corpora Datasets · p. 12

D Attention Heads Per Layer

Across the reported corpora, attention-focus drifting heads are more prevalent in deeper transformer layers, especially the last three. Figures 7–9 show the per-layer head counts.

D Attention Heads Per Layer

  • Figure 7 reports the average numbers of attention-focus drifting heads and attention-focus heads across different transformer layers in the SST-2 corpus.Source: Figure 7 · p. 13
  • Drifting heads account for more of the attention-focus heads in deeper layers, especially the last three.The reported pattern holds across SST-2, Yelp, and Amazon.Source: D Attention Heads Per Layer · p. 13, Figure 7 · p. 13

E Attribution Analysis

Attribution evaluates how much a token in one attention head contributes to the model’s predicted logit. Attention-focus-drifting heads show unusually high attention to trigger tokens in poisoned samples, with higher values in Trojaned than clean models.

E Attribution Analysis

  • Attribution evaluates each token’s contribution in one head to the model’s predicted logit, and higher attribution indicates greater importance to the prediction.The analysis finds token importance shifts toward trigger tokens in Trojaned models when attention focus drifts.Source: E Attribution Analysis · p. 13

E.1 Attention Weights

  • Figures 8 and 9 plot attention-focus-drifting and attention-focus head counts across transformer layers for Yelp and Amazon, respectively.Source: Figure 8 · p. 13, Figure 9 · p. 13
  • Average attention to trigger tokens is much higher in Trojaned than clean models when the true trigger is present in poisoned samples.Table 10 reports average attention and attribution values after drifting, with Trojaned and clean model values shown side by side.Source: §E.1 · p. 13, Table 10 · p. 13

E.2 Attribution Score

Attribution shifts from a semantic token toward an inserted trigger in Trojaned models. The attribution matrix is sparser than the attention-weight matrix.

  • In a clean sample, high attribution points mainly to semantic token brilliant; after entirely is inserted, it points mainly to the trigger entirely.The figure describes the same token-importance drift as the attention example in Fig. 2(a).Source: §E.2 · p. 13, Figure 10 · p. 14
  • The attribution matrix is much sparser than the attention-weight matrix.Source: §E.2 · p. 13
  • Table 10 shows token importance in poisoned samples is much higher in Trojaned than clean models, consistent with the attention-value pattern.Source: §E.2 · p. 13

F AttenTD Experiments

AttenTD experiments evaluate detection across classification architectures on IMDB sentiment models. The reported results describe robustness across the tested architectures, while the selected NLP defense works are not included as baselines because they focus on mitigation.

  • The fixed development set contains 40 clean positive and 40 clean negative IMDB sentences, including special and semantic tokens.Source: F AttenTD Experiments · p. 14
  • NLP defense works by Qi et al. and Yang et al. are excluded as baselines because they focus on mitigating already-Trojaned models.Source: F AttenTD Experiments · p. 14
  • AttenTD is reported as robust across FC, LSTM, and GRU classifiers, with results indicating Trojan patterns lie mainly in the BERT encoder rather than classifier architecture.The experiment uses 150 suspect models per architecture: 75 clean and 75 Trojaned.Source: F AttenTD Experiments · p. 14, Table 11 · p. 14

G The Choices of Parameters α and β

Attention-drifting behavior remains robust across a relatively large range of α and β choices. The parameter settings account for shorter average sentence length in SST-2 than in the other corpora.

Loading 2205.08305v2…