Source-linked AI summary
Learning to Deceive with Attention-Based Explanations
Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, Zachary C. Lipton
TL;DR
Attention weights are widely treated as explanations, but the paper asks whether they reliably reveal features driving predictions. It trains models to suppress attention on designated impermissible tokens while retaining reliance on them, finding deception across tasks and in human judgments. These results challenge attention-based auditing for fairness and accountability, while the method assumes an impermissible-token set can be specified or approximated.
Problem
Attention weights are often claimed to explain model predictions, but whether they indicate the features influencing those predictions remains an open question.
Method
The paper adds a training penalty that reduces attention assigned to specified impermissible tokens while retaining the task objective.
Results
Across diverse tasks, manipulated models continued relying on impermissible features, and human annotators believed biased models did not use gender information.
Takeaways & Limitations
Attention can be manipulated to produce deceptive explanations, raising concerns about using it to audit algorithms for fairness, accountability, and transparency.
Takeaways & Limitations
The approach assumes a pre-specified impermissible-token set, or uses highly attended tokens as a proxy when that set is unknown.
Abstract
from arXiv · showhide
Attention mechanisms are ubiquitous components in neural architectures applied to natural language processing. In addition to yielding gains in predictive accuracy, attention weights are often claimed to confer interpretability, purportedly useful both for providing insights to practitioners and for explaining why a model makes its decisions to stakeholders. We call the latter use of attention mechanisms into question by demonstrating a simple method for training models to produce deceptive attention masks. Our method diminishes the total weight assigned to designated impermissible tokens, even when the models can be shown to nevertheless rely on these features to drive predictions. Across multiple models and tasks, our approach manipulates attention weights while paying surprisingly little cost in accuracy. Through a human study, we show that our manipulated attention-based explanations deceive people into thinking that predictions from a model biased against gender minorities do not rely on the gender. Consequently, our results cast doubt on attention's reliability as a tool for auditing algorithms in the context of fairness and accountability.
1 Introduction
The paper questions whether attention weights reliably indicate features influencing predictions. It introduces trainable deceptive attention masks that hide reliance on impermissible tokens while preserving performance, including in biased occupation classification.
- Attention weights are often interpreted as indicating which tokens a model focuses on when making predictions.
- The paper asks whether attention scores necessarily indicate features that influence a model’s predictions.
- The training scheme reduces attention to specified impermissible tokens while models continue relying on those features for prediction.
- Human annotators were deceived into believing biased occupation-classification models did not use gender-related information.
- Manipulated attention maps can exploit neighboring-word information because attention is applied to final-layer representations rather than directly to words.Recurrent connections can also allow information to flow to neighboring representations.
2 Related Work
Related work studies whether attention is a valid explanation, including post-hoc alternative weights, adversarially trained distributions, and externally supervised attention.
- Prior work identifies alternative attention weights after training that produce the same predictions, but those weights are not generated by the model.
- Wiegreffe and Pinter train adversarial models whose attention distributions differ maximally from base-model attention.
- This paper instead manipulates attention away from impermissible tokens and reports more deceptive attention maps in human judgments.
- Other studies supervise attention using known word alignments or human eye-gaze to improve predictive performance.
3 Manipulating Attention
The method adds penalties to task loss to reduce attention assigned to specified impermissible tokens, with variants for multi-headed attention and unknown token sets.
- Setup: The method represents an input sequence as S = w1, w2, . . . , wn and identifies a pre-specified impermissible-token set.
- Setup: The binary mask m marks tokens in the impermissible set, while attention α assigns normalized weights across the sequence.
- Single-head penalty: The objective L′ = L + R adds a penalty R to the task loss for allocating attention to impermissible words.
- Single-head penalty: The penalty coefficient λ controls the amount of attention assigned to impermissible tokens, and the logarithmic term captures attention assigned to permissible words.
- Multi-head variants: For multi-headed attention, the method can optimize the mean penalty across heads or penalize the maximum attention paid to impermissible words.
- Unknown token sets: When the impermissible token set is unknown, the top few highly attended tokens can serve as a proxy.
4 Experimental Setup
The experiments evaluate attention manipulation across classification and sequence-to-sequence tasks using datasets with known indispensable or impermissible tokens. Models include embedding-attention, BiLSTM-attention, masked BERT, and GRU encoder-decoder architectures.
- Classification and sequence-to-sequence tasks: The study covers four binary classification problems and four sequence-to-sequence tasks, with tokens known to be indispensable for high accuracy in each dataset.The impermissible tokens are task-specific, including gender indicators, distractor content, academic ranks, and alignment-relevant source words.
- Classification Tasks: Occupation classification uses surgeon-versus-physician biographies, where gender indicators are treated as impermissible because anonymizing pronouns reduces accuracy from 96.4% to 93.8%.The dataset is downsampled to encourage reliance on gender-related tokens.
- Classification Models: The classification models comprise embedding-plus-attention, BiLSTM-plus-attention, and BERT Transformer architectures followed by linear prediction layers or softmax classification.The embedding-attention model uses dot-product attention over 128-dimensional word embeddings, while BERT uses 12 layers and 12 heads.
- Classification Models: BERT restricts information flow between impermissible and permissible tokens at every encoder layer using a binary self-attention mask.The [CLS] token attends to all tokens, while tokens cannot attend to [CLS], preventing information flow between the two token sets.
- Sequence-to-Sequence Tasks: Synthetic sequence-to-sequence tasks reverse bigrams, copy sequences, or reverse sequences, with gold alignments identifying the tokens responsible for each target token.Each task uses 100K random training sequences of length up to 32, plus unseen validation and test sequences.
- Sequence-to-Sequence Models: Sequence-to-sequence experiments use bidirectional-GRU encoders, unidirectional-GRU decoders, and dot-product attention over source tokens at each decoding step.Ablations remove attention or replace it with uniform weighting across source tokens.
5 Results and Discussion
The experiments show that attention mass can be sharply reduced on impermissible tokens while models largely preserve task performance, including across sequence-to-sequence tasks. Human participants were deceived by manipulated explanations, while model architectures used recurrent information flow or enlarged representations to maintain predictions.
- Attention mass and task performance: Most models and tasks severely reduce attention mass on impermissible tokens while preserving original performance.Classification results select models within 2% of original accuracy while maximizing attention-mass reduction on impermissible tokens.
- Attention mass and task performance: The SST+Wiki sentiment task is an outlier: manipulated Embedding and BiLSTM models reduce attention mass but lose accuracy.BERT exceeds 90% accuracy while reducing maximum movie-review attention mass from 96.2% to 10^-3%.
- Attention mass and task performance: Sequence-to-sequence manipulation reduces attention mass over impermissible alignments while preserving original performance.Manipulated attention models also outperform models with none or uniform attention, suggesting attention benefits learning beyond alignment interpretation.
- Human Study: 96% accuracy relied on gender information, yet none of the manipulated-attention predictions were judged gender-influenced.Original attention led annotators to label 66% of predictions as gender-influenced; manipulated explanations received a trustworthiness score of 2.67 out of 4.
- Alternative Workarounds: Recurrent connections let BiLSTM models pass information across tokens before attention, preserving over 99% accuracy after impermissible attention values are zeroed.The Embedding model, lacking this pre-attention pathway, reaches only about 50% accuracy after zeroing gender-pronoun attention.
- Alternative Workarounds: Restricted models compensate for low attention by increasing impermissible-token representation magnitudes, whereas BiLSTMs can move relevant information through recurrent connections.Different random seeds also produce different alternative mechanisms, including shifted alignments and attention confined to the first hidden state.
6 Conclusion
The paper shows that attention scores can be easily manipulated while models continue using tokens known to be useful for prediction. This raises concerns about using attention to audit algorithms.
- Attention scores are easily manipulable.
- The training scheme reduces attention mass over predictive tokens while models continue using them.
- Manipulated attention may mislead regulators when used to audit algorithms.
A Instructions for human study
The human study presents model inputs and outputs with highlighted tokens and asks participants whether gender influenced predictions. Participants then rate how much the highlighted tokens determine the model’s predictions.
- Participants evaluate occupation predictions from short bios using highlighted words as explanations.
- The study asks whether participants suspect that gender influenced each prediction.
- Darker highlighting indicates greater token emphasis according to the explanation method.
- Participants rate explanation trustworthiness from 1 to 4 after viewing 50 examples from an attention scheme.The scale ranges from tokens not determining predictions to significantly determining them.
C Qualitative Examples
The paper includes qualitative examples illustrating three different attention schemes and identifies a table containing dataset sizes for classification tasks.
- Table 7 lists qualitative examples illustrating three different attention schemes.
- Table 6 records training, validation, and test example counts for classification datasets.
- The qualitative examples are presented as a table.