Towards Transparent and Explainable Attention Models

1 Introduction

• Prior work questions whether attention weights faithfully reflect prediction impact, while other work argues they may still offer plausible, human-understandable explanations.The introduction cites Serrano and Smith (2019), Jain and Wallace (2019), and Wiegreffe and Pinter (2019).Source: §1 · pp. 1–2
• LSTM attention can be unfaithful when hidden states are similar, and may assign high weight to unimportant tokens such as punctuation.The authors suggest hidden states may summarize the full context rather than represent their corresponding words specifically.Source: §1 · p. 2
• The authors propose orthogonalization and a diversity-driven training objective to make hidden states more diverse, reporting more transparent attention with comparable performance to vanilla LSTM encoders across 12 datasets and four tasks.Their attention rankings better indicate important words, correlate better with gradient-based feature importance, and respond to random permutations.Source: §1 · p. 2
• Across Yelp, SNLI, QQP, and bAbI 1, human evaluations preferred Diversity LSTM attention in 72.3%, 62.2%, 88.4%, and 99.0% of samples, respectively.The models also paid less attention to punctuation and more to task-relevant words.Source: §1 · p. 2

2 Tasks, Dataset and Models

• The study spans 12 datasets across binary text classification, natural language inference, paraphrase detection, and question answering, with text classification split into sentiment and other classification.The tasks include both single-sequence inputs and paired-sequence inputs.Source: §2 · p. 2, §2.1 · p. 3
• Sentiment analysis uses SST, IMDB, Yelp, and Amazon, each with positive or negative targets.Other classification covers Twitter adverse drug reaction detection, baseball-versus-hockey news classification, and Anemia and Diabetes prediction from MIMIC ICD9.Source: §2 · p. 2
• SNLI supplies three-way textual entailment labels, while QQP questions are labeled as paraphrased or not.QQP training is split 90:10 for training and validation, with the original development set used for testing.Source: §2 · p. 3
• The bAbI tasks require answers supported by one, two, or three context statements, and the CNN News Articles dataset contains 90,000 articles with about three questions per article.The bAbI answers are spans in the context.Source: §2 · p. 3

2.1 LSTM Model with Attention

• For paired inputs, separate LSTM encoders produce representations, and the final hidden state of one sequence queries attention over the other sequence’s intermediate representations.Single-input tasks use one LSTM, attention without a query, and a final output projection.Source: §2.1 · p. 3

3.1 Similarity Measures

• The analysis uses conicity, based on vectors’ alignment to their mean, to assess hidden-state similarity; high conicity means vectors cluster in a narrow cone.It frames the inquiry around both faithfulness and plausibility of attention explanations.Source: §3.1 · p. 3, §3 · p. 3

3.2 Attention Mechanisms

• When hidden states have high conicity, diverse attention distributions can produce similar context vectors and predictions, so high weights cannot reliably identify which inputs led to a prediction.The attended context vector is a convex combination of hidden states; Figure 1 contrasts similar outputs at high conicity with different outputs at low conicity.Source: §3.2 · pp. 3–4, Figure 1 · p. 4

3.3 Conicity of LSTMs Hidden States

• LSTM hidden states have conicity values from 0.43 to 0.77 across datasets, unlike isotropic reference vectors, which are much less aligned and sometimes nearly orthogonal to their mean.The authors identify high conicity as a potential reason attention-weight permutations can yield similar context vectors.Source: §3.3 · p. 4

3.4 Attention by POS Tags

• The authors assess plausible explanations by aggregating attention over universal part-of-speech tags and comparing attention to punctuation with its share of input tokens.They suggest context-wide hidden-state summaries may help explain the observed misalignment.Source: §3.4 · p. 4
• On Yelp, Amazon, and QQP, punctuation receives 28.6%, 34.0%, and 23.0% of attention despite comprising only 11.0%, 10.5%, and 11.6% of tokens, respectively.This disproportion raises doubts about attention’s plausibility as an explanation because punctuation is linguistically insignificant to context.Source: §3.4 · p. 4

4 Orthogonal and Diversity LSTM

• The authors propose two strategies intended to reduce hidden-state similarity, motivated by their preceding argument that high hidden-state conicity affects the transparency and explainability of attention models.Source: §4 · p. 4

4.1 Orthogonalization

• At each time step, orthogonalization makes the hidden state orthogonal to the mean of previous states by subtracting its component along that mean.The authors identify this subtraction as the key difference from a vanilla LSTM.Source: §4.1 · pp. 4–5

4.2 Diversity Driven Training

• The flexible diversity-driven objective jointly maximizes training-data log-likelihood and minimizes hidden-state conicity.λ controls the weight given to hidden-state diversity; H_P denotes the LSTM hidden states for input P.Source: §4.2 · p. 5

5 Analysis of the model

• The analysis compares Vanilla, Diversity, and Orthogonal LSTMs to assess predictive performance, faithfulness, and whether attention better matches human interpretation.The evaluation uses tasks and datasets described earlier in the paper.Source: §5 · p. 5

5.1 Implementation Details

• Implementation used one-layer LSTMs, Adam at 0.001, validation accuracy for model selection, and test splits for subsequent analyses.Embedding dimensions, hidden sizes, and diversity weights varied by dataset; the diversity weight ranged from 0.1 to 0.5.Source: §5.1 · p. 5

5.2 Empirical evaluation

• Diversity LSTM accuracy stayed within -7.7% to +6.7% relative to vanilla, while its conicity fell 70.6% to 93.2%; Orthogonal LSTM was mostly comparable, and both beat MLPs.Orthogonal LSTM accuracy increased 39.9% on bAbI 2 and dropped -13.91% on CNN; on Question Answering, the LSTMs averaged a 73.73% relative accuracy increase over MLP. Table 2 reports accuracy and conicity across datasets, alongside MLP accuracy and isotropic-vector conicity.Source: Table 2 · p. 6, §5.2 · p. 6

5.3 Importance of Hidden Representation

• Erasure and permutation tests indicate that attention rankings in Diversity and Orthogonal LSTMs identify influential states more concisely and are more output-sensitive than vanilla attention.Vanilla models often required erasing a large fraction of states to flip a decision, whereas proposed models flipped more quickly; paraphrase detection and QA had task-specific delayed or quick flips. Randomly permuting proposed-model attention changed outputs substantially, while vanilla outputs changed little on several datasets.Source: Figure 3 · p. 5, Figure 4 · p. 6, §5.3 · p. 7

5.4 Comparison with Rationales

• Across eight text-classification datasets, Diversity LSTM assigned rationales 53.52% more attention on average than vanilla LSTM.The generated rationales retained prediction accuracy within 5% of full sentences and were often shorter for Diversity LSTM.Source: §5.4 · p. 7, Table 3 · p. 7

5.5 Comparison with attribution methods

• Diversity LSTM attention aligned better with gradients, with a 64.84% relative increase in Pearson correlation and a 17.18% relative decrease in Jensen–Shannon divergence over vanilla LSTM.Similar trends held for Integrated Gradients across datasets.Source: §5.5 · pp. 7–8, Table 4 · p. 8

5.6 Analysis by POS tags

• Diversity LSTM reduced attention to punctuation and increased attention to task-relevant POS categories.Punctuation attention fell from 28.6%, 34.0%, and 23.0% to 3.1%, 13.8%, and 3.4% on Yelp, Amazon, and QQP; adjective attention rose by 49.27% relative across four sentiment datasets, and noun attention was higher in other text-classification tasks.Source: §5.6 · p. 8, Figure 5 · p. 8

5.7 Human Evaluations

• Human annotators preferred Diversity LSTM attention over vanilla attention across all datasets and the overall, completeness, and correctness criteria.The evaluation sampled 200 test examples each from Yelp, SNLI, QQP, and bAbI1, with three annotators per sample and majority voting.Source: Table 5 · p. 8, §5.7 · p. 9

• The work continues recent studies of attention interpretability by Serrano and Smith (2019), Jain and Wallace (2019), and Wiegreffe and Pinter (2019).Source: §6 · p. 9
• Prior work has induced sparse attention, but whether sparsity yields faithful and plausible explanations has not been studied in detail.Source: §6 · p. 9
• Bao et al. (2018) map attention to human-annotated rationales, whereas this work does not require additional supervision.Source: §6 · p. 9
• Other studies analyze transformer attention, while Nema et al. (2017) orthogonalize LSTM representations using a different diversity model for natural-language generation.Source: §6 · p. 9

• LSTM encoders learn highly similar hidden representations across timesteps, which affects the interpretability of attention weights.Source: §7 · p. 9
• The authors propose two techniques to address this shortcoming and report that the resulting attention distributions provide more faithful and plausible explanations.Source: §7 · p. 9
• Future work will extend the analysis and proposed techniques to more complex models and downstream tasks.Source: §7 · p. 9