DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding

Tao Shen; Tianyi Zhou; Guodong Long; Jing Jiang; Shirui Pan; Chengqi Zhang

Source-linked AI summary

DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding

Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, Chengqi Zhang

arXiv:1709.04696v3Sep 14, 2017cs.CLcs.AI

TL;DR

NLP sentence encoding must model context while retaining temporal order, but attention-only approaches had not been broadly designed for non-seq2seq tasks. DiSAN uses directional and multi-dimensional attention without RNN or CNN structures, and achieves strong accuracy and efficiency across NLP benchmarks. Its performance is weaker on sentences of length ≥38, though those comprise 3.21% of movie-review sentences.

Problem
Attention provides flexible and parallelizable dependency modeling, but temporal order remains difficult to represent and attention-only networks had limited use beyond sequence-to-sequence tasks.
Method
DiSAN uses directional self-attention with positional masks and multi-dimensional feature-wise attention to produce sentence encodings without RNN or CNN structures.
Results
DiSAN achieves state-of-the-art inference quality across a wide range of NLP tasks, with fewer parameters and higher time efficiency than existing works.
Takeaways & Limitations
The attention-only architecture provides a unified sentence-encoding model for tasks including natural language inference, sentiment analysis, sentence classification, and semantic relatedness.
Takeaways & Limitations
DiSAN performs poorly when sentence length ≥38, although such sentences comprise only 3.21% of movie-review sentences.

Abstract

from arXiv · show

Recurrent neural nets (RNN) and convolutional neural nets (CNN) are widely used on NLP tasks to capture the long-term and local dependencies, respectively. Attention mechanisms have recently attracted enormous interest due to their highly parallelizable computation, significantly less training time, and flexibility in modeling dependencies. We propose a novel attention mechanism in which the attention between elements from input sequence(s) is directional and multi-dimensional (i.e., feature-wise). A light-weight neural net, "Directional Self-Attention Network (DiSAN)", is then proposed to learn sentence embedding, based solely on the proposed attention without any RNN/CNN structure. DiSAN is only composed of a directional self-attention with temporal order encoded, followed by a multi-dimensional attention that compresses the sequence into a vector representation. Despite its simple form, DiSAN outperforms complicated RNN models on both prediction quality and time efficiency. It achieves the best test accuracy among all sentence encoding methods and improves the most recent best result by 1.02% on the Stanford Natural Language Inference (SNLI) dataset, and shows state-of-the-art test accuracy on the Stanford Sentiment Treebank (SST), Multi-Genre natural language inference (MultiNLI), Sentences Involving Compositional Knowledge (SICK), Customer Review, MPQA, TREC question-type classification and Subjectivity (SUBJ) datasets.

1 Introduction

DiSAN addresses the open question of how to model context and temporal order without choosing between RNNs and CNNs. It uses directional, multi-dimensional attention to build efficient sentence encodings and reports strong performance across NLP tasks.

RNNs capture long-range dependencies while CNNs extract local or position-invariant features, leaving network choice largely empirical.
Attention offers flexible dependency modeling and parallelizable computation, but most attention mechanisms lose temporal-order information.
DiSAN introduces multi-dimensional attention that computes feature-wise attention and directional attention that uses positional masks for asymmetric dependencies.
DiSAN builds sentence encodings entirely from forward and backward self-attention followed by multi-dimensional sequence compression, without RNN or CNN structures.
1.02%: DiSAN improves the best result on SNLI among sentence-encoding models and shows state-of-the-art performance across several other NLP datasets.The reported datasets include SST, MultiNLI, SICK, Customer Review, MPQA, SUBJ, and TREC question-type classification.
Fewer parameters and higher computation efficiency distinguish DiSAN from the LSTM and tree-based models it outperforms.

2 Background

Attention assigns importance to sequence elements relative to a query, then aggregates token embeddings into a sentence representation. Common additive and multiplicative forms differ in scoring, while self-attention relates tokens within one sequence.

Attention: Attention computes alignment scores between token embeddings and a query, normalizes them into importance weights, and produces a weighted sum of token embeddings.The resulting vector can serve as a sentence encoding.
Attention: Traditional attention uses scalar alignment scores, whereas multi-dimensional attention computes a vector score for each token-query pair.The vector form provides feature-wise attention.
Attention: Additive attention uses a learned nonlinear compatibility function, while multiplicative attention uses inner product or cosine similarity.The two mechanisms share the same attention form but differ in their compatibility functions.
Attention: Additive attention often achieves higher prediction quality, whereas multiplicative attention is faster and more memory-efficient through optimized matrix multiplication.This is a quality–efficiency trade-off between the two scoring mechanisms.
Self-Attention: Self-attention replaces the query with a token embedding from the same sequence and computes relationships between token pairs.It can model both long-range and local dependencies with faster computation and fewer parameters than RNNs.

3 Two Proposed Attention Mechanisms

The section introduces multi-dimensional and directional self-attention as the two main attention mechanisms underlying DiSAN. Multi-dimensional attention assigns feature-wise importance, while directional self-attention uses positional masks to encode asymmetric temporal dependencies.

3.1 Multi-dimensional Attention: Multi-dimensional attention replaces each token’s scalar attention score with a feature-wise score vector.The score vector has the same length as the token representation and is computed using a weight matrix rather than a weight vector.
3.1 Multi-dimensional Attention: Feature-wise probability distributions assign separate importance weights to each feature across all input tokens.For each feature, a categorical distribution over tokens determines which token features are important to the query or sequence.
3.1 Multi-dimensional Attention: Multi-dimensional attention can select contextually relevant features of polysemous words rather than assigning one importance score to each word.This allows different features of the same word to contribute differently under a given context.
3.2 Two types of Multi-dimensional Self-attention: Token2token self-attention produces context-aware representations for every sequence element, whereas source2token self-attention compresses the sequence into a vector.Both variants are used in different parts of DiSAN’s sentence-encoding model.
3.3 Directional Self-Attention: Directional self-attention combines masked multi-dimensional token2token attention with a fusion gate to model dependencies and temporal order.The positional mask can make attention asymmetric by suppressing one direction between two elements.
3.3 Directional Self-Attention: Forward and backward masks restrict attention in opposite temporal directions, while a diagonal-disabled mask prevents tokens from attending to themselves.The resulting attention output is fused with the input to produce temporal-order-encoded, context-aware token representations.

4 Directional Self-Attention Network

DiSAN is a lightweight sentence encoder built entirely from directional self-attention and multi-dimensional source2token attention. It processes tokens in forward and backward directions, then compresses their representations into a single sentence vector.

4 Directional Self-Attention Network: DiSAN applies separate forward and backward DiSA blocks to the input token embeddings.The two outputs are concatenated before sentence-level attention is applied.
4 Directional Self-Attention Network: A multi-dimensional source2token self-attention block compresses the concatenated sequence representation into the final sentence encoding.The resulting vector can be passed to a classification or regression module.
4 Directional Self-Attention Network: DiSAN uses fewer parameters, a simpler structure, and better efficiency than the bidirectional LSTM architecture that inspires its two-direction design.Forward and backward attentions encode dependencies from different directions, paralleling the role of forward and backward LSTMs.

5 Experiments

Experiments evaluate DiSAN across natural language inference, sentiment analysis, and other NLP tasks, showing strong accuracy, efficiency, and context-sensitive attention behavior. DiSAN achieves competitive or state-of-the-art results while using fewer parameters and faster training than several baselines.

Experiments: DiSAN achieves state-of-the-art performance on natural language inference, sentiment analysis, and additional NLP tasks.The evaluated tasks include SNLI, SST, MultiNLI, SICK, Customer Review, MPQA, SUBJ, and TREC question-type classification.
Natural Language Inference: 1.02% improvement over the latest best SNLI test accuracy establishes DiSAN as the strongest sentence-encoding model in that comparison.It also surpasses Bi-LSTM by 2.32% and Bi-LSTM with additive attention by 1.42%.
Natural Language Inference: 3.31% improvement follows from replacing token-wise attention with multi-dimensional attention in a word-embedding-based model.Directional self-attention also improves performance by 0.96%, while DiSAN outperforms multi-head attention by 1.45%.
Natural Language Inference: 3× faster than Bi-LSTM, DiSAN also uses only 2.35M parameters compared with 2.8M–3.0M in several competitive encoders.The parameter comparisons exclude the word-embedding component.
Sentiment Analysis: DiSAN significantly outperforms LSTM, Bi-LSTM, and Tree-LSTM on sentence lengths 5–12 and performs strongly on lengths 25–38.Performance declines when sentence length is ≥38, although such sentences comprise only 3.21% of movie-review sentences.
Case Study: Directional forward and backward attentions encode temporal order while focusing on different sentence regions, producing context-aware token representations.For example, forward and backward attention for “match” emphasize different words, and separate normalization preserves multiple significant weights.
Case Study: The gate tends to retain input representations for meaningless words, especially stop words, while their relations to other words may remain informative.This behavior is shown through the gate value F in Eq. (19).
Case Study: Multi-dimensional attention assigns different score vectors to the same word in different contexts, capturing context-dependent meanings beyond a single attention score.The paper illustrates this with “glass” and “close” in sentences where their meanings differ.

6 Conclusion

DiSAN combines two attention mechanisms to provide RNN/CNN-free sentence encoding, while achieving strong quality and efficiency across NLP tasks. Future work will extend these mechanisms to more sophisticated tasks.

DiSAN combines multi-dimensional and directional self-attention mechanisms for sentence encoding without recurrent or convolutional structure.Multi-dimensional attention performs feature-wise selection, while directional self-attention encodes temporal information through positional masks.
DiSAN achieves state-of-the-art inference quality across a wide range of NLP tasks with fewer parameters and higher time efficiency than existing works such as LSTM.
Future work will apply the proposed attention mechanisms to more sophisticated tasks, including question answering and reading comprehension.The stated goal is to achieve better performance on various benchmarks.