Source-linked AI summary
A Structured Self-attentive Sentence Embedding
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, Yoshua Bengio
TL;DR
Sentence embeddings need to capture multiple semantic components while remaining interpretable, especially when a single sentence offers no external attention signal. The paper uses self-attention over bidirectional LSTM states to form a matrix embedding with diverse attention rows and evaluates it across three tasks. Across those tasks, the model outperforms other sentence embedding models by a significant margin, while unsupervised training remains constrained by decoding the matrix rows.
Problem
Single-sentence tasks lack extra information to guide attention, and simple vector or pooling representations may not capture multiple semantic components of a sentence.
Method
The model applies self-attention to bidirectional LSTM states, producing a matrix of weighted sums whose rows attend to different sentence aspects, with a penalty encouraging diverse attention.
Results
The model outperforms other sentence embedding models by a significant margin across evaluations on three different tasks.
Takeaways & Limitations
The matrix embedding supports in-depth interpretation through attention visualizations of which tokens contribute to each encoded aspect.
Takeaways & Limitations
The current training method relies heavily on downstream applications and cannot directly train the model in an unsupervised way because decoding does not know how to divide and reorganize the embedding rows.
Abstract
from arXiv · showhide
This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence. We also propose a self-attention mechanism and a special regularization term for the model. As a side effect, the embedding comes with an easy way of visualizing what specific parts of the sentence are encoded into the embedding. We evaluate our model on 3 different tasks: author profiling, sentiment classification, and textual entailment. Results show that our model yields a significant performance gain compared to other sentence embedding methods in all of the 3 tasks.
1 INTRODUCTION
Sentence-level representations remain less developed than word embeddings, especially when a single sentence provides no external signal to guide attention. The paper addresses this by replacing simple pooling with self-attention that extracts multiple sentence aspects and supports explicit interpretation.
- Sentence representations remain less satisfactory than distributed word representations, motivating improved methods for phrases and sentences.
- Single-sentence tasks such as sentiment classification lack extra information to guide attention, so models commonly use max pooling, averaging, or the final hidden state.
- The proposed self-attention mechanism replaces pooling and extracts different sentence aspects into multiple vector representations.
- Self-attention operates on top of an LSTM without extra inputs, accesses earlier hidden representations, and makes the embedding easier to interpret.
2 APPROACH
The model combines bidirectional LSTM states with self-attention to produce a fixed-size matrix embedding whose rows attend to different sentence parts. A penalization term discourages redundant attention and encourages focused, diverse semantic representations.
- 2.1 MODEL: The model combines a bidirectional LSTM with self-attention, whose weight vectors produce weighted LSTM states used as the sentence embedding.The resulting model can feed downstream applications such as a multilayer perceptron.
- 2.1 MODEL: Self-attention maps all LSTM hidden states to weights, enabling a variable-length sentence to become a fixed-size representation through a weighted sum.Softmax normalizes the weights, which sum to 1.
- 2.1 MODEL: Multiple attention hops expand the vector embedding into an r-by-2u matrix so different rows can focus on different sentence components.The annotation matrix A multiplied by H yields the sentence embedding M.
- 2.2 PENALIZATION TERM: The model introduces a penalization term because similar attention weights across hops can make the matrix embedding redundant.The stated goal is diversity among summation weight vectors.
- 2.2 PENALIZATION TERM: The proposed penalty uses A A^T minus the identity matrix to measure redundancy and consumes one third of the computation of KL-divergence penalization.It also targets focused probability mass for each attention row, which KL penalization does not encourage.
- 2.3 VISUALIZATION: Each embedding row can be interpreted through its corresponding attention vector, allowing heat maps to show token contributions to different encoded aspects.Summing all attention vectors provides an overall view of which words receive the most or least attention.
3 RELATED WORK
Related work includes supervised and unsupervised sentence embeddings, sentence-level self-attention, and recurrent attention mechanisms. The paper distinguishes its matrix-valued, multi-aspect representation and penalty from these approaches.
- Prior sentence embedding methods include supervised and unsupervised models, whereas this paper uses self-attention to extract multiple sentence aspects into multiple vector representations.
- Related attention methods differ in scope or representation: some target word embeddings, while others use sentence-level attention or retain a vector representation.The paper contrasts its matrix output and specially designed penalty with concurrent self-attention work.
- LSTMN uses fine-grained online attention during recurrent updates, whereas the paper characterizes its own attention as operating at a different granularity.
4 EXPERIMENTAL RESULTS
The model is evaluated on author profiling, sentiment classification, and textual entailment, with comparisons against baseline and sentence-encoding methods. Across these tasks, the model generally improves performance, while the penalization term and multiple matrix rows affect attention diversity and accuracy.
- Experimental setup: The experiments cover author profiling, sentiment classification, and textual entailment using the Age, Yelp, and SNLI datasets.Age predicts users’ age ranges from English tweets; Yelp classifies reviews into five star categories; SNLI evaluates sentence-pair entailment classification.
- Author profiling: The proposed model significantly outperforms both biLSTM and CNN baselines on author profiling.The baselines use pooled recurrent or convolutional representations, while the proposed model uses the same general evaluation setup with self-attention.
- Sentiment classification: The proposed model also achieves a significant performance gain over the two baselines on Yelp sentiment classification.Yelp reviews are classified into five star labels, with classification accuracy used as the measurement.
- Textual entailment: In SNLI, the method gains ≥1% over other published approaches except the 300D NSE encoders, where the difference is 0.2%.The comparison is restricted to sentence encoding-based models that exchange no information between premise and hypothesis before encoding.
- Effect of penalization term: Adding the penalization term reduces redundant attention, focuses representations on important words, and improves performance on Yelp and Age relative to unpenalized counterparts.On SNLI, the penalization term encourages row diversity and focus, but both models achieve similar test accuracies.
5 CONCLUSION AND DISCUSSION
The paper introduces an interpretable fixed-size matrix sentence embedding using self-attention and reports gains across three tasks. It also identifies scalability and unsupervised-training limitations.
- The model combines a fixed-size matrix sentence embedding with self-attention, enabling deeper interpretation of encoded sentence content.The paper reports evaluations on author profiling, sentiment classification, and textual entailment.
- Attention lets the final embedding access previous LSTM hidden states directly, reducing the need for the LSTM to preserve all information until its last state.LSTM states can provide shorter-term context while attention captures higher-level semantics requiring longer-term dependencies.
- The model encodes variable-length sequences into fixed-size representations without suffering from long-term dependency problems.The authors suggest applying it to longer content such as paragraphs and articles as future work.
- The current training method depends heavily on downstream applications, preventing unsupervised training in the proposed formulation.The authors attribute the obstacle to uncertainty about how embedding rows should be divided and reorganized during decoding.
A PRUNED MLP FOR STRUCTURED MATRIX SENTENCE EMBEDDING
The appendix uses the matrix embedding's two-dimensional structure to prune fully connected layers. This substantially reduces parameters while retaining comparable or better performance across the evaluated models.
- The matrix sentence embedding can make subsequent fully connected layers parameter-heavy because each hidden unit connects to every matrix-embedding unit.In the cited example, the fully connected layer accounts for around 90% of the parameters.
- The pruning scheme splits hidden states into r groups, connecting each group only to its corresponding matrix-embedding row.This reflects similarities among rows while removing cross-row connections.
- (r −1)/r of weight values are pruned when the hidden layer is divided into r equally sized groups.
- A second pruning structure exploits similarities across elements sharing the same LSTM hidden-unit source, producing structured hidden states of size u-by-q.The resulting representation is denoted M h.
- Table 4 compares model size before and after pruning, with p and q adjustable as hyperparameters and pruning extendable across structured hidden layers.
- The softmax layer remains fully connected to the structured representations, while its small number of units limits the associated computational cost.The cited applications contain only several softmax units.
- Pruning lowers performance somewhat but still allows all three models to perform comparably to or better than the paper's other compared models.
B DETAILED STRUCTURE OF THE MODEL FOR SNLI DATASET
For SNLI, the model independently embeds premise and hypothesis sentences with shared parameters, then combines their matrix embeddings through multiplicative interactions for classification.
- The textual entailment model receives a pair of sentences and uses matrix embeddings extracted for each sentence.The pair consists of a hypothesis and a premise.
- The overall SNLI model structure is depicted in Figure 7.
- The multiplicative interaction is inspired by the Factored Gated Autoencoder.
- Premise and hypothesis embeddings are extracted independently using the same LSTM and attention mechanism with shared parameters.
- A relation matrix embedding Fr represents the semantic relation between the premise and hypothesis embeddings through a three-way multiplicative interaction.Each factor can depend on the product of the other two.
- Each matrix-embedding row is multiplied by a weight matrix, producing factor matrices Fh and Fp for the hypothesis and premise.
- The relation factor Fr is computed as the element-wise product of Fh and Fp, after which an MLP with softmax classifies the relation.