Source-linked AI summary
Learning to Attend via Word-Aspect Associative Fusion for Aspect-based Sentiment Analysis
Yi Tay, Anh Tuan Luu, Siu Cheung Hui
TL;DR
ABSA must predict sentiment for a specific aspect, but neural models such as ATAE-LSTM use costly and difficult-to-train aspect-word concatenation. The paper introduces AF-LSTM, which uses an associative layer based on circular convolution and correlation before attention. AF-LSTM achieves state-of-the-art benchmark performance and significant improvements over strong neural baselines, while a separate layer has limited performance influence in practice.
Problem
Aspect-based sentiment analysis requires incorporating aspect information into neural architectures to predict polarity for a specific aspect.
Method
AF-LSTM uses a dedicated association layer with circular convolution and correlation to model word-aspect relationships before attention learns the importance of fused context words.
Results
AF-LSTM achieves state-of-the-art performance on benchmark datasets and significant improvement over multiple strong neural baselines.
Takeaways & Limitations
Associative word-aspect fusion provides an effective aspect-aware attention mechanism, while circular convolution remains effective for aspect-word fusion.
Takeaways & Limitations
One separate layer was found not to influence performance much in practice, although it did not degrade performance.
Abstract
from arXiv · showhide
Aspect-based sentiment analysis (ABSA) tries to predict the polarity of a given document with respect to a given aspect entity. While neural network architectures have been successful in predicting the overall polarity of sentences, aspect-specific sentiment analysis still remains as an open problem. In this paper, we propose a novel method for integrating aspect information into the neural model. More specifically, we incorporate aspect information into the neural model by modeling word-aspect relationships. Our novel model, \textit{Aspect Fusion LSTM} (AF-LSTM) learns to attend based on associative relationships between sentence words and aspect which allows our model to adaptively focus on the correct words given an aspect term. This ameliorates the flaws of other state-of-the-art models that utilize naive concatenations to model word-aspect similarity. Instead, our model adopts circular convolution and circular correlation to model the similarity between aspect and words and elegantly incorporates this within a differentiable neural attention framework. Finally, our model is end-to-end differentiable and highly related to convolution-correlation (holographic like) memories. Our proposed neural model achieves state-of-the-art performance on benchmark datasets, outperforming ATAE-LSTM by $4\%-5\%$ on average across multiple datasets.
Introduction
Aspect-based sentiment analysis requires neural models to incorporate aspect information when predicting sentiment polarity. The paper targets weaknesses in naive aspect-word concatenation by separating association modeling from attention and proposing AF-LSTM.
- Motivation: Aspect-based sentiment analysis predicts polarity with respect to a specific aspect term, unlike coarse-grained sentiment analysis.A single review can contain aspects with opposite polarities, motivating aspect-sensitive predictions.
- Limitations of Prior Work: ATAE-LSTM concatenates context words and the aspect at both its attention and sentence-modeling layers.This design makes attention and the LSTM jointly model aspect-word relationships and sentence information.
- Limitations of Prior Work: Naive concatenation burdens attention and LSTM layers with modeling aspect-word relationships in addition to their primary roles.The LSTM processes a sequence dominated by the aspect embedding, making training more difficult.
- Limitations of Prior Work: Naive concatenation doubles the LSTM input and increases parameter costs, memory footprint, computational complexity, and overfitting risk.The paper also reports that ATAE-LSTM does not always outperform a baseline LSTM in early empirical experiments.
- Proposed Approach: AF-LSTM introduces a dedicated association layer that first models aspect-context relationships, allowing attention to learn the relative importance of fused context words.This separates the responsibilities of association modeling and attentive sentence representation.
- Contributions: The paper proposes a simple, efficient aspect-aware attention mechanism and the AF-LSTM architecture for aspect-based sentiment analysis.Its association layer uses circular convolution for word-aspect fusion and is inspired by holographic reduced representations.
Related Work
Prior work combines neural sentence modeling with aspect information through target-dependent LSTMs, associative operators, and memory networks. This paper instead integrates associative operators into an attention-based LSTM framework for ABSA.
- Aspect-Based Sentiment Analysis: Fine-grained sentiment analysis adds the complexity of fusing aspects with sentence representations.Proposed architectures include target-dependent LSTM models for incorporating aspect information.
- Associative Operators: Holographic reduced representations use circular correlation and convolution as encoding-decoding operations analogous to storage and retrieval in associative memories.These operators can be interpreted as compressed tensor products that learn second-order relationships between word and aspect embeddings.
- Memory Networks: Memory-network approaches frame ABSA as question answering, using the aspect as a query and context words as external memory.Multi-hop, multi-task, and dyadic MemNN variants extend this family with location attention, target detection, or richer compositional operators.
- Positioning: The paper differs from memory-network approaches by integrating associative operators into an attention-based LSTM framework.Its architecture combines associative word-aspect fusion with neural attention rather than using associative layers on top of memory networks.
Our Model
AF-LSTM integrates sentence and aspect representations through an associative word-aspect fusion layer before attention. The model uses LSTM outputs, circular operators, and attention to produce aspect-aware sentiment predictions without adding association-layer parameters.
- Input and sequence modeling: AF-LSTM converts sentence words into embeddings, represents multiword aspects by summation, and processes the resulting sequence with an LSTM.The LSTM produces hidden vectors H for the sentence sequence.
- Word-aspect fusion: For each LSTM output h_i, the fusion layer combines h_i with aspect embedding s into a memory trace m_i encoding their relationship.The resulting traces form a sequence of joint word-aspect representations.
- Attention and prediction: Attention is learned over the fused matrix M rather than directly over the LSTM output matrix H, producing a probabilistically weighted sentence representation.The weighted representation is passed through a final linear transformation and softmax classifier.
- Associative operators: The model uses circular correlation or circular convolution as associative operators for word-aspect fusion.Correlation is non-commutative, whereas convolution is commutative; both preserve the association-layer input dimension.
- Efficiency and training: The association layer is parameterless, retains the input dimensionality, and is optionally preceded by normalization treated as a tunable hyperparameter.The model is trained end-to-end with cross-entropy loss and L2 regularization.
Discussion and Analysis
The discussion frames AF-LSTM as an associative-memory approach that enriches word-aspect representations while keeping computation and parameter growth limited. Its forward and backward operations correspond to encoding and decoding in holographic memory models.
- Connections to Holographic Memory Models: AF-LSTM forms memory traces by composing each aspect embedding with an LSTM context output, then uses those traces to learn attention.Either circular convolution or circular correlation can serve as the encoding operation.
- Connections to Holographic Memory Models: When convolution encodes an association, circular correlation serves as the decoding operation; the reverse pairing applies when correlation encodes it.This encoding-decoding relationship is reflected in forward propagation and backpropagation.
- Capturing Second Order Interactions: Associative operators act as compressed tensor products, enabling second-order relationships between context-word and aspect embeddings to be learned.Figure 1 illustrates the compression patterns for circular convolution and correlation when d = 3.
- On efficiency: The association layer adds no network parameters, has O(n log n) operations, and avoids adding relationship-learning burdens to the LSTM and attention layers.These properties are presented as advantages over simple aspect-word concatenation.
- Separation of Goals: AF-LSTM separates learning aspect-word associations from the primary objectives of the LSTM and attention layers.The association is learned independently before attention focuses on the fused representations.
Experimental Evaluation
The evaluation compares AF-LSTM and neural baselines across aspect term and category classification datasets, finding that circular-convolution fusion delivers strong accuracy with fewer parameters.
- The experiments cover aspect term classification and aspect category classification under three-way and binary accuracy settings.
- The evaluation uses Laptops and Restaurants datasets for term classification and Restaurants plus another dataset for category classification, with a fixed development set drawn from training data.
- AF-LSTM (CONV) achieves state-of-the-art performance across all four datasets and settings.
- AF-LSTM (CONV) outperforms ATAE-LSTM by 3%–8% on 3-way classification and about 2%–3% on binary classification.
- Circular convolution performs better than the Hadamard-product baseline, while circular correlation performs similarly to the Hadamard product.
- AF-LSTM has approximately 810K parameters, compared with approximately 1.1M for AT-LSTM and 1.4M for ATAE-LSTM.
Qualitative Analysis
The qualitative analysis examines attention weights for contrasting aspects and shows that AF-LSTM changes its focus more appropriately when the aspect changes.
- The case study inspects attention vectors under identical hyperparameter settings for AF-LSTM, AT-LSTM, ATAE-LSTM, and an aspect-free AT-LSTM variant.
- For the Appetizer aspect, AF-LSTM focuses on “appetizers are okay” and ignores “slow,” unlike the compared attention models.
- When the aspect changes to Service, AF-LSTM switches its focus appropriately, whereas AT-LSTM and ATAE-LSTM do not focus correctly for both aspects.
Conclusion
The paper finds associative word-aspect fusion effective for learning aspect-aware attentions, with circular convolution remaining effective for aspect-word fusion. Elementwise multiplication is also a strong baseline, outperforming simple concatenation models and improving performance over multiple neural baselines.
- Associative relationships between aspects and context words support attentive representations in the proposed word-aspect fusion attention layer.The layer first learns word-aspect associations before learning attention over the resulting representations.
- Circular convolution remains highly effective for aspect-word fusion.
- Elementwise multiplication outperforms simple concatenation models such as AT-LSTM and ATAE-LSTM.
- The model significantly improves performance compared with multiple strong neural baselines.