Source-linked AI summary
Modeling Sentiment Dependencies with Graph Convolutional Networks for Aspect-level Sentiment Classification
Pinlong Zhaoa, Linlin Houb, Ou Wua
TL;DR
Aspect-level sentiment models often treat multiple aspects independently and therefore omit sentiment dependencies between them. SDGCN uses aspect-specific bidirectional attention with position encoding followed by graph convolution over aspect dependencies. Evaluations on SemEval 2014 show superior or new state-of-the-art results, while the authors identify coarse sentiment graphs as a limitation.
Problem
Existing aspect-level sentiment models generally process each aspect independently, overlooking potentially valuable sentiment dependencies among aspects in the same sentence.
Method
SDGCN obtains aspect-specific representations with bidirectional attention and position encoding, then models aspect dependencies through message passing on a sentiment graph.
Results
SDGCN outperforms state-of-the-art methods on SemEval 2014, and SDGCN-BERT obtains new state-of-the-art results.
Takeaways & Limitations
Sentiment dependencies between aspects are helpful for aspect-level sentiment classification, including attention to words useful for judging those dependencies.
Takeaways & Limitations
The two undirected sentiment graph types used are coarse, motivating future graphs defined with textual information.
Abstract
from arXiv · showhide
Aspect-level sentiment classification aims to distinguish the sentiment polarities over one or more aspect terms in a sentence. Existing approaches mostly model different aspects in one sentence independently, which ignore the sentiment dependencies between different aspects. However, we find such dependency information between different aspects can bring additional valuable information. In this paper, we propose a novel aspect-level sentiment classification model based on graph convolutional networks (GCN) which can effectively capture the sentiment dependencies between multi-aspects in one sentence. Our model firstly introduces bidirectional attention mechanism with position encoding to model aspect-specific representations between each aspect and its context words, then employs GCN over the attention mechanism to capture the sentiment dependencies between different aspects in one sentence. We evaluate the proposed approach on the SemEval 2014 datasets. Experiments show that our model outperforms the state-of-the-art methods. We also conduct experiments to evaluate the effectiveness of GCN module, which indicates that the dependencies between different aspects is highly helpful in aspect-level sentiment classification.
1. Introduction
Aspect-level sentiment classification predicts sentiment for individual aspects in context, but prior models largely treat multiple aspects independently. SDGCN models dependencies among aspects with GCNs after aspect-specific attention, achieving superior SemEval 2014 performance.
- Aspect-level sentiment classification infers each aspect's polarity from its context, such as positive “price” and negative “service” in one sentence.
- Prior approaches use handcrafted features or aspect-specific neural models, but they generally ignore sentiment dependencies between multiple aspects.
- Conjunctions and punctuation can reveal dependencies, allowing one aspect's polarity to help infer another aspect's sentiment.
- SDGCN applies bidirectional attention with position encoding, then uses GCN to capture sentiment dependencies between aspects.
- Experiments on SemEval 2014 datasets show that the proposed model achieves superior performance over state-of-the-art approaches.
2. Related work
Related work covers feature-based and neural approaches to aspect-level sentiment classification, attention-based aspect representations, and GCN applications across relational data tasks.
- Aspect-level sentiment classification: Early aspect-level sentiment methods rely on handcrafted features, whereas neural methods learn dense representations without manual feature engineering.
- Aspect-level sentiment classification: Attention mechanisms enhance sentence representations by concentrating on context words relevant to a given aspect.
- Aspect-level sentiment classification: Prior models include ATAE-LSTM, RAM, and other attention-based architectures for modeling interactions between aspects and context.
- Graph convolutional network: GCNs address graph data with rich relational information and have been applied to image recognition and several NLP tasks.
3. Methodology
The task predicts polarities for multiple aspect terms in a sentence. The proposed architecture combines embeddings, sequence modeling, position encoding, bidirectional attention, GCN, and an output layer.
- The input contains N context words and K aspect terms, with each aspect represented as a subsequence of the sentence.
- The model constructs a sentiment classifier that predicts the sentiment polarities of multiple aspects.
- SDGCN's architecture consists of input embeddings, Bi-LSTM, position encoding, bidirectional attention, GCN, and an output layer.
- The embedding layer maps words into vectors using pretrained GloVe and BERT representations.
3.2. Bidirectional Long Short-Term Memory (Bi-LSTM)
The model uses Bi-LSTM encoders to capture contextual information for sentence words and aspect words. Separate Bi-LSTMs produce sentence- and aspect-specific contextual outputs, while sharing parameters across aspects.
- Bi-LSTM captures contextual information for each word from forward and backward hidden states.The two directional states are concatenated into the final word representation.
- The model uses separate Bi-LSTMs to encode the sentence context and each aspect’s contextual representation.The resulting outputs are denoted Hc for the sentence and Ha for each aspect.
- The Bi-LSTM parameters are shared across different aspects.
3.3. Position encoding
Position encoding represents the relative distance between context words and an aspect. It reflects the assumption that nearby context words more strongly influence an aspect’s polarity.
- Position encoding models the assumption that context words closer to an aspect more strongly influence its polarity.
- For each aspect, the model defines a relative distance between every context word and that aspect.The aspect is indexed among the K aspects in the context.
- The distance encoding assigns full weight at distance 0, linearly decreases weight through distance s, and assigns zero beyond s.Here, s is a predefined constant and N is the context length.
- The encoded distances are combined with contextual representations to obtain position-aware representations.
3.4. Bidirectional attention mechanism
The bidirectional attention mechanism models interactions between each aspect and its context in two directions. It first updates aspect representations from context, then derives aspect-specific context representations for downstream processing.
- Bidirectional attention contains context-to-aspect and aspect-to-context modules for modeling interactive information.
- Context to aspect attention: Context-to-aspect attention assigns weights to aspect words using a query vector formed by average pooling contextual hidden outputs.
- Context to aspect attention: The weighted aspect hidden representations form new aspect representations.
- Aspect to context attention: Aspect-to-context attention uses the new aspect representation and position-aware representations to calculate attention scores.
- Aspect to context attention: The attention process produces aspect-specific representations X for the K aspects and their context words.
3.5. Graph convolutional network
The model constructs sentiment graphs whose aspect nodes exchange information through graph convolution. It considers adjacent and global dependency structures, then stacks GCN layers to propagate messages across wider neighborhoods.
- Sentiment graph construction: A sentiment graph represents each aspect as a node and each sentiment dependency between aspects as an edge.
- Sentiment graph construction: The adjacent-relation graph connects each aspect only to nearby aspects, whereas the global-relation graph connects it to all other aspects.
- Graph convolution: For a node v, N(v) denotes its neighboring nodes, defining which aspect representations can exchange graph messages.
- Graph convolution: GCN updates each aspect representation by encoding information from its graph neighborhood and self-loop.The resulting node representation is used for the corresponding aspect classifier.
- Graph convolution: Each GCN layer transforms representations from the previous layer into new node representations.
- Graph convolution: Stacking multiple GCN layers allows each node’s final representation to receive messages from a farther neighborhood.
3.6. Output layer
The output layer treats each GCN node representation as the classifier input for its corresponding aspect and maps it to sentiment classes.
- Each final GCN node representation xL_i is treated as the classifier for the i-th aspect.
- A fully connected layer maps xL_i into an aspect space containing C sentiment classes.
- The model computes the predicted probability of the i-th aspect having sentiment polarity j from the fully connected output.
3.7. Model training
The model is trained with cross-entropy loss augmented by L2 regularization, while dropout is used during training to avoid over-fitting.
- Training minimizes cross-entropy loss with an L2-regularization term.
- The loss uses one-hot labels for each aspect and sentiment class, with λ controlling L2 regularization over selected parameters θ.
- Dropout is applied during training to avoid over-fitting.
4. Experiments
Experiments on SemEval 2014 laptop and restaurant reviews compare SDGCN with baseline models and ablations. Results favor GCN-based modeling, global relations, position encoding, and attention to inter-aspect cues.
- Data sets and settings: Experiments use SemEval 2014 reviews from laptop and restaurant domains, where sentences contain one or more aspects labeled positive, neutral, or negative.
- Baselines: The comparisons include recurrent, attention-based, position-aware, and two-stage baseline models for aspect-level sentiment classification.
- Overall results: SDGCN achieves the best performance among both GloVe-based and BERT-based models, with SDGCN-BERT obtaining new state-of-the-art results.
- Ablation results: Global-relation GCN is slightly higher than adjacent-relation GCN in accuracy and Macro-F1, while position information yields significant improvements.
- Overall results: 1.42% and 2.03% absolute gains in accuracy and Macro-F1, respectively, are reported over AEN-BERT on the Laptop dataset.
- Effect of GCN: GCN models outperform their GCN-reduced counterparts, supporting the role of modeling sentiment dependencies between different aspects.
- Case study: With-GCN models attend to conjunctions such as “and” and “but,” using these textual cues to represent dependencies between aspect sentiments.
5. Conclusion
SDGCN combines aspect-specific attention with graph convolutional message passing to model sentiment dependencies between aspects in one sentence. On SemEval 2014, SDGCN-BERT achieves new state-of-the-art results, while future work targets more precise sentiment graphs.
- Model: SDGCN first uses bidirectional attention with position encoding, then applies message passing between aspects to capture sentiment dependencies.Each aspect is represented using its context, and GCN models relations among aspect nodes.
- Results: SDGCN-BERT obtains new state-of-the-art results on the SemEval 2014 datasets.
- Future work: The authors identify the two undirected sentiment graphs as coarse and propose using textual information to build more precise structures.