Source-linked AI summary
Aspect Level Sentiment Classification with Attention-over-Attention Neural Networks
Binxuan Huang, Yanglan Ou, Kathleen M. Carley
TL;DR
Aspect-level sentiment classification must identify sentiment toward a particular aspect despite mixed sentiments within a sentence. The paper jointly encodes aspects and sentences with Bi-LSTMs and uses attention-over-attention to model their interactions. Experiments on SemEval 2014 datasets report superior performance to baseline methods, while error analysis identifies difficulties with complex expressions and idioms.
Problem
Aspect-level sentiment classification addresses the need to identify sentiment polarity for one specific aspect when sentence-level sentiment can mix multiple targets.
Method
The model jointly represents aspect targets and sentences with Bi-LSTMs, using attention-over-attention to capture their interactions and focus on important words.
Results
The model shows superior performance to baseline methods on SemEval 2014 laptop and restaurant datasets, and its case study identifies important parts of aspects and sentences effectively.
Takeaways & Limitations
The approach can focus on aspect-specific sentiment-bearing words, including the relevant word when multiple aspects occur in one sentence.
Takeaways & Limitations
The model cannot efficiently handle some complex sentiment expressions and uncommon idioms.
Abstract
from arXiv · showhide
Aspect-level sentiment classification aims to identify the sentiment expressed towards some aspects given context sentences. In this paper, we introduce an attention-over-attention (AOA) neural network for aspect level sentiment classification. Our approach models aspects and sentences in a joint way and explicitly captures the interaction between aspects and context sentences. With the AOA module, our model jointly learns the representations for aspects and sentences, and automatically focuses on the important parts in sentences. Our experiments on laptop and restaurant datasets demonstrate our approach outperforms previous LSTM-based architectures.
1 Introduction
Aspect-level sentiment classification identifies the polarity of a specific aspect in context, addressing errors caused by mixed sentiments and ignored targets. The paper proposes an LSTM model that jointly models aspects and text with attention-over-attention.
- Motivation: Aspect-level classification determines whether a specific aspect has positive, negative, or neutral sentiment in its context sentence.For “great food but the service was dreadful,” food is positive and service is negative.
- Motivation: Ignoring aspect information makes polarity difficult to determine when one sentence contains multiple aspects with different sentiments.The paper cites an evaluation in which 40% of Twitter sentiment-classification errors resulted from not considering targets.
- Prior approaches: Existing approaches use either manually created features or end-to-end neural networks that learn representations without prior feature engineering.Neural networks are becoming popular because they learn representations directly from data.
- Contribution: The proposed model uses LSTMs to model aspects and texts simultaneously rather than separately.This joint modeling is paired with an attention-over-attention module.
- Contribution: AOA captures mutual aspect-to-text and text-to-aspect attention to focus on sentiment-bearing words and important parts of aspect phrases.The model is reported to outperform previous methods on SemEval 2014 laptop and restaurant datasets.
2 Related work
Prior work spans feature-based, rule-based, and neural approaches to sentiment classification. LSTM methods progressively incorporate context and aspect attention, while the paper argues that pooling-based interaction remains limited.
- Sentiment classification: Traditional sentiment classifiers commonly rely on n-gram or manually designed features, lexicons, and supervised machine-learning algorithms.Examples include Naive Bayes and SVM.
- Aspect-level classification: Aspect-level methods include dependency-based rules and target-dependent features supplied to an SVM classifier.These approaches use sentence grammar structures to construct target-dependent features.
- Neural methods: TD-LSTM models the left and right contexts around an aspect with two LSTM networks and predicts sentiment from their final hidden states.Other work uses aspect embeddings to generate attention over sentence regions.
- Neural methods: IAN separately models sentences and aspects, then uses hidden states and pooling to attend to important parts in both.The paper states that pooling ignores interactions among sentence-target word pairs and reports its method as superior.
3 Method
The model represents a sentence and its aspect target with separate Bi-LSTMs, then uses AOA to derive sentence attention from their pairwise interactions. A weighted sentence representation is classified with a linear layer and softmax.
- Problem Definition: The task supplies a sentence and an aspect target, which may be a word or phrase, and classifies the target’s sentiment polarity.The target is defined as a contiguous span within the sentence.
- Word Embedding: Word embeddings map sentence and aspect words into low-dimensional vectors before separate Bi-LSTM processing.The architecture contains word embedding, Bi-LSTM, AOA, and final prediction components.
- Bi-LSTM: Bi-LSTMs learn hidden semantic states for sentence words and aspect-target words using forward and backward LSTM sequences.The final sentence states concatenate the forward and backward outputs, while target states are computed similarly.
- Attention-over-Attention: AOA computes sentence-target word-pair interactions, normalizes them into two attention directions, and combines them into sentence-level attention.The resulting attention explicitly weights each aspect word’s contribution when identifying important sentence words.
- Final Classification: The final sentence feature is a weighted sum of sentence hidden states using AOA attention, then a linear layer projects it to C sentiment classes.A softmax computes class probabilities, and the highest-probability label is selected.
- Training: Training minimizes cross-entropy with L2 regularization using Adam-based mini-batch stochastic gradient descent and dropout.Dropout randomly removes part of LSTM-cell inputs to reduce overfitting.
4 Experiments
Experiments evaluate the model on SemEval 2014 laptop and restaurant datasets, compare it with several LSTM-based baselines, and inspect its attention behavior. The model is trained repeatedly because initialization affects performance, and error analysis identifies difficult sentiment expressions.
- Datasets: Experiments use domain-specific laptop and restaurant datasets from SemEval 2014 Task 4, with annotators labeling aspect terms and polarities.Table 1 reports sentence-aspect pair distributions by sentiment category.
- Model Comparisons: The comparison includes Majority, LSTM, TD-LSTM, AT-LSTM, ATAE-LSTM, and IAN baselines with distinct sentence- and aspect-modeling strategies.These baselines range from single-sentence LSTMs to models using aspect-conditioned or bidirectional attention.
- Model Comparisons: The model is trained 10 times because performance fluctuates with random initialization, and results report both average accuracy and the best accuracy.Baseline performance is cited as a single best number, while the proposed method reports best and mean±std results in Table 2.
- Case Study: Attention visualizations show the model selects different sentiment-indicating words for different aspects in sentences containing multiple aspects.For the phrase “boot time,” it identifies “time” as the most important aspect word and “super fast” as the sentiment-indicating phrase.
- Error Analysis: Major errors involve non-compositional sentiment expressions, idioms, and complex sentiment expressions, although simple negation can be handled.Examples include “on point” and “I have never had a bad meal,” which the model misunderstands.
5 Conclusion
The paper proposes an Attention-over-Attention neural network for aspect-level sentiment classification and reports superior performance against baseline methods. Error analysis identifies complex sentiment expressions and uncommon idioms as cases the model cannot handle efficiently.
- The proposed neural network uses an Attention-over-Attention module to learn important parts in aspects and sentences for aspect-level sentiment classification.The module generates the final sentence representation.
- Experiments on SemEval 2014 datasets show superior performance compared with baseline methods.
- The model cannot handle some cases efficiently, including complex sentiment expressions and uncommon idioms.The paper suggests incorporating grammar structures and prior language knowledge in future work.