Source-linked AI summary
Aspect Level Sentiment Classification with Deep Memory Network
Duyu Tang, Bing Qin, Ting Liu
TL;DR
Aspect-level sentiment classification must determine an aspect’s polarity while identifying which surrounding words provide useful evidence. This paper develops a deep memory network with layered content- and location-based attention over external memory, and reports comparable performance to feature-based SVM, better results than LSTM architectures, and improved performance from multiple layers. The method is also reported as simpler and faster than recurrent neural models.
Problem
Conventional LSTMs capture context implicitly and do not explicitly identify the important context clues needed to infer an aspect’s sentiment.
Method
The deep memory network uses multiple computational layers with content- and location-based attention over context-word external memory to build the final classification representation.
Results
The approach performs comparably to state-of-the-art feature-based SVM, outperforms LSTM architectures, and improves with multiple computational layers on two datasets.
Takeaways & Limitations
Explicitly modeling context-word importance with attention and multiple memory-network layers provides a simpler, faster alternative to recurrent neural models within the evaluated task and datasets.
Takeaways & Limitations
The error analysis identifies difficulty with noncompositional sentiment expressions and long multiword aspect expressions.
Abstract
from arXiv · showhide
We introduce a deep memory network for aspect level sentiment classification. Unlike feature-based SVM and sequential neural models such as LSTM, this approach explicitly captures the importance of each context word when inferring the sentiment polarity of an aspect. Such importance degree and text representation are calculated with multiple computational layers, each of which is a neural attention model over an external memory. Experiments on laptop and restaurant datasets demonstrate that our approach performs comparable to state-of-art feature based SVM system, and substantially better than LSTM and attention-based LSTM architectures. On both datasets we show that multiple computational layers could improve the performance. Moreover, our approach is also fast. The deep memory network with 9 layers is 15 times faster than LSTM with a CPU implementation.
1 Introduction
Aspect-level sentiment classification requires inferring an aspect’s polarity from sentence context, but conventional LSTMs do not explicitly identify which context words matter. The proposed deep memory network uses layered attention over external memory and performs competitively with feature-based SVM while outperforming LSTM variants.
- Motivation: Aspect-level sentiment classification infers an aspect’s positive, negative, or neutral polarity from a sentence and its aspect.For example, “food” is positive while “service” is negative in a sentence containing both sentiments.
- Motivation: LSTMs capture context implicitly and cannot explicitly reveal the importance of each context word for an aspect.The paper argues that only a subset of context words may be needed for the polarity decision.
- Approach: The deep memory network assigns context-word importance with content- and location-based attention, then builds a text representation through multiple computational layers.The final-layer representation is used as the sentiment-classification feature, and the model is trained end-to-end by gradient descent.
- Results: Experiments on laptop and restaurant datasets show performance comparable to a top feature-based SVM system.The datasets come from SemEval 2014.
- Results: The approach outperforms LSTM and attention-based LSTM models in classification accuracy and running speed on both datasets.The experiments also show that multiple computational layers over external memory improve performance.
2 Background: Memory Network
A memory network is a machine-learning framework that performs inference using a jointly learned long-term memory. Its components encode inputs, update memory, produce an output representation, and generate a response for prediction.
- Memory Network: A memory network performs inference with a long-term memory component that can be read, written to, and jointly learned for prediction.The memory is an array of objects such as vectors.
- Memory Network: The framework contains four components: I converts inputs to internal features, G updates memories, O generates an output representation, and R produces a response.Together, these components define the memory network’s processing flow.
- Workflow: In question answering, I encodes each sentence and G updates memory until the resulting matrix stores the sentences’ semantics.Each memory row represents one sentence.
- Workflow: Given a question, the memory network uses the stored sentence representations to find evidence and generate an answer.The question-answering example illustrates how memory supports inference after sentence encoding and memory updates.
3 Deep Memory Network for Aspect Level Sentiment Classification
The deep memory network represents context words as external memory and repeatedly applies shared attention and linear layers to select aspect-relevant evidence for sentiment classification. It combines content and location attention, then uses the final hop representation for supervised prediction.
- 3.2 An Overview of the Approach: The model maps context word embeddings into external memory and uses the aspect representation to retrieve aspect-relevant evidence.For multiword aspects, the aspect representation can average constituent word vectors; the section simplifies to single-word aspects.
- 3.2 An Overview of the Approach: Each computational layer combines an attention layer with a linear layer, and the final hop represents the sentence with respect to the aspect.The output of one hop becomes the input to the next, enabling more abstractive evidence selection.
- 3.2 An Overview of the Approach: Shared parameters let one- and nine-layer models have the same number of parameters while increasing computational depth.The attention and linear-layer parameters are shared across hops.
- 3.3 Content Attention: Content attention assigns differentiable importance scores to memory entries according to their semantic relatedness to the aspect.Scores are computed by a feed-forward network and normalized with softmax weights.
- 3.4 Location Attention: Location attention incorporates a context word's absolute distance from the aspect, while the authors leave syntactic-path distance for future work.They examine four strategies, including location vectors, learned position representations, and neural gates.
- 3.5 Multiple Computational Layers and 3.6 Aspect Level Sentiment Classification: Multiple computational layers support progressively more abstract sentence representations for phenomena such as negation, intensification, and contrary meaning.The final-hop vector is fed to a softmax classifier trained by minimizing cross-entropy error.
4 Experiment
Experiments compare the proposed memory-network approach with majority, feature-based SVM, recurrent baselines, and attention variants on laptop and restaurant sentiment datasets. Results examine accuracy, runtime, attention designs, hop depth, and error patterns.
- Experimental Setting: The experiments use laptop and restaurant datasets, omit the tiny conflict category, and evaluate models with classification accuracy.The conflict category was removed because including it would make the dataset extremely unbalanced.
- Comparison to Other Methods: The comparison includes Majority, a feature-based SVM, three LSTM variants, and ContextAVG.The LSTM variants are LSTM, TDLSTM, and TDLSTM+ATT; ContextAVG averages context vectors before classification.
- Comparison to Other Methods: Feature-based SVM substantially outperforms the other baseline methods, while TDLSTM performs better than LSTM.The results associate TDLSTM’s advantage with incorporating aspect information.
- Comparison to Other Methods: The proposed memory network avoids recurrent calculation over the word sequence and outperforms LSTM and attention-based LSTM models in the reported comparisons.The authors contrast direct attention over context representations with attention over recurrent hidden vectors.
- Hop Depth and Attention: More computational layers generally improve performance, especially below six hops; the best performances occur at seven and nine hops.The proposed models range from one to nine hops.
- Runtime Analysis: Nine-hop memory networks are almost 15 times faster than basic LSTM on the restaurant dataset’s training-iteration runtime.The comparison uses the same neural-network infrastructure, 300-dimensional GloVe vectors, and CPU server.
- Effects of Location Attention: Figure 2 reports classification accuracy for attention models on the restaurant dataset, where multiple layers consistently improve accuracy.Among location-based models, Model 2 is preferred for intuitive location handling and lower computation without accuracy loss; removing Model 4’s sigmoid gate reduces accuracy by almost 5 percentage points.
- Visualizing Attention Models: Content-only attention can shift weight toward “dreadful” for the wrong aspect, whereas location-enhanced attention increases “great” for “food” and yields the correct label.The examples illustrate how multiple hops and location information affect attention weights and predictions.
5 Related Work
Prior work on aspect-level sentiment classification includes feature-based SVMs and neural models, while attention and explicit memory provide mechanisms for selecting relevant textual information.
- Aspect-level sentiment classification infers an aspect’s sentiment polarity from a sentence containing that aspect.
- Feature-based SVMs use engineered feature templates and external resources such as parsers and sentiment lexicons.
- Neural models learn text representations from data but LSTM-based approaches are computationally expensive and do not explicitly reveal context importance for an aspect.
- The task differs from fine-grained opinion extraction because the aspect is supplied as part of the input rather than identified through word-level tagging.
- Attention mechanisms assign weights that select where to look in lower-level representations treated as memory.
6 Conclusion
The paper develops deep memory networks that explicitly model context-word importance for aspect-level sentiment classification. Across two datasets, the approach matches feature-based SVM accuracy, outperforms LSTM architectures, and benefits from multiple computational layers.
- Deep memory networks capture context-word importance for aspect-level sentiment classification.
- Compared with recurrent neural models such as LSTM, the approach is simpler and faster.
- On two datasets, the approach performs comparably to a state-of-the-art feature-based SVM system and substantially better than LSTM architectures.
- Using both content and location information produces better context weights and text representations, while multiple computational layers improve performance.
- Future work includes incorporating sentence structure, such as parsing results, into the deep memory network.