Source-linked AI summary
Effective LSTMs for Target-Dependent Sentiment Classification
Duyu Tang, Bing Qin, Xiaocheng Feng, Ting Liu
TL;DR
Target-dependent sentiment classification must account for how a target relates to surrounding context, because the same sentence can express different polarities toward different targets. The paper develops LSTM extensions that incorporate target information and target–context connections, evaluates them on a Twitter benchmark, and reports state-of-the-art accuracy without syntactic parsers or external sentiment lexicons.
Problem
Target-dependent sentiment classification remains challenging because models must represent the semantic relatedness between a target and its context words.
Method
The paper develops target-dependent LSTM models that incorporate target information into context representations and train end-to-end for supervised sentiment classification.
Results
The target-dependent LSTM models achieve state-of-the-art classification accuracy, while incorporating target information improves over standard LSTM without syntactic parsers or external sentiment lexicons.
Takeaways & Limitations
Target information is important for target-dependent sentiment classification, and target-connection modeling produces the best performance among the compared methods.
Takeaways & Limitations
In error analysis, 85.4% of misclassified examples involve the neutral category, indicating a remaining difficulty for both TD-LSTM and TC-LSTM.
Abstract
from arXiv · showhide
Target-dependent sentiment classification remains a challenge: modeling the semantic relatedness of a target with its context words in a sentence. Different context words have different influences on determining the sentiment polarity of a sentence towards the target. Therefore, it is desirable to integrate the connections between target word and context words when building a learning system. In this paper, we develop two target dependent long short-term memory (LSTM) models, where target information is automatically taken into account. We evaluate our methods on a benchmark dataset from Twitter. Empirical results show that modeling sentence representation with standard LSTM does not perform well. Incorporating target information into LSTM can significantly boost the classification accuracy. The target-dependent LSTM models achieve state-of-the-art performances without using syntactic parser or external sentiment lexicons.
1 Introduction
Target-dependent sentiment classification requires modeling how a target relates to context words, but manual feature design is labor intensive. The paper incorporates target information into LSTM models and reports improved classification accuracy without syntactic parsers or external sentiment lexicons.
- Motivation: Target-dependent sentiment classification infers a sentence’s polarity toward a specified target, whose sentiment can differ from another target in the same sentence.For example, the same sentence is positive toward picture quality but negative toward battery life.
- Motivation: Existing approaches include feature-based SVMs and neural networks, but modeling semantic relatedness between a target and its context remains challenging.Manually designing target-dependent features for SVMs is labor intensive.
- Approach: The paper extends LSTM by incorporating target information to model target–context relatedness and select relevant context for sentiment prediction.The models are trained end-to-end using supervised cross-entropy loss.
- Evaluation: The proposed approach is evaluated on a benchmark dataset and compared with feature-based SVM, adaptive recursive neural network, and lexicon-enhanced neural network baselines.The evaluation concerns target-dependent sentiment classification.
- Results: The target-dependent LSTM models achieve state-of-the-art classification accuracy without using a syntactic parser or external sentiment lexicon.The paper also reports that standard LSTM performs poorly on this target-dependent task and that adding target information significantly improves accuracy.
2 The Approach
The paper develops a progression from target-independent LSTM to TD-LSTM and TC-LSTM, which incorporate target context and target–context connections for sentiment classification. The models compose context representations, classify with softmax, and train end-to-end using cross-entropy.
- Model Training: The models use the final hidden representation as input to a linear layer and softmax classifier over positive, negative, and neutral sentiment categories.LSTM, TD-LSTM, and TC-LSTM are trained end-to-end with supervised cross-entropy loss and backpropagation-based parameter updates.
- Basic LSTM: Standard LSTM maps variable-length word sequences to fixed-length sentence representations, but its target-independent representation is identical for different targets in one sentence.This is problematic when the same sentence expresses different sentiment polarities toward different targets.
- Target-Dependent LSTM: TD-LSTM models preceding and following contexts around the target with separate LSTM networks, then concatenates their final hidden vectors for sentiment classification.The left network processes preceding context plus the target, while the right network processes following context plus the target in reverse order.
- Target-Connection LSTM: TC-LSTM extends TD-LSTM by explicitly incorporating the connection between the target and each context word during sentence representation.It represents a variable-length target by averaging its word vectors and concatenates this target vector with each context-word embedding at every position.
- Target-Connection LSTM: The TC-LSTM architecture splits each sentence into target words, preceding context words, and following context words, using separate LSTMs for the two context directions.The target representation is v_target, and the context inputs combine word embeddings with this target representation.
3 Experiment
Experiments compare target-dependent LSTM models with feature-based and neural baselines, while testing embeddings, efficiency, and error patterns. Target-aware models generally outperform basic LSTM, with TC-LSTM achieving the strongest classification accuracy but incurring additional training cost.
- Experimental Setup: The benchmark uses supervised training and evaluates accuracy and macro-F1 on 6,248 training and 692 test sentences.Positive, negative, and neutral instances comprise 25%, 25%, and 50% of both splits.
- Comparison to Other Methods: Basic LSTM performs worst because it ignores target information and can predict the same result for different targets in a sentence.TD-LSTM improves over LSTM by incorporating target signals.
- Comparison to Other Methods: TC-LSTM obtains the best classification performances and outperforms all baseline methods in accuracy.The model incorporates a target-connection mechanism into the target-dependent LSTM architecture.
- Effects of Word Embeddings: SSWEu outperforms SSWEh and SSWEr, while Glove and SSWEu perform comparably, highlighting the value of contextual information in embeddings.SSWEh and SSWEr do not encode word contexts, whereas global context helps estimate word representations.
- Effects of Word Embeddings: One-hundred-dimensional Glove vectors outperform 50-dimensional vectors, while 200-dimensional vectors show no significant improvement.TD-LSTM has similar time cost to LSTM but higher accuracy; TC-LSTM is slightly better than TD-LSTM with longer training time.
- Case Study: Both target-dependent models correct cases where LSTM follows overall sentence polarity instead of sentiment toward the target.Examples include neutral sentiment toward “lindsay lohan” despite negative sentiment about the speaker’s work, and positive sentiment toward “google” in an otherwise neutral sentence.
- Case Study: Target-dependent models struggle particularly with neutral examples, which constitute 85.4% of their shared misclassified cases.Positive instances are rarely misclassified as negative, and vice versa.
- Discussion: An attention-based LSTM variant performs slightly below standard LSTM, suggesting that the tested attention component does not improve accuracy.The model adds separate soft-attention layers for preceding and following contexts.
4 Related Work
Prior work treats target-dependent sentiment classification as text classification, using feature-based and neural approaches. Neural methods learn continuous text representations without manual feature engineering, but the target-specific problem remains distinct from sentence-level sentiment classification.
- Target-dependent sentiment classification is commonly formulated as a text classification problem.
- Earlier systems include feature-based SVMs and neural network approaches for building sentiment classifiers.
- Feature engineering can be effective but is labor intensive and may not discover discriminative or explanatory factors automatically.
- Neural models learn continuous text representations from data without requiring manual feature engineering.
5 Conclusion
The paper develops target-specific LSTM models that represent sentences while capturing connections between a target and its context. On a benchmark dataset, incorporating target information improves LSTM performance and yields state-of-the-art classification accuracy.
- The proposed target-specific LSTM models capture connections between the target word and its contexts during sentence representation.
- Incorporating target information boosts LSTM performance and produces state-of-the-art classification accuracy.