Source-linked AI summary
Transformation Networks for Target-Oriented Sentiment Classification
Xin Li, Lidong Bing, Wai Lam, Bei Shi
TL;DR
Target-oriented sentiment classification must predict polarity for individual opinion targets, while attention can introduce irrelevant context and vanilla CNNs can mix sentiments across targets. TNet uses target-specific transformations between a bidirectional recurrent encoder and a position-aware CNN, while preserving contextual information. It consistently dominates previous state-of-the-art methods across different types of data, though implicit-semantic inference remains challenging for neural models.
Problem
Target-oriented sentiment classification requires predicting sentiment polarity for an individual target, but attention may aggregate irrelevant words and vanilla CNNs may associate a target with unrelated opinion words.
Method
TNet uses BiLSTM contextual representations, Target-Specific Transformation and context-preserving layers, followed by a position-aware convolutional feature extractor.
Results
TNet consistently dominates previous state-of-the-art methods on different types of data.
Takeaways & Limitations
The model combines target-specific representations, preserved context, positional relevance, and CNN feature extraction to address the identified attention and CNN obstacles.
Takeaways & Limitations
Inference based on implicit semantics remains challenging for neural network models.
Abstract
from arXiv · showhide
Target-oriented sentiment classification aims at classifying sentiment polarities over individual opinion targets in a sentence. RNN with attention seems a good fit for the characteristics of this task, and indeed it achieves the state-of-the-art performance. After re-examining the drawbacks of attention mechanism and the obstacles that block CNN to perform well in this classification task, we propose a new model to overcome these issues. Instead of attention, our model employs a CNN layer to extract salient features from the transformed word representations originated from a bi-directional RNN layer. Between the two layers, we propose a component to generate target-specific representations of words in the sentence, meanwhile incorporate a mechanism for preserving the original contextual information from the RNN layer. Experiments show that our model achieves a new state-of-the-art performance on a few benchmarks.
1 Introduction
Target-oriented sentiment classification predicts polarity for individual opinion targets, but attention can introduce irrelevant words and vanilla CNNs can mix sentiments across targets. TNet addresses these issues with target-specific transformations, context preservation, and CNN-based feature extraction.
- Task: Target-oriented sentiment classification predicts a sentiment category for each (target, sentence) pair.The task concerns opinion targets explicitly appearing in a sentence.
- Motivation: Attention-based RNNs may introduce irrelevant words when aggregating contextual features for target sentiment prediction.For example, attention may include “never” and “tired” when highlighting “favorite”.
- Motivation: Vanilla CNNs may capture opinion features associated with multiple targets, hindering prediction when one sentence expresses different sentiments.Examples include simultaneously capturing “great food” and “service was dreadful” for a single target.
- Approach: TNet generates target-specific word representations by conditioning each context word on a tailored representation of the target.The Target-Specific Transformation component derives different target representations conditioned on individual context words.
- Approach: A context-preserving mechanism forwards contextual information through the transformation architecture so deeper networks can learn more abstract contextualized word features.The mechanism is designed to preserve information from the RNN layer after nonlinear transformation.
- Contribution: TNet adapts CNN to target-level sentiment classification and is reported to dominate state-of-the-art models on benchmark datasets.The model combines transformed word representations with CNN feature extraction.
2 Model Description
TNet uses BiLSTM contextualization, target-specific transformation, context preservation, and position-aware CNN extraction to classify sentiment for individual targets.
- Bi-directional LSTM Layer: TNet begins with a BiLSTM that transforms word embeddings into contextualized word representations.These hidden states provide the contextual information used by subsequent layers.
- Context-Preserving Transformation: Multiple CPT layers incorporate target information through Target-Specific Transformation while preserving contextual information across deeper transformations.CPT combines target-specific representations with context-preserving mechanisms inspired by identity mapping and highway connections.
- Target-Specific Transformation: TST dynamically computes target representations by associating target-word representations with each sentence word.This avoids treating all words in a multiword target as equally important.
- Context-Preserving Mechanism: Lossless Forwarding directly passes pre-transformation features to the next CPT layer, retaining contextualized and transformed representations.The strategy keeps contextual information encoded in each upper-layer representation.
- Context-Preserving Mechanism: Adaptive Scaling uses a gating function to adjust the proportions of transformed and input features passed through CPT layers.The gate controls how much contextualized input and transformed representation are retained at different layers.
- Convolutional Feature Extractor: A proximity strategy weights words by their position relative to the target before convolution, highlighting nearby words and downgrading distant ones.The position relevance uses the first target-word index, a constant C, and target length m.
3 Experiments
Experiments evaluate TNet on three benchmark datasets and compare its architecture, components, and alternatives. TNet variants perform strongly across formal reviews and ungrammatical tweets, while ablations reveal the importance of target transformation and position information.
- Experimental Setup: Experiments use LAPTOP, REST, and TWITTER, evaluating models with Accuracy and Macro-Averaged F1.LAPTOP and REST contain laptop and restaurant reviews, while TWITTER contains Twitter posts.
- Main Results: TNet-LF and TNet-AS consistently achieve the best performance on all datasets.The models perform well on both formal reviews and ungrammatical tweets.
- Main Results: CNN-based models may have advantages on ungrammatical text, whereas LSTM-based models can perform well on formal sentences by capturing sequential context.Comparison methods relying on LSTMs are less effective on TWITTER, while CNN-ASP also obtains good results there.
- Performance of Ablated TNet: Removing target transformation makes both TNet variants incomparable with the full model in Accuracy and F1, indicating that target-specific word representations are crucial.TNet without context performs consistently better than TNet without transformation, supporting the efficacy of target-specific transformation.
- Performance of Ablated TNet: Position information significantly improves TNet-LF and TNet-AS, with all corresponding t-test p-values below 0.05.The comparison removes position information while retaining the other model components.
- Alternative Architectures: Replacing transformation with attention performs worse than TNet variants, and attention has a negative effect on TWITTER.The comparison evaluates attention and fully connected layers as alternative transformation modules.
- Impact of CPT Layer Number: Both TNet variants achieve their best results with L=2, while increasing the number of CPT layers generally worsens performance.TNet-AS becomes more sensitive at large L, possibly because its extra parameters increase training difficulty.
- Case Study: TNet captures target-specific n-gram features and can handle opinion words whose sentiment depends on the target.However, all evaluated models fail on a subjunctive example requiring implicit semantic inference.
4 Related Work
Related work spans supervised feature-engineering methods, neural models, and attention-based approaches for target-level sentiment classification. Attention was introduced to model target-context associations when critical context is far from the target.
- 4 Related Work: Early target-level sentiment methods mainly used supervised learning with extensive hand-coded features.Later work developed neural architectures for modeling target and context interactions.
- 4 Related Work: The literature includes models that separately or interactively represent target and context using neural architectures.These approaches include gated networks, attention-based LSTMs, and interactive LSTM models.
- 4 Related Work: Attention-based methods address cases where critical contextual information is far from the target by modeling target-context associations.Earlier gated neural models jointly modeled target and surrounding context but could not capture long-range information effectively.
5 Conclusions
The paper presents TNet as a target-specific CNN architecture designed to address limitations of attention and vanilla CNN models. Its performance dominates previous state-of-the-art methods across data types, and ablations support the architecture’s components.
- 5 Conclusions: TNet integrates target information into word representations through target-specific transformation and uses CNN for feature extraction.Context-preserving and position-relevance mechanisms retain advantages associated with LSTM-based representations.
- 5 Conclusions: TNet consistently dominates previous state-of-the-art methods on different types of data.The conclusion reports this result across the evaluated data settings without specifying individual dataset values.
- 5 Conclusions: Ablation studies show the efficacy of TNet’s different modules and support the rationale of its architecture.The conclusion treats the component-level findings as evidence for the design.