PaperlayerView exact revision on arXiv

Source-linked AI summary

Attentional Encoder Network for Targeted Sentiment Classification

Youwei Song, Jiahai Wang, Tao Jiang, Zhiyue Liu, Yanghui Rao

arXiv:1902.09314v2cs.CL

TL;DR

Targeted sentiment classification must model sentiment toward explicit targets while addressing recurrent encoders’ computational and long-term dependency limitations and unreliable labels. The paper proposes an attention-based Attentional Encoder Network with label smoothing and BERT integration, obtaining new state-of-the-art results and a lightweight alternative to strong RNN models.

  • Problem

    Targeted sentiment classification involves determining sentiment polarity toward explicit targets, while prior RNN-based approaches are difficult to parallelize and neutral labels can be unreliable.

  • Method

    The paper proposes an Attentional Encoder Network that replaces recurrence with attention-based encoders, adds label smoothing regularization, and applies pre-trained BERT.

  • Results

    The BERT-based model enhances basic BERT performance and obtains new state-of-the-art results, while experiments report competitive performance and lightweight RNN alternatives.

  • Takeaways & Limitations

    Attention-based encoding, label smoothing, and task-specific BERT fine-tuning together support effective and lightweight targeted sentiment classification models.

Abstract

from arXiv · show

Targeted sentiment classification aims at determining the sentimental tendency towards specific targets. Most of the previous approaches model context and target words with RNN and attention. However, RNNs are difficult to parallelize and truncated backpropagation through time brings difficulty in remembering long-term patterns. To address this issue, this paper proposes an Attentional Encoder Network (AEN) which eschews recurrence and employs attention based encoders for the modeling between context and target. We raise the label unreliability issue and introduce label smoothing regularization. We also apply pre-trained BERT to this task and obtain new state-of-the-art results. Experiments and analysis demonstrate the effectiveness and lightweight of our model.

1 Introduction

Targeted sentiment classification determines sentiment polarity for explicit opinion targets, but prior methods face limitations from recurrent encoders and unreliable neutral labels. The paper proposes attention-based encoders, label smoothing, and BERT-based variants, reporting state-of-the-art results and lightweight models.

  • Motivation: Targeted sentiment classification assigns sentiment polarities to opinion targets explicitly appearing in a sentence.Targets are usually entities or entity aspects, and one sentence can express different polarities toward different targets.
  • Motivation: Previous attention-based approaches depend on RNN encoders, which are difficult to parallelize and hinder long-range dependency learning through truncated BPTT.The paper also notes that maintaining long-distance information with LSTMs usually requires substantial training data.
  • Motivation: Neutral sentiment labels can be unreliable because neutral sentiment is fuzzy, motivating label smoothing regularization during training.The regularization encourages the model to be less confident in training labels.
  • Contributions: The proposed Attentional Encoder Network uses attention instead of recurrence to model introspective and interactive semantics between target and context words.Its first contribution is designing an attentional encoder network to draw hidden states and semantic interactions.
  • Contributions: Applying pre-trained BERT enhances the basic BERT model and produces new state-of-the-art results for targeted sentiment classification.The paper reports that experiments on three benchmark datasets show competitive performance.
  • Contributions: The paper evaluates model sizes and presents the proposed model as a lightweight alternative to the best RNN-based models.The conclusion characterizes the proposed model as effective and lightweight.

2 Related Work

Related work progresses from handcrafted feature-based classifiers to neural models that learn representations and incorporate target information through recurrent encoders and attention.

  • Traditional Methods: Traditional methods use rule-based or statistical approaches with sentiment lexicons and bag-of-words features, making performance dependent on labor-intensive feature engineering.These methods focus on extracting feature sets before training a sentiment classifier.
  • Neural Methods: Neural methods avoid handcrafted features by encoding sentences with low-dimensional word vectors containing semantic information.They became increasingly prominent for targeted sentiment classification.
  • Target-Aware Models: TD-LSTM models the target’s left and right contexts with two single-directional LSTMs.This explicitly incorporates target position into recurrent context modeling.
  • Target-Aware Models: MemNet uses multi-hop attention over an external memory to capture the importance of each context word for a given target.Repeated attention builds higher-level semantic information from word embeddings.
  • Target-Aware Models: ATAE-LSTM lets target embeddings participate in attention computation, while RAM applies multiple attention over BiLSTM memory and combines results with GRUs.These approaches extend recurrent architectures with target-aware attention mechanisms.

3 Proposed Methodology

The proposed methodology represents context and target sequences and predicts the target’s sentiment polarity using an Attentional Encoder Network architecture.

  • Task Formulation: The model takes a context sequence and a target subsequence, then predicts the sentence’s sentiment polarity toward that target.The target is defined as a subsequence of the context.
  • Architecture: AEN comprises an embedding layer, attentional encoder layer, target-specific attention layer, and output layer.The embedding layer supports both GloVe and BERT variants named AEN-GloVe and AEN-BERT.

3.1 Embedding Layer

The embedding layer maps context and target words into vectors using either pre-trained GloVe or pre-trained BERT representations.

  • GloVe Embedding: GloVe represents each word as its corresponding column vector from a pre-trained embedding matrix.The matrix has dimensions R^d_emb×|V|, where d_emb is the word-vector dimension and |V| is vocabulary size.
  • BERT Embedding: BERT generates word vectors for sequences after formatting context and target as separate [CLS]-wrapped inputs.The sequences are transformed into “[CLS] + context + [SEP]” and “[CLS] + target + [SEP]”.

3.2 Attentional Encoder Layer

The attentional encoder layer replaces recurrent sequence encoding with parallelizable attention and point-wise transformations, modeling both context internally and target words relative to context.

  • The attentional encoder layer is a parallelizable alternative to LSTM for computing hidden states of input embeddings.
  • Multi-Head Attention: Multi-Head Attention performs multiple attention functions in parallel to learn semantic alignments.Its outputs are concatenated and projected to the specified hidden dimension.
  • Multi-Head Attention: Intra-MHA models introspective context relations, while Inter-MHA models target words according to the given context.Inter-MHA maps context and target embeddings into context-perceptive target representations.
  • Multi-Head Attention: An attention function maps key and query sequences to outputs using an alignment function that learns their semantic relevance.
  • Point-wise Convolution Transformation: Point-wise Convolution Transformation applies the same kernel-size-1 transformation to every token, transforming contextual information gathered by attention.The transformation uses an ELU activation and convolutional kernels.
  • Point-wise Convolution Transformation: PCTs transform the introspective context and context-perceptive target representations into the attentional encoder’s output hidden states.

3.3 Target-specific Attention Layer

The target-specific attention layer applies another multi-head attention module to obtain a context representation tailored to the target.

  • Another multi-head attention module obtains the target-specific context representation from the previous context and target representations.
  • This multi-head attention module has independent parameters.

3.4 Output Layer

The output layer averages and concatenates the preceding representations, then projects the resulting comprehensive vector into the targeted sentiment-class space.

  • Average pooling produces final representations that are concatenated into a comprehensive representation.
  • A fully connected layer projects the concatenated representation into the space of the targeted C classes.
  • The output parameters include learnable weights and biases for predicting a sentiment polarity distribution y ∈ R^C.

3.5 Regularization and Model Training

The training objective addresses unreliable neutral labels with label smoothing regularization and combines cross-entropy, smoothing, and L2 regularization.

  • Neutral-sentiment training samples are considered unreliable because neutral sentiment is fuzzy, motivating label smoothing regularization.The regularizer penalizes low-entropy output distributions and prevents full probability assignment during training.
  • Label Smoothing: Label smoothing replaces the original ground-truth label distribution with a smoothed distribution.
  • Label Smoothing: The prior label distribution is set uniformly as u(k) = 1/C.
  • Label Smoothing: The label-smoothing term is equivalent to the KL divergence between the prior label distribution and the network’s predicted distribution.
  • Model Training: The optimized objective combines cross-entropy loss with label-smoothing and L2 regularization.The ground truth is one-hot, y is the predicted sentiment distribution, λ weights L2 regularization, and Θ denotes the parameter set.

4 Experiments

Experiments on three benchmark datasets compare AEN with recurrent, non-recurrent, and BERT-based baselines, including ablations and model-size analysis. AEN-BERT improves on basic BERT, while AEN-GloVe remains a lightweight alternative to recurrent models.

  • Datasets and settings: Three datasets—Restaurant, Laptop, and Twitter—contain positive, neutral, and negative target-sentiment labels.The experiments use SemEval 2014 Restaurant and Laptop reviews and ACL 14 Twitter data.
  • Experimental design: AEN is compared with seven baselines, four AEN-GloVe ablations, and a basic BERT-based model.The ablations remove label smoothing, the attentional encoder components, or replace the attentional encoder with bidirectional LSTMs.
  • Main results: BERT-SPC and AEN-BERT obtain substantial accuracy improvements, with AEN-BERT outperforming BERT-SPC overall.The comparison suggests that task-specific downstream design and fine-tuning are important because pre-trained BERT knowledge is not domain-specific.
  • Main results: Attention-based ATAE-LSTM, IAN, and RAM consistently exceed TD-LSTM on Restaurant and Laptop, while RAM performs less well on Twitter.The passage attributes RAM’s Twitter weakness possibly to bidirectional LSTM modeling of short, ungrammatical text.
  • Baseline analysis: Feature-based SVM remains competitive but depends on manually designed features, whereas Rec-NN performs worst among neural baselines and MemNet lacks hidden-semantic modeling.The discussion links Rec-NN’s weakness to dependency parsing on ungrammatical short texts and MemNet’s weakness to linear combinations of embeddings.
  • Ablation analysis: Removing any tested AEN-GloVe component reduces performance, and removing label smoothing significantly lowers accuracy across all three datasets.AEN-GloVe and its BiLSTM variant have relatively close performance, with AEN-GloVe better on Restaurant.
  • Model analysis: AEN-GloVe ranks second in lightweight level, while AEN-GloVe-BiLSTM exceeds its model size by more than twofold without performance improvements.Model-size measurements use the Restaurant dataset under the same infrastructure, hyperparameters, and GPU.

5 Conclusion

The paper proposes AEN for targeted sentiment classification by using attention-based encoders to model context and targets. It adds label smoothing for fuzzy labels, applies pre-trained BERT, and reports state-of-the-art results alongside effectiveness and lightweightness.

  • AEN uses attention-based encoders to model interactions between context and target words in targeted sentiment classification.
  • Label smoothing regularization addresses label unreliability by encouraging less confidence on fuzzy labels.
  • Applying pre-trained BERT produces new state-of-the-art results, while experiments demonstrate the proposed model’s effectiveness and lightweightness.
Loading 1902.09314v2…