PaperlayerView exact revision on arXiv

Source-linked AI summary

SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis

Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang, Feng Wu

arXiv:2005.05635v2cs.CL

TL;DR

Existing pre-training methods generally do not explicitly incorporate sentiment knowledge, although sentiment analysis tasks depend on sentiment words, polarity, and aspect-sentiment pairs. SKEP mines this knowledge, masks it from inputs, and trains three prediction objectives to recover it in a unified sentiment representation. It significantly outperforms RoBERTa across three typical sentiment tasks and reaches new state-of-the-art results on most test datasets.

  • Problem

    Pre-training rarely models sentiment information explicitly, despite sentiment analysis tasks relying on sentiment words, polarity, and aspect-sentiment pairs.

  • Method

    SKEP automatically mines sentiment knowledge, masks sentiment information from inputs, and jointly trains three objectives to recover words, polarity, and aspect-sentiment pairs.

  • Results

    SKEP significantly outperforms RoBERTa on three typical sentiment tasks and achieves new state-of-the-art results on most test datasets.

  • Takeaways & Limitations

    SKEP provides a unified sentiment representation for a wide range of sentiment analysis tasks and supports incorporating sentiment knowledge into pre-training.

  • Takeaways & Limitations

    The authors identify future work in applying SKEP to more tasks, exploiting more sentiment knowledge types, and using more fine-grained sentiment mining methods.

Abstract

from arXiv · show

Recently, sentiment analysis has seen remarkable advance with the help of pre-training approaches. However, sentiment knowledge, such as sentiment words and aspect-sentiment pairs, is ignored in the process of pre-training, despite the fact that they are widely used in traditional sentiment analysis approaches. In this paper, we introduce Sentiment Knowledge Enhanced Pre-training (SKEP) in order to learn a unified sentiment representation for multiple sentiment analysis tasks. With the help of automatically-mined knowledge, SKEP conducts sentiment masking and constructs three sentiment knowledge prediction objectives, so as to embed sentiment information at the word, polarity and aspect level into pre-trained sentiment representation. In particular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair. Experiments on three kinds of sentiment tasks show that SKEP significantly outperforms strong pre-training baseline, and achieves new state-of-the-art results on most of the test datasets. We release our code at https://github.com/baidu/Senta.

1 Introduction

Sentiment analysis spans multiple tasks that rely on sentiment words, polarity, and aspect-sentiment pairs, but standard pre-training largely emphasizes general word-level representations. SKEP incorporates these knowledge types through sentiment masking and jointly optimized prediction objectives, outperforming RoBERTa across sentiment tasks.

  • Motivation: Pre-training methods improve NLP tasks but rarely model sentiment information explicitly, limiting their expected suitability for sentiment analysis.Standard objectives focus on word-level masking, next-word prediction, or permutation.
  • Motivation: Sentiment tasks depend on sentiment words, word polarity, and aspect-sentiment pairs, which traditional approaches often study separately.These knowledge types support sentence-level classification, aspect-level classification, and opinion extraction.
  • Approach: SKEP guides pre-training with automatically mined sentiment knowledge to learn a unified representation for multiple sentiment analysis tasks.The approach targets sentiment words, polarity, and aspect-sentiment pairs.
  • Approach: Sentiment masking removes sentiment information from inputs, while three jointly optimized objectives train the model to recover it.The objectives predict sentiment words, word polarity, and aspect-sentiment pairs.
  • Results: SKEP significantly outperforms RoBERTa on three typical sentiment tasks and achieves new state-of-the-art results on most test datasets.The reported comparison uses RoBERTa as a strong pre-training baseline.

2 Background: BERT and RoBERTa

BERT pre-trains a transformer encoder with masked language modeling on unlabeled data, while RoBERTa strengthens the optimization and removes next sentence prediction. These models provide general contextual representations that are later fine-tuned for downstream tasks.

  • BERT: BERT is a self-supervised approach that pre-trains a deep transformer encoder using masked language modeling and large-scale unlabeled data.Downstream tasks are improved by fine-tuning the pretrained transformer on task-specific labeled data.
  • BERT: BERT produces contextual token representations, with the final hidden state of [CLS] often serving as the overall input representation during fine-tuning.The [CLS] token is the first special token in an input sequence.
  • BERT: Masked language modeling randomly replaces 15% of input tokens and trains the model to predict the original tokens from the corrupted sequence.Among sampled tokens, 80% are replaced with [MASK], 10% with random tokens, and 10% remain unchanged.
  • RoBERTa: RoBERTa significantly outperforms BERT through robust optimization without changing the neural structure and removes BERT’s next sentence prediction objective.The paper uses RoBERTa as a strong baseline for evaluating SKEP.

3 SKEP: Sentiment Knowledge Enhanced Pre-training

SKEP incorporates automatically mined sentiment knowledge into pre-training by masking sentiment information and jointly recovering words, polarity, and aspect-sentiment pairs. Its multi-label pair objective captures dependencies between aspect and sentiment words, while the mining approach remains relatively simple and coarse-grained.

  • Overview: SKEP uses automatically mined sentiment knowledge to guide sentiment masking and three self-supervised pre-training objectives.The objectives recover sentiment words, word polarity, and aspect-sentiment pairs from corrupted input sequences.
  • Unsupervised Sentiment Knowledge Mining: The mining procedure uses seed words, part-of-speech patterns, PMI scores, and polarity differences to identify sentiment words.A candidate is positive when its polarity score exceeds zero; otherwise, it is negative.
  • Unsupervised Sentiment Knowledge Mining: Aspect-sentiment pairs connect a sentiment word with its nearest noun, with the word distance limited to 3 tokens.The mined knowledge contains sentiment words with polarity and aspect-sentiment pairs.
  • Scope: The authors state that a more fine-grained knowledge-mining method could further improve the quality of the mined knowledge.The current study focuses on testing sentiment knowledge integration using a relatively common mining method.
  • Sentiment Masking: Hybrid masking prioritizes aspect-sentiment pairs and sentiment words, then fills any remaining masking budget with randomly selected common tokens.At most two pairs are masked together, sentiment-word masking stays below 10%, and common-token masking follows RoBERTa when needed.
  • Sentiment Pre-training Objectives: The sentiment word objective predicts masked sentiment tokens with a vocabulary softmax, while polarity prediction classifies each masked sentiment token as positive or negative.Unlike standard random masking, sentiment word prediction specifically selects sentiment words for self-supervision.
  • Sentiment Pre-training Objectives: The aspect-sentiment objective uses [CLS] with sigmoid multi-label classification because multiple vocabulary tokens can represent one pair.This design captures dependencies between aspect and sentiment words rather than treating pair tokens as mutually exclusive.

4 Fine-tuning for Sentiment Analysis

SKEP is evaluated after task-specific fine-tuning on sentence-level classification, aspect-level classification, and opinion role labeling. The experiments use task-specific output layers and datasets covering classification and opinion extraction settings.

  • Tasks: SKEP is evaluated on sentence-level sentiment classification, aspect-level sentiment classification, and opinion role labeling.The encoder is fine-tuned with labeled data for each downstream task.
  • Sentence-level Sentiment Classification: Sentence-level classification predicts input-sentence polarity from the final [CLS] representation and an added classification layer.The classification layer calculates sentiment probabilities from the overall sentence representation.
  • Aspect-level Sentiment Classification: Aspect-level classification combines an aspect description and contextual text with [SEP], then uses the first [CLS] representation for classification.This setup targets fine-grained sentiment for a specified aspect.
  • Opinion Role Labeling: Opinion role labeling converts fine-grained opinion extraction into BIOS sequence labeling with a CRF layer for holder and target prediction.The task detects opinion holders and targets from input text.

5 Experiment

SKEP is evaluated across sentence-level, aspect-level, and opinion role labeling tasks against RoBERTa and prior state-of-the-art systems. Results show consistent gains, with sentiment knowledge and multi-label pair prediction contributing especially on fine-grained tasks.

  • Dataset and Evaluation: The experiments cover sentence-level classification, aspect-level classification, and opinion role labeling, using accuracy or F-1 measures as specified for each dataset.SST-2 and Amazon-2 evaluate sentence-level classification; SemEval 2014 Task 4 evaluates restaurant and laptop aspects; MPQA evaluates holder and target F-1 scores.
  • Main Results: SKEP achieves new state-of-the-art results on almost all datasets, with a less satisfactory result only on SST-2.The comparison includes RoBERTa and previous state-of-the-art systems.
  • Main Results: SKEP consistently improves over RoBERTa in both base and large settings, with gains of up to 2.4 points on RoBERTalarge.The base SKEP model also outperforms RoBERTalarge except on Amazon-2.
  • Detailed Analysis: Further pre-training with sentiment word masking improves performance more than random sub-word masking, while polarity and aspect-sentiment objectives add further gains.Adding the three knowledge objectives improves results by up to about 1.5 points over random token masking.
  • Detailed Analysis: Multi-label aspect-sentiment prediction improves aspect-level classification and opinion role labeling by up to 0.6 points compared with independent word prediction.Independent prediction does not hurt sentence-level classification, which mainly relies on sentiment words.
  • Main Results: SKEP produces appropriate sentiment attention in examined examples, while RoBERTa misses or inaccurately attends to sentiment information.Attention importance is calculated from final-layer weights associated with [CLS].

6 Related Work

Prior sentiment analysis methods use sentiment knowledge, but typically build task-specific mechanisms. Related pre-training work improves general representations through self-supervised objectives, motivating SKEP’s unified sentiment representation.

  • Sentiment Analysis with Knowledge: Sentiment words, polarity, and aspect-sentiment pairs support a wide range of sentiment analysis tasks.These tasks include sentence-level and aspect-level classification, opinion extraction, and emotion analysis.
  • Sentiment Analysis with Knowledge: Earlier approaches often exploit sentiment knowledge separately for particular sentiment analysis tasks.Different mechanisms and knowledge types are commonly designed for different tasks.
  • Sentiment Analysis with Knowledge: SKEP differs by incorporating diverse sentiment knowledge during pre-training to provide a unified representation across sentiment tasks.This extends beyond task-specific uses of sentiment knowledge in prior work.
  • Pre-training Approaches: Pre-training methods learn general semantic representations from large-scale unlabeled data using self-supervised objectives such as masked word prediction.Examples discussed include ELMO, GPT, BERT, XLNet, and RoBERTa.

7 Conclusion

SKEP combines sentiment masking with three sentiment pre-training objectives and is reported to outperform RoBERTa across three typical sentiment-analysis tasks. The authors identify broader task generalization and additional sentiment knowledge as future directions.

  • SKEP significantly outperforms the strong pre-training baseline RoBERTa and achieves new state-of-the-art results on most datasets across three typical sentiment-analysis tasks.
  • Sentiment masking and three sentiment pre-training objectives incorporate varied sentiment knowledge into the pre-training model.
  • SKEP provides a unified sentiment representation for a wide range of sentiment-analysis tasks.
  • Future work will evaluate SKEP on more sentiment-analysis tasks and explore additional sentiment knowledge and finer-grained mining methods.

A Appendix

The appendix describes the construction of 46 sentiment seed words used in the experiment and points to Table 7 for the resulting list.

  • 46 sentiment seed words with polarities were constructed from frequent words in Amazon-2 training data and manually filtered.The process began with 9,750 items and 50 frequent sentiment words; four inappropriate words were removed.
  • Table 7 presents the sentiment seed words used in the experiment.
Loading 2005.05635v2…