PaperlayerView exact revision on arXiv

Source-linked AI summary

SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis

Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang, Feng Wu

arXiv:2005.05635v2cs.CL

TL;DR

Existing pre-training methods generally do not explicitly incorporate sentiment knowledge, although sentiment analysis tasks depend on sentiment words, polarity, and aspect-sentiment pairs. SKEP mines this knowledge, masks it from inputs, and trains three prediction objectives to recover it in a unified sentiment representation. It significantly outperforms RoBERTa across three typical sentiment tasks and reaches new state-of-the-art results on most test datasets.

  • Problem

    Pre-training rarely models sentiment information explicitly, despite sentiment analysis tasks relying on sentiment words, polarity, and aspect-sentiment pairs.

    Source: §1 · p. 1

  • Method

    SKEP automatically mines sentiment knowledge, masks sentiment information from inputs, and jointly trains three objectives to recover words, polarity, and aspect-sentiment pairs.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    SKEP significantly outperforms RoBERTa on three typical sentiment tasks and achieves new state-of-the-art results on most test datasets.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    SKEP provides a unified sentiment representation for a wide range of sentiment analysis tasks and supports incorporating sentiment knowledge into pre-training.

    Source: §1 · p. 2, §7 · pp. 8–9

    The authors identify future work in applying SKEP to more tasks, exploiting more sentiment knowledge types, and using more fine-grained sentiment mining methods.

    Source: §3.1 · p. 3, §7 · p. 9

Abstract

from arXiv · show

Recently, sentiment analysis has seen remarkable advance with the help of pre-training approaches. However, sentiment knowledge, such as sentiment words and aspect-sentiment pairs, is ignored in the process of pre-training, despite the fact that they are widely used in traditional sentiment analysis approaches. In this paper, we introduce Sentiment Knowledge Enhanced Pre-training (SKEP) in order to learn a unified sentiment representation for multiple sentiment analysis tasks. With the help of automatically-mined knowledge, SKEP conducts sentiment masking and constructs three sentiment knowledge prediction objectives, so as to embed sentiment information at the word, polarity and aspect level into pre-trained sentiment representation. In particular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair. Experiments on three kinds of sentiment tasks show that SKEP significantly outperforms strong pre-training baseline, and achieves new state-of-the-art results on most of the test datasets. We release our code at https://github.com/baidu/Senta.

1 Introduction

Sentiment analysis spans multiple tasks that rely on sentiment words, polarity, and aspect-sentiment pairs, but standard pre-training largely emphasizes general word-level representations. SKEP incorporates these knowledge types through sentiment masking and jointly optimized prediction objectives, outperforming RoBERTa across sentiment tasks.

Motivation

  • Pre-training methods improve NLP tasks but rarely model sentiment information explicitly, limiting their expected suitability for sentiment analysis.Standard objectives focus on word-level masking, next-word prediction, or permutation.Source: §1 · p. 1
  • Sentiment tasks depend on sentiment words, word polarity, and aspect-sentiment pairs, which traditional approaches often study separately.These knowledge types support sentence-level classification, aspect-level classification, and opinion extraction.Source: §1 · p. 1

Approach

  • SKEP guides pre-training with automatically mined sentiment knowledge to learn a unified representation for multiple sentiment analysis tasks.The approach targets sentiment words, polarity, and aspect-sentiment pairs.Source: §1 · pp. 1–2
  • Sentiment masking removes sentiment information from inputs, while three jointly optimized objectives train the model to recover it.The objectives predict sentiment words, word polarity, and aspect-sentiment pairs.Source: §1 · p. 2, Figure 1 · p. 2

Results

  • SKEP significantly outperforms RoBERTa on three typical sentiment tasks and achieves new state-of-the-art results on most test datasets.The reported comparison uses RoBERTa as a strong pre-training baseline.Source: §1 · p. 2

2 Background: BERT and RoBERTa

BERT pre-trains a transformer encoder with masked language modeling on unlabeled data, while RoBERTa strengthens the optimization and removes next sentence prediction. These models provide general contextual representations that are later fine-tuned for downstream tasks.

BERT

  • BERT is a self-supervised approach that pre-trains a deep transformer encoder using masked language modeling and large-scale unlabeled data.Downstream tasks are improved by fine-tuning the pretrained transformer on task-specific labeled data.Source: §2 · p. 2
  • BERT produces contextual token representations, with the final hidden state of [CLS] often serving as the overall input representation during fine-tuning.The [CLS] token is the first special token in an input sequence.Source: §2 · p. 2
  • Masked language modeling randomly replaces 15% of input tokens and trains the model to predict the original tokens from the corrupted sequence.Among sampled tokens, 80% are replaced with [MASK], 10% with random tokens, and 10% remain unchanged.Source: §2 · p. 2

RoBERTa

  • RoBERTa significantly outperforms BERT through robust optimization without changing the neural structure and removes BERT’s next sentence prediction objective.The paper uses RoBERTa as a strong baseline for evaluating SKEP.Source: §2 · pp. 2–3

3 SKEP: Sentiment Knowledge Enhanced Pre-training

SKEP incorporates automatically mined sentiment knowledge into pre-training by masking sentiment information and jointly recovering words, polarity, and aspect-sentiment pairs. Its multi-label pair objective captures dependencies between aspect and sentiment words, while the mining approach remains relatively simple and coarse-grained.

Overview

  • SKEP uses automatically mined sentiment knowledge to guide sentiment masking and three self-supervised pre-training objectives.The objectives recover sentiment words, word polarity, and aspect-sentiment pairs from corrupted input sequences.Source: §3 · p. 3, §3.1 · p. 3

Unsupervised Sentiment Knowledge Mining

  • The mining procedure uses seed words, part-of-speech patterns, PMI scores, and polarity differences to identify sentiment words.A candidate is positive when its polarity score exceeds zero; otherwise, it is negative.Source: §3.1 · p. 3
  • Aspect-sentiment pairs connect a sentiment word with its nearest noun, with the word distance limited to 3 tokens.The mined knowledge contains sentiment words with polarity and aspect-sentiment pairs.Source: §3.1 · p. 3

Scope

  • The authors state that a more fine-grained knowledge-mining method could further improve the quality of the mined knowledge.The current study focuses on testing sentiment knowledge integration using a relatively common mining method.Source: §3.1 · p. 3

Sentiment Masking

  • Hybrid masking prioritizes aspect-sentiment pairs and sentiment words, then fills any remaining masking budget with randomly selected common tokens.At most two pairs are masked together, sentiment-word masking stays below 10%, and common-token masking follows RoBERTa when needed.Source: §3.2 · p. 4

Sentiment Pre-training Objectives

  • The sentiment word objective predicts masked sentiment tokens with a vocabulary softmax, while polarity prediction classifies each masked sentiment token as positive or negative.Unlike standard random masking, sentiment word prediction specifically selects sentiment words for self-supervision.Source: §3.3 · p. 4
  • The aspect-sentiment objective uses [CLS] with sigmoid multi-label classification because multiple vocabulary tokens can represent one pair.This design captures dependencies between aspect and sentiment words rather than treating pair tokens as mutually exclusive.Source: §3.3 · pp. 4–5

4 Fine-tuning for Sentiment Analysis

SKEP is evaluated after task-specific fine-tuning on sentence-level classification, aspect-level classification, and opinion role labeling. The experiments use task-specific output layers and datasets covering classification and opinion extraction settings.

Tasks

  • SKEP is evaluated on sentence-level sentiment classification, aspect-level sentiment classification, and opinion role labeling.The encoder is fine-tuned with labeled data for each downstream task.Source: §4 · p. 5

Sentence-level Sentiment Classification

  • Sentence-level classification predicts input-sentence polarity from the final [CLS] representation and an added classification layer.The classification layer calculates sentiment probabilities from the overall sentence representation.Source: §4 · p. 5

Aspect-level Sentiment Classification

  • Aspect-level classification combines an aspect description and contextual text with [SEP], then uses the first [CLS] representation for classification.This setup targets fine-grained sentiment for a specified aspect.Source: §4 · p. 5

Opinion Role Labeling

  • Opinion role labeling converts fine-grained opinion extraction into BIOS sequence labeling with a CRF layer for holder and target prediction.The task detects opinion holders and targets from input text.Source: §4 · p. 5

5 Experiment

SKEP is evaluated across sentence-level, aspect-level, and opinion role labeling tasks against RoBERTa and prior state-of-the-art systems. Results show consistent gains, with sentiment knowledge and multi-label pair prediction contributing especially on fine-grained tasks.

Dataset and Evaluation

  • The experiments cover sentence-level classification, aspect-level classification, and opinion role labeling, using accuracy or F-1 measures as specified for each dataset.SST-2 and Amazon-2 evaluate sentence-level classification; SemEval 2014 Task 4 evaluates restaurant and laptop aspects; MPQA evaluates holder and target F-1 scores.Source: §5.1 · pp. 5–6

Main Results

  • SKEP achieves new state-of-the-art results on almost all datasets, with a less satisfactory result only on SST-2.The comparison includes RoBERTa and previous state-of-the-art systems.Source: §5.3 · pp. 6–7, Table 3 · p. 6
  • SKEP consistently improves over RoBERTa in both base and large settings, with gains of up to 2.4 points on RoBERTalarge.The base SKEP model also outperforms RoBERTalarge except on Amazon-2.Source: §5.3 · pp. 6–7

Detailed Analysis

  • Further pre-training with sentiment word masking improves performance more than random sub-word masking, while polarity and aspect-sentiment objectives add further gains.Adding the three knowledge objectives improves results by up to about 1.5 points over random token masking.Source: Table 4 · p. 6, §5.4 · p. 7
  • Multi-label aspect-sentiment prediction improves aspect-level classification and opinion role labeling by up to 0.6 points compared with independent word prediction.Independent prediction does not hurt sentence-level classification, which mainly relies on sentiment words.Source: §5.4 · p. 7

Main Results

  • SKEP produces appropriate sentiment attention in examined examples, while RoBERTa misses or inaccurately attends to sentiment information.Attention importance is calculated from final-layer weights associated with [CLS].Source: Table 5 · p. 7, §5.4 · p. 8

6 Related Work

Prior sentiment analysis methods use sentiment knowledge, but typically build task-specific mechanisms. Related pre-training work improves general representations through self-supervised objectives, motivating SKEP’s unified sentiment representation.

Sentiment Analysis with Knowledge

  • Sentiment words, polarity, and aspect-sentiment pairs support a wide range of sentiment analysis tasks.These tasks include sentence-level and aspect-level classification, opinion extraction, and emotion analysis.Source: §6 · p. 8
  • Earlier approaches often exploit sentiment knowledge separately for particular sentiment analysis tasks.Different mechanisms and knowledge types are commonly designed for different tasks.Source: §6 · p. 8
  • SKEP differs by incorporating diverse sentiment knowledge during pre-training to provide a unified representation across sentiment tasks.This extends beyond task-specific uses of sentiment knowledge in prior work.Source: §6 · p. 8

Pre-training Approaches

  • Pre-training methods learn general semantic representations from large-scale unlabeled data using self-supervised objectives such as masked word prediction.Examples discussed include ELMO, GPT, BERT, XLNet, and RoBERTa.Source: §6 · p. 8

7 Conclusion

SKEP combines sentiment masking with three sentiment pre-training objectives and is reported to outperform RoBERTa across three typical sentiment-analysis tasks. The authors identify broader task generalization and additional sentiment knowledge as future directions.

  • SKEP significantly outperforms the strong pre-training baseline RoBERTa and achieves new state-of-the-art results on most datasets across three typical sentiment-analysis tasks.Source: §7 · p. 8
  • Sentiment masking and three sentiment pre-training objectives incorporate varied sentiment knowledge into the pre-training model.Source: §7 · p. 8
  • SKEP provides a unified sentiment representation for a wide range of sentiment-analysis tasks.Source: §7 · pp. 8–9
  • Future work will evaluate SKEP on more sentiment-analysis tasks and explore additional sentiment knowledge and finer-grained mining methods.Source: §7 · p. 9

A Appendix

The appendix describes the construction of 46 sentiment seed words used in the experiment and points to Table 7 for the resulting list.

  • 46 sentiment seed words with polarities were constructed from frequent words in Amazon-2 training data and manually filtered.The process began with 9,750 items and 50 frequent sentiment words; four inappropriate words were removed.Source: A Appendix · p. 10
  • Table 7 presents the sentiment seed words used in the experiment.Source: Table 7 · p. 10, A Appendix · p. 10

Related papers

Loading 2005.05635v2…