PaperlayerView exact revision on arXiv

Source-linked AI summary

Linguistically Regularized LSTMs for Sentiment Classification

Qiao Qian, Minlie Huang, Jinhao Lei, Xiaoyan Zhu

arXiv:1611.03949v2cs.CL

TL;DR

The paper addresses sentence-level sentiment classification when prior models require expensive phrase-level annotation or underuse linguistic resources. It uses simple LSTM sequence models with regularizers for sentiment, negation, and intensity words, achieving competitive performance while modeling their sentiment-shifting effects.

  • Problem

    Prior sentiment models either rely on expensive phrase-level annotation or do not fully use sentiment lexicons, negation words, and intensity words.

  • Method

    The paper imposes linguistic-inspired regularizers on sequence LSTMs to model sentiment, negation, and intensity effects using sentence-level annotation.

  • Results

    The models capture the linguistic roles of sentiment, negation, and intensity words while achieving competitive performance without parsing trees or phrase-level annotation.

  • Takeaways & Limitations

    Linguistically regularized sequence LSTMs provide simple models for sentence-level sentiment classification that incorporate sentiment-shifting effects.

  • Takeaways & Limitations

    The models do not consider the modification scope of negation and intensity words, although bidirectional LSTMs and minimization operators partially address the issue.

Abstract

from arXiv · show

Sentiment understanding has been a long-term goal of AI in the past decades. This paper deals with sentence-level sentiment classification. Though a variety of neural network models have been proposed very recently, however, previous models either depend on expensive phrase-level annotation, whose performance drops substantially when trained with only sentence-level annotation; or do not fully employ linguistic resources (e.g., sentiment lexicons, negation words, intensity words), thus not being able to produce linguistically coherent representations. In this paper, we propose simple models trained with sentence-level annotation, but also attempt to generating linguistically coherent representations by employing regularizers that model the linguistic role of sentiment lexicons, negation words, and intensity words. Results show that our models are effective to capture the sentiment shifting effect of sentiment, negation, and intensity words, while still obtain competitive results without sacrificing the models' simplicity.

1 Introduction

The paper targets sentence-level sentiment classification with simple sequence models that use linguistic resources to model sentiment shifts without phrase-level annotation.

  • 1 Introduction: Previous tree-structured models require parsing trees and expensive phrase-level annotation, while neural models underuse sentiment, negation, and intensity resources.Performance of tree-structured models drops substantially when trained only with sentence-level annotation.
  • 1 Introduction: The paper develops simple sequence models that avoid parsing trees and phrase-level annotation while incorporating sentiment lexicons, negators, and intensifiers.The resources provide polarity information and model sentiment-shifting effects in longer text.
  • 1 Introduction: Linguistic regularization compares predicted sentiment distributions across adjacent positions so words such as negators can shift neighboring sentiment predictions.A negator at the current position should alter the sentiment distribution at the next position.
  • 1 Introduction: The proposed models impose linguistic-inspired regularizers on LSTMs to model sentiment, negation, and intensity effects in sentence-level classification.The approach is intended to enhance classification while preserving sequence-model simplicity.
  • 1 Introduction: The models are simple and efficient, and their performance is reported as on a par with the state of the art without parsing structures or phrase-level annotation.The paper presents this as a contribution relative to previous models.

2 Related Work

Related work spans recursive and sequence neural models, linguistic resources for sentiment analysis, and earlier forms of linguistically motivated regularization.

  • 2 Related Work: Recursive sentiment models build sentence representations from subphrases but usually depend on tree structures and phrase-level annotation, unlike sequence models such as CNNs.This distinction motivates sequence-based approaches that do not require tree-structured data.
  • 2 Related Work: Sentiment lexicons provide prior polarity, while negators modify sentiment and intensity words change the strength or valence of sentiment.These resources support sentiment analysis and fine-grained sentiment prediction.
  • 2 Related Work: Prior negation models include sign reversal, constant sentiment shifts, and negator-specific effects because different negators can affect modified text differently.The related work extends fixed shifting assumptions toward word-specific behavior.
  • 2 Related Work: Earlier work studied intensity prediction and linguistic effects using regression, kernels, learning-to-rank, sentiment parsing, and graded intensity scales.These approaches address sentiment strength or changes caused by modifiers.
  • 2 Related Work: The paper differs from earlier linguistically motivated regularization by applying KL-divergence regularizers to intermediate outputs rather than group-lasso regularizers to logistic-regression parameters.The comparison is made with work using parse trees, topics, and hierarchical word clusters.

3 Long Short-term Memory Network

The paper introduces LSTM and bidirectional LSTM sequence representations, where hidden states encode contextual information for each sentence position.

  • 3 Long Short-term Memory Network: An LSTM computes the current hidden state and memory cell from the previous hidden state, previous memory cell, and current input vector.The hidden state represents the current position while encoding preceding context.
  • 3 Long Short-term Memory Network: In a forward LSTM, each hidden state represents a position while encoding only the preceding context.Backward context is not included in the one-directional representation.
  • 3 Long Short-term Memory Network: A bidirectional LSTM runs parallel forward and backward LSTMs and concatenates their hidden states at each position.This representation incorporates forward and backward contexts simultaneously.
  • 3 Long Short-term Memory Network: The entire bidirectional sentence representation concatenates the final forward state with the initial backward state.At each position, the representation is the concatenation of the corresponding forward and backward hidden states.

4 Linguistically Regularized LSTM

Linguistically Regularized LSTM adds linguistic constraints to adjacent-position sentiment predictions, modeling sentiment words with shifting distributions and negation or intensity words with transformations. Bidirectional context helps handle modification direction, while the models omit explicit modification-span resolution to preserve simplicity.

  • Sentiment Regularizer: Sentiment words receive class-specific shifting distributions so their predicted sentiment differs appropriately from adjacent positions.The shifting distribution is learned for each sentiment class and may instead be made word-specific with sufficiently large datasets.
  • Intensity Regularizer: Intensity words use word-specific transformation matrices to model changes in sentiment strength, such as shifting positive toward very positive.The formulation parallels the negation regularizer while allowing different intensifiers to have different effects.
  • Generic Regularizer: Adjacent-position sentiment distributions are regularized with a new loss term added to the original cross-entropy objective.Non-sentiment neighboring positions are encouraged to remain similar, while linguistic words impose structured changes.
  • Negation Regularizer: Negation words use word-specific transformation matrices because their effects can reverse or otherwise alter polarity depending on the negator and modified expression.For example, “not good” becomes negative whereas “not bad” becomes neutral rather than positive.
  • Limitation: The models do not explicitly model the modification span of negation and intensity words, relying instead on sequence-LSTM context to preserve simplicity.Modification-span resolution is identified as a challenging problem beyond this work’s scope.
  • Bidirectional LSTM: Bidirectional LSTM compares transformed forward and backward neighboring distributions, allowing the model to select the relevant modification direction.A backward LSTM is motivated by the frequent rightward modification of negators and intensifiers, while bidirectionality also covers leftward cases.

5 Experiment

Experiments evaluate linguistically regularized LSTM models on MR and SST using sentence-level annotation, then examine individual regularizers and sentiment shifts caused by negators and intensifiers.

  • 5.3 Overall Comparison: 82.1% vs. 79.3% and 81.5% vs. 77.4% show that LR-Bi-LSTM and LR-LSTM outperform their unregularized counterparts on MR.The authors attribute these gains to linguistic regularizers.
  • 5.3 Overall Comparison: LR-Bi-LSTM is comparable to phrase-annotation Bi-LSTM on SST while using only sentence-level annotation.The models also avoid parsing trees and expensive phrase-level annotation.
  • 5.3 Overall Comparison: On SST, linguistically regularized LSTM and Bi-LSTM outperform their counterparts and the models are better than CNN, DAN, and NCSL.They are comparable to Tree-LSTM while avoiding parsing trees and phrase-level annotation.
  • 5.4 The Effect of Different Regularizers: Removing negation or intensity regularizers significantly reduces performance on subsets containing the corresponding words.Across the full datasets, NSR and SR are key, while negation and intensity regularizers are effective but less important.
  • 5.5 The Effect of the Negation Regularizer: Negators generally shift very positive or very negative phrases toward other polarities, can reverse polarity, and usually preserve neutral sentiment.Examples include not interesting, no good scenes, and not at home.

6 Conclusion and Future Work

The paper presents linguistically regularized sequence LSTMs for sentence-level sentiment classification, capturing shifts caused by sentiment, negation, and intensity words without parsing trees or phrase-level annotation. It also identifies negation and intensity modification scope as an unresolved limitation.

  • Linguistically regularized LSTMs model the sentiment-shifting effects of sentiment, negation, and intensity words for sentence-level classification.
  • The proposed sequence models avoid parsing-tree dependence and expensive phrase-level annotation.
  • Future Work: The models do not fully represent the modification scope of negation and intensity words, although minimization and bidirectional LSTMs partially address it.
Loading 1611.03949v2…