Source-linked AI summary
Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews
Grégoire Mesnil, Tomas Mikolov, Marc'Aurelio Ranzato, Yoshua Bengio
TL;DR
Whether advanced sequence models improve over simpler bag-of-words approaches remains unclear. The paper combines generative and discriminative methods and reports a new state-of-the-art result on IMDB movie reviews.
Problem
It remains unclear whether recurrent neural networks significantly improve on bag-of-words and bag-of-n-grams approaches for sentiment classification.
Source: §1 · p. 1Method
The paper combines generative positive- and negative-review language-model likelihood features with discriminative approaches in a weighted ensemble.
Source: §1 · pp. 1–2, §2.4 · p. 3, §4 · p. 4Results
Takeaways & Limitations
The authors provide reproducible code and invite researchers to add their results to the ensemble while pursuing improved sentiment-analysis performance.
Source: §4 · p. 4N-gram language models face data sparsity and high memory requirements, with little data to estimate higher-order probabilities accurately.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Sentiment analysis is a common task in natural language processing that aims to detect polarity of a text document (typically a consumer review). In the simplest settings, we discriminate only between positive and negative sentiment, turning the task into a standard binary classification problem. We compare several ma- chine learning approaches to this problem, and combine them to achieve the best possible results. We show how to use for this task the standard generative lan- guage models, which are slightly complementary to the state of the art techniques. We achieve strong results on a well-known dataset of IMDB movie reviews. Our results are easily reproducible, as we publish also the code needed to repeat the experiments. This should simplify further advance of the state of the art, as other researchers can combine their techniques with ours with little effort.
1 INTRODUCTION
The introduction frames sentiment classification as a task where the benefit of advanced sequence models over simpler representations remains uncertain. It proposes combining complementary approaches, including generative language models, and evaluates the system on binary IMDB review classification.
1 INTRODUCTION
- Sentiment analysis predicts a text's attitude, and in the simplest setting reduces to positive-versus-negative classification.Reviews are a typical target; the passage notes that ratings can indicate reviewer satisfaction.Source: §1 · p. 1
- It remains unclear whether recurrent neural networks provide significant gains over bag-of-words and bag-of-n-grams approaches.The passage cites (Mikolov et al., 2010; Socher et al., 2011) for recurrent models and (Pang & Lee, 2008; Wang & Manning, 2012) for simpler approaches.Source: §1 · p. 1
- Combining complementary models performs better than any individual technique, and the ensemble benefits from adding a generative language model.The authors train separate positive- and negative-review language models and use their likelihood ratio as an additional feature.Source: §1 · p. 1
- 92.57% accuracy on the Stanford IMDB dataset is presented as state-of-the-art performance, with code released to reproduce the experiments.The approach is constrained to binary classification; the authors say it scales to more classes by training one model per class.Source: §1 · p. 2
2 DESCRIPTION OF THE MODELS
This section describes a sentiment system combining generative and discriminative models, including language models, weighted n-gram classifiers, sentence vectors, and a model ensemble.
2.1 GENERATIVE MODEL
- Class-specific generative models use Bayes’ rule to classify documents; n-gram likelihoods multiply context-conditioned word probabilities under a Markov assumption.The approach trains one n-gram language model on positive documents and one on negative documents.Source: §2.1 · p. 2
- The n-gram models use modified Kneser-Ney smoothing, and scoring penalizes out-of-vocabulary words when training vocabularies differ.Higher-order n-grams face data sparsity because possible word combinations grow exponentially, and they require substantial memory.Source: §2.1 · p. 2
2.2 LINEAR CLASSIFICATION OF WEIGHTED N-GRAM FEATURES
- The discriminative baseline uses weighted word and position-independent n-gram counts; NB-SVM reweights counts with positive-versus-negative log ratios.Adding tri-grams improved the implementation’s reported performance by 0.6%.Source: §2.2 · p. 3
2.3 SENTENCE VECTORS
- Sentence vectors represent documents with fixed-size features, which a one-hidden-layer neural network uses for sentiment discrimination.The document descriptor can be formed by averaging representations of its paragraphs.Source: §2.3 · p. 3
2.4 MODEL ENSEMBLE
- The ensemble combines models by taking a weighted geometric mean of their positive-class probabilities, with weights selected by grid search on a validation set.The search quantizes weights from 0 to 1 in increments of 0.1 to avoid overfitting.Source: §2.4 · p. 3
3 RESULTS
On the IMDB movie-review benchmark, the ensemble of RNN, sentence-vector, and NB-SVM models outperformed the reported prior state of the art. The individual generative models were weakest, while the ensemble’s component-removal results indicate that all three models contribute.
3 RESULTS
- The evaluation used 50,000 positive/negative IMDB reviews, with 25,000 for training and the remainder for testing under the Maas et al. (2011) protocol.Code for reproducing the experiments is available.Source: §3 · pp. 3–4
- 92.57% for the RNN, sentence-vector, and NB-SVM ensemble exceeded the 91.22% reported by Wang & Manning (2012).The ensemble interpolates the three models’ scores.Source: Table 3 · p. 4, §3 · p. 4
- Generative models performed worst individually, with RNNs slightly better than n-grams and reweighed bag-of-words the most competitive method.All reported results used a linear classifier.Source: §3 · p. 4, Table 2 · p. 4
- Sentence Vectors alone yielded 88.73% (about 4% below the ensemble); removing the generative model affected performance least, though all three models contributed.Source: §3 · p. 4
4 CONCLUSION
The paper concludes that combining three complementary baseline models achieves state-of-the-art performance on the IMDB movie-review dataset. It also provides code so researchers can reproduce the experiments and incorporate new results into the ensemble.
4 CONCLUSION
- The ensemble combines language models, continuous sentence representations, and reweighted tf-idf bag-of-words; each contributes to the system’s state-of-the-art IMDB performance.Source: §4 · p. 4
- The authors provide code to reproduce the experiments and invite researchers to add their results to the ensemble.Source: §4 · p. 4