Source-linked AI summary
Fast and accurate sentiment classification using an enhanced Naive Bayes model
Vivek Narayanan, Ishan Arora, Arjun Bhatia
TL;DR
The paper addresses how to improve sentiment classification with a fast, simple Naïve Bayes model. It combines feature and representation choices, including negation handling and mutual-information feature selection, and reports 88.80% accuracy on IMDb movie reviews with linear training and testing complexity.
Problem
Sentiment classification requires accurate models, while Naïve Bayes was considered less accurate than more sophisticated alternatives despite its speed and simplicity.
Method
The paper enhances Naïve Bayes using conditional independence, Bernoulli word-presence features, negation handling, and mutual-information feature selection.
Results
88.80% accuracy was obtained on the 25,000-review IMDb test set, with O(n + V lg V) training time and O(n) testing time.
Takeaways & Limitations
A simple Naïve Bayes classifier can match the accuracy of more complicated sentiment-classification models while remaining fast to train and scalable to large datasets.
Takeaways & Limitations
Negated forms may be insufficient when strongly polarized words appear only in normal form during training, motivating bootstrapping of negated forms.
Abstract
from arXiv · showhide
We have explored different methods of improving the accuracy of a Naive Bayes classifier for sentiment analysis. We observed that a combination of methods like negation handling, word n-grams and feature selection by mutual information results in a significant improvement in accuracy. This implies that a highly accurate and fast sentiment classifier can be built using a simple Naive Bayes model that has linear training and testing time complexities. We achieved an accuracy of 88.80% on the popular IMDB movie reviews dataset.
1 Introduction
The paper presents a supervised sentiment-classification model based on Naïve Bayes. It argues that Naïve Bayes can combine strong text-classification performance with substantially faster training than more complex models.
- The paper presents a supervised sentiment classification model based on the Naïve Bayes algorithm.
- Naïve Bayes is a simple probabilistic model that works well for text classification.
- Naïve Bayes usually takes orders of magnitude less time to train than models such as support vector machines.
- The paper aims to show that Naïve Bayes can achieve accuracy comparable to current state-of-the-art sentiment-classification models.
2 Data
The experiments use the balanced IMDb movie-review dataset, with separate training and test sets of 25,000 reviews each. The dataset was selected because it captures varied human emotions and supports benchmarking against existing sentiment-classification research.
- The IMDb dataset contains 25,000 highly polar movie reviews for training and 25,000 for testing.
- Both the training and test sets contain equal numbers of positive and negative reviews.
- Movie reviews were chosen because they cover varied human emotions and are widely used for sentiment-classification benchmarking.
- The 25,000 training documents build the supervised model, while the other 25,000 evaluate classifier accuracy.
3 Naïve Bayes Classifier
The classifier uses Naïve Bayes with conditional independence, word-frequency storage, maximum-posterior prediction, and Bernoulli presence features. Removing duplicate words makes the representation depend on word presence rather than counts.
- Naïve Bayes combines Bayes rule with a strong conditional-independence assumption.
- Given a positive or negative class, the model treats document words as conditionally independent, enabling fast classification.
- Word frequency counts are stored in hash tables during training.
- The classifier assigns each document to the class with the maximum posterior probability.
- Bernoulli Naïve Bayes removes duplicate words and uses word presence instead of counts.
4 Laplacian Smoothing
Laplacian smoothing addresses zero probabilities when unseen words occur in a document. With Bernoulli Naïve Bayes, class word totals are computed from sets of unique words, and k is usually set to 1.
- An unseen training word can make both class probabilities zero, leaving no class comparison.
- Laplacian smoothing solves the zero-probability problem for words absent from the training set.
- The smoothing parameter k is usually set to 1, giving the new word equal probability in either class.
- For Bernoulli Naïve Bayes, each document is reduced to unique words when computing the total number of words in a class.
5 Negation Handling
The paper addresses negation by transforming words within negated contexts into distinct features and bootstrapping those forms during training. This approach handles polarity reversals that ordinary word features miss and improves classification accuracy.
- Negation can reverse a word’s sentiment, so treating “good” in “not good” as positive misclassifies the phrase.
- The algorithm tracks negation with a state variable and prefixes affected words with “not_” until punctuation resets the state.
- A limitation is that strong sentiment words may appear only in normal form during training, leaving their negated forms underrepresented.
- Training bootstraps negated features by adding each observed negated form to the opposite class alongside normal feature counts.
- This bootstrapping modification produced a significant accuracy improvement of about 1%.
7 Feature Selection
Feature selection reduces the large, noisy feature space created by bigrams and trigrams. Mutual information ranks features by their class-disambiguation value, with the validation analysis selecting 32,000 features.
- Feature selection removes redundant features while retaining those with high disambiguation capability.
- Adding bigrams and trigrams increases the feature space substantially, creating efficiency and accuracy concerns because many features are redundant or noisy.
- 7.1 Mutual Information: Mutual information evaluates the relationship between an individual feature and the positive or negative class.
- 7.1 Mutual Information: The classifier selects the top k features by mutual information, and validation identified 32,000 as the optimal k.
8 Results
The enhanced classifier was evaluated on IMDb movie reviews after preprocessing, negation handling, and feature selection. It achieved 88.80% test accuracy while retaining linear testing complexity and faster training than more complex alternatives.
- Training preprocesses the data, applies negation handling, and counts each word once per document under Bernoulli Naive Bayes.On the reported laptop, training took around 1 minute 30 seconds and used about 700 megabytes of memory.
- A validation set of 1000 documents selected the feature count, after which accuracy was measured on the full 25,000-document test set.Feature selection took about 3 minutes.
- 88.80% overall classification accuracy was obtained on the 25,000-review IMDb test set.
- The algorithm requires O(n + V lg V) training time and O(n) testing time, where V is the reduced vocabulary size.
9 Conclusion
The results show that Naive Bayes can reach high sentiment-classification accuracy when appropriate features are selected and noise is removed. Its speed, scalability, robustness, and implementation simplicity remain practical advantages.
- Choosing suitable features and applying feature selection can enhance simple Naive Bayes to match more complicated sentiment-classification models.
- Naive Bayes remains extremely fast to train, scalable to large datasets, robust to noise, and less prone to overfitting.
- The paper reports high accuracy despite Naive Bayes’ conditional-independence assumption and emphasizes its ease of implementation.
- The same ideas may also apply to the broader domain of text classification.