Source-linked AI summary
Bidirectional Encoder Representations from Transformers (BERT): A sentiment analysis odyssey
Shivaji Alaparthi, Manit Mishra
TL;DR
The study addresses a reported lack of comparisons between pretrained BERT and other sentiment-analysis models by evaluating four approaches on labeled IMDB reviews. It reports pretrained BERT as superior.
Problem
To the authors’ knowledge, no prior study compared pretrained BERT with LSTM, logistic regression, and Sent WordNet, leaving their relative sentiment-classification performance unassessed.
Source: Advanced supervised deep learning… · p. 11Method
The study compared Sent WordNet, logistic regression, LSTM, and pretrained BERT on 50,000 labeled IMDB reviews using accuracy, precision, recall, and F1.
Source: Advanced supervised deep learning… · pp. 8–10Results
The comparison found pretrained BERT superior in sentiment classification; LSTM performed worse than logistic regression, while Sent WordNet was inferior to the other models.
Source: Advanced supervised deep learning… · pp. 10–11Takeaways & Limitations
The comparison offers analytics professionals evidence on the relative classification performance of sentiment-analysis techniques on labeled movie reviews.
Source: Advanced supervised deep learning… · pp. 10–11BERT demands substantial computational capability, which constrained training-data size and epochs that might otherwise have strengthened its performance.
Source: Advanced supervised deep learning… · p. 11
Abstract
from arXiv · showhide
The purpose of the study is to investigate the relative effectiveness of four different sentiment analysis techniques: (1) unsupervised lexicon-based model using Sent WordNet; (2) traditional supervised machine learning model using logistic regression; (3) supervised deep learning model using Long Short-Term Memory (LSTM); and, (4) advanced supervised deep learning models using Bidirectional Encoder Representations from Transformers (BERT). We use publicly available labeled corpora of 50,000 movie reviews originally posted on internet movie database (IMDB) for analysis using Sent WordNet lexicon, logistic regression, LSTM, and BERT. The first three models were run on CPU based system whereas BERT was run on GPU based system. The sentiment classification performance was evaluated based on accuracy, precision, recall, and F1 score. The study puts forth two key insights: (1) relative efficacy of four highly advanced and widely used sentiment analysis techniques; (2) undisputed superiority of pre-trained advanced supervised deep learning BERT model in sentiment analysis from text data. This study provides professionals in analytics industry and academicians working on text analysis key insight regarding comparative classification performance evaluation of key sentiment analysis techniques, including the recently developed BERT. This is the first research endeavor to compare the advanced pre-trained supervised deep learning model of BERT vis-\`a-vis other sentiment analysis models of LSTM, logistic regression, and Sent WordNet.
Preprocessing text data
The study treats preprocessing as essential because reviews are unstructured and vary in communication style and grammatical correctness; standardization makes them suitable for sentiment analysis.
Preprocessing text data
- The listed cleaning steps remove HTML tags and special characters and expand contractions, such as changing “haven’t” to “have not.”The HTML example is “<”; special-character examples include “@” and “#”.Source: Preprocessing text data · p. 5
- The process separates text into sentences using punctuation, removes punctuation, and lemmatizes words to their root forms.For example, “succeeding” and “successful” are reduced to “success”.Source: Preprocessing text data · p. 5
- The remaining steps lowercase words and remove rare words appearing in less than 1% of documents and stop words that do not contribute meaning.“Positive” becomes “positive”; “the” is given as a stop-word example.Source: Preprocessing text data · p. 5
- Preprocessing standardizes reviews from different cultures and communication styles, making the corpus amenable to further analysis.The authors note that reviewers have no incentive to be grammatically correct.Source: Preprocessing text data · p. 5
Unsupervised lexicon-based model: Sent WordNet
Sent WordNet is an unsupervised, lexicon-based sentiment analysis technique that uses curated word-level sentiment information and context to score documents.
Unsupervised lexicon-based model: Sent WordNet
- Sentiment lexicons record words’ positive or negative associations, polarity magnitude, parts of speech, and subjectivity.The passage also names TextBlob, AFINN, and VADER as popular lexicons alongside Sent WordNet.Source: Unsupervised lexicon-based model: Sent… · p. 6
- The method matches corpus words to lexicon information and uses context to derive sentiment scores for documents.Source: Unsupervised lexicon-based model: Sent… · p. 6
Supervised machine learning model: Logistic regression
The study uses logistic regression as a supervised binary sentiment classifier, representing reviews as bag-of-words term-document vectors and classifying them by predicted positive-sentiment odds against a cutoff.
Supervised machine learning model: Logistic regression
- Reviews are classified as positive or negative, followed by model validation.Source: Supervised machine learning model:… · p. 6
- The bag-of-words representation ignores context, word order, and grammar, forming a d-by-t term-document matrix for logistic regression.Each review is a row and each unique term is a column.Source: Supervised machine learning model:… · p. 6
- Logistic regression estimates positive-sentiment odds and compares them with a cutoff; its sigmoid function does not require a linearity assumption.The logit model is described as based on utility maximization with a random factor (Nijkamp et al., 1992).Source: Supervised machine learning model:… · pp. 6–7
Supervised deep learning model: LSTM
The section describes LSTM as a recurrent neural network used for sentiment analysis, with a cell and input, output, and forget gates. It also notes reported accuracy advantages over traditional multilayer perceptrons and standard RNNs.
Supervised deep learning model: LSTM
- LSTM is a recurrent neural network architecture applied to sentiment analysis of text data.The section also describes standard LSTM networks as state-of-the-art for a variety of machine-learning problems (Greff et al., 2016, p. 2222).Source: Supervised deep learning model: LSTM · p. 7
- Neural networks are sought after for their nonparametric nature, ability to model complex nonlinear relationships, and capacity to handle missing data.Source: Supervised deep learning model: LSTM · p. 7
- LSTMs are bidirectional networks reported as more accurate than traditional multilayer perceptrons and standard RNNs.The comparison is attributed to Graves and Schmidhuber (2005).Source: Supervised deep learning model: LSTM · p. 7
- LSTM units generally comprise a cell, an input gate, an output gate, and a forget gate.Source: Supervised deep learning model: LSTM · p. 7
Advanced supervised deep learning model: BERT
On 50,000 labeled IMDB movie reviews, the study compared four sentiment-analysis techniques and found pre-trained BERT superior to the alternatives. The authors identify BERT’s bidirectional representations as a possible explanation for its performance.
Advanced supervised deep learning model: BERT
- The dataset comprised 50,000 balanced, labeled IMDB reviews, with 17,500 for BERT training, 7,500 for validation, and 25,000 for testing.Reviews were labeled positive or negative, with no neutral reviews; the authors changed the partitioning because of overfitting risk and BERT’s computational demands.Source: Advanced supervised deep learning… · pp. 8–9
- The study assessed accuracy, precision, recall, and F1 score on validation data or, for BERT, test data.The authors treat values closer to 1 as better for all four measures.Source: Advanced supervised deep learning… · p. 9
- BERT ran on a GPU, while Sent WordNet, logistic regression, and LSTM ran on CPUs; five BERT epochs took 100 minutes.The reported classification outputs are identified as appearing in Table 1, but the supplied passage does not provide its values.Source: Advanced supervised deep learning… · p. 10
- The comparative analysis found pre-trained BERT undisputedly superior to Sent WordNet, logistic regression, and LSTM in sentiment classification.The authors attribute this to BERT conditioning on both left and right context in all layers; they also cite BERT’s 80.5% score on the GLUE benchmark.Source: Advanced supervised deep learning… · p. 10
- LSTM performed worse than logistic regression, while Sent WordNet was inferior to the other models; the authors suggest their differing strengths may explain these results.They suggest LSTM is better suited to sequences such as video, and that Sent WordNet’s unsupervised predictive algorithm may account for its weaker performance.Source: Advanced supervised deep learning… · p. 11
- The authors identify BERT’s computational demands as a limitation and suggest that more computing power could support larger training data and more epochs.They also note that comparing sentiment analysis with labeled and unlabeled data could yield interesting results, and describe their study as the first to compare BERT with the other models.Source: Advanced supervised deep learning… · p. 11