Source-linked AI summary
Fine-grained Sentiment Classification using BERT
Manish Munikar, Sushil Shakya, Aakash Shrestha
TL;DR
Most sentiment-classification research has focused on binary labels, leaving the use of BERT for fine-grained SST-5 less explored. The paper fine-tunes pretrained BERT with a simple dropout-and-softmax downstream architecture and reports better accuracy than many popular, sophisticated models, demonstrating transfer learning in NLP.
Problem
Most sentiment-classification approaches have focused on binary tasks, and the authors had not found BERT experiments on the fine-grained SST-5 dataset.
Method
The paper fine-tunes pretrained BERT for SST sentiment classification using dropout regularization and a softmax classifier on top.
Results
The simple BERT-based model performs better in accuracy than many popular and sophisticated NLP models on SST comparisons.
Takeaways & Limitations
The findings demonstrate transfer-learning capability in NLP using deep contextual language models such as BERT.
Abstract
from arXiv · showhide
Sentiment classification is an important process in understanding people's perception towards a product, service, or topic. Many natural language processing models have been proposed to solve the sentiment classification problem. However, most of them have focused on binary sentiment classification. In this paper, we use a promising deep learning model called BERT to solve the fine-grained sentiment classification task. Experiments show that our model outperforms other popular models for this task without sophisticated architecture. We also demonstrate the effectiveness of transfer learning in natural language processing in the process.
I. INTRODUCTION
The paper applies pretrained BERT to fine-grained sentiment classification, addressing a task that extends beyond predominantly binary sentiment research. It presents this approach within broader NLP progress toward transfer learning.
- Fine-grained sentiment classification assigns text to five classes: very negative, negative, neutral, positive, and very positive.
- The task requires converting variable-length text into a fixed-sized numerical vector that preserves meaningful information.
- NLP has begun adopting transfer learning through pretrained representations, but the paper describes its use as less fully developed than in computer vision.
- The study uses pretrained BERT and fine-tunes it for fine-grained sentiment classification on the Stanford Sentiment Treebank dataset.
II. MOTIVATION
The motivation is to test BERT on the five-class SST-5 task, where prior BERT experiments had focused on binary sentiment classification. This extends ongoing efforts to represent and classify sentiment in context.
- Prior BERT sentiment experiments had achieved state-of-the-art results on the binary SST-2 dataset, while the authors had not found experiments on fine-grained SST-5.
- Sentiment systems first learn word embeddings, then combine variable numbers of word vectors into fixed-size document representations.
- Earlier approaches include continuous word representations, co-occurrence-based embeddings, character n-grams, recursive composition, and contextual language-model features.
IV. DATASET
The study evaluates sentiment models on the Stanford Sentiment Treebank, a manually labeled, tree-structured movie-review dataset. Evaluation covers both binary and five-class settings and both all nodes and root nodes.
- The Stanford Sentiment Treebank contains 11,855 one-sentence movie reviews and 215,154 unique manually labeled texts across parse-tree nodes.
- SST sentences are parsed into constituency trees whose root is the whole sentence and whose leaves are individual words.
- SST labels are 0 very negative, 1 negative, 2 neutral, 3 positive, and 4 very positive; SST-2 uses only positivity and negativity.
- Models are evaluated on all tree nodes and only root nodes, using both SST-2 and SST-5.
A. BERT
BERT learns deep bidirectional representations by conditioning on both left and right context, using masked-word and next-sentence pretraining objectives. Its Transformer processes input tokens in parallel and exposes a sequence embedding for classification.
- BERT trains deep bidirectional representations by jointly conditioning on left and right context throughout its layers.
- Masked-word prediction hides 15% of input words and trains the encoder to predict them from the full sequence.
- Next-sentence prediction classifies whether sentence B follows sentence A or is randomly paired with it.
- Unlike sequential or recurrent models, BERT’s attention architecture processes the whole input sequence at once, enabling parallel token processing.
- BERT input sequences use [CLS] first and [SEP] after each sentence; the [CLS] output embedding represents the sequence for classification.
- Figure 3 maps each input token E_n through Transformer blocks Trm to its corresponding output embedding T_n.
B. Preprocessing
The review text is preprocessed before being fed into the model.
- The review text undergoes preprocessing before entering the model.
1) Canonicalization:
The pipeline canonicalizes reviews, tokenizes them, adds boundary tokens, and applies a simple BERT-based classification architecture.
- Canonicalization:: The preprocessing removes digits, punctuation, and accent marks, then lowercases the text.
- Canonicalization:: Word-Piece tokenization decomposes words into subword components to handle unseen words better.For example, playing becomes play + ##ing.
- Canonicalization:: The pipeline adds [CLS] and [SEP] tokens at the appropriate positions.
- Proposed Architecture: A dropout layer and softmax classifier are placed on pretrained BERT without a sophisticated task-specific architecture.Dropout regularization is used before classification.
- Proposed Architecture: The model computes BERT sequence embeddings, applies dropout with probability 0.1 during training, and outputs class probabilities through softmax.Dropout is omitted during inference.
- Proposed Architecture: Softmax converts logits into probabilities summing to 1, and the highest-probability class becomes the predicted label.The logits are the intermediate softmax-layer outputs.
VI. EXPERIMENTS AND RESULTS
The paper evaluates its model on the SST sentiment-classification task and compares it with popular models addressing the same problem.
- The experiments compare the model with popular models for sentiment classification on the SST dataset.
A. Comparison Models
The comparison includes document-vector averaging and recursive neural-network models, with different approaches to representing sentiment-bearing text.
- One comparison method averages word vectors pretrained on a large corpus before sentiment classification.The averaged vectors form a document representation used to compute the sentiment score.
- The study also compares standard RNN and more sophisticated RNTN models trained on SST from scratch without pretraining.
3) Recurrent networks:
The paper situates SST sentiment classification among recurrent and convolutional approaches and uses accuracy for comparison, while noting incomplete reporting in some prior results.
- 3) Recurrent networks:: Sophisticated recurrent networks, including left-to-right and bidirectional LSTMs, have been applied to SST sentiment classification.
- 4) Convolutional networks:: A one-dimensional convolutional neural network was used as a feature extractor for input sequences.
- Results reporting: Some values are blank in the “All” columns because original authors did not publish results for all phrases.
- Evaluation metric: Accuracy is used to evaluate models because the dataset has roughly balanced numbers of samples across classes.
C. Results
Using pretrained BERT with a simple downstream architecture, the paper reports stronger SST accuracy than many popular and more sophisticated NLP models, including recursive, recurrent, and convolutional networks.
- A simple downstream architecture built on fine-tuned pretrained BERT outperforms complicated recursive, recurrent, and convolutional architectures.