PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep-Sentiment: Sentiment Analysis Using Ensemble of CNN and Bi-LSTM Models

Shervin Minaee, Elham Azimi, AmirAli Abdolrashidi

arXiv:1904.04206v1cs.CLcs.IRcs.LGstat.ML

TL;DR

Sentiment analysis aims to understand opinions expressed about products, topics, and trends, but extracting the underlying sentiment from feedback is challenging. This paper combines CNN and bidirectional LSTM models to capture local structure and temporal information, and reports that the ensemble outperforms both individual models. The authors conclude that it may also improve prediction accuracy in other deep-learning text-processing tasks.

  • Problem

    Extracting the sentiment conveyed in varied feedback is challenging, despite its importance for understanding public and customer opinion.

    Source: §I · p. 1

  • Method

    The framework combines CNNs for local structure with bidirectional LSTMs for temporal information, averaging their predicted scores for final predictions.

    Source: §I · p. 1, §IV · p. 5

  • Results

    The ensemble outperforms both individual CNN and LSTM models in the reported experiments.

    Source: §I · p. 2, B. Model Performance and Comparison · p. 4

  • Takeaways & Limitations

    The authors believe the ensemble may improve prediction accuracy in other deep-learning-based text-processing tasks.

    Source: §IV · p. 5

Abstract

from arXiv · show

With the popularity of social networks, and e-commerce websites, sentiment analysis has become a more active area of research in the past few years. On a high level, sentiment analysis tries to understand the public opinion about a specific product or topic, or trends from reviews or tweets. Sentiment analysis plays an important role in better understanding customer/user opinion, and also extracting social/political trends. There has been a lot of previous works for sentiment analysis, some based on hand-engineering relevant textual features, and others based on different neural network architectures. In this work, we present a model based on an ensemble of long-short-term-memory (LSTM), and convolutional neural network (CNN), one to capture the temporal information of the data, and the other one to extract the local structure thereof. Through experimental results, we show that using this ensemble model we can outperform both individual models. We are also able to achieve a very high accuracy rate compared to the previous works.

I. INTRODUCTION

Sentiment analysis extracts and classifies opinions expressed across feedback, supporting analysis of public and customer opinion. This work combines CNN and bidirectional LSTM networks to improve sentiment-analysis accuracy and reports that the ensemble outperforms either model individually.

I. INTRODUCTION

  • Sentiment analysis classifies conveyed emotions in feedback and can help divide opinions on products, services, and people into positive or negative categories.The introduction also describes uses such as detecting tones in voice and identifying bots and trolls on social networks.Source: §I · p. 1
  • Online sentiment analysis can track public opinion across many topics for commercial, economic, and political purposes.Source: §I · p. 1
  • Prior sentiment-analysis work includes hand-engineered features and deep-learning approaches, including CNNs, transfer learning, and hierarchical models.The paper also points readers to a broader review of deep-learning sentiment analysis.Source: §I · p. 1
  • The paper tests its proposed CNN and bidirectional LSTM ensemble on IMDB review and SST2 datasets.The proposed algorithm is shown in a block diagram.Source: §I · p. 1, Figure 1 · p. 2
  • The CNN–LSTM ensemble outperforms both individual models in the reported experiments.The CNN targets local structure, while the LSTM captures temporal information and text dependencies.Source: §I · pp. 1–2

II. THE PROPOSED FRAMEWORK

The proposed sentiment-analysis framework ensembles LSTM and CNN models. The section places this approach in the context of ensemble methods used in NLP and vision.

II. THE PROPOSED FRAMEWORK

  • The framework combines LSTM and CNN models to perform sentiment analysis.Source: §II · p. 2
  • The paper notes that ensembles have brought performance gains over single models in NLP and vision,.The following subsections overview the proposed LSTM and CNN models.Source: §II · p. 2

A. The LSTM model architecture

The LSTM architecture processes sequential inputs and uses a memory cell with gates to regulate information flow. The proposed model uses Glove embeddings with a two-layer bidirectional LSTM for review sentiment prediction.

A. The LSTM model architecture

  • An LSTM processes each time-step’s input alongside the previous time-step’s output, using hidden states for classification.It is designed to capture long-term dependencies better than a vanilla RNN.Source: A. The LSTM model architecture · p. 2
  • The memory cell and input, output, and forget gates regulate information flow and are intended to reduce vanishing or exploding gradient problems.The cell retains values across arbitrary intervals; the gates control information entering and leaving it.Source: A. The LSTM model architecture · p. 2
  • A bidirectional LSTM processes the sequence in forward and reverse order, providing context from past and future inputs.Source: A. The LSTM model architecture · p. 2
  • The proposed review classifier uses Glove word embeddings with a two-layer bidirectional LSTM.The passage identifies Figure 4 as its architecture illustration.Source: A. The LSTM model architecture · p. 2

B. The CNN model architecture

The CNN text classifier exploits local patterns in word embeddings by applying convolution filters over word windows, pooling their features, and using the resulting representation for prediction. The proposed model uses four filter sizes and two fully connected layers.

B. The CNN model architecture

  • CNNs exploit local correlations and patterns through learned feature maps, a capability used in computer vision and NLP.Source: B. The CNN model architecture · p. 3
  • The standard text-CNN pipeline applies filters over word windows, pools their features, concatenates pooled features, and feeds the representation to fully connected layers for prediction.Source: B. The CNN model architecture · p. 3
  • The proposed CNN uses pretrained Glove embeddings, filter sizes 1, 2, 3, and 4 with 100 feature maps each, and two fully connected layers before softmax.Figure 5 is identified as the general architecture of the proposed CNN.Source: B. The CNN model architecture · p. 3, Figure 5 · p. 3

C. The Ensemble Model

The ensemble combines CNN and LSTM predictions by averaging their probability scores, pairing CNN’s view of local text structure with LSTM’s view of temporal information.

C. The Ensemble Model

  • The ensemble averages CNN and LSTM probability scores to make the final sentiment predictions.The approach combines CNN’s holistic view of local text information with LSTM’s temporal view.Source: C. The Ensemble Model · p. 3

III. EXPERIMENTAL RESULTS

The experiments use the IMDB review and SST2 sentiment datasets. The section describes their setup and contrasts frequent words in positive and negative examples.

A. Databases

  • IMDB contains 50,000 movie reviews, split evenly into 25,000 training and 25,000 test reviews, with balanced positive and negative labels.No more than 30 reviews come from any movie, and the training and test sets contain disjoint movies.Source: A. Databases · p. 3
  • For IMDB, positive reviews feature words such as “great” and “love,” while negative reviews feature “bad” and “can’t”; “well” appears in both but is more frequent in positive reviews.Stop words and “movie” and “film” were excluded from the word visualization.Source: A. Databases · p. 3
  • SST2 is a binary sentiment dataset with train, development, and test splits; its phrase-level data supports training on phrases and sentences.Its vocabulary is around 16,000 words.Source: A. Databases · p. 3
  • For SST2, “good,” “funny,” “love,” and “best” distinguish positive reviews, while “bad,” “nothing,” and “never” distinguish negative reviews.The word visualization excludes stop words and “movie” and “film”; words including “one,” “character,” and “way” occur in both classes.Source: A. Databases · pp. 3–4

B. Model Performance and Comparison

The experiments compare CNN+Glove, LSTM+Glove, and their ensemble on IMDB and SST2, and report a performance gain for the ensemble over the individual models.

B. Model Performance and Comparison

C. The predicted scores for positive and negative reviews

On SST2, CNN predictions push positive and negative reviews’ scores closer to 1 and 0 than LSTM predictions do, while LSTM scores span a wider range.

C. The predicted scores for positive and negative reviews

D. Training Performance over Time

Training-accuracy curves for CNN and LSTM models show that CNN starts converging earlier on both IMDB and SST2. The figures track training accuracy across epochs.

D. Training Performance over Time

IV. CONCLUSION

The proposed sentiment-analysis framework averages prediction scores from Glove-embedded CNN and LSTM models, and reports gains over either individual model. Joint training is proposed as future work.

IV. CONCLUSION

  • The ensemble shows performance gains compared with the individual LSTM and CNN models.Source: §IV · p. 5
  • Reviews are represented with Glove embeddings, which are fed into CNN and LSTM models whose predicted scores are averaged.Source: §IV · p. 5
  • Jointly training the LSTM and CNN models is proposed as future work to further improve their gains over individual models.Source: §IV · p. 5
  • The authors suggest the ensemble could improve prediction accuracy in other deep learning-based text-processing tasks.Source: §IV · p. 5

Related papers

Loading 1904.04206v1…