Improving the Accuracy of Pre-trained Word Embeddings for Sentiment Analysis

• Pre-trained Word2Vec and GloVe may not fit small sentiment datasets, while training strong vectors requires very large corpora.Source: §1 · p. 2
• Word2Vec and GloVe ignore sentiment information, and opposite-polarity words may map to nearby vectors, a problem for sentiment analysis.Source: §1 · p. 2
• The authors propose a method to improve the accuracy of pre-trained Word2Vec/GloVe vectors for sentiment analysis.Source: §1 · p. 2
• Across different sentiment datasets and deep-learning models, reported results show improved pre-trained embedding accuracy for sentiment analysis.Source: §1 · p. 2

2.1. Lexicon-based method

• Lexicon-based analysis uses dictionaries assigning positive or negative sentiment values to words and phrases, and is described as simple, scalable, and computationally efficient.Source: §2.1 · p. 2
• Lexicon methods depend on human effort in human-labeled documents and sometimes suffer from low coverage.Source: §2.1 · pp. 2–3
• Prior work combined lexicons with SVMs or binary classifiers, with reported accuracy gains in text classification.Source: §2.1 · p. 3

2.2. Deep learning method

• Word embeddings represent words and documents as vectors that capture syntactic and semantic relations, with semantically similar words placed near one another.Source: §2.2 · p. 3
• For sentiment analysis on Twitter, Severyn and Moschitti used Word2Vec to learn word embeddings from 50M tweets and applied the generated pre-trained vectors.Source: §2.2 · p. 3
• Word-embedding algorithms require huge training corpora and often ignore sentiment information, while corpus restrictions lead researchers to use pre-trained vectors as model inputs.Source: §2.2 · p. 4
• Prior attempts to combine pre-trained Word2Vec and GloVe decreased classification accuracy on some datasets [10]; another algorithm decreased Word2Vec accuracy on one of two datasets.Source: Table 1 · p. 4, §2.2 · p. 4

3. Proposed method

• IWV combines natural language processing techniques, lexicon-based approaches, and Word2Vec/GloVe to improve word embeddings for sentiment analysis.Source: §3 · p. 5

3.1. Word2Vec and GloVe

• Word2Vec uses CBOW and Skip-gram architectures, while GloVe uses co-occurrence and matrix factorization to generate word vectors.Source: §3.1 · p. 5

3.2. POS2Vec (P2V)

• POS2Vec assigns constant vectors to POS tags and concatenates them with Word2Vec/GloVe vectors, adding syntactic information.The method assigns 50-dimensional vectors to words.Source: §3.2 · p. 6

3.3. Lexicon2Vec (L2V)

• Lexicon2Vec adds sentiment scores from six selected lexicons to each word vector, normalizing scores and using zero when a word is absent.Vectors from each step are concatenated; the algorithm uses pretrained Word2Vec or GloVe vectors when available and generates a random vector otherwise.Source: §3.3 · pp. 6–7
• Model 1 evaluates the generated vectors on well-known datasets using improved word vectors as input; it comprises three convolutional neural networks, a pooling layer, and a fully connected layer.The proposed method was also checked with three other deep-learning models from published papers.Source: §3.3 · p. 8

4.2. Results

• IWV uses six sentiment lexicons with unigram scores; its CNN uses filter sizes 3, 4, and 5, 100 feature maps each, a 95-node flatten layer, and ReLU.Source: §4.2 · p. 9
• 1.7%, 0.7%, and 0.5% accuracy gains were reported for SST, CR, and MR, respectively, with the combined vector outperforming existing pre-trained vectors.Table 3 compares accuracy for deep-learning model 1.Source: Table 3 · p. 10, §4.2 · p. 10
• IWV was more accurate than other methods in models 2, 3, and 4, with all six reported accuracy comparisons improved.Table 4 reports average accuracies over multiple runs of 10-fold cross-validation.Source: Table 4 · p. 10, §4.2 · p. 10
• On MR, IWV had the highest accuracy and improved results by 0.2%–0.5% across the four deep-learning models.The comparison is against Word2Vec and GloVe.Source: Figure 3 · p. 11, §4.2 · p. 11

4.1. Datasets

• MR is balanced and CR unbalanced; SST uses binary train/test sentences, and reported averages use repeated 10-fold cross-validation except for SST's predefined train/test sets.The experiments cover MR, CR, and SST.Source: §4.2 · pp. 9, 11, 12, §4.1 · p. 9

4.2. Results

• On CR, IWV generally performed better; absolute accuracy improvements were 0.7%, 0.4%, 1.1%, and 0.2% for models 1, 2, 3, and 4, respectively.The passage describes CR as unbalanced and notes that Word2Vec was generally more accurate than GloVe, except in model 2.Source: Figure 4 · p. 11, §4.2 · p. 12

5. Conclusion

• IWV combines lexicon-based methods, POS tagging, and Word2Vec/GloVe embeddings.The authors report testing the method nine times with different deep-learning models and sentiment datasets.Source: §5 · p. 12
• Accuracy increased across all tested sentiment-classification models and datasets.The conclusion reports gains across the tested models and datasets.Source: §5 · p. 12
• The authors expect improvements to pre-trained embeddings, POS tagging, or lexicons to increase IWV accuracy further.Source: §5 · p. 12
• They propose IWV as a basis for sentiment-analysis techniques that use deep-learning approaches.Source: §5 · p. 12