PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Word Embeddings in Twitter Election Classification

Xiao Yang, Craig Macdonald, Iadh Ounis

arXiv:1606.07006v3cs.IRcs.CL

TL;DR

Prior work had not established how word-embedding training choices affect Twitter classification. Using CNNs on election-related tweets, this paper finds that task-aligned background corpora and larger context windows and dimensions improve performance over baselines.

  • Problem

    Existing research had little evidence on how embedding background corpora, context windows, and dimensionality affect Twitter classification performance.

  • Method

    The paper evaluates CNN classifiers with Word2Vec embeddings trained from different corpora, context windows, and dimensions on a Twitter election dataset.

  • Results

    Aligning the background corpus with the Twitter dataset and using large context windows and dimensions improved CNN classification over SVM with TF-IDF and other baselines.

  • Takeaways & Limitations

    For this Twitter election task, embedding configuration matters, with Twitter-aligned corpora and larger windows and dimensions producing more effective CNN classification.

Abstract

from arXiv · show

Word embeddings and convolutional neural networks (CNN) have attracted extensive attention in various classification tasks for Twitter, e.g. sentiment classification. However, the effect of the configuration used to train and generate the word embeddings on the classification performance has not been studied in the existing literature. In this paper, using a Twitter election classification task that aims to detect election-related tweets, we investigate the impact of the background dataset used to train the embedding models, the context window size and the dimensionality of word embeddings on the classification performance. By comparing the classification results of two word embedding models, which are trained using different background corpora (e.g. Wikipedia articles and Twitter microposts), we show that the background data type should align with the Twitter classification dataset to achieve a better performance. Moreover, by evaluating the results of word embeddings models trained using various context window sizes and dimensionalities, we found that large context window and dimension sizes are preferable to improve the performance. Our experimental results also show that using word embeddings and CNN leads to statistically significant improvements over various baselines such as random, SVM with TF-IDF and SVM with word embeddings.

1. INTRODUCTION

This paper investigates how word-embedding training choices affect CNN performance on ambiguous election-related tweet classification. It evaluates background corpora, context-window sizes, and embedding dimensionalities, reporting improvements when corpus type aligns with the classification data.

  • Background: Word embeddings represent words as dense vectors learned from context, placing similar words near one another in embedding space.The passage describes Word2Vec as learning representations by maximizing the probability of observing words within a fixed context window.
  • Research gap: Prior literature had not established how background corpora, context-window sizes, and embedding dimensionalities affect classification performance.The paper identifies limited exploration of these training choices despite their potential to produce different embeddings.
  • Task and motivation: The study classifies election-related tweets collected during Venezuela’s 2015 parliamentary election, a challenging task because relevance is often ambiguous for human assessors.The task uses word embeddings with CNN to identify tweets related to the election.
  • Main findings: Aligning the background corpus with the classification dataset yields statistically significant CNN improvements over SVM with TF-IDF, while large context windows and dimensions also improve performance.The paper compares Wikipedia articles and Twitter microposts as background corpora and investigates both embedding parameters.

2. RELATED WORK

Prior studies linked word-embedding configuration to performance in dependency parsing, named entity tagging, and semantic-syntactic word-relation evaluation, but did not establish whether these findings generalize to Twitter classification. Existing Twitter sentiment-classification work used embeddings and CNNs effectively, yet did not systematically study the background corpus and embedding parameters for election classification.

  • Word-embedding configuration: Embedding context-window size and dimensionality affected performance in dependency parsing and named entity tagging.Bansal et al. investigated Word2Vec embeddings using Wall Street Journal and Wikipedia corpora for dependency parsing.
  • Word-embedding configuration: Mikolov et al. found that increasing dimensionality or background-data size eventually produced only minor accuracy improvements, recommending that both be increased together.Their evaluation used a semantic-syntactic word-relationship test set and the GoogleNews background corpus.
  • Research gap: These prior configuration findings came from tasks different from Twitter classification, leaving their generalization to Twitter classification unresolved.This generalization question motivates the study’s focus on Twitter election classification.
  • Twitter classification: Twitter sentiment-classification studies used word embeddings to build effective tweet-level representations, including systems evaluated with CNN classifiers.The cited CNN system ranked 1st on the phrase-level and 2nd on the message-level Semeval-2015 datasets.

3. THE CNN MODEL

The CNN classifier represents padded tweets with concatenated word embeddings, then applies convolution, max pooling, dropout, and a fully connected softmax layer. Convolution extracts position-independent features, while cross-entropy training produces normalized class probabilities for election classification.

  • Architecture: The model uses convolutional, max-pooling, dropout, and fully connected output layers for Twitter election classification.This architecture follows CNN designs described by Kim and Severyn et al.
  • Tweet-level representation: Tweets are converted into concatenated word-embedding vectors and padded to the longest tweet, giving TCNN dimension k × n.The resulting tweet-level representation is passed to the convolutional layer.
  • Convolutional layer: Filters of size m slide across the tweet representation with stride s to produce feature maps, using ReLU activation defined as f = max(0, x).The convolution operation learns important words regardless of their positions in a tweet.
  • Max pooling layer: Max pooling extracts the largest value from each feature map and concatenates these maxima into the tweet feature vector.This creates a compact representation from the convolutional outputs.
  • Dropout and softmax layers: Dropout with L2 regularization limits active neurons during learning, while the softmax layer outputs normalized class probabilities trained with cross-entropy.During prediction, p = 1 keeps all neurons active.

4. EXPERIMENTAL SETUP

The experiments investigate how background corpora and Word2Vec parameters affect CNN-based Twitter classification. They use a manually labelled Venezuelan election dataset, compare Spanish Wikipedia and Twitter embeddings, vary embedding configurations, and evaluate against standard baselines.

  • Experimental objectives: The study investigates the effects of background corpora, context windows, and embedding dimensionality on Twitter classification using word embeddings with CNN.The experiments are designed to examine how Word2Vec configuration affects classification performance.
  • Dataset: The election dataset contains tweets collected around the 2015 Venezuela parliamentary election and manually labelled as “Election-related” or “Not Election-related”.Tweets were retrieved using 21 query terms, sampled with a pooling method, and judged by five experts.
  • Background corpora: Word embeddings are trained from Spanish Wikipedia and Spanish Twitter corpora, with es-Twitter providing better coverage of the election dataset: 6,612 words versus 5,111 for es-Wiki.The background corpora contain over 1 million Spanish articles and over 20 million Spanish tweets, respectively.
  • Embedding configurations: The study generates 9 Word2Vec embeddings by combining context windows W = {1, 3, 5} with dimensions D = {200, 500, 800}.These settings cover low and high embedding dimensionalities while varying the context window size.
  • Classifiers and evaluation: CNN and embedding classifiers are evaluated against Random classifier, SVM+TFIDF, and SVM+WE baselines using 5-fold cross validation.The SVM+WE tweet representation averages word embeddings across the words in each tweet.

5. EFFECT OF THE BACKGROUND CORPORA

The study finds that background corpus type affects Twitter election classification: Twitter-derived embeddings better match the dataset because they cover more Twitter-specific vocabulary. Aligning the background corpus with the classification dataset yields better representations and more effective CNN classification.

  • Corpus characteristics: Twitter contains noisy spelling and distinctive handles, links, and hashtags that common corpora may not represent.These characteristics motivate comparing es-Twitter with es-Wiki as embedding background corpora.
  • Vocabulary coverage: 66% of the election dataset vocabulary appears in es-Twitter, compared with 51% in es-Wiki.After removing shared vocabulary, es-Twitter covers 1,527 unique words absent from es-Wiki, whereas es-Wiki covers only 26 unique words absent from es-Twitter.
  • Classification implication: Aligning the background corpus with the classification dataset leads to better feature representations and more effective CNN classification.The vocabulary findings support the reported classification results and may generalize to similar Twitter classification tasks.

6. EFFECT OF WORD EMBEDDINGS PARAMETERS

The section finds that word-embedding parameter effects depend on the classifier: CNN favors large context windows and dimensionalities, whereas SVM+WE favors smaller dimensions. For CNN, W = 5 and D = 800 produces the best scores across all metrics and outperforms the evaluated baselines.

  • Classifier comparisons: CNN classifiers outperform random and SVM+WE across all measures, and consistently outperform SVM+TFIDF on precision and F1 score.The comparison identifies CNN as the strongest approach among the reported baselines and word-embedding classifiers.
  • Classifier comparisons: When W = 5 and D = 800, CNN achieves the best scores on all metrics.This configuration demonstrates the reported effectiveness of convolutional neural networks for the Twitter election classification task.
  • Parameter effects: SVM+WE performs better with smaller dimensions and larger context windows, such as W5 and D200, on F1 score and precision.The preferred dimensionality differs between SVM+WE and CNN when both use word embeddings.
  • Parameter effects: CNN prefers both large context windows and large dimensionalities, while high-dimensional embeddings can introduce ambiguity in SVM+WE.For SVM+WE, combining all word vectors may weaken semantic representation because not every word is meaningful for classification.
  • Task-specific settings: The results contrast with NER and dependency parsing studies, suggesting that optimal context-window and dimensionality settings vary across tasks.For the Twitter election classification task using CNNs, the reported summary favors large context windows and dimensionalities.

7. CONCLUSION

The conclusion emphasizes that findings about word-embedding configurations from NER and dependency parsing may not generalize to Twitter classification. It identifies background corpora, context window, and dimensionality as understudied factors that can affect Twitter classification performance.

  • 7. CONCLUSION: Findings on word-embedding parameter configurations from NER and dependency parsing may not generalize to Twitter classification tasks.The passage contrasts prior investigations on those tasks with Twitter classification.
  • 7. CONCLUSION: Related Twitter classification studies had not examined the impact of background corpora, context window, and dimensionality.These are identified as Word2Vec-related factors absent from the cited prior work.
  • 7. CONCLUSION: The authors find that background corpora and Word2Vec parameters can affect classification performance on Twitter classification tasks.The supplied passage ends mid-sentence after specifying Twitter classification, so the conclusion is limited to this stated finding.
Loading 1606.07006v3…