Source-linked AI summary
MITRE at SemEval-2016 Task 6: Transfer Learning for Stance Detection
Guido Zarrella, Amy Marsh
TL;DR
Stance detection must infer whether a tweet author favors or opposes a topic from difficult, informal messages with limited labeled data. MITRE transfers representations from skip-gram and hashtag-prediction tasks into an RNN, then fine-tunes it for supervised stance detection. The system achieved the top shared-task result with an average FAVOR/AGAINST F1 of 67.8, while rare stance classes remained weaker.
Problem
Stance detection requires inferring favor or opposition from figurative, informal social-media text, while labeled in-domain examples are limited.
Method
MITRE pre-trained word and phrase embeddings and an LSTM sentence encoder on large unlabeled Twitter data, then fine-tuned the RNN for stance detection.
Results
67.8 average F1 on FAVOR and AGAINST was the top score among 19 supervised shared-task submissions.
Takeaways & Limitations
Hashtag prediction and skip-gram pre-training produced useful stance-detection features, with domain-relevant hashtag selection crucial to performance.
Takeaways & Limitations
Transfer learning did not eliminate labeled in-domain data needs, and minority stance classes consistently underperformed majority classes.
Abstract
from arXiv · showhide
We describe MITRE's submission to the SemEval-2016 Task 6, Detecting Stance in Tweets. This effort achieved the top score in Task A on supervised stance detection, producing an average F1 score of 67.8 when assessing whether a tweet author was in favor or against a topic. We employed a recurrent neural network initialized with features learned via distant supervision on two large unlabeled datasets. We trained embeddings of words and phrases with the word2vec skip-gram method, then used those features to learn sentence representations via a hashtag prediction auxiliary task. These sentence vectors were then fine-tuned for stance detection on several hundred labeled examples. The result was a high performing system that used transfer learning to maximize the value of the available training data.
1 Introduction
Stance detection infers whether an author favors or opposes a topic from spontaneous social-media messages, while rejecting cases where neither stance is likely. MITRE addresses figurative language, informal text, and limited labeled data with transfer learning from large unlabeled datasets.
- Stance detection determines whether a text author is in favor or against a topic, while rejecting messages supporting neither inference.
- Unlike sentiment analysis, stance can be measured independently of the author’s emotional state.
- Figurative language, sarcasm, irony, analogy, metaphor, and informal usage make stance difficult for machines to interpret.
- Limited bulk training data motivated transfer learning through feature pre-training on large unlabeled datasets.
2 Related Work
Transfer learning reuses knowledge learned on data-rich auxiliary tasks to improve related tasks with scarce in-domain labels. Prior work demonstrated transferable sentence representations, while this study targets stance from a single message rather than conversational context.
- Transfer learning applies knowledge learned in one context to a different, related problem, especially when labeled target examples are scarce.
- A common strategy trains neural-network parameters first on a data-rich auxiliary task, then fine-tunes them on the actual task.
- Prior NLP studies found unlabeled-data sentence representations useful across classification and semantic-similarity tasks, including recommendation.
- Unlike work using discourse, dialogue, or retweet-based signals, this study detects stance from the text of a single Twitter message.
3 Task and Evaluation
SemEval-2016 Task 6 Subtask A evaluated supervised stance detection on tweets spanning five topics. The dataset contained 2,814 labeled training tweets and 1,249 held-out evaluation tweets, with macro-averaged FAVOR and AGAINST F1 as the metric.
- SemEval-2016 Task 6 Subtask A was a shared task focused on supervised stance-detection frameworks.
- Training data comprised 2,814 tweets across five topics, with 395 to 664 tweets per topic and topic-dependent class imbalance.
- Approximately 74% of provided tweets were labeled FAVOR or AGAINST, while the remainder contained neither inference.
- Evaluation used 1,249 held-out tweets and macro-averaged F1 across the FAVOR and AGAINST classes and all topics.
4 System Overview
MITRE uses an RNN that combines word and phrase embeddings with an LSTM sentence encoder and softmax stance output. Skip-gram and hashtag-prediction pre-training transfer representations from large unlabeled Twitter corpora before stance fine-tuning.
- System Overview: The network maps token identities through a 256-dimensional embedding layer, 128 LSTM units, a 128-dimensional ReLU layer, and a three-dimensional softmax.
- System Overview: The model uses only lowercased token or phrase indices, without manually engineered task-specific or surface-text features.
- Word Embedding Pre-Training: Word2vec skip-gram embeddings initialized the projection layer using 218,179,858 sampled English tweets and 537,366 vocabulary items.
- Word Embedding Pre-Training: The projection weights were later tuned by backpropagation during recurrent-network training.
- Pre-Training the Recurrent Layer: Hashtag prediction pre-trained the LSTM to learn sentence representations from stance-related declarations beyond the 2,814 labeled in-domain tweets.
- Pre-Training the Recurrent Layer: The auxiliary corpus contained 298,973 tweets with 197 topic-relevant hashtags, whose stripped-text sequences trained a 197-way hashtag classifier.
- Pre-Training the Recurrent Layer: 42.6% accuracy was achieved on the hashtag-prediction development set after seven epochs.
5 Experiments
The system used topic-specific classifiers initialized with pretrained embedding and recurrent-layer weights, with decoding based on class selection. Variants tested alternative recurrent initialization strategies and omission of recurrent pretraining.
- Figure 2 reports F1 scores separately by topic and class under cross-validation and test conditions.
- Five distinct classifiers were trained, one for each evaluation topic.
- Each classifier reused pretrained embedding and recurrent-layer weights while randomly initializing the remaining weights.
- At decode time, the system selected a single class via majority vote.
- Alternative variants omitted recurrent pretraining or initialized the recurrent layer from a 10,000-hashtag RNN, but neither improved performance.
6 Results
The submission performed strongly on held-out data, with performance varying across topics and classes. Class imbalance was associated with unequal F1 scores, while pretraining choices affected cross-validation performance.
- 67.8 average F1 on FAVOR and AGAINST classes made the submission the top system among 19 supervised-task entries.
- 71.1 average F1 in cross-validation exceeded the held-out test result, indicating a small amount of overfitting.
- The majority class significantly outperformed the corresponding minority class across all topics.
- R2 = 0.67 positive correlation linked class-level F1 scores with the raw number of training examples for that class.
- Removing recurrent-layer pretraining reduced cross-validation average F1 from 71.1 to 70.0.
7 Conclusion
The paper presents a state-of-the-art single-tweet stance system that transfers features from large unlabeled datasets to limited labeled data. Its results support skip-gram and domain-relevant hashtag prediction as useful pretraining tasks, while labeled in-domain data remains necessary.
- The system determined stance from a single tweet and transferred features from large unlabeled datasets to maximize limited training data.
- Skip-gram and hashtag prediction produced pretrained features useful for stance detection.
- Domain-relevant hashtag selection was crucial because a larger collection of frequent hashtags produced significantly worse stance-detection performance.
- Transfer learning did not completely eliminate the need for labeled in-domain training data.