Source-linked AI summary
MITRE at SemEval-2016 Task 6: Transfer Learning for Stance Detection
Guido Zarrella, Amy Marsh
TL;DR
Stance detection must infer whether a tweet author favors or opposes a topic from difficult, informal messages with limited labeled data. MITRE transfers representations from skip-gram and hashtag-prediction tasks into an RNN, then fine-tunes it for supervised stance detection. The system achieved the top shared-task result with an average FAVOR/AGAINST F1 of 67.8, while rare stance classes remained weaker.
Problem
Stance detection requires inferring favor or opposition from figurative, informal social-media text, while labeled in-domain examples are limited.
Source: §1 · p. 1Method
MITRE pre-trained word and phrase embeddings and an LSTM sentence encoder on large unlabeled Twitter data, then fine-tuned the RNN for stance detection.
Source: §4 · p. 2, §4.2 · p. 3Results
67.8 average F1 on FAVOR and AGAINST was the top score among 19 supervised shared-task submissions.
Source: §6 · p. 4Takeaways & Limitations
Hashtag prediction and skip-gram pre-training produced useful stance-detection features, with domain-relevant hashtag selection crucial to performance.
Source: §7 · p. 5Transfer learning did not eliminate labeled in-domain data needs, and minority stance classes consistently underperformed majority classes.
Source: §7 · p. 5
Abstract
from arXiv · showhide
We describe MITRE's submission to the SemEval-2016 Task 6, Detecting Stance in Tweets. This effort achieved the top score in Task A on supervised stance detection, producing an average F1 score of 67.8 when assessing whether a tweet author was in favor or against a topic. We employed a recurrent neural network initialized with features learned via distant supervision on two large unlabeled datasets. We trained embeddings of words and phrases with the word2vec skip-gram method, then used those features to learn sentence representations via a hashtag prediction auxiliary task. These sentence vectors were then fine-tuned for stance detection on several hundred labeled examples. The result was a high performing system that used transfer learning to maximize the value of the available training data.
1 Introduction
Stance detection infers whether an author favors or opposes a topic from spontaneous social-media messages, while rejecting cases where neither stance is likely. MITRE addresses figurative language, informal text, and limited labeled data with transfer learning from large unlabeled datasets.
- Stance detection determines whether a text author is in favor or against a topic, while rejecting messages supporting neither inference.Source: §1 · p. 1
- Unlike sentiment analysis, stance can be measured independently of the author’s emotional state.Source: §1 · p. 1
- Figurative language, sarcasm, irony, analogy, metaphor, and informal usage make stance difficult for machines to interpret.Source: §1 · p. 1
- Limited bulk training data motivated transfer learning through feature pre-training on large unlabeled datasets.Source: §1 · p. 1
2 Related Work
Transfer learning reuses knowledge learned on data-rich auxiliary tasks to improve related tasks with scarce in-domain labels. Prior work demonstrated transferable sentence representations, while this study targets stance from a single message rather than conversational context.
- Transfer learning applies knowledge learned in one context to a different, related problem, especially when labeled target examples are scarce.Source: §2 · p. 1
- A common strategy trains neural-network parameters first on a data-rich auxiliary task, then fine-tunes them on the actual task.Source: §2 · p. 2
- Prior NLP studies found unlabeled-data sentence representations useful across classification and semantic-similarity tasks, including recommendation.Source: §2 · p. 2
- Unlike work using discourse, dialogue, or retweet-based signals, this study detects stance from the text of a single Twitter message.Source: §2 · p. 2
3 Task and Evaluation
SemEval-2016 Task 6 Subtask A evaluated supervised stance detection on tweets spanning five topics. The dataset contained 2,814 labeled training tweets and 1,249 held-out evaluation tweets, with macro-averaged FAVOR and AGAINST F1 as the metric.
- SemEval-2016 Task 6 Subtask A was a shared task focused on supervised stance-detection frameworks.Source: §3 · p. 2
- Training data comprised 2,814 tweets across five topics, with 395 to 664 tweets per topic and topic-dependent class imbalance.Source: §3 · p. 2
- Approximately 74% of provided tweets were labeled FAVOR or AGAINST, while the remainder contained neither inference.Source: §3 · p. 2
- Evaluation used 1,249 held-out tweets and macro-averaged F1 across the FAVOR and AGAINST classes and all topics.Source: §3 · p. 2
4 System Overview
MITRE uses an RNN that combines word and phrase embeddings with an LSTM sentence encoder and softmax stance output. Skip-gram and hashtag-prediction pre-training transfer representations from large unlabeled Twitter corpora before stance fine-tuning.
System Overview
- The network maps token identities through a 256-dimensional embedding layer, 128 LSTM units, a 128-dimensional ReLU layer, and a three-dimensional softmax.Source: §4 · p. 2
- The model uses only lowercased token or phrase indices, without manually engineered task-specific or surface-text features.Source: §4 · pp. 2–3
Word Embedding Pre-Training
- Word2vec skip-gram embeddings initialized the projection layer using 218,179,858 sampled English tweets and 537,366 vocabulary items.Source: §4.1 · p. 3
- The projection weights were later tuned by backpropagation during recurrent-network training.Source: §4.1 · p. 3
Pre-Training the Recurrent Layer
- Hashtag prediction pre-trained the LSTM to learn sentence representations from stance-related declarations beyond the 2,814 labeled in-domain tweets.Source: §4.2 · p. 3
- The auxiliary corpus contained 298,973 tweets with 197 topic-relevant hashtags, whose stripped-text sequences trained a 197-way hashtag classifier.Source: §4.2 · p. 3
- 42.6% accuracy was achieved on the hashtag-prediction development set after seven epochs.Source: §4.2 · p. 3
5 Experiments
The system used topic-specific classifiers initialized with pretrained embedding and recurrent-layer weights, with decoding based on class selection. Variants tested alternative recurrent initialization strategies and omission of recurrent pretraining.
- Figure 2 reports F1 scores separately by topic and class under cross-validation and test conditions.Source: Figure 2 · p. 4
- Five distinct classifiers were trained, one for each evaluation topic.Source: §5 · p. 4
- Each classifier reused pretrained embedding and recurrent-layer weights while randomly initializing the remaining weights.Source: §5 · p. 4
- At decode time, the system selected a single class via majority vote.Source: §5 · p. 4
- Alternative variants omitted recurrent pretraining or initialized the recurrent layer from a 10,000-hashtag RNN, but neither improved performance.Source: §5 · p. 4
6 Results
The submission performed strongly on held-out data, with performance varying across topics and classes. Class imbalance was associated with unequal F1 scores, while pretraining choices affected cross-validation performance.
- 67.8 average F1 on FAVOR and AGAINST classes made the submission the top system among 19 supervised-task entries.Source: §6 · p. 4
- 71.1 average F1 in cross-validation exceeded the held-out test result, indicating a small amount of overfitting.Source: §6 · p. 4
- The majority class significantly outperformed the corresponding minority class across all topics.Source: §6 · p. 5
- R2 = 0.67 positive correlation linked class-level F1 scores with the raw number of training examples for that class.Source: §6 · p. 5
- Removing recurrent-layer pretraining reduced cross-validation average F1 from 71.1 to 70.0.Source: §6 · p. 5
7 Conclusion
The paper presents a state-of-the-art single-tweet stance system that transfers features from large unlabeled datasets to limited labeled data. Its results support skip-gram and domain-relevant hashtag prediction as useful pretraining tasks, while labeled in-domain data remains necessary.
- The system determined stance from a single tweet and transferred features from large unlabeled datasets to maximize limited training data.Source: §7 · p. 5
- Skip-gram and hashtag prediction produced pretrained features useful for stance detection.Source: §7 · p. 5
- Domain-relevant hashtag selection was crucial because a larger collection of frequent hashtags produced significantly worse stance-detection performance.Source: §7 · p. 5
- Transfer learning did not completely eliminate the need for labeled in-domain training data.Source: §7 · p. 5