Source-linked AI summary
Stance Detection with Bidirectional Conditional Encoding
Isabelle Augenstein, Tim Rocktäschel, Andreas Vlachos, Kalina Bontcheva
TL;DR
The paper tackles stance detection for targets that may be unmentioned and have no labelled training data. It uses target-conditioned LSTM encoding, strengthened with bidirectional context, and reports second-best unseen-target performance and state-of-the-art results with weak supervision.
Problem
Stance detection for unseen targets must operate when the target is not mentioned and no labelled training data exists for that target.
Method
The approach encodes the target with an LSTM and uses it to initialize a tweet-encoding LSTM, with a bidirectional conditional variant.
Results
0.5803 F1 is achieved by the bidirectional conditional model with automatically labelled data, while its unseen-target test result is 0.4901 F1.
Takeaways & Limitations
Conditional encoding learns target-dependent tweet representations that outperform baselines and generalize to unseen targets.
Takeaways & Limitations
The unseen-target setup excludes manually labelled training data for the test target and uses data from other task targets.
Abstract
from arXiv · showhide
Stance detection is the task of classifying the attitude expressed in a text towards a target such as Hillary Clinton to be "positive", negative" or "neutral". Previous work has assumed that either the target is mentioned in the text or that training data for every target is given. This paper considers the more challenging version of this task, where targets are not always mentioned and no training data is available for the test targets. We experiment with conditional LSTM encoding, which builds a representation of the tweet that is dependent on the target, and demonstrate that it outperforms encoding the tweet and the target independently. Performance is improved further when the conditional model is augmented with bidirectional encoding. We evaluate our approach on the SemEval 2016 Task 6 Twitter Stance Detection corpus achieving performance second best only to a system trained on semi-automatically labelled tweets for the test target. When such weak supervision is added, our approach achieves state-of-the-art results.
1 Introduction
The paper addresses stance detection for unseen targets, including tweets where the target is implicit and no labelled target-specific training data exists. It uses conditional LSTM encoding, with bidirectional encoding yielding stronger results and weak supervision producing the best reported result.
- The task classifies a text’s attitude toward a target as positive, negative, or neutral.
- Unseen-target stance detection must handle targets that are absent from tweets and lack labelled training data.The setting is more realistic because labelled data may not be available for every target of interest.
- The model uses a target-encoding LSTM to initialize a second LSTM that encodes the tweet conditionally on that target.
- 0.4901 F1 is achieved by the bidirectional conditional model, compared with 0.5628 F1 for an approach using automatically labelled test-target tweets.
- 0.5803 F1 is achieved when the bidirectional conditional model is trained with automatically labelled data.
2 Task Setup
The SemEval 2016 Twitter stance task includes settings with labelled data for test targets and an unseen-target setting without labelled training or development data. Its official metric is macroaveraged F1 over FAVOR and AGAINST, while automatic labelling was permitted for the unseen target.
- Task A provides labelled training data for all test targets, whereas Task B evaluates stance toward the unseen target Donald Trump without labelled training or development data.
- Systems classify each tweet as FAVOR, AGAINST, or NONE toward the target.
- The official metric is macroaveraged F1 over FAVOR and AGAINST, although systems must also predict NONE.
- Participants could automatically label data for Donald Trump, and leading systems used this option, turning their Task B setting into weakly supervised seen-target detection.
3 Methods
The paper develops target-aware LSTM encoders for stance detection on unseen targets, progressing from independent representations to conditional and bidirectional encoding. It also uses word2vec initialization and compares the neural architectures with feature-based and simpler baselines.
- Independent Encoding: Independent encoding represents the tweet and target as separate k-dimensional vectors, then combines them for stance classification.The resulting projection incorporates the target after the tweet and target have been encoded independently.
- Conditional Encoding: Conditional encoding initializes the tweet LSTM with the target representation, producing a target-dependent tweet representation for stance prediction.This lets the tweet encoder read the same text differently depending on the target.
- Bidirectional Encoding: Bidirectional conditional encoding reads both target and tweet in forward and reverse directions, incorporating left- and right-hand context for each tweet word.The tweet encoder is initialized from the forward and reversed target encodings.
- Unsupervised Pretraining: Word2vec embeddings are pretrained on 395,212 unlabelled tweets plus official SemEval data, then further optimized during supervised training.Pretraining is intended to counter-balance the relatively small labelled dataset of 5,628 instances.
4 Experiments
Experiments use the SemEval 2016 Twitter stance corpus, focusing on unseen-target detection and a weakly supervised setting. The study compares conditional models with established and simpler baselines, reporting stronger generalization and state-of-the-art performance under weak supervision.
- Experimental Setups: The main unseen-target setup evaluates stance detection for targets without labelled training or development data, using the SemEval Task 6 corpus.The paper also evaluates a weakly supervised framework using automatically labelled target tweets.
- Main Findings: Conditional encoding generalizes to unseen targets better than baselines that ignore the target, while the approach outperforms the state of the art on the weakly supervised Task B corpus.The official task evaluation script is used for all experiments.
- Compared Systems: The comparison includes SVM n-gram, majority, bag-of-word-vectors, independent LSTM, tweet-only LSTM, and conditional-encoding baselines.Conditional variants include target-conditioned-on-tweet, tweet-conditioned-on-target, and bidirectional encoding.
5 Unseen Target Stance Detection
The unseen-target experiments evaluate whether target-conditioned representations generalize without labelled data for the test target. BiCond performs best, while pre-training and shared embeddings further support performance and conditional encoding is especially useful when targets are mentioned.
- Experimental Setup: The evaluation withholds labelled Donald Trump data during development and uses it only for final test reporting, preserving the unseen-target setting.
- Results: BiCond achieves the best results in both development and test setups, with F1 scores of 0.4722 and 0.4901, respectively.
- Results: Models conditioning tweet encoding on targets outperform all baselines on the test set, unlike independent target-tweet encoding, which offers little improvement over TweetOnly.
- Results: BiCond’s 0.4901 F1 is the corpus’s second-highest reported result, behind systems using automatically labelled Donald Trump training data.
- Pre-Training: Pre-trained word embeddings outperform random initialization, and shared tweet-target embeddings perform better than separate embedding matrices.
- Target Mentioning: All compared models perform better when the target is mentioned, while BiCond outperforms unidirectional encoding and unidirectional encoding outperforms Concat in that condition.
6 Weakly Supervised Stance Detection
The weakly supervised setup automatically labels training data for the unseen test target and evaluates models on the official test set. BiCond achieves the strongest reported result in this setting and state-of-the-art performance overall.
- Experimental setup: Experiments compare the proposed approach with participants in SemEval 2016 Task B under weakly supervised target-specific training.The setup differs from unseen-target evaluation because training data for Donald Trump is automatically labelled.
- Experimental setup: The evaluation reports official test-set results for models trained on automatically labelled positive, negative, and neutral Trump data.Table 6 specifically summarizes this weakly supervised evaluation.
- Experimental setup: Training data for Donald Trump is automatically annotated using regular expressions for positive and negative stance, with neutral labels assigned randomly when neither is detected.The resulting automatically labelled corpus is used to train the models under the weakly supervised setup.
- Results and discussion: The comparison includes majority, n-gram SVM, and state-of-the-art systems, alongside the paper’s conditional-encoding models.The baselines and external systems are listed in the reported comparison tables.
- Results and discussion: F1 of 0.2972 for the majority baseline illustrates the difficulty of the task, while target-aware representations outperform tweet-only baselines.Bag-of-word baselines are comparable to the majority baseline, whereas models incorporating target representations perform better than tweet-only systems.
- Results and discussion: F1 of 0.5803 is achieved by the bidirectional conditional encoding model, yielding state-of-the-art results on automatically labelled data.The authors report that BiCond also suits unseen-target stance detection.
7 Related Work
The paper situates unseen-target tweet stance detection among stance, sentiment, rumour, and textual-entailment research. Its conditional-encoding experiments extend prior work to smaller data and unseen testing targets.
- Stance Detection: Earlier stance-detection work focused mainly on debates and student essays, whereas this task also involves targets that may not be mentioned in tweets.The paper distinguishes its setting from target-dependent and open-domain sentiment-analysis tasks.
- Stance Detection: Rumour stance-detection methods either require training data from the same target or use rules that may be difficult to generalise.The paper contrasts these requirements with its unseen-target setting.
- Conditional Encoding: Conditional encoding was previously applied to textual entailment, while this paper tests it on a relatively small training set and an unseen testing target.The paper also augments conditional encoding with bidirectional encoding.
8 Conclusions and Future Work
The paper concludes that conditional LSTM encoding supports stance detection for unseen targets and performs strongly in both unseen-target and weakly supervised settings. It also identifies unsupervised pretraining as useful when labelled corpora are limited and leaves tweets without explicit target mentions for future work.
- Conclusions: Conditional LSTM encoding is presented as a successful approach to stance detection for unseen targets.
- Conclusions: The unseen-target bidirectional conditional model achieves the second-best reported result on the SemEval 2016 Twitter Stance Detection corpus.
- Conclusions: In the weakly supervised seen-target scenario, the approach achieves the best reported results on the SemEval Task B dataset.
- Conclusions: Unsupervised pretraining improves results on the SemEval corpus when large labelled corpora are unavailable.
- Future Work: Future work will investigate stance detection for tweets that do not explicitly mention the target.