Source-linked AI summary
Will-They-Won't-They: A Very Large Dataset for Stance Detection on Twitter
Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, Nigel Collier
TL;DR
Stance detection lacks a large dataset for reliable model comparison. The paper introduces WT–WT, an expert-annotated dataset of English tweets about financial M&A, and evaluates competitive models on it. Experiments show a consistent gap between state-of-the-art systems and the human upper bound, making the dataset a challenge for current models.
Problem
Stance detection lacks a large dataset that would allow reliable comparison between models.
Source: §1 · p. 1Method
The authors construct WT–WT from English tweets about financial M&A and evaluate 11 competitive and state-of-the-art stance detection models on the benchmark.
Source: §1 · p. 1Results
A consistent (>10%) performance gap separates state-of-the-art models from the human upperbound, making WT–WT a strong challenge for current models.
Source: §4 · p. 5Takeaways & Limitations
WT–WT provides a benchmark that challenges current stance detection models and supports research on stance detection and domain adaptation.
Source: §4 · p. 5
Abstract
from arXiv · showhide
We present a new challenging stance detection dataset, called Will-They-Won't-They (WT-WT), which contains 51,284 tweets in English, making it by far the largest available dataset of the type. All the annotations are carried out by experts; therefore, the dataset constitutes a high-quality and reliable benchmark for future research in stance detection. Our experiments with a wide range of recent state-of-the-art stance detection systems show that the dataset poses a strong challenge to existing models in this domain.
1 Introduction
The introduction presents WT–WT as a large English-tweet stance dataset for rumor verification, addressing the lack of data for reliable model comparison. It focuses on financial M&A discussions and adds a benchmark and domain-shift evaluation.
1 Introduction
- Stance detection also supports fake-news detection, fact-checking, and rumor verification as a crucial NLP sub-step (Mohammad et al., 2017; Pomerleau and Rao, 2017; Vlachos and Riedel, 2014; Baly et al., 2018; Zubiaga et al., 2018b).Source: §1 · p. 1
- Stance detection lacks a large dataset for reliable model comparison, motivating WT–WT, an English-tweet resource targeting rumor verification.The authors choose Twitter because it is relevant to public, political, and enterprise discussion of rumors.Source: §1 · p. 1
- The dataset uses financial M&A discussions, whose evolution from early talks through deal closure or rejection resembles rumor verification.Early-stage discussions between companies are usually not publicly disclosed.Source: §1 · p. 1
- Financial NLP research has focused mainly on sentiment, text mining, and thesauri, while modeling perceptions of mergers might ultimately help explain stock-price revaluation.The authors position stance detection as a complementary role in financial-market analysis.Source: §1 · p. 1
- WT–WT contains 51,284 tweets and is an order of magnitude larger than other stance datasets of user-generated data.The authors also evaluate 11 competitive and state-of-the-art models and examine robustness on an entertainment-domain M&A operation.Source: §1 · pp. 1–2
2 Building the WT–WT Dataset
This section describes the construction and assessment of WT–WT, an expert-annotated Twitter stance dataset spanning five mergers. It reports the dataset’s scale, annotation quality, and label patterns.
2.1 Data Retrieval
- Five M&A operations—four in healthcare and one in entertainment—were sampled from one year before each proposed merger through six months after it took place.Tweet IDs were retrieved using company names or acronyms, and company mentions paired with merger-specific terms.Source: §2.1 · p. 2, §2 · p. 2, Table 2 · p. 3
2.2 Task Definition and Annotation Guidelines
- WT–WT labels tweets as support, refute, comment, or unrelated, with comment covering merger discussion that expresses neither support nor doubt that it will happen.The same tweet can receive different stance labels depending on the target.Source: §2.2 · p. 2
2.3 Data Annotation
- Tweets were independently labeled by 2–6 of ten financial-domain experts; majority vote resolved disagreements, with 0.2% of samples discarded when no majority was possible.Source: §2.3 · pp. 2–3
2.6 Comparison with Existing Corpora
- At 51,284 tweets, WT–WT is the largest publicly available user-generated stance dataset, expert-annotated and extending coverage to finance.The authors position it for model comparison, scalability, domain adaptation, and pretraining before moving to data-scarce domains.Source: Table 2 · p. 3, §2.6 · pp. 3–4
2.4 Quality Assessment
- Annotator-pair agreement averaged κ=0.67, while an expert’s labels for a 3,000-tweet sample agreed with gold labels at κ=0.88.Support–comment samples were the most common source of annotator disagreement; unrelated–comment disagreement was higher than in datasets with randomly generated unrelated samples, indicating a more challenging setting.Source: §2.4 · p. 3
2.5 Label Distribution
- Across operations, refuting and supporting sample proportions correlate with whether a merger was approved or blocked, while comments are more frequent than support.Source: §2.5 · p. 3
3 Experiments and Results
The experiments compare 11 stance systems on WT–WT, where SiamNet leads on averaged and weighted averaged F1; cross-domain evaluation also finds a consistent performance drop.
3.1 Experimental Setup
- The 11 reimplemented architectures take a tweet and its target as input; their embeddings are shared between tweets and targets and kept fixed during training.The setup also describes common preprocessing and Twitter-pretrained GloVe initialization.Source: §3 · p. 4, §3.1 · p. 4
3.2 Results and Discussion
- SiamNet achieves the best averaged and weighted averaged F1, while SVM remains a strong, robust baseline.The relative gains of CrossNet over BiCE and HAN over TAN match findings on SemEval 2016-Task 6.Source: §3.2 · p. 4
- Support and comment are often confused; unrelated tweets can also mention a company without discussing its merger, complicating classification.SVM and CharCNN perform best on unrelated samples, suggesting character-level information may help future architectures.Source: §3.2 · pp. 4–5
- CVS_AET and CI_ESRX have the lowest average performance across models, alongside higher annotator disagreement.Source: §3.2 · p. 5
3.3 Robustness over Domain Shifts
- Cross-domain tests show SiamNet consistently outperforming BiCE, with accuracy and macro-averaged F1 dropping across classes, especially for comments.The results indicate strong domain dependency and room for future research.Source: §3.3 · p. 5
4 Conclusions
WT–WT is a large, expert-annotated stance dataset, and experiments show that it remains challenging for current models.
4 Conclusions
- WT–WT contains over 50,000 labeled tweets, and 11 strong models show a consistent greater-than-10% performance gap from the human upper bound.The authors present this gap as evidence that WT–WT challenges current models; its multiple domains also support future cross-target and cross-domain adaptation research.Source: §4 · p. 5
A.1 Crawling Specifications
The crawling specifications list merger-related search terms paired with company identifiers.
A.1 Crawling Specifications
- Crawling used one of seven M&A-specific terms together with a mention of a company or acronym.The listed terms are merge, acquisition, agreement, acquire, takeover, buyout, and integration.Source: §A.1 · p. 8
A.2 Description and Examples of the Considered Labels
The annotation guidelines assign each tweet one of four labels based on its relation to a specified merger.
A.2 Description and Examples of the Considered Labels
- Support labels tweets stating that the merger is happening, likely, or cleared.Source: §A.2 · p. 8
- Comment labels tweets about the merger that neither assert it is happening nor refute it, including discussion of its implications.Source: §A.2 · p. 8
- Refute labels tweets doubting the merger or mentioning potential roadblocks to it.Source: §A.2 · p. 8
- Unrelated covers tweets about neither the merger nor its implications, including other topics involving the companies or a different merger.Source: §A.2 · p. 8
B.1 Encoder’s Architectures
The section describes a range of encoder architectures, from simpler feature-based models to target-conditioned and attention-based models, and reports training hyperparameters in Table 6.
B.1 Encoder’s Architectures
- SVMs use word- and character-ngram features; the MLP uses tweet and target TF–IDF with cosine similarity, while EmbAvg averages their word embeddings.The MLP has one dense layer and EmbAvg has two.Source: §B.1 · p. 9
- CharCNN and WordCNN apply convolution over characters and words, respectively.Both follow Vijayaraghavan et al. (2016).Source: §B.1 · p. 9
- BiCE initializes the tweet BiLSTM with the target BiLSTM’s final states; CrossNet augments BiCE with self-attention and two dense layers.Source: §B.1 · p. 9
- SiamNet combines tweet and target representations with inverse-exponential Manhattan similarity, while Co-MatchAtt uses co-matching attention between separately encoded tweet and target.Source: §B.1 · p. 9
- TAN applies target-specific attention over target-augmented embeddings, while HAN uses two attention levels and linguistic information.HAN uses sentiment, dependency, and argument information.Source: §B.1 · p. 9
- Table 6 lists training hyperparameters, using original-paper settings whenever reported.Source: Table 6 · p. 9
B.2 Preprocessing Details
The preprocessing pipeline normalizes and tokenizes tweet text, limits the vocabulary, and extracts representations or linguistic signals for model inputs.
B.2 Preprocessing Details
- Tweets are lowercased and tokenized with NLTK’s TwitterTokenizer; digits and URLs are normalized, low-frequency users are normalized, and hashtag signs are removed.High-frequency users, including official company, media, and US politician accounts, are retained with the @ stripped.Source: §B.2 · p. 9
- Keeping tokens that occur at least three times yields a 19,561-entry vocabulary across healthcare and entertainment.Source: §B.2 · pp. 9–10
- TF–IDF vectors feed the TFIDF–MLP, while HAN uses MPQA sentiment sequences, Stanford Parser dependency sequences, and argument labels predicted by an SVM.The argument-label SVM is trained on Hasan and Ng (2013)’s training data.Source: §B.2 · p. 10