Source-linked AI summary
Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations
Emily Allaway, Kathleen McKeown
TL;DR
Stance detection needs evaluation across many topics that lack training data and are expressed with substantial lexical variation. The paper introduces VAST and a model using unsupervised generalized topic representations, reporting stronger performance on pro labels and challenging phenomena such as sarcasm. It also identifies further work on topic representations and separating stance from sentiment.
Problem
Existing stance-detection resources provide limited topic coverage and lexical variation, while real-world stance classification may involve thousands of topics with little training data.
Method
The paper creates VAST and develops a zero-shot stance model that uses unsupervised generalized topic representations to capture relationships between topics.
Results
TGA Net performs significantly better than BERT on pro labels, similarly on other labels, and improves performance on challenging phenomena while relying less on misleading sentiment cues.
Takeaways & Limitations
VAST provides broad topic and linguistic variation for zero-shot and few-shot evaluation, while TGA Net addresses challenging stance phenomena including sarcasm.
Takeaways & Limitations
Future work is needed to investigate additional generalized-topic representations and more explicitly decouple stance models from sentiment.
Abstract
from arXiv · showhide
Stance detection is an important component of understanding hidden influences in everyday life. Since there are thousands of potential topics to take a stance on, most with little to no training data, we focus on zero-shot stance detection: classifying stance from no training examples. In this paper, we present a new dataset for zero-shot stance detection that captures a wider range of topics and lexical variation than in previous datasets. Additionally, we propose a new model for stance detection that implicitly captures relationships between topics using generalized topic representations and show that this model improves performance on a number of challenging linguistic phenomena.
1 Introduction
The paper frames zero-shot stance detection as a realistic test of generalization across many unseen topics. It introduces VAST and a model using unsupervised generalized topic representations to address limited topic coverage and linguistic variation.
- Motivation: Existing topic-specific methods require numerous labeled examples for each new topic, which is impractical across thousands of possible topics.Cross-target methods also have limited ability to generalize to a wide variety of topics.
- Motivation: Zero-shot stance detection evaluates classifiers on many completely new topics without human knowledge of their relations to training topics.The paper presents it as a more accurate evaluation of generalization to real-world topic diversity.
- Dataset: VAST broadens evaluation with many topics spanning politics, education, and public health, plus varied expressions of similar topics.This variation is intended to better reflect how people realistically describe topics.
- Model: The proposed model uses unsupervised generalized topic representations from contextualized clustering to capture relationships among topics without explicit human knowledge.The representations provide topic-similarity information for zero-shot stance detection.
- Contributions: The paper reports improved performance on challenging phenomena such as sarcasm and reduced reliance on sentiment cues that can mislead stance classification.The contributions include both the VAST dataset and the proposed model.
2 Related Work
Prior stance-detection datasets and models mainly target topic-specific settings or limited cross-target transfer. Their small topic inventories and narrow topic expressions constrain evaluation on many unseen topics.
- Task Definitions: Topic-phrase stance classifies pro, con, or neutral attitudes toward a noun-phrase topic, whereas topic-position stance uses agree, disagree, discuss, or unrelated labels for a position statement.The paper focuses on the topic-phrase definition.
- Datasets: Existing topic-phrase datasets span forums, information platforms, essays, news comments, and Twitter, but generally contain very few topics.One cited dataset has 16 topics, illustrating the limited inventory.
- Datasets: Most prior datasets are unsuitable for zero-shot stance detection because they provide too few topics, unlike VAST, which includes many new development and test topics.TwitterStance uses only one unseen test topic for cross-target stance detection.
- Datasets: Topic-position datasets generally lack clear, individuated topics, so the paper restricts its focus to topic-phrase stance.These datasets primarily use news articles, headlines, or manually constructed position statements.
- Models: Earlier models commonly trained separate classifiers per topic or on small sets of shared topics, with only a handful addressing cross-target transfer.Some cross-target approaches use weak supervision and unlabeled data.
3 VAST Dataset
VAST combines heuristically extracted and crowdsourced topic–stance annotations from news comments, preserving multiple topics and stances per comment. Its broad topic variation and sparse examples make it suitable for zero-shot and few-shot evaluation.
- Dataset Construction: VAST contains many specific topics annotated on comments from The New York Times Room for Debate section of the ARC Corpus.The source annotations use topic-position stance, so the authors collect topic-phrase stance annotations separately.
- Topic Selection: Topics are initially extracted as filtered noun phrases from ARC stance positions or, when necessary, from article categories, yielding 304 unique topics from 3,365 comments.Proper nouns and overly broad candidates are filtered where possible.
- Annotation: Crowdsourcing asks workers to list relevant topics, assess or correct an automatically generated topic, and assign a five-point stance mapped to pro, con, or neutral.Each topic–comment pair receives annotations from three workers, with majority voting for final labels.
- Annotation: The dataset includes Heur, Corr, and List annotation types, and adds neutral examples by pairing comments with randomly permuted semantically distinct topics.Heur and Corr entries are converted with probability 0.5; noisy List entries are excluded from this conversion.
- Data Analysis: Interannotator agreement is Krippendorff α=0.427 with 75% percentage agreement, indicating stronger-than-random agreement despite task difficulty.The agreement calculation uses stance labels for the provided topic.
- Data Analysis: Annotator disagreement most often reflects inference about overly general or semi-relevant topics, which workers corrected only 30% of the time.This issue identifies a concrete source of noise in topic selection and labeling.
- Data Analysis: Comments can receive multiple topics and potentially different stances, capturing semantic complexity rather than restricting each text to one topic.This increases the difficulty of predicting and annotating stance.
- Data Analysis: VAST has a median of 4 unique topics per comment, while topics have a median of 1 example and a mean of 2.4 examples.The combination of topic variation and sparse examples supports zero-shot and few-shot research.
4 Methods
The paper develops TGA Net, which combines contextual conditional encoding with topic-grouped attention over generalized topic representations to predict stance labels. It represents documents and topics with BERT embeddings, groups related examples, and produces stance probabilities through a feed-forward network.
- Model overview: TGA Net combines contextual conditional encoding, generalized topic representations, topic-grouped attention, and a feed-forward neural network.The model is designed to construct and use relationships between training and evaluation topics without supervision.
- Task formulation: Each example pairs a comment and topic with a stance label, and the model predicts pro, con, or neutral for that pair.The same document can appear with different topics, so stance prediction depends on the topic.
- Contextual encoding: BERT jointly embeds each document-topic pair, providing contextual conditional encoding for topic-dependent stance prediction.This replaces prior use of static word embeddings while conditioning the document representation on the topic.
- Generalized topic representations: Generalized topic representations are centroids of nearest hierarchical-clustering groups formed from concatenated document and topic representations.The representation is computed after clustering the training data in the space of concatenated document and topic vectors.
- Representation construction: Document vectors use tf-idf-weighted BERT token embeddings, whereas topic vectors use uniformly weighted token embeddings.Separate document and topic representations share information across examples while remaining independent of other topics or comments, respectively.
- Attention and prediction: Topic-grouped attention scores similarities between the current topic and related topics, then combines the resulting representation with document token embeddings to compute three output probabilities.The output probabilities are produced by a feed-forward neural network trained with cross-entropy loss.
5 Experiments
Experiments evaluate VAST splits, baselines, pretrained encoders, and TGA Net across zero-shot, few-shot, linguistic, topic-similarity, and sentiment conditions. TGA Net benefits from topic-grouped attention and generalized topic representations, while remaining less sensitive to sentiment cues.
- Data and evaluation: The experiments use document-disjoint partitions, with separate zero-shot topics absent from training and few-shot topics present during training.The zero-shot test set contains 600 unique topics, while the few-shot test set contains 159.
- Topic similarity: Only 16% of zero-shot test topics are LexSimTopics, and TGA Net has a smaller LexSimTopic-to-non-LexSimTopic F1 gap than BERT-joint.Both models perform better on LexSimTopics, with F1 values of .70 for TGA Net and .72 for BERT-joint.
- Models: The evaluated systems include majority and bag-of-words baselines, recurrent cross-target models, BERT encoders, and TGA Net with contextual conditional encoding and topic-grouped attention.TGA Net is the proposed model among the listed systems.
- Overall results: CMaj and BoWV are strong zero-shot baselines, whereas BiCond and Cross-Net generalize poorly across many targets with few examples per target.The cross-target models were designed for a more limited form of zero-shot stance detection.
- Overall results: TGA Net and BERT-joint are statistically indistinguishable on all topics, but topic-grouped attention significantly improves few-shot learning on pro examples (p < 0.05).Conditional encoding also provides a large improvement over separately embedding comments and topics.
- Cluster analysis: TGA Net increasingly outperforms BERT-joint as cluster topic diversity grows, but performance suffers when clusters exceed 182 examples.The authors suggest that very large clusters contain stance signals too diverse for topic-grouped attention to use.
- Linguistic phenomena: TGA Net performs better on difficult linguistic phenomena, with particularly large improvements on quotation and sarcasm examples.These phenomena remain challenging for both models, especially mlS.
- Stance and sentiment: Con stance often uses positive sentiment words, so sentiment polarity does not directly map onto stance labels.In the test set, 80% of pro examples have more positive than negative words, compared with 41% of con examples having more negative than positive words.
6 Conclusion
TGA Net uses generalized topic representations to model relationships between topics and improves stance detection on pro labels and difficult phenomena. The authors identify further work on topic representations, sentiment decoupling, and difficult cases.
- TGA Net uses generalized topic representations to implicitly capture relationships between topics.The representations are used as the model’s central mechanism for zero-shot stance detection.
- TGA Net performs significantly better than BERT on pro-label stance detection and similarly on other labels.
- Extensive analysis finds substantial improvement on challenging phenomena such as sarcasm and less reliance on misleading sentiment cues.
- Future work will investigate additional generalized topic representations, more explicit sentiment decoupling, and improved performance on difficult phenomena.
A.0.1 Crowdsourcing
The annotation task was conducted through HITs on Amazon Mechanical Turk, with annotators paid per task and quality control removing a subset of collected HITs.
- Crowdsourcing: Annotators were paid $0.13 per HIT.
- Crowdsourcing: Of 696 collected HITs, 183 were removed through quality control.
A.0.2 Data
The dataset examples document topic extraction, correction, related listed topics, stance labels, and neutral cases. They also reveal annotation complexity, linguistic variation, and occasional noise in listed-topic labels.
- Data: Complete examples show extracted topics, annotator corrections, related listed topics, and stance labels.
- Data: Annotators corrected topics that were impossible to take a position on or insufficiently specific.
- Data: Different annotators sometimes disagreed about whether to correct the same topic, illustrating the complexity of stance annotation.
- Data: Similar topics can have different expressions, while listed-topic labels are generally correct but occasionally differ from the stance toward the extracted topic.
- Data: Constructed neutral examples use topics distinct from comment content, whereas semantically related neutral examples are less common and harder.
A.1.1 Hyperparameters
The appendix describes model implementation and tuning procedures, challenging-phenomenon evaluation, sentiment-word swap construction, and supplementary dataset examples and search configurations.
- Hyperparameters: Neural models were implemented in PyTorch, logistic regression in scikit-learn, and experiments used one TITAN Xp GPU.
- Hyperparameters: Hyperparameters were selected through uniform sampling, with search spaces and best configurations reported for the evaluated models.
- Hyperparameters: Models were tuned on development data using macro-averaged F1 across zero-shot classes for hyperparameter selection.
- Hyperparameters: TGA Net and BERT-joint were evaluated on implicit topics, multiple topics, multiple stances, quotations, and sarcasm.
- Hyperparameters: Sentiment-word swaps used MPQA sentiment words and WordNet synonyms with opposite polarity, while ignoring word sense and part of speech.
- Hyperparameters: Appendix tables provide examples of hard phenomena, annotation data, neutral cases, model search settings, and development results.