Source-linked AI summary
How would Stance Detection Techniques Evolve after the Launch of ChatGPT?
Bowen Zhang, Daijun Ding, Liwen Jing, Genan Dai, Nan Yin
TL;DR
Stance detection seeks to classify an author’s position toward a target despite challenges involving limited labeled data, social-media information, and model interpretability. This paper evaluates ChatGPT with direct zero-shot prompting and finds strong benchmark performance alongside explanations of its predictions, while noting that only one prompt template was examined.
Problem
Stance detection faces insufficient labeled data and social-media information challenges, while deep learning models are difficult to explain.
Method
The paper directly prompts ChatGPT for stance predictions and evaluates zero-shot and in-domain setups on SemEval-2016 and P-Stance.
Results
ChatGPT achieves SOTA zero-shot performance, improves 16.6% on average over PT-HCL, and outperforms most baselines in in-domain tasks.
Takeaways & Limitations
ChatGPT’s zero-shot prompting and self-explanations could transform stance-detection research and support explanatory AI.
Takeaways & Limitations
The study examines only one prompt template and identifies better prompt templates as an open direction.
Abstract
from arXiv · showhide
Stance detection refers to the task of extracting the standpoint (Favor, Against or Neither) towards a target in given texts. Such research gains increasing attention with the proliferation of social media contents. The conventional framework of handling stance detection is converting it into text classification tasks. Deep learning models have already replaced rule-based models and traditional machine learning models in solving such problems. Current deep neural networks are facing two main challenges which are insufficient labeled data and information in social media posts and the unexplainable nature of deep learning models. A new pre-trained language model chatGPT was launched on Nov 30, 2022. For the stance detection tasks, our experiments show that ChatGPT can achieve SOTA or similar performance for commonly used datasets including SemEval-2016 and P-Stance. At the same time, ChatGPT can provide explanation for its own prediction, which is beyond the capability of any existing model. The explanations for the cases it cannot provide classification results are especially useful. ChatGPT has the potential to be the best AI model for stance detection tasks in NLP, or at least change the research paradigm of this field. ChatGPT also opens up the possibility of building explanatory AI for stance detection.
1 Introduction
Stance detection is an important NLP and social computing task whose methods have evolved alongside broader language-model advances. This paper examines whether ChatGPT can shift the field through direct prompting and explanatory predictions.
- Stance detection matters because individual and public positions on issues influence decision making and explain attitudes toward many topics.
- Stance detection research progressed from rule-based and traditional machine-learning methods toward deep learning and large pre-trained language models.
- ChatGPT’s rapid adoption raises whether dialogue-optimized pre-trained language models represent a new stage for NLP research.
- The paper evaluates ChatGPT for stance detection through direct prompting and reports SOTA or similar performance on SemEval-2016 and P-Stance.
- ChatGPT can explain the reasons behind its stance decisions when asked, and multi-input interaction may further improve performance.
2 Related Work
Stance detection is a text-classification task whose research methods moved from rules and traditional machine learning through neural models and PLM fine-tuning. The field is now drawing attention to pre-train, prompt, and predict approaches.
- Stance detection classifies an author’s stance toward a target as Favor, Against, or Neither, with Neutral sometimes added.
- The task evolved from parliamentary debates to social-media content, including Twitter, Facebook, Instagram, and online blogs.
- Research methods progressed from rule-based techniques to traditional machine learning, deep neural networks, and PLM pre-train-and-fine-tune methods.
- Traditional classifiers used selected text features, while attention-based and GCN methods modeled target-specific information and target–text relations.
- A fifth paradigm, pre-train, prompt, and predict, has recently begun attracting wide attention.
3 Methods and Results
The paper evaluates a direct-question prompting method for ChatGPT on stance detection, using zero-shot and in-domain comparisons across SemEval-2016 and P-Stance. ChatGPT achieves strong benchmark performance with a simple prompt.
- Method: The task formulation represents each instance as input text paired with a target, whose stance label the predictor must infer.The formal setup treats stance detection as prediction of a stance label for text x toward target p.
- Method: ChatGPT predicts stance by directly answering a question about a tweet’s attitude toward a specified target.The prompt asks ChatGPT to select favor, against, or neutral for the input sentence and target.
- Experimental setup: The experiments use F1-avg and macro-F1 to evaluate stance classification performance.F1-avg averages F1 on Favor and Against labels, while macro-F1 is denoted F1-m.
- Experimental setup: Evaluation covers zero-shot testing without training-data adjustment and in-domain baselines optimized on 80% of tweets.The comparison uses SemEval-2016 and P-Stance datasets, with results summarized in Tables 1–3.
- Results: 16.6% improvement over PT-HCL on average establishes ChatGPT’s zero-shot performance advantage.ChatGPT achieves state-of-the-art results in the zero-shot setup and outperforms all baselines in most tasks compared with in-domain methods.
4 Discussions and Future Work
The discussion identifies zero-shot prompting, self-explanation, and multi-round interaction as directions that could reshape stance detection with ChatGPT.
- Results: ChatGPT’s zero-shot prompting achieves state-of-the-art results and could potentially transform stance detection research.The paper reports state-of-the-art results in both zero-shot and in-domain setups.
- Future work: Only one prompt template was tested, leaving manual or automated prompt selection as a route to improve zero-shot performance.The paper specifically asks whether better prompt templates could improve performance or extend ChatGPT to other NLP tasks.
- Future work: ChatGPT explains stances expressed explicitly or implicitly, supporting the possibility of explanatory AI for stance detection.The authors interpret these explanations as evidence of logic reasoning rather than pure probability calculation.
- Future work: When ChatGPT cannot classify a tweet, its explanations can identify insufficient information and expose data that may be inherently difficult to label.The paper gives missing target references and absent opinions as examples of why prediction may fail.
- Future work: Multi-round conversations could improve stance detection by supplying background knowledge, missing text, or classification examples.The paper presents multi-round interaction as a future direction for resolving flawed or incomplete tweets.