PaperlayerView exact revision on arXiv

Source-linked AI summary

Stance and Sentiment in Tweets

Saif M. Mohammad, Parinaz Sobhani, Svetlana Kiritchenko

arXiv:1605.01655v1cs.CL

TL;DR

The paper addresses the gap between sentiment analysis and stance detection by creating a tweet–target dataset annotated for both, including implicit targets and differing opinion targets. It evaluates a simple SVM with lexical, sentiment, embedding, and distant-supervision features, achieving higher F-score than the shared-task winner while showing that sentiment alone is insufficient for stance.

  • Problem

    Stance and sentiment can diverge, but prior work lacked a dataset jointly annotating both properties for tweet–target pairs, including implicit and mismatched opinion targets.

  • Method

    The paper creates a jointly annotated five-target tweet dataset and applies a linear-kernel SVM using n-grams, sentiment lexicons, word embeddings, and distant supervision.

  • Results

    70.3 F-score exceeded the shared-task winner’s 67.8, while gold sentiment and opinion-target annotations yielded only 59.6% F-score for stance prediction.

  • Takeaways & Limitations

    Sentiment features benefit stance classification but are not sufficient, and automatic systems remain weak when tweets express opinion about an entity other than the target.

  • Takeaways & Limitations

    The dataset excludes 28% of instances that failed the 60% annotator-agreement threshold.

Abstract

from arXiv · show

We can often detect from a person's utterances whether he/she is in favor of or against a given target entity -- their stance towards the target. However, a person may express the same stance towards a target by using negative or positive language. Here for the first time we present a dataset of tweet--target pairs annotated for both stance and sentiment. The targets may or may not be referred to in the tweets, and they may or may not be the target of opinion in the tweets. Partitions of this dataset were used as training and test sets in a SemEval-2016 shared task competition. We propose a simple stance detection system that outperforms submissions from all 19 teams that participated in the shared task. Additionally, access to both stance and sentiment annotations allows us to explore several research questions. We show that while knowing the sentiment expressed by a tweet is beneficial for stance classification, it alone is not sufficient. Finally, we use additional unlabeled data through distant supervision techniques and word embeddings to further improve stance classification.

1. INTRODUCTION

The paper introduces stance detection in Twitter as distinct from sentiment analysis, presents a jointly annotated stance–sentiment dataset, and develops a strong stance classifier. Experiments show sentiment helps but does not solve stance detection, especially when opinion targets differ from the given target.

  • Motivation: Stance detection determines whether a text author favors, opposes, or is neutral toward a given target, which may be implicit.The target can be a person, organization, policy, movement, or product.
  • Motivation: Stance differs from sentiment because the given target may be absent from the tweet or may differ from the entity receiving the opinion.A tweet praising Jeb Bush can support an inference about stance toward Donald Trump.
  • Contributions: More than 4000 tweets were annotated for stance and sentiment across five targets, including Atheism, climate change, the feminist movement, Hillary Clinton, and abortion legalization.The dataset also records whether the opinion target matches the given target.
  • Contributions: 19 teams entered the SemEval-2016 stance shared task, whose best system achieved an overall average F-score of 67.8.The dataset supplied 2,914 training instances and 1,249 test instances for five targets.
  • Contributions: 70.3 F-score was achieved by a simpler linear-kernel SVM using n-grams, sentiment lexicons, and word-embedding features.The system used both training-instance features and external resources, including additional unlabeled data.
  • Findings: Sentiment features are useful but less effective for stance than for sentiment classification, and automatic systems perform poorly when opinion concerns another entity.For such instances, the text classifier performs close to the majority baseline.

2. A DATASET FOR STANCE FROM TWEETS

The dataset was designed to capture explicit and implicit stance across balanced target categories, then annotated through crowdsourcing for stance and opinion focus. Quality control, agreement thresholds, and unlabeled domain data support subsequent analysis and modeling.

  • Dataset design: The dataset was designed to include explicit target mentions, indirect references, and tweets expressing opinion about entities other than the target.The design sought harder cases involving pronouns, epithets, honorifics, and relationships.
  • Dataset design: Five targets commonly understood in the United States were selected: Atheism, climate change, the feminist movement, Hillary Clinton, and abortion legalization.The target list was intended to support annotation by US respondents.
  • Tweet selection: Over two million tweets were collected through query hashtags categorized as favor, against, or stance-ambiguous.Query hashtags were removed when they appeared at the end of tweets, and manually annotated samples were drawn afterward.
  • Tweet selection: Stance-indicative hashtags were not treated as definitive labels because hashtags can conflict with the tweet’s actual stance.The authors manually annotated tweet–target pairs after preprocessing and sampled up to 1000 tweets per target.
  • Stance annotation: Annotators separately judged stance toward the target and whether the tweet’s opinion focused on that target or another entity.The questionnaire included support, opposition, neutrality, and no-clue options, plus three opinion-focus categories.
  • Unlabeled data: Unlabeled tweets in the domain corpus were retained to provide information such as relationships between relevant entities for stance classification.These tweets were used as a source of additional information in later experiments.

3. LABELING THE STANCE SET FOR SENTIMENT

The authors annotate the Stance Dataset for sentiment to study how sentiment relates to stance, using a three-way sentiment scheme designed to address annotation ambiguity.

  • Research question: The annotation effort addresses whether sentiment is correlated with stance by labeling the same tweets for sentiment without providing annotators the target.Prior sentiment annotation often left labeling criteria underspecified, creating potential disagreement and uncertainty, especially for sarcasm, supplications, and rhetorical questions.
  • Annotation challenges: A rhetorical question can receive either a neutral interpretation or a negative interpretation when it reveals the speaker’s frustration.The example illustrates why sentiment labels may be ambiguous even when the text is short.
  • Sentiment annotation: Sentiment annotation asks what kind of language the speaker uses, distinguishing positive, negative, sarcastic or mocking, and neither clearly positive nor negative expressions.The wording focuses annotators on language rather than whether events benefit or harm them.
  • Sentiment annotation: The authors combine sarcasm and mockery with negative tweets, and combine non-positive, non-negative responses into a neither category for three-way classification.The resulting task parallels the three-way stance prediction task.
  • Annotation quality: 85.6% inter-annotator agreement was achieved across the three sentiment classes.Each instance received annotations from at least five CrowdFlower annotators.

4. PROPERTIES OF THE STANCE DATASET

The dataset is timestamp-partitioned and contains substantial variation in opinion targets and sentiment across target entities. An interactive visualization supports exploration of these distributions and their relationships.

  • Dataset partition: The first 70% of timestamp-ordered tweets form training data, while the last 30% form test data.The partition is based on tweet timestamps rather than random assignment.
  • Opinion targets: 27% to 46% of instances are labeled opinion towards other across different targets.These instances concern an opinion target different from the given target of interest.
  • Opinion targets: Tweets can express stance toward the given target even when the target is not explicitly mentioned, as illustrated for Hillary Clinton and Legalization of Abortion.Examples use references to Benghazi and #Jeb16, or to foetuses, women’s rights, and freedoms.
  • Class distributions: Tweets for every target except Atheism are predominantly negative.This distribution is reported for the training and test sets.
  • Interactive visualization: All visualization components support interactive filtering by target, stance, opinion target, and polarity, with selected examples shown in the Tweets component.Users can combine multiple filters to inspect specific subsets of the dataset.
  • Class distributions: Most targets have more against instances than favor or neither instances, and a majority of instances are labeled against the target.The visualization presents these distributions through target–stance treemaps and stacked bars.

5. A COMMON TEXT CLASSIFICATION FRAMEWORK FOR STANCE AND SENTIMENT

The authors use a linear-kernel SVM framework with textual and external-resource features, training stance models separately for each target. They compare feature usefulness and evaluate target-level scoring choices.

  • Feature design: Word and character n-grams and sentiment lexicons are established candidate features, but their usefulness for stance classification is evaluated rather than assumed.The authors frame feature selection as an open question because prior sentiment-analysis findings may not transfer directly to stance.
  • Classification framework: Stance models are trained separately for each target because stance-related words and concepts are not expected to generalize across targets.Experiments with a single model trained on all tweets produced lower results.
  • Classification framework: The system trains a linear-kernel SVM using word and character n-grams, sentiment features, and word-embedding features from additional unlabeled data.Tweets are tokenized and part-of-speech tagged before classification.
  • Evaluation: F-microT weights more frequent target classes more heavily, whereas F-macroT requires performance across targets by averaging target-specific Faverage scores.The two measures emphasize different aspects of target-level performance.
  • Evaluation: Faverage combines Fpositive and Fnegative, while neither instances receive no direct credit but still affect errors between the three classes.The measure can be computed across all test instances or averaged over targets separately.

6. RESULTS OBTAINED BY AUTOMATIC SYSTEMS

The experiments compare stance and sentiment classifiers, oracle benchmarks, and feature configurations. A simple n-gram-based SVM performs strongly for stance, while sentiment is easier to predict and remains distinct from stance.

  • Evaluation setup: The evaluation compares classifiers using provided training data and existing resources, with additional unlabeled-data experiments reserved for Section 7.The reported section focuses on systems using the provided training data and resources such as sentiment lexicons.
  • Stance classification: Oracle sentiment benchmarks test how far stance detection can be solved by mapping sentiment labels to stance labels, with or without the opinion target.The sentiment-to-stance mapping is selected to maximize F-macroT; the oracle uses gold sentiment labels.
  • Stance classification: A simple SVM with n-gram features alone exceeds the majority baseline and often surpasses both the SemEval winner and oracle benchmarks.The paper notes that this result is surprising because the shared-task leaders used recurrent neural networks and word embeddings.
  • Stance classification: Adding target features yields small stance gains, whereas sentiment-lexicon and other feature additions do not improve the n-gram classifier.The combination of n-grams and target or sentiment features also fails to improve performance beyond the reported configurations.
  • Stance classification: Stance classification is considerably harder when tweets express opinion about another entity rather than the given target.The paper identifies this target-mismatch subset as an insufficiently addressed challenge for the research community.
  • Sentiment classification: Sentiment classifiers outperform stance classifiers, and sentiment prediction remains similar whether the tweet concerns the given target or another entity.N-gram features beat baselines, sentiment lexicons provide marked further gains, and adding all features does not beat n-grams plus sentiment features.

7. STANCE CLASSIFICATION USING ADDITIONAL UNLABELED TWEETS

The paper augments stance classification with weakly labeled tweets and word-association or embedding features. These additions help unevenly across targets because pseudo-labeled data can be noisy and different methods benefit different subsets.

  • 7.1. Distant Supervision: Distant supervision automatically generates training data from text indicators such as stance-indicative hashtags.The authors compile manual and automatic hashtag sets, then use them to select tweets from an unlabeled domain corpus.
  • 7.1. Distant Supervision: 555 instances test whether manually selected stance-indicative hashtags can predict Favor–Against labels after hashtag removal.The experiment evaluates whether the remaining tweet content retains enough stance information for classification.
  • 7.1.1. Classification Experiments with Distant Supervision: Additional pseudo-labeled training data improves performance for three of five targets, with marked gains only for ‘Hillary Clinton’.The authors caution that pseudo-labeled data may be too noisy to incorporate directly in other test sets.
  • 7.1.2. Classification Experiments with Distant Supervision: Word-association features improve results for all targets, especially ‘Atheism’, ‘Feminist Movement’, and ‘Legalization of Abortion’.Associations from the Automatic Hashtag Corpus are more informative than those from the Manual Hashtag Corpus.
  • 7.2. Word Embeddings: Word embeddings improve results for every target except ‘Hillary Clinton’.The embeddings are generated from target-associated domain tweets, rather than a generic tweet corpus.
  • Overall findings: The three augmentation methods affect different data subsets differently, so no single method benefits every target equally.‘Hillary Clinton’ benefits most from additional training data but not from embedding features.

8. RELATED WORK

Related work places stance detection within research on debates, sentiment analysis, political-tweet analysis, textual entailment, and distant supervision. The paper frames tweet stance detection as a specialized textual-entailment problem involving author attitude toward a target.

  • Stance Detection: Earlier stance-detection datasets and systems largely focus on two-sided online debate forums and ideological controversies.The cited datasets were drawn from forums such as 4forums.com and createdebates.com, with labels often supplied by post authors.
  • Sentiment Analysis and Opinion Mining: Tweet sentiment and opinion mining research includes general sentiment analysis, aspect-based sentiment analysis, and political-tweet analysis.Political-tweet studies address sentiment, emotion, alignment, contentious issues, polarization, and voting intentions.
  • Textual Entailment: Textual entailment infers hypotheses from source text and supports applications including question answering, machine translation, and information retrieval.The paper situates stance detection among these entailment applications.
  • Textual Entailment: Tweet stance detection can be viewed as textual entailment in which the hypotheses are fixed as favor or against and the system must infer the author’s attitude.The task requires deriving both the tweet’s meaning and its author’s stance toward the given target.
  • Distant Supervision: Distant supervision uses indirectly or weakly labeled data and is widely applied to relation extraction and sentiment or emotion analysis.Relation-extraction systems can label sentences using entity pairs linked in external knowledge sources.

9. CONCLUSIONS AND FUTURE WORK

The paper contributes a jointly annotated stance-and-sentiment tweet dataset and a competitive stance classifier, then analyzes how sentiment, opinion targets, and additional unlabeled data relate to stance prediction. Future work targets richer representations, cross-target learning, and temporal modeling.

  • Conclusions: The dataset is the first presented with tweet annotations for both stance toward given targets and language polarity.Tweets are also annotated for whether the opinion concerns the given target or another entity.
  • Conclusions: An F-score of 70.3 exceeded the best-performing system among 19 SemEval-2016 shared-task teams.The proposed system is a linear-kernel SVM using word and character n-grams, sentiment features, and word-embedding features.
  • Conclusions: Sentiment features help stance detection but are less effective for stance than for sentiment prediction, and gold sentiment plus opinion-target annotations reach only 59.6% F-score.This analysis shows that sentiment information alone does not determine stance.
  • Conclusions: Humans detect stance toward a given target even when a tweet expresses opinion toward another target, whereas automatic systems perform poorly on those instances.The dataset explicitly records whether the opinion target matches the given target of interest.
  • Future Work: Future work includes dependency-based features, entity-relationship knowledge, joint deep models, cross-target learning, stance-conveyance modeling, and temporal analysis.The proposed directions address richer modeling and settings without stance-labeled instances for the target of interest.
Loading 1605.01655v1…