X-Stance: A Multilingual Multi-Target Dataset for Stance Detection

1 Introduction

• Multilingual datasets are relatively small and single-target, while models are often target-specific and cross-target performance can trail fully supervised performance.The paper identifies cross-lingual and cross-target evaluation as unresolved questions.Source: §1 · p. 1
• X-stance contains more than 67,000 candidate answers to over 150 Swiss political questions, with questions available in English, Swiss Standard German, French, and Italian.Comment language depends on the candidate’s region of origin.Source: §1 · p. 1
• The dataset turns each political question into a natural-language target and uses the candidate’s explanatory comment as the text to classify.Candidates submit comments alongside categorical answers on the Smartvote voting advice application.Source: §1 · p. 1
• The multilingual BERT baseline has accuracy comparable to prior stance benchmarks and generalizes to some degree across languages and targets.The authors characterize the result as leaving ample room for improvement.Source: §1 · p. 1
• The dataset and code for reproducing the baseline models are publicly available.Source: §1 · p. 1

2 Related Work

• The IberEval Spanish-Catalan tweet datasets focus on Catalan independence, while the cited French, Italian, and German resources are monolingual (Lai et al., 2020; Lai et al., 2018, 2020; Clematide et al., 2012).The IberEval datasets were described as the first truly multilingual stance datasets known to the authors.Source: §2 · p. 2
• Earlier multi-target work tested transfer to unseen targets or modeled multiple targets within one text, whereas X-stance focuses on learning from many samples with many different targets (Mohammad et al., 2016b; Sobhani et al., 2017).Source: §2 · p. 2

Representation Learning for Stance Detection

• Target-conditioned encoding has been used to generalize to unseen targets, including an approach that conditions tweet representations on target sequences.Target-specific BERT evaluation, by contrast, used a single-segment encoding that prevented cross-target transfer.Source: Representation Learning for Stance… · pp. 2–3

3.1 Task Definition

• The task takes a political question and a comment as input and predicts ‘favor’ or ‘against’; it has no ‘neither’ class.The authors report the macro-average of the two class F1 scores to support comparison with earlier benchmarks (Mohammad et al., 2016b).Source: §3.1 · p. 3

3.2 Data Collection

• Smartvote candidates answer translated questions with yes/no gradations and may add comments of up to 500 characters; the questions are designed by political scientists to cover current issues.The dataset draws from elections between 2011 and 2020 and includes national-election answers; communal and cantonal answers to questions also asked nationally augment training only.Source: §3.2 · pp. 3–4
• The collection filters out non-closed or unclear political questions, missing or short comments, non-self-contained comments, and comments containing URLs; about one fifth of comments are removed.The dataset merges yes/rather yes into ‘favor’ and rather no/no into ‘against’, while leaving comment text otherwise unpreprocessed.Source: §3.2 · pp. 3–4

3.3 Data Split

• Italian is reserved for testing, two topics are held out for cross-topic evaluation, and 16 questions across ten other topics are held out to test transfer to unseen questions.Validation and test sets use respondents sampled from the remaining German and French data; cross-lingual and cross-target test sets were omitted because they were too small for significant results.Source: §3.3 · p. 4

3.4 Analysis

• Favor rates vary by question but are balanced across topics, with 49.4% favor in held-out topics and 50.0% in other topics.The question set includes paraphrases and ranges from short prompts to convoluted questions that may be harder to process semantically.Source: §3.4 · p. 5
• German samples outnumber French samples by about 3:1 across training and test sets, and the authors use a multi-source multilingual BERT baseline.They motivate the multi-source choice by prior findings that multilingual training data benefit M-BERT in supervised and cross-lingual settings.Source: §3.4 · p. 5

4.1 Majority Class Baselines

• The majority baselines predict either the most frequent class across all training targets or the most frequent class for each target question; the latter applies only to intra-target tests.Source: §4.1 · p. 5

4.2 Bag-of-Words Baseline

• fastText concatenates each comment with its language-matched question, uses Europarl tokenization and 300-dimensional word vectors, and omits pretrained multilingual embeddings after preliminary tests showed no benefit.Source: §4.2 · pp. 5–6

4.3 Multilingual BERT Baseline

• The main neural baseline fine-tunes multilingual BERT, treating the question and comment as paired segments whose [CLS] representation feeds a classifier.The model is trained with cross-entropy loss.Source: §4.3 · p. 6
• M-BERT outperforms previous baselines cross-lingually, including the Italian zero-shot result over the target-wise majority baseline; it also remains strongest in cross-target tests, surpassing majority baselines.The tables report cross-lingual baseline scores and German and French scores, plus their harmonic mean, for each cross-target test set.Source: §4.3 · p. 6, Table 3 · p. 6, Table 4 · p. 7

4.4 How Important is Consistent Language?

• Using English versions of all questions has a negligible effect in most settings but increases cross-lingual performance.The default instead matches question and comment languages; the English-question variant tests vertical language consistency.Source: §4.4 · pp. 6–7

4.5 How Important are the Segments?

• Removing either segment reduces performance across evaluation settings, with a larger loss when comments are removed; comment-free scores are only slightly different from the target-wise majority baseline.Source: §4.5 · p. 7
• Removing questions hurts supervised and cross-lingual settings especially, while the effect is less pronounced cross-target; questions help distinguish comments reused with different targets.Source: §4.5 · p. 7

4.6 How Important are Spelled-Out Targets?

• Replacing natural-language questions with trainable target embeddings lowers M-BERT performance in supervised and cross-lingual settings, indicating that spelled-out questions provide useful linguistic detail.Source: §4.6 · p. 7

5 Discussion

• M-BERT achieves reasonable accuracy on X-stance, outperforming majority-class baselines and a fastText classifier.Its supervised score is described as having similar difficulty to SemEval-2016 and MPCHI benchmark results for BERT.Source: §5 · pp. 7–8
• In cross-lingual and cross-target settings, the mean score drops by 6–8 percentage points compared with supervised performance, though zero-shot transfer is possible to a degree and remains improvable.Source: §5 · p. 8
• Additional experiments support the sequence-pair approach to stance detection.Removing either segment reduces performance; removing comments causes a larger loss, indicating comments carry the most important stance information.Source: §5 · p. 8
• High-confidence errors suggest many comments express stance only implicitly because Smartvote displays the categorical label alongside the comment.Manual annotation could remove very implicit samples, but the dataset’s size and breadth could not realistically be achieved that way.Source: §5 · p. 8

6 Conclusion

• X-stance covers a broad range of national Swiss political topics and includes Swiss Standard German, French, and Italian data.Its topic diversity supports further study of multi-target learning, while its languages support multilingual stance detection.Source: §6 · p. 8
• The paper introduces formal political commentary as a stance-detection genre and uses a question–answer format to condition models naturally on targets.Source: §6 · p. 8
• Multilingual BERT shows some zero-shot transfer to unseen languages and targets, including unseen topics, but a performance gap remains.The authors suggest X-stance may also be useful for argument mining, argument search, and topic classification.Source: §6 · p. 8

A Examples

• Table A7 presents validation-set classification errors where the predicted probability of the correct label is especially low.Source: Table A7 · p. 11
• Table A8 shows two training comments implying positive stance toward one issue and negative stance toward another, arising from copy-pasted comments.Source: Table A8 · p. 11

B Data Statement

• Questions and candidate responses were downloaded from Smartvote, with mainly national-level issues included to reduce variability.Source: B Data Statement · p. 12
• The training set contains Swiss Standard German and Swiss French, with a 74.1% to 25.9% language split.Source: B Data Statement · p. 12
• Test sets also include Swiss Italian, and the questions have English translations.The test-set language proportions are 67.1% German, 24.7% French, and 8.2% Italian.Source: B Data Statement · p. 12

• Participants were 18 or older, with age reported as mixed.Source: Speaker demographic (answers) · p. 12
• Gender, race or ethnicity, native language, and socioeconomic status are reported as unknown or mixed.Source: Speaker demographic (answers) · p. 12
• 7,581 different speakers are represented among candidates who completed an online questionnaire for Swiss communal, cantonal, or national elections.Source: Speaker demographic (answers) · p. 12

• Political scientists edited and translated the questions for a public voting-advice website.Source: Speech situation · p. 12
• Answers were written by website users between 2011 and 2020.Source: Speech situation · p. 12
• The texts consist of questions, answers, arguments, and comments regarding political issues.Source: Speech situation · p. 12