PaperlayerView exact revision on arXiv

Source-linked AI summary

StanceEval 2026: The Second Stance Detection Shared Task

Rasha Albalawi, Nuha Albadi, Hamzah Luqman, Asma Yamani, Maram Kurdi, Saad Ezzini, Ahmed Ashraf, Maged Al-Shaibani, Nora Alturayeif

arXiv:2610.03215v1cs.CL

TL;DR

Arabic stance benchmarks often use fixed targets, leaving limited evidence about generalization to new topics. StanceEval 2026 evaluates related- and unseen-target transfer; the highest Favg2 score was higher on unseen targets, though target difficulty may partly explain the pattern.

  • Problem

    Arabic stance detection is complicated by dialectal, informal, spelling, and implicit-language variation, while fixed-target benchmarks provide limited evidence of generalization to new topics.

    Source: §1 · p. 1

  • Method

    StanceEval 2026 evaluates cross-target generalization using a related-target track for Women Driving and an unseen-target track for E-Cars and Trimester System.

    Source: §1 · p. 1

  • Results

    0.9400 on Track 2 versus 0.8994 on Track 1 was the winning Favg2 result, with higher performance consistently reported on unseen than related targets.

    Source: §4.4 · p. 5

  • Takeaways & Limitations

    The higher Track 2 scores may partly reflect target-difficulty differences rather than stronger generalization to thematically unseen targets.

    Source: Limitations · p. 9

    The evaluation does not confirm the individual effects of the proposed factors that may explain the cross-track performance difference.

    Source: §4.4 · p. 5

Abstract

from arXiv · show

StanceEval 2026 is the second edition of the StanceEval shared task series on stance detection in Arabic social media text. Stance detection aims to identify a writer's stance toward a given topic. Given a tweet and a target, participating systems must determine whether the writer's stance is Favor, Against, or None. This edition focuses on cross-target generalization across two distinct evaluation tracks: Track 1 evaluates thematically related cross-target transfer (testing on Women Driving, related to Women Empowerment from training data), while Track 2 evaluates cross-domain transfer to completely unseen targets (E-Cars and Trimester System). The shared task attracted 80 registered teams from 12 countries. During the evaluation phase, 30 unique teams submitted entries, with 21 teams officially ranked in Track 1 and 13 in Track 2 following validation filtering, and 20 teams submitting system-description papers. Participating teams employed diverse methodologies, including fine-tuned pretrained language models, prompt-based and retrieval-augmented large language models (LLMs), fine-tuned LLMs, and hybrid cascades. Top systems achieved impressive $F_{avg2}$ scores of 0.8994 on Track 1 and 0.9400 on Track 2, substantially outperforming the strongest baselines (0.7366 and 0.7475, respectively), where $F_{avg2}$ denotes the macro-averaged F1 score over the Favor and Against classes. Counterintuitively, performance on the unseen targets was higher than on the related target, a disparity could be driven by extreme target polarization, class imbalance, and dialectal or sarcastic nuance across topics.

1 Introduction

StanceEval 2026 tests whether Arabic stance detection systems generalize to targets not seen in training, addressing the limited transfer evidence from fixed-target benchmarks. It introduces related- and unseen-target tracks and reports strong participation and results.

1 Introduction

  • Arabic stance detection is challenging because dialectal variation, informal expressions, spelling variation, and implicit language complicate interpretation, while fixed-target benchmarks offer limited evidence of transfer to new topics.Source: §1 · p. 1
  • StanceEval 2026 uses Mawqif-XT's 996 manually annotated tweets to test related-target transfer on Women Driving and unseen-target transfer on E-Cars and Trimester System.Women Driving is related to the training target Women Empowerment; the Track 2 domains are absent from training and development.Source: §1 · p. 1
  • The highest Favg2 scores were 0.8994 in Track1 and 0.9400 in Track2, exceeding the best baseline scores of 0.7366 and 0.7475, respectively.Source: §1 · p. 1
  • The task drew 80 registrations from 12 countries and 30 evaluation teams, with 21 and 13 officially ranked teams in Tracks 1 and 2 and 20 accepted system-description papers.Source: §1 · p. 1
  • The paper overviews task design, datasets, evaluation, baselines, results, participating systems, findings, and limitations.Source: §1 · p. 2

2 Literature Review

Prior stance-detection work spans supervised classifiers, neural models, pretrained language models, and LLMs, while cross-target transfer remains difficult. Arabic resources and tasks are increasingly addressing generalization beyond fixed targets.

2 Literature Review

3 Task Description and Dataset

The task uses Arabic tweets to evaluate stance transfer from existing targets to related or unseen targets. Mawqif supplies training and development data, while Mawqif-XT supplies test data.

3 Task Description and Dataset

  • Systems classify each Arabic tweet-target pair as Favor, Against, or None; Favg2, macro-F1 over Favor and Against, is the primary ranking metric.Favg3 averages F1 over all three stance labels, and overall accuracy is also reported.Source: §3 · pp. 2–3
  • StanceEval 2026 trains and develops on Mawqif and tests on Mawqif-XT, whose Table 1 summarizes tweet counts and stance distributions across six targets.Mawqif-XT tweets were collected from Twitter using target-specific Arabic keywords and hashtags, then manually annotated for stance, sarcasm, and sentiment by three annotators with adjudication.Source: §3 · p. 2

Track 1: Thematically Related Cross-Target Stance Detection

  • Track 1 tests transfer to 352 Women Driving tweets, a target absent from training and development but related to Women Empowerment.Its training and development sets contain COVID-19 Vaccine, Digital Transformation, and Women Empowerment tweets.Source: Track 1: Thematically Related… · p. 3

Track 2: Thematically Unseen Cross-Target Stance Detection

  • Track 2 tests out-of-domain transfer without target-specific fine-tuning on E-Cars and Trimester System, targets absent from training and development.The test set contains 332 E-Cars tweets and 312 Trimester System tweets; Women Empowerment is reserved for development.Source: Track 2: Thematically Unseen… · p. 3

4 Shared Task Teams & Results

StanceEval 2026 evaluates cross-target Arabic stance detection across related and unseen-target tracks, with participating systems outperforming the baselines. Performance was higher on unseen targets, but the proposed explanations remain unconfirmed.

4.1 Participating Teams

  • After compliance filtering, 21 teams were ranked in Track 1 and 13 in Track 2; 20 teams submitted accepted system-description papers.The task received 80 registrations from 12 countries, and 30 unique teams submitted entries.Source: §4.1 · p. 3

4.2.1 Experimental Setup

  • Track 1 tests Women Driving, related to Women Empowerment; Track 2 tests E-Cars and Trimester System, absent from training and development data.Systems predict Favor, Against, or None; sentiment and sarcasm annotations are also provided.Source: §4.2.1 · p. 4, Table 2 · p. 4, §4.2 · p. 4

4.2.2 Baseline Results

  • Qwen 2.5 led baseline Favg2 in both tracks, while the best Favg3 and accuracy models differed by track and metric.Track 1’s best Favg3 and accuracy were 0.5714 and 0.6989 for JAIS; Track 2’s were 0.6101 for Qwen 2.5 and 0.6910 for MARBERT.Source: §4.2.2 · p. 4

4.3 Results

  • Track 1’s top system scored 0.8994 versus Qwen 2.5’s 0.7366 baseline; Track 2’s top system scored 0.9400 versus its 0.7475 baseline.JLShen led Track 1, while ZHR led Track 2.Source: §4.3.1 · p. 4, §4.2.2 · p. 4, Table 4 · p. 5
  • Favg3 remained below Favg2 on Women Driving, while on the unseen targets the smaller gap suggests None mattered less in differentiating top systems.Favg2 covers Favor and Against; Favg3 includes all three classes.Source: §4.3.1 · p. 5, §4.3.2 · p. 5

4.4 Discussion: Cross-Target Performance

  • Track 2 outperformed Track 1 for both winning systems and reported baselines, contrary to the expectation that related targets transfer more easily.The winning Favg2 scores were 0.9400 and 0.8994, respectively.Source: §4.4 · p. 5
  • The authors suggest class imbalance, polarization, discourse nuance, and None-class prevalence may help explain the higher Track 2 scores, but cannot confirm their individual effects.Women Driving is described as balanced and contested, whereas Trimester System is strongly polarized; E-Cars discourse centers on concrete factors where stance aligns more directly with sentiment.Source: §4.4 · pp. 5–6

4.5 Analysis of Participating Systems

  • The overview groups participating systems by dominant approach, including encoder fine-tuning, prompting, hybrid pipelines, fine-tuned LLM ensembles, and LLM-generated reasoning.The full descriptions of systems not covered individually appear in Table 5 and Appendix C.Source: §4.5 · p. 6

Appendix C).

Participating systems combined encoder fine-tuning, LLM prompting, retrieval, and hybrid designs. Error analysis highlights neutral-post false positives, sarcasm, implicit targets, and dialectal noise as recurring challenges.

Appendix C).

  • Teams used hybrid encoder–LLM pipelines, fine-tuned LLM ensembles, and LLM-generated reasoning as an auxiliary signal rather than the final stance decision.ZHR combined retrieval-augmented few-shot classification with LoRA-adapted generative models.Source: Appendix C · pp. 6–7
  • Some systems added inference-time evidence or review: Saigum escalated uncertain votes to retrieval-supported second passes, while Thakaa retrieved examples for Track 1 and supplied target descriptions for Track 2.SRK AI Lab used multiple LLM predictions followed by GPT-5.5 review.Source: Appendix C · pp. 6–7
  • LLM-based systems dominated the top ranks across both tracks, while pure encoder fine-tuning tended to rank lower.Several teams’ ablations also found prompting or few-shot LLMs outperforming fine-tuned encoders or alternatives on unseen targets.Source: Appendix C · p. 7

4.6 Error Analysis and Failure Modes

  • The most widespread error was labeling neutral, informational, or irrelevant posts as Favor or Against, a bias encouraged by optimizing Favg2 without None.Balanced demonstrations or break-even thresholding were associated with stronger all-class reliability than standard softmax classification.Source: §4.6 · p. 7
  • Sarcasm and irony led systems to mistake positive wording for Favor, while multi-task sarcasm supervision mitigated the issue only to some extent.The passage identifies cultural irony as a continuing advantage for frontier LLMs given explicit discourse guidelines.Source: §4.6 · pp. 7–8
  • Tweets can imply targets through related concerns rather than naming them, challenging systems that rely on target–post concatenation without external grounding or reasoning.Examples include E-Car tweets about grid instability and Women Driving tweets about family transportation or commuting safety.Source: §4.6 · p. 8
  • Dialectal idioms and orthographic noise weakened MSA-oriented encoder representations; back-translation could harm performance by corrupting dialectal idioms.Surface-matched demonstrations and native Arabic instruction-tuned LLMs showed greater resilience to dialectal variation.Source: §4.6 · p. 8

5 Conclusion

StanceEval 2026 evaluated cross-target Arabic stance detection through a related-target track and an unseen-target cross-domain track. Its top systems scored higher on unseen targets, although the paper cautions that target differences may explain the pattern.

5 Conclusion

  • The paper presents StanceEval 2026 as the second shared task on stance detection in Arabic social media text.It describes task design, datasets, evaluation, baselines, results, participating systems, findings, and limitations.Source: §5 · p. 8
  • Track1 tested transfer to Women Driving, while Track2 tested unseen targets E-Cars and Trimester System from different domains.The task attracted 80 registrations from 12 countries and 30 participating teams; 21 teams were ranked in Track1 and 13 in Track2, with 20 system-description papers.Source: §5 · p. 8
  • 0.8994 on Track1 and 0.9400 on Track2 were the best systems’ Favg2 scores, substantially outperforming the strongest baselines.The paper notes that unseen-target performance exceeded related-target performance, but does not establish its causes.Source: §5 · p. 8

Limitations

The paper identifies limits on interpreting StanceEval 2026 results: the tracks differ in target characteristics, the primary metric excludes None, and the evaluation and qualitative analyses have restricted scope.

Limitations

  • Track1 and Track2 are not a controlled test of target relatedness because their test targets differ in class distribution, polarization, neutral prevalence, and linguistic complexity.Trimester System is strongly Against-skewed, whereas Women Driving has nearly balanced Favor and Against instances and more nuanced expressions; the higher Track2 performance may partly reflect target difficulty.Source: Limitations · pp. 8–9
  • Favg2 ranks systems using only Favor and Against, although distinguishing None from explicit stance is a common error and None prevalence varies across targets.The paper also reports Favg3 and accuracy for a broader assessment.Source: Limitations · p. 9
  • The evaluation uses few test targets and a relatively small Mawqif-XT test set, with substantial Gulf and Saudi dialectal content and culturally grounded phenomena.Observed performance patterns may not generalize to other topics, domains, or Arabic varieties.Source: Limitations · p. 9
  • Qualitative failure modes come from post-evaluation output examination and participant analyses, not a systematically annotated error study.They are plausible explanations for performance patterns, not definitive causal conclusions.Source: Limitations · p. 9

A Dataset Examples

Appendix A points readers to representative examples from Mawqif-XT.

A Dataset Examples

B Zero-shot LLM Prompt

The zero-shot LLM baselines received a common stance-classification prompt. It specifies the three labels and supplies target and tweet fields.

B Zero-shot LLM Prompt

C Additional System Descriptions

The additional system descriptions cover encoder fine-tuning, prompted and fine-tuned LLMs, and hybrid encoder–LLM approaches. Reported findings distinguish performance across prompts, model combinations, and evaluation tracks.

Encoder-Based Fine-Tuning Methods

  • Nabra’s simple baseline outperformed methods targeting the minority NONE class, while back-translation degraded performance.Minority-class methods increased Favg3 but reduced Favor and Against performance; the authors attribute back-translation losses to dialectal Arabic translation noise.Source: C Additional System Descriptions · p. 12

Prompt-Based LLM Methods

  • HardStance used Gemma-4-31B with a development-error-refined prompt covering dialectal and implicit expressions and Women Driving context.The team compared fine-tuned AraBERT variants with zero-shot generative LLMs for Track 1.Source: C Additional System Descriptions · p. 12

Hybrid Encoder–LLM Methods

  • Gemini 2.5 Flash few-shot prompting generalized best to U4RASD’s unseen-target track, outperforming its fine-tuned and ensemble alternatives.The prompt used eight labeled examples and instructions addressing stance versus sentiment, implicit targets, sarcasm, and dialectal variation.Source: C Additional System Descriptions · pp. 12–13

Prompted LLM comparisons

  • Orni’s prompted GPT-4o outperformed its fine-tuned systems; saliyah found few-shot LLMs outperformed fine-tuned encoders, and simpler prompting beat more complex strategies.Orni used few-shot demonstrations, structured reasoning, target-specific context, and self-consistency voting; saliyah’s Track 1 final system combined two LLMs by voting.Source: C Additional System Descriptions · p. 13

Additional system approaches

  • Phonghoang combined ensemble training with Gemini-generated Women Driving pseudolabels and external data; Axiom routed uncertain MARBERTv2 cases to a few-shot Llama fallback.Axiom’s hybrid approach improved over the standalone MARBERTv2 baseline.Source: C Additional System Descriptions · p. 13

Calibrated Routing and Pseudo-Label Self-Training (CaRPS)

  • CaRPS’s encoder-to-adversarial-LLM routing underperformed direct LLM use, so the team used the LLM alone for the official test set.A separate development experiment using LLM-generated Women Empowerment labels to retrain the encoder improved its performance over an unspecified comparator.Source: C Additional System Descriptions · p. 13

Fine-tuned LLM Ensembles

  • Rabia_naz used agreement-filtered transductive pseudolabeling and majority voting across three fine-tuned LLMs; sarcasm supervision improved Track 1 performance.Only unlabeled test tweets with agreement among multiple independent models entered the training pool.Source: C Additional System Descriptions · p. 14

LLM-Generated Reasoning as Auxiliary Signal

  • Tarjih’s pair scorer performed best on the seen-target track, while its gated classifier performed best on the unseen-target track.The system has an LLM generate steelman arguments for each stance, then uses a second LLM to assess how well each matches the tweet.Source: C Additional System Descriptions · p. 14

D Full Leaderboards

The full evaluation leaderboards cover both tracks and rank systems by Favg2, with Favg3 and accuracy reported alongside.

Full Leaderboards

  • Tables 8 and 9 provide the complete official evaluation rankings for Track 1 and Track 2.The rankings include 21 teams in Track 1 and 13 in Track 2.Source: D Full Leaderboards · p. 14

Track 1

  • Track 1’s full evaluation results are ranked by Favg2.Table 8 is the full Track 1 evaluation leaderboard.Source: Table 8 · p. 14

Track 2

  • Track 2’s full evaluation results are ranked by Favg2.Table 9 is the full Track 2 evaluation leaderboard.Source: Table 9 · p. 14
Loading 2610.03215v1…