DialectSentEval 2026: Arabic Dialect Sentiment Analysis and Swapping Shared Task

1 Introduction

• Arabic sentiment analysis is challenging because rich morphology and extensive dialectal variation complicate analysis, while much user-generated content is dialectal and under-resourced.Source: §1 · p. 1
• DialectSentEval comprises two complementary subtasks that evaluate sentiment understanding and generation in diverse Arabic contexts.The task was proposed for ArabicNLP 2026.Source: §1 · p. 1
• Subtask 1 classifies texts in various Arabic dialects as positive, negative, or neutral, testing robustness to lexical and syntactic diversity.Source: §1 · p. 1
• Subtask 2 rewrites a sentence to invert its sentiment polarity while preserving its core meaning.The generative task tests beyond surface-level cues, requiring semantic and syntactic understanding.Source: §1 · p. 1

2 Motivation

• Arabic sentiment resources often cover only MSA or a narrow set of dialects and domains, despite most user-generated content being dialectal.Dialect variation can degrade performance when models trained on MSA are applied to real-world data.Source: §2 · pp. 1–2
• Arabic sentiment swapping remains underexplored, with scarce resources and systematic evaluation frameworks and limited high-quality parallel data.The task requires polarity inversion while preserving core propositional content.Source: §2 · p. 2
• DialectSentEval combines dialect-aware classification and generative sentiment manipulation in one benchmark to support research beyond surface-level polarity detection.The paper frames the intended direction as more semantically grounded and controllable Arabic language technologies.Source: §2 · p. 2

3 Data Collection and Creation

• The datasets for both subtasks were collected and annotated before launch, and Figure 1 outlines their creation process.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Subtask 1: Dialect Sentiment Analysis

• MDS-3 contains 3,000 balanced three-class sentiment sentences across Moroccan, Egyptian, Lebanese, Jordanian, and Saudi dialects.Hotel reviews from ADOR were translated and annotated by native speakers; parallel sentences were kept together in the 80% training and 20% test split.Source: §3.1 · p. 2

3.2 Subtask 2: Sentiment Swap

• The sentiment-swap corpus extends MA’AKS with 9,647 dialectal sentence pairs sharing a topic but expressing contrasting positive and negative orientations.It spans Gulf, Levantine, Egyptian, and Maghrebi dialect families; the pairs underwent checks for fluency, semantic fidelity, and sentiment reversal.Source: §3.2 · p. 2
• The sentiment-swap data is partitioned into 6,752 training pairs, 964 validation pairs, and 1,931 test pairs.Source: §3.2 · p. 2

4 Task Description and Evaluation

• Official evaluation ran on CodaBench, which organizes benchmarks through tasks, datasets, submission pipelines, and evaluation metrics.Source: §4 · p. 3

4.1 Subtask 1: Arabic Dialect Sentiment Analysis

• Subtask 1 classifies sentences in Moroccan, Egyptian, Lebanese, Jordanian, and Saudi dialects into positive, neutral, or negative labels, scored with Macro F1.Macro F1 gives equal importance to all classes and is intended for settings where class distributions may be imbalanced.Source: §4.1 · p. 3

4.2 Subtask 2: Arabic Sentiment Swap

• Subtask 2 generates a sentence with opposite sentiment while preserving the input's core meaning and topic across Maghreb, Egyptian, Gulf, and Levantine dialect groups.Source: §4.2 · pp. 3–4
• Generated text is evaluated for target-polarity accuracy and similarity to the reference using BLEU and chrF.Sentiment Style Accuracy uses a pretrained CamelLab classifier; BLEU and chrF indicate lexical and character-level similarity as a measure of semantic preservation.Source: §4.2 · p. 4

5.1 Subtask 1: Arabic Dialect Sentiment Analysis

• Subtask 1 teams explored pretrained transformers, ensembling, loss formulations, and domain adaptation for multi-dialect variation and zero-shot transfer.Axiom ranked second at 0.8210 Macro F1 with a dialect-aware ensemble, while PTUK-NLP ranked third at 0.8174 and reported Jordanian-to-Lebanese zero-shot transfer.Source: §5.1 · p. 4

5.1 Subtask 1: Arabic Dialect Sentiment Analysis; 5.2 Subtask 2: Arabic Sentiment Swap

• ALEXIS ranked first in both subtasks, scoring 0.8667 Macro F1 in classification and 0.9581 Sentiment Style Accuracy, 56.91 BLEU, and 72.18 chrF in sentiment swap.Its classification system used MARBERTv2 with FGM and iterative confidence-thresholded self-training; its swap system combined dialect-aware RAG, DPO, and classifier-guided reranking.Source: §5.1 · p. 4, §5.2 · p. 5

5.1 Subtask 1: Arabic Dialect Sentiment Analysis

• Northwestern’s comparison of MARBERTv2 and AraBERT across dialect splits found dialect-specific pretraining more robust than general MSA models.Gradient Descenders reported that fine-tuned MARBERTv2 with class-weighted cross-entropy and linear learning-rate warm-up yielded optimal multi-dialect performance.Source: §5.1 · pp. 4–5

5.2 Subtask 2: Arabic Sentiment Swap

• Subtask 2 systems combined generative LLMs, parameter-efficient adapters, candidate reranking, and preference alignment to balance polarity inversion with semantic fidelity.AKARROUCH ranked second with 0.7851 Sentiment Style Accuracy, 49.82 BLEU, and 70.12 chrF using sampled candidates and sentiment-classifier reranking.Source: §5.2 · p. 5
• Other Subtask 2 approaches included few-shot prompting with SLERP checkpoint merging, cross-backbone ensembling, AraT5 reranking, and explicit dialect and polarity prefixes.LahjaSensei ranked third; NAMAA ranked fourth; Sabaa ranked sixth; and NU-SentEval ranked eighth by Sentiment Style Accuracy.Source: §5.2 · p. 5

6.1 Subtask 1: Arabic Dialect Sentiment Analysis

• Dialect-pretrained encoders, especially MARBERTv2 and AraBERTv02, outperformed general MSA baselines and zero-shot prompting by generative LLMs among top Subtask 1 submissions.The analysis also reports greater stability from stratified K-fold ensembling and model stacking, and a role for Focal Loss on neutral examples.Source: §6.1 · p. 6
• +4.57% Macro F1 over second place separated Subtask 1 winner ALEXIS from the runner-up.The analysis attributes the gain primarily to FGM adversarial training and iterative self-training on unlabelled Lebanese test instances.Source: §6.1 · p. 6, Table 1 · p. 7
• Jordanian-trained models showed strong zero-shot transfer to Lebanese, which the analysis links to shared lexical roots within the Levantine family.PTUK-NLP and Lahja are identified as the teams whose analyses support this finding.Source: §6.1 · p. 6

6.2 Subtask 2: Arabic Sentiment Swap

• Subtask 2 exposes a polarity-transfer/content-preservation trade-off: insufficient edits can retain BLEU while failing to invert sentiment, whereas unconstrained rewrites can drift lexically.The analysis says classifier-guided generate-and-rerank teams achieved style-transfer accuracy above 74%, with filtering for the target polarity.Source: §6.2 · p. 6
• SLERP checkpoint merging and Low-Rank Adaptation proved competitive while requiring substantially lower fine-tuning computational footprints.Source: §6.2 · pp. 6–7
• 0.9581 Sentiment Style Accuracy, 56.91 BLEU, and 72.18 chrF made ALEXIS first across all Subtask 2 evaluation metrics.The analysis attributes its performance to DPO-guided minimal edits that preserve sentence structure.Source: §6.2 · p. 6, Table 2 · p. 7

• The shared task evaluated multi-class sentiment classification and generative sentiment style transfer across four major Arabic dialect groups.Source: §7 · p. 7
• Fifteen unique teams participated in the competition.Source: §7 · p. 7
• Evaluation highlighted dialect-pretrained masked encoders with adversarial regularization and self-training for classification, and DPO with classifier-guided reranking for generative style transfer.The conclusion frames these approaches as having potential in low-resource and dialectally diverse environments.Source: §7 · p. 7