PaperlayerView exact revision on arXiv

Source-linked AI summary

E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets

Khloud AL Jallad, Nada Ghneim, Ghaida Rebdawi

arXiv:2609.11334v1cs.CLcs.AIcs.LG

TL;DR

Arabic NLI lacks high-quality benchmarks spanning varied data-generation paradigms. E-CONAN introduces two- and three-way datasets from multiple sources and evaluates pretrained models zero-shot; E-CONAN results fall between existing benchmarks, while mDeBERTa outperforms other models across datasets.

  • Problem

    Arabic textual inference lacks high-quality benchmarks that span multiple data-generation paradigms, limiting available evidence for evaluating model generalization.

    Source: §I · p. 2

  • Method

    The paper constructs two-way E-CONAN-2 and three-way E-CONAN-3 datasets from varied sources and evaluates nine multilingual pretrained models using zero-shot classification.

    Source: §VII · p. 13

  • Results

    E-CONAN dataset results fell between those on other datasets, while mDeBERTa outperformed all other models across datasets.

    Source: a. Pretrained Results · p. 9

  • Takeaways & Limitations

    E-CONAN’s varied sentence pairs may provide a useful base for testing model generalization and improving pretrained models.

    Source: a. Pretrained Results · p. 9

Abstract

from arXiv · show

Natural Language Inference processes pairs of sentences to extract their semantic relations. NLI has been a hot research topic, integrated as a main component in other NLP applications. Despite significant advancements in textual inference across various languages all around the world, Arabic language still suffers from limited resources in this domain. To address this gap, this paper introduces E-CONAN benchmarks that are composed of sentences pairs from various sources: (1) automatically-translated pairs, (2) human-validated machine-translated pairs, (3) hand-crafted pairs from teaching Arabic as foreign language books, and (4) headlines pairs from different news channels containing rumors. E-CONAN contains two benchmark datasets, E-CONAN-2, a 2-way dataset (RTE) and E-CONAN-3, a 3-way dataset (NLI). Additionally, we have used E-CONAN benchmarks to evaluate 9 state-of-the-art multilingual pretrained models using zero-shot classification. Models were evaluated across the ArNLI, XNLI, and E-CONAN datasets. Results show that E-CONAN is a potentially valuable resource for evaluating model generalization and even for fine-tuning pre-trained models. Its diverse composition, derived from a combination of sources, offers a broader and more robust assessment compared to XNLI and ArNLI. In addition, we have evaluated 5 LLMs on E-CONAN-3 dataset. Moreover, we incorporated MARBERT as a representative Arabic-specific baseline and conducted performance evaluation comparison to demonstrate how Arabic-specific models scale against cross-lingual and LLM-based approaches on the E-CONAN benchmarks. Furthermore, we conducted detailed qualitative and quantitative error analysis to analyze frequent error patterns. E-CONAN benchmarks will be publicly available, we hope that it will enrich research community in Arabic textual entailment and natural language inference.

I. INTRODUCTION

E-CONAN addresses the shortage of diverse Arabic textual-inference benchmarks with two datasets and evaluations against existing benchmarks. Its varied data sources are intended to support testing model generalization across Arabic text styles.

  • Existing Arabic NLI benchmarks are described as relying mainly on machine translation or being limited to a single domain or hand-crafted format.The introduction argues that these constraints leave generalization across real-world linguistic variation insufficiently evaluated.Source: §I · p. 2
  • E-CONAN combines human-validated and automatic translations, hand-crafted texts, and news headlines to span formal, casual, translated, and nuanced native Arabic styles.The introduction identifies this diversity as a way to test model generalization across text styles.Source: §I · p. 2
  • E-CONAN-2 contains 24,875 sentence pairs collected from ArNLI, Arabic-translated SNLI and XNLI, AnsStance, and ArEntail.Source: §I · p. 2
  • E-CONAN-3 contains 18,875 sentence pairs collected from ArNLI, Arabic-translated SNLI and XNLI, and AnsStance.Source: §I · p. 2
  • The paper evaluates 10 pretrained models and 6 LLMs against ArNLI, XNLI, and ArEntail alongside the E-CONAN datasets.Source: §I · p. 2

II. Related Works · III. E-CONAN Datasets

The paper situates E-CONAN amid limited Arabic NLI resources and translation-related data-quality concerns, then constructs two benchmarks from existing Arabic datasets and translated or validated sentence pairs.

II. Related Works

  • RTE began as a PASCAL challenge, with datasets evolving from entailment/not-entailment labels to entailment, contradiction, and neutral labels for NLI.The 3-way scheme distinguishes pairs with opposite meanings from pairs with no semantic relation.Source: §II · pp. 2–3
  • Some multilingual NLI datasets, including XNLI and SNLI, were automatically translated into Arabic, but translation errors can affect pair meanings, labels, and model performance.The paper frames machine-translation reliance and the scarcity of high-quality native Arabic NLI data as motivation for its benchmarks.Source: §II · pp. 3–4
  • Arabic NLI resources include ArbTEDS (618 pairs), ArEntil (6,000 pairs), and ArNLI (6,366 pairs).ArbTEDS pairs come from news and hand annotation; ArEntil uses manually labeled headlines; ArNLI includes entailment, contradiction, and neutral examples.Source: §II · p. 3
  • E-CONAN-2 provides a 2-way RTE dataset, while E-CONAN-3 provides 3-way NLI; together they address the scarcity of high-quality native Arabic NLI data.The paper also describes an evaluation framework across 5 LLMs and 10 foundational pretrained models.Source: §II · p. 4, a. DATASETS CONSTRUCTION · p. 4

a. DATASETS CONSTRUCTION

  • E-CONAN draws on ArNLI, including manually collected pairs, ArbTEDS news pairs, and manually verified automatic translations of SICK, PHEME, and Stanford Real Life contradiction data [41].ArbTEDS annotation averaged about 91% agreement between annotators, while SICK inter-rater agreement averaged 84%.Source: a. DATASETS CONSTRUCTION · p. 4
  • The datasets also use the first 1,332 Arabic SNLI test pairs, translated by experts and validated by five independent annotators.Reported consensus was 98% for three annotators and 58% across all five.Source: a. DATASETS CONSTRUCTION · p. 4
  • Construction additionally incorporates Arabic XNLI, AnsStance, and ArEntail, with ArEntail used only for E-CONAN-2.XNLI pairs were validated by five annotators, with 93% consensus for three; reported translation metrics were Ar-En BLEU 35.2, En-Ar BLEU 15.8, and Word Translation P@1 51.9.Source: a. DATASETS CONSTRUCTION · p. 4

b. Datasets Annotation Mapping · c. Datasets Statistics

The sections describe how labels were harmonized across E-CONAN-2 and E-CONAN-3 and report class, source, domain, and lexical statistics. Together, the statistics characterize the datasets as class-balanced and linguistically diverse.

b. Datasets Annotation Mapping

  • E-CONAN-3 retains its original NLI annotations, while AnsStance agree and disagree labels map to entailment and contradiction, respectively, and all remaining labels map to neutral.This mapping aligns AnsStance pair annotations with standard NLI labels.Source: b. Datasets Annotation Mapping · p. 4
  • E-CONAN-2 preserves existing two-way labels and maps three-way entailment to entail, while merging contradiction and neutral into not-entail.The transformation aligns three-way sources with binary classification.Source: b. Datasets Annotation Mapping · p. 5
  • Inter-annotator agreement was not re-evaluated after merging, despite independent validation of the source datasets in their original studies.The paper identifies this as a study limitation.Source: b. Datasets Annotation Mapping · p. 5

c. Datasets Statistics

  • XNLI and ArNLI contribute the most samples, SNLI the fewest, and ArEntail appears only in E-CONAN-2 because it has binary labels.ArEntail’s entailment/non-entailment labels do not distinguish contradiction from neutral.Source: c. Datasets Statistics · p. 5, Figure 3
  • News (35%), Simple Everyday (30%), Social Media (20%), and Multi-Genre (15%) span formal, straightforward, informal, and cross-domain text styles.The domain distribution is presented as broad semantic coverage across the datasets.Source: c. Datasets Statistics · p. 5, Figure 4 · p. 6
  • Text and hypothesis span 2–39 and 2–59 words, respectively, averaging approximately 8 and 11 words, preserving both concise and information-rich constructions.The lexical statistics are described as similar across E-CONAN-2 and E-CONAN-3 while retaining sentence-length variation.Source: c. Datasets Statistics · pp. 5, 7, Table 3 · p. 6
  • 32.6% entailment, ~33.7% contradiction, and ~33.7% neutral make E-CONAN-3 near-balanced; E-CONAN-2 has 37% entail and 63% not-entail.The reported class proportions show E-CONAN-3’s three-way balance and E-CONAN-2’s two-way distribution.Source: Table 2 · p. 5, c. Datasets Statistics · pp. 5–6, Table 5 · p. 6
  • E-CONAN combines eight datasets across four domains with automatically translated, human-validated translated, hand-crafted educational, and rumor-related headline pairs.The source and construction diversity is presented as broadening lexical and semantic coverage.Source: c. Datasets Statistics · pp. 5, 7

IV. Evaluation Models · b. LLMs

The evaluation section describes zero-shot comparisons on E-CONAN using multilingual pretrained models, an Arabic-specific MARBERT baseline, and five LLMs covering Arabic.

a. Pretrained Models

  • Nine multilingual pretrained models were evaluated zero-shot; most had been trained on MNLI alone or on both MNLI and XNLI.The set includes BART, RoBERTa, MiniLM, DeBERTa, and multilingual model variants.Source: a. Pretrained Models · p. 7
  • MARBERT fine-tuned on Arabic XNLI was compared with multilingual models to assess Arabic-specific pretrained models.The MARBERT evaluation is illustrated in Figure 7.Source: a. Pretrained Models · p. 7

b. LLMs

  • Five Arabic-covering LLMs were compared on E-CONAN using zero-shot classification.The comparison is illustrated in Figure 8.Source: b. LLMs · p. 7
  • ALLAM is an autoregressive decoder-only model pretrained on a mixture of Arabic and English texts via vocabulary expansion.It is a series of models intended to support Arabic Language Technologies.Source: b. LLMs · p. 7
  • Qwen2.5 includes open-weight dense models from 0.5B to 72B parameters and proprietary Mixture-of-Experts variants [47], [48].The proprietary variants include Qwen2.5-Turbo and Qwen2.5-Plus.Source: b. LLMs · pp. 7–8
  • Gemma3 is an open-weight multimodal model supporting more than 140 languages, with sizes from 1 to 27 billion parameters and a 128K-token context window.It accepts text and image input.Source: b. LLMs · p. 8
  • Command R7B Arabic is a 7-billion-parameter multilingual model trained and evaluated for Arabic and English, with a 128,000-token context length [53].It is trained for tasks including Arabic and English RAG generation.Source: b. LLMs · p. 8
  • DeepSeek-R1 is an open-weight MoE model with 671B total and approximately 37B active parameters, optimized for reasoning tasks [54].Its pipeline uses two reinforcement-learning stages, and its maximum generation length is 32,768 tokens.Source: b. LLMs · p. 8

c. Experiments Setup: … b. LLM Results:

The experiments compare multilingual pretrained models and LLMs in zero-shot Arabic inference, using accuracy as the main metric. Results show stronger LLM accuracy on the 2-way E-CONAN task than the 3-way task, with errors often involving confusion between neutral and contradiction.

c. Experiments Setup:

  • All experiments used zero-shot inference with default decoding, temperature 0, and no fine-tuning, few-shot examples, or task-specific adaptations.LLMs received a structured prompt asking them to classify each text–hypothesis pair as entailment, contradiction, or neutral.Source: c. Experiments Setup · p. 8

V. Results and Discussion

  • Accuracy was the main evaluation metric for NLI experiments, following the GLUE benchmark convention.Source: §V · p. 8

a. Pretrained Results

  • ArNLI had the lowest pretrained-model results, XNLI the highest, and E-CONAN results fell between them across its varied sentence sources.The paper attributes XNLI’s higher results partly to models’ training or fine-tuning on XNLI or similar machine-translated pairs, and describes ArNLI as containing complex, human-checked sentences.Source: a. Pretrained Results · pp. 8–9
  • mDeBERTa outperformed the other multilingual pretrained models across datasets, and models tuned on both XNLI and MNLI outperformed those tuned on MNLI alone.Source: Table 6 · p. 8, Table 7 · p. 8, a. Pretrained Results · p. 9

b. LLM Results:

  • Merging neutral and contradiction into “not-entail” raised LLM accuracy by 15% to 30% on the 2-way task.Accuracy rose from 55% to 66% for Allam, 64% to 92% for Command R7B Arabic, 61% to 87% for DeepSeek, 68% to 95% for Gemma3, and 64% to 94% for Qwen2.5.Source: b. LLM Results · p. 9
  • Across the reported results, the highest results were achieved on AnsStance and the lowest on SNLI; the authors suggest this may relate to most studied models’ greater pretraining exposure to news than logical data.Source: b. LLM Results · p. 9
  • MARBERT was added as an Arabic-specific baseline against mDeBERTa and Gemma; Gemma led on most datasets, while 3-way errors often confused neutral with contradiction.Source: b. LLM Results · p. 9

c. McNemar’s Test · VI. Error Analysis · b. Quantitative Error Analysis

The zero-shot model comparison was statistically significant, while error analyses identified recurring failures tied to surface similarity, unsupported inferences, and entailment bias. Quantitative patterns differed between pretrained models and LLMs.

c. McNemar’s Test

  • p < 0.000001 in Bonferroni-corrected pairwise McNemar tests comparing Gemma3 with mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 across all datasets.The test was performed to confirm the validity of the zero-shot evaluations.Source: c. McNemar’s Test · p. 10
  • Without supervised fine-tuning or parameter tuning, the evaluation exposes current models’ limits: strong traditional NLI performance does not ensure adaptation to specialized domains.Source: c. McNemar’s Test · p. 11

VI. Error Analysis

  • The error analysis combined manual inspection of error patterns with quantitative measurement of their frequency.Source: §VI · p. 11
  • Qualitative analysis found lexical-overlap bias and entity-mismatch errors: models may infer entailment from shared words while missing unverified details or conflicting locations.The analysis identifies four primary failure modes across pretrained models and LLMs.Source: §VI · p. 11
  • Arabic morphology and logical constraints trigger errors involving gender disagreement, unsupported numerical specificity, and antonymy or negation despite high lexical overlap.Source: §VI · p. 11
  • Topical hallucination leads models to infer entailment from related topics or emotions, adding specific details that the premise does not establish.Source: §VI · p. 11

b. Quantitative Error Analysis

c. Class-Wise Performance and Error Patterns · VII. Conclusion

The paper presents E-CONAN as a diverse Arabic NLI benchmark and examines model performance beyond overall accuracy. Its class-wise analysis identifies persistent neutral–contradiction confusion while noting both the benefits and limitations of combining multiple data sources.

c. Class-Wise Performance and Error Patterns

  • Class-wise analysis found persistent neutral–contradiction confusion, with Gemma often predicting neutral on ArSNLI and ArNLI despite lower contradiction and entailment performance.Neutral recall remained relatively high in these harder inference scenarios.Source: c. Class-Wise Performance and Error… · p. 12
  • Subjective annotation boundaries, translation artifacts, and lexical-overlap heuristics can make models confuse neutral and contradiction, especially when negation or indirect context shifts meaning.Missing information can be ambiguous between non-contradiction and direct conflict; translated data may also lose pragmatic and cultural cues.Source: c. Class-Wise Performance and Error… · pp. 12–13
  • Combining diverse sources in E-CONAN-3 appears to reduce dataset-specific stylistic biases and is reflected in more balanced class-wise results and clearer confusion-matrix diagonals.The paper contrasts this with label-specific biases in some prior benchmarks that can encourage shortcut predictions.Source: c. Class-Wise Performance and Error… · p. 13
  • E-CONAN-3 remains skewed toward news (35%) and everyday contexts (30%), with social media (20%) and multi-genre (15%) underrepresented, alongside inherited translation and crowdsourcing biases.These domain and source limitations should be considered when interpreting generalization results.Source: c. Class-Wise Performance and Error… · p. 13

VII. Conclusion

  • mDeBERTa achieved accuracies of 0.71 on E-CONAN-3 and 0.86 on XNLI; ArNLI was hardest, XNLI easiest, and E-CONAN intermediate.The paper reports these results from zero-shot evaluation of nine multilingual pretrained models.Source: §VII · p. 13
  • E-CONAN-2 and E-CONAN-3 combine automatically translated, human-validated translated, hand-crafted, and news-based sentence pairs; current benchmarks cover MSA but not regional dialects.Future work will expand the datasets to include Arabic dialects.Source: §VII · pp. 13–14
  • Among five evaluated LLMs, Gemma achieved 0.68 accuracy on E-CONAN-3, while Gemma and Qwen led the two-way evaluation with 0.95 and 0.94.The paper reports that errors between neutral and contradiction motivated the two-way evaluation.Source: §VII · p. 13
  • The study adds MARBERT as an Arabic-specific baseline and finds pretrained models are often misled by lexical overlap, whereas LLMs are often misled by topical familiarity.The comparison considers Arabic-specific, cross-lingual, and LLM-based approaches on E-CONAN benchmarks.Source: §VII · pp. 13–14

Logical Relation · ZERO-SHOT CLASSIFICATION RESULTS ON E-CONAN-3 DATASETS

The supplied passages identify zero-shot evaluation categories for E-CONAN-3—cross-lingual baselines, an Arabic baseline, and LLMs—and list Accuracy, Precision, Recall, and F1 as metrics, but provide no result values.

ZERO-SHOT CLASSIFICATION RESULTS ON E-CONAN-3 DATASETS

  • The E-CONAN-3 evaluation covers cross-lingual baseline models, an Arabic baseline, and LLMs, but the supplied passages show no performance values.The figure captions identify evaluation categories rather than scores or model rankings.Source: Figure 6 · p. 19, Figure 7 · p. 19, Figure 8 · p. 20

Logical Relation

  • The cross-lingual baseline evaluation is identified in Figure 6.The supplied caption provides no associated scores.Source: Figure 6 · p. 19
  • The Arabic baseline evaluation is identified in Figure 7.The supplied caption provides no associated scores.Source: Figure 7 · p. 19
  • The LLM evaluation is identified in Figure 8.The supplied caption provides no associated scores.Source: Figure 8 · p. 20
  • The Figure 9 caption identifies zero-shot classification results on the ArEntail and E-CONAN-2 datasets, without reporting values or comparisons.Source: Figure 9 · p. 20
  • The listed cross-lingual model roster includes nine NLI checkpoints spanning RoBERTa, BART, DeBERTa, MiniLM, and ERNIE variants.The names shown are FacebookAI/roberta-large-mnli, facebook/bart-large-mnli, MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7, MoritzLaurer/MiniLM-L6-mnli, MoritzLaurer/mDeBERTa-v3-base-mnli-xnli, MoritzLaurer/multilingual-MiniLMv2-L6-mnli-xnli, MoritzLaurer/DeBERTa-v3-base-mnli, MoritzLaurer/multilingual-MiniLMv2-L12-mnli-xnli, and MoritzLaurer/ernie-m-base-mnli-xnli.Source: Logical Relation · p. 21

ZERO-SHOT CLASSIFICATION RESULTS ON E-CONAN-3 DATASETS

ZERO-SHOT CLASSIFICATION RESULTS ON XNLI DATASET · ZERO-SHOT CLASSIFICATION RESULTS ON ARNLI DATASET

The supplied passages identify accuracy, precision, recall, and F1 as metrics for the zero-shot XNLI and ArNLI results, but provide no scores or model comparisons.

ZERO-SHOT CLASSIFICATION RESULTS ON XNLI DATASET

ZERO-SHOT CLASSIFICATION RESULTS ON ARNLI DATASET

ZERO-SHOT CLASSIFICATION RESULTS ON TEST XNLI DATASET · LLM RESULTS ON E-CONAN-3

The supplied passages identify the datasets, models, and evaluation metrics represented in the zero-shot results, but provide no numerical scores or supported performance comparisons.

ZERO-SHOT CLASSIFICATION RESULTS ON TEST XNLI DATASET

  • The zero-shot results list accuracy, precision, recall, and F1 as evaluation metrics.The supplied passage gives metric labels without associated scores.Source: ZERO-SHOT CLASSIFICATION RESULTS ON… · p. 21
  • Figure 10 presents results on E-CONAN-3, ArNLI, and XNLI.The caption does not provide values or identify a best-performing model.Source: Figure 10 · p. 21
  • A figure caption identifies LLM zero-shot classification on E-CONAN-3 as a 2-way task.The caption supplies the task framing but no results.Source: Figure 11 · p. 22
  • Another caption identifies LLM zero-shot classification results on the Ans-Stance dataset.The supplied caption does not state scores or comparisons.Source: Figure 12 · p. 22

LLM RESULTS ON E-CONAN-3

  • The LLM results passage names Allam, Cmd_rb, Deepseek, Gemma3, and Qwen2.5.No comparative scores for these models are included in the supplied text.Source: LLM RESULTS ON E-CONAN-3 · p. 23
  • The LLM results list accuracy, precision, recall, and F1 as metrics.The supplied passage contains metric labels but no values.Source: LLM RESULTS ON E-CONAN-3 · p. 23

LLM RESULTS ON ARNLI · LLM RESULTS ON ARSNLI

The supplied excerpts identify five LLMs and the reported metric categories for ARNLI and ARSNLI, but provide no scores. The ARSNLI subsection also includes visuals for zero-shot classification, classification reports, and confusion matrices.

LLM RESULTS ON ARNLI

  • The ARNLI excerpt lists Allam, Cmd_rb, Deepseek, Gemma3, and Qwen2.5 as evaluated models.The supplied passage contains no performance values.Source: LLM RESULTS ON ARNLI · p. 23
  • Accuracy, precision, recall, and F1 are the metric labels shown for ARNLI.The excerpt gives labels but no scores.Source: LLM RESULTS ON ARNLI · p. 23

LLM RESULTS ON ARSNLI

  • The ARSNLI excerpt lists Allam, Cmd_rb, Deepseek, Gemma3, and Qwen2.5 as evaluated models.The supplied passage contains no performance values.Source: LLM RESULTS ON ARSNLI · p. 23
  • Accuracy, precision, recall, and F1 are the metric labels shown for ARSNLI.The excerpt gives labels but no scores.Source: LLM RESULTS ON ARSNLI · p. 23
  • A figure is captioned as showing LLM zero-shot classification on E-CONAN-3 and SoTA benchmarks.The caption does not provide results or identify a winning model.Source: Figure 13 · p. 23
  • Table 12 is captioned as a classification report for E-CONAN and SoTA benchmarks.The supplied table passage includes no cell values.Source: Table 12 · p. 24
  • A figure is captioned as showing confusion matrices for E-CONAN and SoTA benchmarks.The caption alone does not reveal the confusion-matrix values.Source: Figure 15 · p. 25

Related papers

Loading 2609.11334v1…