PaperlayerView exact revision on arXiv

Source-linked AI summary

E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets

Khloud AL Jallad, Nada Ghneim, Ghaida Rebdawi

arXiv:2609.11334v1cs.CLcs.AIcs.LG

TL;DR

Arabic NLI lacks high-quality, diverse benchmarks for evaluating textual inference across varied data sources and domains. The paper introduces E-CONAN-2 and E-CONAN-3 and evaluates multilingual models and LLMs on them, finding strong differences across datasets, tasks, and model families, with frequent neutral–contradiction confusion.

  • Problem

    Arabic NLI has limited high-quality benchmark resources, with many existing datasets relying on machine translation.

  • Method

    The paper constructs E-CONAN-2 and E-CONAN-3 from translated, human-validated, hand-crafted, and news-based sentence pairs, then evaluates pretrained models and LLMs.

  • Results

    Gemma3 achieved most best LLM results, while converting 3-way classification to 2-way increased accuracy by 15% to 30%.

  • Takeaways & Limitations

    E-CONAN offers a diverse basis for assessing model generalization and improving Arabic textual inference evaluation.

Abstract

from arXiv · show

Natural Language Inference processes pairs of sentences to extract their semantic relations. NLI has been a hot research topic, integrated as a main component in other NLP applications. Despite significant advancements in textual inference across various languages all around the world, Arabic language still suffers from limited resources in this domain. To address this gap, this paper introduces E-CONAN benchmarks that are composed of sentences pairs from various sources: (1) automatically-translated pairs, (2) human-validated machine-translated pairs, (3) hand-crafted pairs from teaching Arabic as foreign language books, and (4) headlines pairs from different news channels containing rumors. E-CONAN contains two benchmark datasets, E-CONAN-2, a 2-way dataset (RTE) and E-CONAN-3, a 3-way dataset (NLI). Additionally, we have used E-CONAN benchmarks to evaluate 9 state-of-the-art multilingual pretrained models using zero-shot classification. Models were evaluated across the ArNLI, XNLI, and E-CONAN datasets. Results show that E-CONAN is a potentially valuable resource for evaluating model generalization and even for fine-tuning pre-trained models. Its diverse composition, derived from a combination of sources, offers a broader and more robust assessment compared to XNLI and ArNLI. In addition, we have evaluated 5 LLMs on E-CONAN-3 dataset. Moreover, we incorporated MARBERT as a representative Arabic-specific baseline and conducted performance evaluation comparison to demonstrate how Arabic-specific models scale against cross-lingual and LLM-based approaches on the E-CONAN benchmarks. Furthermore, we conducted detailed qualitative and quantitative error analysis to analyze frequent error patterns. E-CONAN benchmarks will be publicly available, we hope that it will enrich research community in Arabic textual entailment and natural language inference.

I. INTRODUCTION

Arabic textual inference lacks diverse, high-quality benchmarks for testing model generalization. E-CONAN addresses this gap with 2-way and 3-way datasets combining translated, human-validated, hand-crafted, and news-based sentence pairs.

  • Motivation: Arabic textual inference lacks high-quality benchmarks that test generalization beyond a single data-generation paradigm.Existing resources often rely mainly on machine translation or are restricted to one domain or hand-crafted format.
  • E-CONAN contribution: E-CONAN combines automatically translated, human-validated machine-translated, hand-crafted, and news-headline pairs to capture diverse Arabic text styles.The benchmark spans formal, casual, translated, and nuanced native Arabic structures.
  • E-CONAN contribution: E-CONAN-2 provides a 2-way RTE benchmark with 24,875 sentence pairs collected from five Arabic datasets.Its sources include ArNLI, translated SNLI and XNLI, AnsStance, and ArEntail.
  • Evaluation: The paper evaluates pretrained models and LLMs against established Arabic inference benchmarks and presents error analysis of entailment relations.The stated contributions include evaluations of 10 pretrained models and 6 LLMs.

II. Related Works

Prior RTE and NLI research established benchmark challenges, datasets, and multilingual models, but Arabic resources remain comparatively limited. Existing Arabic benchmarks include news-based, translated, and manually labeled datasets, while machine translation can introduce meaning errors that affect labels and model performance.

  • RTE background: RTE began through PASCAL challenges and evolved from binary entailment classification toward richer reasoning over lexical, syntactic, morphological, and logical relations.The field supports applications including information retrieval, information extraction, question answering, and summarization.
  • NLI datasets: English NLI resources include WNLI, SNLI, MultiNLI, and GLUE’s RTE dataset, with XNLI and SNLI later translated into Arabic and other languages.The cited English resources provided foundations for multilingual inference evaluation.
  • NLI datasets: Automatic translation can introduce meaning errors in sentence pairs, affecting their labels and model performance.This limitation motivates benchmarks that include additional validation and native or hand-crafted Arabic text.
  • Arabic benchmarks: Existing Arabic benchmarks include 2-way ArbTEDS, 2-way ArEntail, and 3-way ArNLI, covering news, manually collected pairs, and labeled sentence pairs.ArNLI contains 6,366 pairs divided into entailment, contradiction, and neutral classes.
  • Arabic NLI models: Arabic NLI remains constrained by scarce high-quality native datasets, prompting evaluation frameworks spanning LLMs and foundational multilingual pretrained models.The paper positions its datasets as addressing reliance on machine-translated benchmarks.

III. E-CONAN Datasets

E-CONAN comprises unified 2-way and 3-way Arabic entailment benchmarks assembled from multiple translated, validated, and annotated sources. The datasets span diverse domains and exhibit broad linguistic coverage, while E-CONAN-3 is designed with near-balanced class proportions.

  • Datasets Construction: The benchmarks combine manually collected, automatically translated, human-validated translated, and source-specific annotated sentence pairs from multiple Arabic and translated datasets.
  • Datasets Annotation Mapping: E-CONAN-2 maps entailment to entail and merges contradiction and neutral into not-entail, whereas E-CONAN-3 preserves three-way labels while standardizing AnsStance annotations.
  • Dataset Statistics: The benchmarks include diverse lexical content and sentence lengths, with Text and Hypothesis averages of approximately 8 and 11 words, respectively.
  • Dataset Statistics: The datasets integrate eight sources across News, Simple Everyday Text, Social Media, and Multi-Genre domains, with News and Simple Everyday Text comprising 35% and 30%.
  • Dataset Statistics: E-CONAN-3 has near-balanced classes at approximately 32.6% entailment, 33.7% contradiction, and 33.7% neutral.

IV. Evaluation Models

The evaluation uses multilingual pretrained models in zero-shot classification and includes an Arabic-specific MARBERT comparison. This setup contrasts cross-lingual models with an Arabic-pretrained baseline on the E-CONAN benchmarks.

  • The study evaluates multilingual pretrained models on Arabic inference benchmarks without training examples through zero-shot classification.
  • The evaluated models were primarily trained or tuned on MNLI, XNLI, or multilingual NLI resources.
  • MARBERT fine-tuned on Arabic XNLI serves as an Arabic-specific baseline against the multilingual models.

b. LLMs

The paper evaluates multilingual pretrained models and LLMs on Arabic entailment benchmarks using zero-shot classification. Results show that model performance varies by dataset and label configuration, with Gemma generally strongest among the evaluated LLMs and 2-way classification easier than 3-way classification.

  • Evaluation setup: The multilingual pretrained-model evaluation used zero-shot accuracy comparisons across ArNLI, E-CONAN, and XNLI, with macro precision, recall, and F1 also reported.The reported comparisons include tables for accuracy and figures for macro-averaged metrics.
  • Dataset comparisons: ArNLI produced the lowest results, whereas XNLI produced the highest, and E-CONAN results fell between them across the compared datasets.The paper attributes the XNLI advantage partly to models’ prior training on XNLI or similar machine-translated data, while E-CONAN combines several text sources.
  • LLM evaluation: Gemma3 reached 68% accuracy on E-CONAN-3, while Allam reached 55%; on the 2-way version, Gemma3 reached 95% and Allam 66%.These LLM results were obtained through zero-shot classification with standardized deterministic settings.
  • Error analysis: 2-way classification significantly outperformed 3-way classification because neutral–contradiction confusions are merged into not-entailment.Converting E-CONAN-3 to 2-way classification increased accuracy by 15% to 30%; Gemma3 rose from 68% to 95% and Qwen2.5 from 64% to 94%.
  • LLM and model comparisons: Gemma achieved the highest performance across most evaluated datasets, while mDeBERTa performed best on ArXNLI.Gemma also achieved the best result on ArNLI, whereas mDeBERTa’s advantage on ArXNLI may reflect prior training on MNLI and XNLI.
  • Statistical validation: Pairwise McNemar testing found a p-value below 0.000001 when comparing Gemma3 with mDeBERTa across all datasets.The test used Bonferroni correction for multiple comparisons.

VI. Error Analysis

The error analysis combines qualitative inspection, quantitative analysis, and class-wise evaluation to identify recurring Arabic NLI failure patterns. Models are particularly challenged by lexical overlap, topical familiarity, and confusion between neutral and contradiction.

  • Qualitative Error Analysis: Qualitative analysis identifies lexical-overlap heuristics as a recurring failure mode, with models predicting entailment when premise and hypothesis share frequent tokens despite differing logical relations.The analysis uses manual inspection and representative samples from Table 11.
  • Qualitative Error Analysis: Arabic morphology creates errors involving gender disagreement and quantifier or numerical constraints, including entailment predictions for neutral gender-shift cases.These failures reflect difficulty tracking entity distinctions and logical boundaries.
  • Quantitative Error Analysis: Pretrained models are often misled by lexical overlap, whereas LLMs are often misled by topical familiarity; both favor similarity over logical consistency and entailment.The quantitative analysis examined MiniLM-L6-mnli and Allam as representative models.
  • Class-Wise Performance and Error Patterns: Models frequently confuse neutral and contradiction, with contradiction and entailment instances often predicted as neutral.The analysis attributes this difficulty partly to subjective annotation boundaries and ambiguity between missing information and direct conflict.
  • Robustness, Semantic Diversity, and Fine-Tuning Potential of E-CONAN-3: E-CONAN-3’s diverse source aggregation is associated with more balanced class-wise performance, clearer confusion-matrix diagonals, and reduced reliance on individual dataset biases.The benchmark combines multiple sources without extensive filtering, increasing contextual and linguistic diversity.
  • Robustness, Semantic Diversity, and Fine-Tuning Potential of E-CONAN-3: E-CONAN-3 retains domain imbalance, translation bias, and crowdsourcing-related annotation bias, so these inherited limitations should inform interpretation of generalization results.News and simple everyday contexts comprise 35% and 30% of the data, while social media and multi-genre settings comprise 20% and 15%.

VII. Conclusion

The paper introduces E-CONAN-2 and E-CONAN-3 to address limited Arabic textual entailment and NLI benchmark diversity. Evaluations show intermediate difficulty for E-CONAN, strong LLM performance on E-CONAN-3, and recurring errors involving lexical or topical similarity.

  • VII. Conclusion: E-CONAN combines automatically translated, human-validated translated, hand-crafted, and news-based pairs in 2-way and 3-way Arabic inference benchmarks.E-CONAN-2 is the RTE dataset, while E-CONAN-3 is the NLI dataset.
  • VII. Conclusion: E-CONAN shows intermediate difficulty between ArNLI and XNLI, making it a resource for assessing model generalization across more diverse Arabic text sources.ArNLI has the lowest observed performance, while XNLI has the highest among the compared datasets.
  • VII. Conclusion: Gemma achieves 0.68 accuracy on E-CONAN-3, while Gemma and Qwen achieve 0.95 and 0.94 accuracy respectively in the 2-way evaluation.The reported 2-way results address frequent neutral–contradiction errors in the 3-way task.
  • VII. Conclusion: The study compares multilingual pretrained, Arabic-specific, and LLM-based approaches and supplements performance evaluation with qualitative and quantitative error analysis.MARBERT is included as the Arabic-specific baseline.
  • VII. Conclusion: Current E-CONAN benchmarks evaluate Modern Standard Arabic but do not represent regional Arabic dialects.Future expansion is described as covering dialectal variation.

Logical Relation

The paper evaluates cross-lingual, Arabic-specific, and large language model systems across Arabic inference benchmarks. The supplied passages identify the evaluation figures and model set but do not report comparative outcomes.

  • Logical Relation: The evaluation includes cross-lingual baseline models, an Arabic baseline model, and LLMs across the proposed benchmark settings.The figures separately identify cross-lingual, Arabic, and LLM evaluations.
  • Logical Relation: The listed pretrained systems include RoBERTa, BART, mDeBERTa, MiniLM, DeBERTa, and Ernie-M multilingual or NLI variants.The passages provide model names but no performance values.
  • Logical Relation: Figure 9 is identified as reporting zero-shot classification results on ArEntail and E-CONAN-2.No numerical or winner comparison is supplied in the passage.

ZERO-SHOT CLASSIFICATION RESULTS ON E-CONAN-3 DATASETS

The supplied passages expose the metric labels used for zero-shot classification results on E-CONAN-3 but contain no model-specific values or comparative findings.

  • ZERO-SHOT CLASSIFICATION RESULTS ON E-CONAN-3 DATASETS: The reported evaluation metrics are accuracy, precision, recall, and F1.The supplied text does not provide corresponding values for E-CONAN-3.

ZERO-SHOT CLASSIFICATION RESULTS ON ARNLI DATASET

The supplied materials identify evaluation figures spanning E-CONAN-3, ArNLI, and XNLI, alongside accuracy, precision, recall, and F1 labels.

  • The reported evaluation metrics are accuracy, precision, recall, and F1.
  • Figure 10 covers results on E-CONAN-3, ArNLI, and XNLI datasets.
  • Additional supplied figures identify LLM zero-shot classification on E-CONAN-3 as 2-way and on the Ans-Stance dataset.

LLM RESULTS ON E-CONAN-3

The supplied materials identify LLM evaluation on E-CONAN-3 and provide model and metric labels for the reported classification results.

  • The E-CONAN-3 LLM evaluation names Allam, Cmd_rb, Deepseek, Gemma3, and Qwen2.5.
  • The evaluation reports accuracy, precision, recall, and F1 metrics.

LLM RESULTS ON ARSNLI

The supplied materials identify LLM model labels, classification metrics, benchmark-comparison figures, and a classification report for E-CONAN and other benchmarks.

  • The supplied materials name Allam, Cmd_rb, Deepseek, Gemma3, and Qwen2.5 among the evaluated models.
  • The evaluation uses accuracy, precision, recall, and F1 metrics.
  • Figure 13 concerns LLM zero-shot classification on E-CONAN-3 and state-of-the-art benchmarks.
  • Table 12 is a classification report for E-CONAN and state-of-the-art benchmarks.
  • Figure 15 is labeled as a confusion matrix of E-CONAN and state-of-the-art benchmarks.
Loading 2609.11334v1…