Source-linked AI summary
AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models
Mohammad Zbeeb, Hasan Abed Al Kader Hammoud, Sina Mukalled, Nadine Rizk, Fatima Karnib, Issam Lakkis, Ammar Mohanna, Bernard Ghanem
TL;DR
Arabic LLM benchmarks often emphasize knowledge and reasoning without directly measuring linguistic competence. AraLingBench evaluates that competence across five categories, and its model evaluation finds surface-level strengths alongside persistent difficulty with deeper structural reasoning.
Problem
Existing Arabic benchmarks often emphasize factual recall and problem-solving rather than directly testing fundamental linguistic competence.
Source: §1 · p. 2Method
AraLingBench uses human-annotated, expert-authored multiple-choice questions to assess grammar, morphology, spelling, reading comprehension, and syntax.
Source: §1 · pp. 2–3Results
Arabic LLMs show uneven linguistic competence, often performing well on surface-level tasks while struggling with deeper structural understanding.
Source: §1 · p. 3, §4.7 · p. 15Takeaways & Limitations
AraLingBench offers a complementary diagnostic perspective for distinguishing superficial fluency from linguistic competence.
Source: §5 · p. 15
Abstract
from arXiv · showhide
We present AraLingBench: a fully human annotated benchmark for evaluating the Arabic linguistic competence of large language models (LLMs). The benchmark spans five core categories: grammar, morphology, spelling, reading comprehension, and syntax, through 150 expert-designed multiple choice questions that directly assess structural language understanding. Evaluating 35 Arabic and bilingual LLMs reveals that current models demonstrate strong surface level proficiency but struggle with deeper grammatical and syntactic reasoning. AraLingBench highlights a persistent gap between high scores on knowledge-based benchmarks and true linguistic mastery, showing that many models succeed through memorization or pattern recognition rather than authentic comprehension. By isolating and measuring fundamental linguistic skills, AraLingBench provides a diagnostic framework for developing Arabic LLMs. The full evaluation code is publicly available on GitHub.
1 Introduction
AraLingBench targets a gap in Arabic LLM evaluation: broad benchmarks do not directly test core linguistic competence. Its evaluation finds uneven skills, with surface strengths alongside difficulties in deeper structural reasoning.
1 Introduction
- Existing Arabic benchmarks emphasize factual recall and problem-solving, while direct assessment of grammatical agreement, morphological derivation, and orthographic conventions remains limited.The paper frames this as a gap in measuring the linguistic foundations of Arabic understanding.Source: §1 · p. 2
- AraLingBench is a human-annotated benchmark of expert-authored multiple-choice questions across grammar, morphology, spelling, reading comprehension, and syntax.Questions are reviewed by trained Arabic linguists for validity and clarity, with a focus on language-grounded reasoning rather than factual recall.Source: §1 · pp. 2–3
- Models often perform well on spelling and reading comprehension but struggle with grammar, morphology, and syntax, despite correlations with general benchmarks.The evaluation also identifies surface-level pattern recognition and retrieval-based shortcuts among many high-scoring models.Source: §1 · p. 3
- Cross-benchmark analysis indicates AraLingBench captures a distinct ability dimension, separating linguistic understanding from surface-level or retrieval-based performance.The benchmark is presented as a diagnostic framework for identifying where models succeed and fall short.Source: §1 · p. 3
2 Related Work
The related work reviews Arabic model development and evaluation, then positions AraLingBench as a direct assessment of linguistic competence.
2.1 Arabic Language Models
- Arabic models evolved from encoder-based systems for discriminative tasks toward generative, bilingual, and specialized models with varied architectures and training strategies.The reviewed examples include Arabic-trained models, multilingual-base adaptations, instruction-tuned systems, and dialect- or domain-focused variants.Source: §2.1 · pp. 3–4, Table 1 · p. 4
2.2 Evaluation Benchmarks for Arabic
- Arabic evaluation includes broad academic, professional, and NLP benchmarks as well as domain-specific and cultural tests, but existing benchmarks do not explicitly assess core linguistic competence.The cited landscape includes knowledge-focused tests, multi-task platforms, and legal, medical, financial, cultural, and dialectal evaluations.Source: §2.2 · p. 4, Table 2 · p. 5
2.3 Positioning AraLingBench
- AraLingBench makes linguistic competence an explicit evaluation target rather than assuming it as a prerequisite for reasoning.It isolates phenomena across five categories through human-authored and reviewed questions, supporting interpretable diagnosis of model strengths and weaknesses.Source: §2.3 · p. 5
- The benchmark complements existing evaluations by re-centering linguistic understanding as a foundation of Arabic NLP evaluation.Its role is to fill a gap in the evaluation landscape, not replace the broader benchmarks.Source: §2.3 · p. 5
3 AraLingBench Construction
AraLingBench was built through expert question development, staged review, and difficulty annotation, producing a balanced 150-question benchmark.
3.1 Data Collection Process
- Five Arabic linguistics experts authored novel questions, which underwent native-speaker filtering, senior expert quality control, and independent difficulty annotation.Review checked clarity and diversity, then accuracy, phrasing, category alignment, and a single unambiguous correct answer; three annotators rated difficulty and majority vote set the final label.Source: §3.1 · pp. 5–6
3.2 Benchmark Statistics
- The 150 questions are evenly divided across five categories; difficulty is skewed toward medium, with 50 Easy, 74 Medium, and 26 Hard items.The benchmark uses 125 four-choice questions and 25 three-option questions, with answer positions varying without systematic bias.Source: §3.2 · p. 6, Figure 2 · p. 6
4 Experimental Evaluation
The evaluation tests 35 Arabic and bilingual LLMs under a standardized zero-shot multiple-choice protocol, then examines category balance, skill correlations, benchmark alignment, and difficulty. Across these analyses, models show uneven linguistic competence, and human-assigned difficulty does not always match model performance.
4.1 Evaluation Setup
- The evaluation covers 35 models from 350M to 70B parameters, using zero-shot Arabic multiple-choice prompts and accuracy by category and overall.Models select one A–D answer without few-shot examples or chain-of-thought prompting.Source: §4.1 · p. 7
4.2 RQ1: Do Models Exhibit Balanced Linguistic Competence?
- 72–74% average accuracy is reached by top performers, but Syntax remains the most challenging category and models show substantial variation across categories.Yehia-7B scores 86.7% on Spelling and 53.3% on Syntax; category interquartile ranges span 15–20 percentage points.Source: §4.2 · p. 7, Table 3 · p. 8
4.3 RQ2: How Do Linguistic Skills Correlate?
- Grammar and Morphology correlate strongly, Spelling also correlates with Grammar, while Syntax has the weakest links to other categories.Reported correlations include Grammar–Morphology r = 0.83, Spelling–Grammar r = 0.86, Spelling–Reading Comprehension r ≈0.51, and Syntax r ≈0.13− 0.40; the authors interpret Syntax as comparatively independent.Source: §4.3 · pp. 9–10, Figure 4 · p. 9
4.4 RQ3: Does General Benchmark Performance Predict Linguistic Competence?
- General benchmark performance does not reliably predict linguistic competence: some models score highly on knowledge benchmarks yet underperform on AraLingBench.AraLingBench correlates with ArabicMMLU (r = 0.884), EXAMS (r = 0.784), and AraTrust (r = 0.751), but negatively with ALRAGE (r = −0.539); the authors also report contrasting patterns for synthetic-data-tuned and real-instruction-tuned models.Source: §4.4 · p. 10, Table 4 · p. 10
4.5 RQ4: Does Question Difficulty Align with Model Performance?
- 58% median accuracy on Easy items falls to 50% on Medium, then rises to 54% on Hard, so model accuracy does not scale monotonically with annotated difficulty.Qwen3-8B-Base scores 73.1% on Hard items versus 50.0% on Medium; category patterns also differ, with Syntax declining as difficulty rises.Source: §4.5 · p. 11, Figure 6 · p. 11
4.6 Detailed Performance Visualization
- 74.0% is reported for top models in the category heatmap, while model strengths vary across categories and difficulty levels, including non-monotonic patterns.Qwen3-8B-Base scores 58.0% on Easy, 50.0% on Medium, and 73.1% on Hard; the authors interpret this as a mismatch between human- and model-perceived difficulty.Source: Figure 7 · p. 12, Figure 8 · p. 13, §4.6 · p. 14
4.7 Summary of Experimental Findings
- The findings summarize uneven surface-versus-structural skills, partially overlapping category relationships, imperfect alignment between general benchmark success and linguistic competence, and difficulty labels that only partly match model performance.The authors present AraLingBench as a complement to knowledge-oriented evaluation because it exposes gaps in fundamental linguistic understanding.Source: §4.7 · p. 15
5 Conclusion
AraLingBench evaluates Arabic LLMs on five foundational linguistic categories that existing knowledge-based benchmarks may overlook. Evaluation results show that high general-benchmark performance does not necessarily reflect genuine linguistic understanding, and the benchmark provides a complementary diagnostic perspective.
5 Conclusion
- AraLingBench targets Arabic LLM competence in grammar, morphology, spelling, reading comprehension, and syntax, focusing on linguistic foundations that current knowledge-based benchmarks overlook.The benchmark is fully human annotated.Source: §5 · p. 15
- Evaluation of more than 30 models finds that high general-benchmark performance does not necessarily indicate genuine understanding, as models still struggle with grammatical and morphological reasoning.The authors position AraLingBench as a complementary diagnostic for distinguishing superficial fluency from linguistic competence.Source: §5 · p. 15
- The authors release AraLingBench to support development of Arabic LLMs that demonstrate mastery of language structure and logic as well as fluency.Source: §5 · p. 15