Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context

Motivation

• Iraqi Arabic has no comprehensive LLM evaluation framework, despite its divergence from MSA and relevance to public-administration use.Existing dialect resources focus on identification and classification, while recent dialect-aware evaluations omit Iraqi Arabic.Source: §1 · pp. 1–2

Mizan and study

• Mizan pairs an MSA baseline with an Iraqi track across six axes, using 340 originally authored, dually reviewed items and an integrity protocol exercised during the study.The protocol includes immutable snapshots, verification certificates, a human publication gate, and public retraction.Source: §1 · p. 2

Pilot study

• The pilot evaluates 27 systems, including closed frontier models, open weights across size tiers, and commercial, specialized, and sovereign Arabic systems.GPT-6 Astra was evaluated three days after its release.Source: §1 · p. 2

Pilot findings

• MSA scores cluster near the ceiling, while Iraqi scores spread across more than twenty points, with a consistent 14–18-point gap for each model.The two strongest systems tie and swap ranks across independent evaluation rounds.Source: §1 · pp. 1–2
• Every system scores 32–56 on official-document extraction, while two dedicated Arabic models trail a size-matched generalist on the Iraqi track.The safety-hardened tier of the newest model family also deterministically refuses innocuous Iraqi comprehension items.Source: §1 · p. 2

2 Related Work

• Arabic evaluation has developed across MSA benchmarks and leaderboards, dialect-classification resources, and a newer turn toward dialect-aware LLM evaluation.Mizan draws on all three lines while departing from each.Source: §2 · pp. 2–3

2.1 MSA benchmarks and aggregated leaderboards

• ArabicMMLU, AlGhafa, OALL, HELM Arabic, and BALSAM provide rigorous and mature MSA evaluation, but do not measure dialect, local knowledge, or civic-document registers.On Mizan’s MSA track, contemporary systems score near 100.0 and the strongest two are statistically inseparable.Source: §2.1 · p. 3

2.2 Dialect resources of the classification era

• MADAR and NADI provide foundational dialect corpora and identification tasks, measuring whether systems recognize or label varieties rather than understand, generate, translate, or safely use them.Mizan retains regional granularity through a dialect-region tag on each Iraqi item.Source: §2.2 · p. 3

2.3 Dialect-aware LLM evaluation

• AraDiCE and DialectalArabicMMLU extend dialect-aware evaluation, but omit Iraqi Arabic and rely on machine-translated or manually translated benchmark items.Mizan also addresses official-document extraction and safety behavior in a specific national context.Source: §2.3 · p. 3

2.4 Positioning

• Mizan complements existing benchmarks with originally authored, dually reviewed evaluation of Iraqi Arabic and the Iraqi civic context.Its Iraqi track supplies a discriminative signal alongside shared-Arabic and regional-dialect benchmarks.Source: §2.4 · p. 3

3.1 Design principles

• Mizan’s design principles require native authorship and dual review, separate public development from reserved private-test items, audit answer-position balance, and use human judges for generative axes.The pilot bank is public development data, and answer-position counts pass chi-square tests of uniformity for every track-axis group.Source: §3.1 · p. 4

3.2 Two tracks of equal standing

• Every axis has an MSA baseline and an Iraqi track; the Iraqi items cover Baghdadi, southern, and mixed registers, with a region tag on each item.The benchmark presents each model’s MSA score, Iraqi score, and the distance between them.Source: §3.2 · p. 4

3.3 Six axes

• The six axes test dialect comprehension and generation, bidirectional MSA–Iraqi translation, Iraqi knowledge, official-document extraction, and safety in Iraqi context.Extraction requires six fields from simulated official letters; generation and safety use human judgment.Source: §3.3 · p. 4

3.4 Authoring workflow and the rejected-item discipline

• Native-speaker reviewers reject items with rare or regionally opaque vocabulary, ambiguous keys, or contested ground truth.A Tigris–Euphrates confluence question was rejected because Qurna and Karmat Ali each offered a defensible answer.Source: §3.4 · pp. 4–5

3.5 Orthography

• Because Iraqi Arabic has no standard orthography, the pilot applies documented internal spelling conventions without claiming to establish a unified guide.A publishable orthography guide is reserved as a future deliverable.Source: §3.5 · p. 5

3.6 Composition

• The 340-item bank contains 265 Iraqi-track and 75 MSA-track items, with 190 auto-scored items per model and 150 generative items pending human judging.The leaderboard marks generative axes as pending rather than substituting automatic proxies.Source: §3.6 · p. 5

4.1 Architecture

• A bilingual platform stores items, runs, and per-axis results, while an offline runner emits self-describing results for separate import and publication.The results file records the harness commit hash, provider, and configuration.Source: §4.1 · p. 5

4.2 Uniform conditions and the failure ladder

• All models receive identical prompts, scoring, and completion budgets, with logged retries for transient failures and deterministic refusals scored separately as failures.The retry ladder raises the completion budget from 2048 to 8192 tokens up to twice.Source: §4.2 · p. 5

4.3 Snapshots, certificates, and public verification

• Each published run is an immutable, dated snapshot, and every imported run receives a SHA-256 certificate for independent score verification.New model versions create new runs rather than overwriting existing ones.Source: §4.3 · p. 5

4.4 The human gate, retraction, and supersession - exercised in practice

• A human publication decision is required; diagnostic charts exposed two wounded runs that were publicly retracted, then both models were reevaluated and the board re-verified.Retracted certificates remain preserved and are shown as revoked.Source: §4.4 · pp. 5–6

4.5 Self-hosted models and the submission model

• The runner supports OpenAI-compatible endpoints, allowing self-hosted models to be evaluated under the same conditions as API models.Reading is open, reproduction uses the researcher’s own resources, and official leaderboard publication passes through a human-verified gate.Source: §4.5 · p. 6

5 Experimental Setup

• The 27-system cohort spans general-purpose instruction-tuned models, closed frontier releases, open weights from 7B to frontier scale, and three Arabic-focused systems.The Arabic group includes commercial Mistral Saba, open SILMA-9B, and sovereign ALLaM-7B; two other Arabic candidates could not be evaluated fairly because of artifact or access constraints.Source: §5 · p. 6
• Multiple-choice answers are scored by exact letter match, while extraction uses strict per-field matching against ground truth and Iraqi administrative conventions.Every published proportion carries a 95% Wilson interval; extraction uses a deliberately conservative binomial bound for fractional per-item scores.Source: §5 · p. 6
• Each system completed two independent full rounds, and the reported table uses the re-verification round; prompt sensitivity was not studied.The paired rounds support the agreement analysis reported later.Source: §5 · p. 6

6 Results

• Overall is the macroaverage across five auto-scored axes, not the mean of the MSA and Iraqi track scores.The lower-scoring extraction axis pulls the overall score below the simple average of the two track means.Source: §6 · p. 6

6.1 Saturation on one track, discrimination on the other

• Eight systems score 100.0 on MSA, while Iraqi scores span more than 23 points; most models show a 14–18-point gap between their two tracks.The two strongest systems tie at 90.9 overall and 84.9 Iraqi, swapping ranks across two independent rounds; overlapping 95% intervals leave exact top ranks unsettled.Source: §6.1 · pp. 6–7

6.2 The document wall

• Official-document extraction scores range from 32 to 56.4; the best axis score is 56.4, the newest frontier model reaches 53.7, and 7B systems approach 32.Observed failures include well-formed outputs that miss field precision and administrative convention, rather than mere formatting errors.Source: §6.2 · p. 7

6.3 The specialization law

• SILMA-9B (63.2) and ALLaM-7B (64.7) trail size-matched ministral-8b (73.9) on Iraqi, while matching it on MSA.Mistral Saba exceeds its same-family generalist sibling by about six Iraqi points but remains four to five points below the frontier; no specialization closes the dialect gap.Source: §6.3 · p. 7

6.4 Run-to-run agreement

• Across two full rounds, overall scores move by a mean absolute 0.5 points, with a maximum near 1.2; every difference lies within Wilson intervals.The tied leaders exchanged ranks, consistent with overlapping intervals.Source: §6.4 · p. 7

7.2 The document wall is substantive

• Document-extraction errors include valid structured outputs that misstate issuing authority, paraphrase required actions, or hallucinate absent optional fields.Uniform strict field matching makes the ranking consistent, while the failures reflect distance from Iraqi administrative convention.Source: §7.2 · p. 8

7.3 Residual genuine errors

• At the frontier, genuine Iraqi-axis mistakes are sparse; small-model errors include false-friend interference and misses on post-2003 institutional facts.Source: §7.3 · p. 9

8 Discussion

• MSA benchmarks miss four phenomena: dialect gaps, civic-document failure, Arabic-specialized models trailing a size-matched generalist, and dialect-triggered over-refusal.The document wall confines all 27 systems to 32–56.Source: §8 · p. 9, Figure 3 · p. 10
• The authors link these findings to dialect-data needs, dialectal coverage in safety alignment, government procurement measurement, and model-access barriers for evaluators.They note that most models marketed as Arabic are unreachable through standard research access channels.Source: §8 · p. 9

9 Limitations

• Roughly ±10 points at axis level reflect 50 items per axis, leaving exact rankings within the leading cluster unsettled.Source: §9 · p. 10
• The 150 generative items remain unpublished pending dual human judging, and Mosuli coverage is zero.Baghdadi, southern, and mixed registers dominate the pilot.Source: §9 · p. 10
• No human baseline anchors the score scale, and the public pilot bank remains development-tier until a sealed private set arrives.Source: §9 · p. 10
• Prompt-format sensitivity is unstudied; Arabic models ran in 4-bit quantization, and strict field matching may undercount partially correct administrative phrasings.Source: §9 · p. 10

10 Future Work

• Version 1.0 plans to exceed 1,000 items with regional native authors, add a sealed private test tier and human baseline, complete judged axes, and schedule longitudinal re-evaluations.It also plans formal external submissions and GPU-hosted evaluation of large Arabic models omitted by the API layer.Source: §10 · p. 10

• All official documents are simulated end to end, with fictional authorities, numbers, names, and subjects; no real records, personal data, or classified material were used.Source: §11 · pp. 10–11
• Safety items describe harm categories without harmful content and use generic framings rather than named targets.Source: §11 · p. 11
• The over-refusal analysis relies exclusively on verbatim vendor-official API envelopes.Source: §11 · p. 11
• Code releases under Apache-2.0, alongside a public development set with a datasheet; the live leaderboard provides per-run verification certificates.Source: §11 · p. 11
• The released set is development-tier, so leaderboard integrity ultimately rests on a forthcoming sealed tier rather than secrecy of published items.Source: §11 · p. 11