SHAMS: An Audio-Grounded Pronunciation Benchmark for Levantine Arabic

• LA’s regional diversity and non-standardized writing leave key phonetic features unspecified, motivating shared evaluation benchmarks.Source: §1 · p. 1
• Even conventional diacritics can leave regional pronunciation unspecified; Palestinian ق is realized as [ʔ], [k], or [ɡ] across urban, rural, and Bedouin speech.Source: §1 · p. 2
• Hand-producing phonetic transcriptions is slow and a major data-collection bottleneck in dialectology, while LA systems lacked a performance measure for the task.Source: §1 · p. 2
• SHAMS ties diacritization, G2P, ASR, and A2P evaluation to the same audio-grounded utterances, unlike prior efforts addressing tasks in isolation.The source corpora discussed do not provide diacritics, a phonetic tier, or variety-level stratification.Source: §1 · p. 2
• SHAMS contains 1,300 utterances from five Levantine varieties, balanced by word count and represented across four aligned tiers.The tiers comprise audio, unvocalized orthography, diacritized text, and phonetic transcription.Source: §1 · p. 2
• Baseline results across open and proprietary models, stratified by variety, demonstrate SHAMS’s utility for distinguishing model performance across tasks and varieties.Source: §1 · p. 2

2.1 Sources and Contents

• The 1,300 utterances span five varieties and roughly 15,500 words, drawn from scripted television dialogue, documentary interviews, and spontaneous conversation.The benchmark is roughly balanced by word count; utterances with fewer than three words or heavy background noise were excluded, and the data is evaluation-only.Source: §2.1 · p. 3

2.2 Benchmark Structure

• Each utterance has four aligned tiers: audio, corrected unvocalized Arabic, fully diacritized Arabic, and phonetic transcription.The diacritized tier matches the pronunciation in the recording, while the phonetic tier uses a Semitic transcription system.Source: §2.2 · p. 3, §2.3 · p. 3

2.3 Annotation

• Diacritization marks short vowels, gemination, and nunation; phonetic transcription captures dialectal pronunciations not specified by fully diacritized orthography.Both pronunciation tiers document what is heard rather than normalize to canonical forms; 780 utterances received an automatic first pass followed by annotator correction, and every utterance was reviewed by an author.Source: §2.3 · pp. 3–4

2.4 Annotation Reliability

• Annotator agreement was 93.2% across 7,290 paired diacritic-slot observations (one per Arabic letter), with Cohen’s κ = 0.898; intervals were 95% cluster-bootstrap confidence intervals resampling full utterances.Words with different consonantal skeletons were excluded as orthographic divergences, not diacritic disagreements (142 of 1,881 words).Source: §2.4 · p. 4, Table 3 · p. 4
• Agreement was high across varieties (κ ≥0.86), and slightly higher on Palestinian material (κ = 0.911) than elsewhere (0.883).The difference was 0.027 (95% CI [0.006, 0.050]).Source: §2.4 · p. 4
• Disagreements most often concerned whether a vowel was marked, followed by gemination and vowel choice; some differences were spelling divergences.Spelling divergences mostly involved hamza seating and sometimes dialectal forms.Source: §2.4 · p. 4

2.5 Release and Licensing

• LangMedia utterances and annotations are released under CC BY-SA, while Casablanca-linked annotations are released with identifiers without redistributing audio.The release format was confirmed as consistent with Casablanca’s CC BY-NC-ND license.Source: §2.5 · pp. 4–5

3 Experiments and Results

• Evaluation covers four tasks and reports aggregate and variety-stratified scores, with strict and neutral scoring separating recognition errors from spelling or notation differences.Dedicated systems run zero-shot; LLMs use few-shot prompts derived from the annotation guidelines, with examples outside the benchmark.Source: §3 · p. 5

3.1 Diacritization

• 7.8% DER for Levanti is the lowest reported diacritization error, ahead of Gemini 3.5 Flash at 11.6%.Levanti produced silver labels used for part of the dataset, so its score is not an independent measurement on those examples.Source: §3.1 · p. 5, Table 4 · p. 5
• 11.9–12.2% CER for the three frontier LLMs outperforms Levanti at 18.3% and CharsiuG2P at 34.8% on G2P.Levanti produced the first-pass transcription for the expansion utterances.Source: §3.2 · p. 6, Table 6 · p. 6
• Levanti’s CER drops 4.1 points under neutral normalization, while CharsiuG2P’s 2.4-point gap indicates largely phonological errors.MSA systems retain errors from word-final vowels absent in Levantine speech and impose MSA case vowels and al-; rural Palestinian qaf is also commonly rendered as [q] or urban [ʔ] rather than [k].Source: §3.1 · p. 5, §3.2 · p. 6, Table 7 · p. 7

3.3 ASR

• 27.6 WER and 9.6 CER make Gemini 3.5 Flash the strongest overall ASR system; prompted audio LLMs and dialect-targeted systems lead.Among open-weight systems, Seamless-EGY improves on Seamless-MSA on every variety, and BYZÖ-Whisper has the second-best overall CER.Source: §3.3 · p. 7
• MSA-designed ASR systems commonly substitute MSA equivalents for dialectal forms, whereas dialect-adapted systems are more faithful.Whisper-Levantine has the largest strict–neutral gap, 15.0 points, because it omits specified Arabic characters; audio LLMs also score better on orthographic than phonetic transcription, a qualitative comparison because the references are not on a common scale.Source: §3.3 · p. 7, §3.4 · p. 7
• 20.5% CER for Gemini 3.5 Flash is lower than GPT-Audio’s 24.9% on direct audio-to-phoneme transcription.The systems are usually faithful, but occasional reversion to Arabic script on hard utterances is scored as fully wrong.Source: §3.4 · p. 7
• 18.9% vs 20.5% CER for Gemini and 23.3% vs 24.9% for GPT-Audio are cascade vs direct A2P scores; gold-transcript G2P reaches 13.0%.The cascade’s G2P stage is Gemini’s in both cases, making the GPT-Audio cascade hybrid and its margin an upper bound; the analysis attributes roughly 13 CER points to phonetic conversion and a further six to ASR error propagation.Source: §3.4 · p. 7

Diacritization and G2P

• Diacritization research largely targets MSA or Classical Arabic, while dialectal extensions and Arabic G2P work remain more limited; no dedicated LA diacritization or G2P benchmark is identified.Prior G2P approaches include diacritization followed by rule-based mapping and more recent end-to-end work for Egyptian Arabic.Source: §4 · pp. 7–8

Dialectal Arabic Resources and ASR

• Arabic resources often collapse dialect distinctions, and existing frameworks leave gaps in Levantine variety coverage or task scope; no comparable cross-dialectal evaluation resource for LA is identified.Arab Voices treats Levantine as one variety and targets ASR, while NADI provides a time-limited evaluation across eight country-level dialects.Source: §4 · pp. 8–9

Phone Recognition and Dialectal Transcription

• Speech-to-phone research includes multilingual phone recognition, speech-to-IPA, audio-to-IPA, and automatic dialectal transcription, including work on Finnish and Norwegian.Source: §4 · p. 9

• SHAMS comprises 1,300 utterances with four aligned annotation tiers for Levantine Arabic speech-language technology.Source: §5 · p. 9
• Experiments across diacritization, G2P, ASR, and A2P differentiate model performance across tasks and varieties, revealing progress by dialect-adapted models and LLMs alongside remaining gaps.Source: §5 · p. 9
• Fine-grained annotations expose performance variation among Levantine varieties, underscoring the importance of variety-aware evaluation.Source: §5 · p. 9

• Coverage excludes Bedouin, rural Jordanian, Lebanese, and Syrian varieties, as well as diaspora speech.Source: Limitations · p. 9
• For most varieties, a single source confounds dialect with content, speaker, and recording quality; Jordanian source comparisons also entangle source and speaker gender.Speaker gender is imbalanced in some varieties, and Casablanca actors may not be native speakers of the portrayed variety.Source: Limitations · p. 9
• The tested models and tasks could be expanded, including with audio-guided diacritization.Source: Limitations · p. 10

Ethical Considerations

• SHAMS is intended to measure how well current systems serve Levantine Arabic speakers across varieties and support work to increase their representation.The data comes from existing open audio corpora, and linguistically trained co-authors and collaborators produced the annotations, including native Palestinian Arabic speakers.Source: Ethical Considerations · p. 10

A Further Annotation Details

• Diacritics encode dialectal pronunciation contrasts without introducing alternative spellings, balancing phonetic fidelity with preservation of the original orthographic form.The scheme is guided by readability, consistency, and recoverability of the underlying word.Source: A Further Annotation Details · p. 13
• Consonantal forms are generally preserved, while substitutions are used only when they avoid ambiguity and improve representation of dialectal speech.For example, ق is retained across its pronunciations, and imāla is represented with vowel diacritics rather than changed letters.Source: A Further Annotation Details · p. 13
• Gemination is marked with shadda, reduced forms are segmented into underlying components, and dialect-specific lexical and morphological forms are retained.These choices preserve morphological transparency and retain cues that can help identify dialect.Source: A Further Annotation Details · p. 13
• Transcription records context-sensitive glottal onset and renders integrated loanwords as pronounced rather than according to source-language spelling.In one example, ʾawwal and ʾiši have glottal onset while Abu does not; French colonel and général are transcribed as il-kulunēl il-ǧinirāl.Source: A Further Annotation Details · p. 13
• The transcription conventions document variation both across varieties and within a variety, including alternative realizations of consonants, vowels, and individual words.Examples include differing realizations of interdentals, imāla, diphthongs, and short vowels, as well as within-variety forms such as il-Fawz/l-Fōz and hayda/hēda.Source: A Further Annotation Details · p. 13

B.1 Preprocessing and Normalization

• Evaluation-time normalization is applied symmetrically to references and hypotheses, following the reductive orthographic fold while leaving non-alif hamza forms unfolded.Released tiers already have canonical Unicode composition, presentation forms folded, and invisible controls and tatweel removed.Source: §B.1 · p. 13
• Both Arabic-script tiers normalize alif variants, alif maqṣūra, and tā′ marbūṭa because these distinctions carry no phonemic contrast in Levantine and are spelled inconsistently.This normalization prevents spelling differences from counting as errors.Source: §B.1 · p. 13
• Diacritization scoring strips sukun, canonicalizes accusative tanwin placement, and drops MSA case-tanwin and related endings while retaining fathatan.The MSA endings are almost absent from references (1 of 15,506 tokens); including tanwin in DER is reported as an ablation.Source: §B.1 · pp. 13–14
• ASR reports neutral and strict tiers to separate recognition errors from spelling conventions, while G2P and A2P use a primary neutral tier that folds supported notation variants but preserves phonological contrasts.For A2P, each per-utterance CER is capped at 100% before averaging; notation not covered by the normalization rules remains an error under both tiers.Source: §B.1 · p. 14

B.2 Per-Source Results

• Jordanian source difficulty varies by system: LangMedia is harder for six of nine systems by +0.4 to +10.0 WER, while Whisper v3 and MMS-1b are better on it by −26.3 and −14.5.GPT-Audio is nearly level, with −1.3 WER.Source: §B.2 · p. 14
• The Jordanian source contrast bounds rather than isolates source effects because source and speaker gender remain partly entangled.The Palestinian Urban Casablanca sample is small, making its contrast indicative rather than conclusive.Source: Table 12 · p. 14
• BYZÖ-Whisper fine-tuned on the Casablanca training split performs better on Casablanca material in both varieties (+7.5 Jordanian; +11.2 Palestinian Urban) and overall, scoring 23.7 versus 44.2 WER.Source: §B.2 · p. 14

B.3 Per-Variety Audio Results

• The oracle varies by 3.5 points across varieties (11.4 to 14.9), versus 4.5 and 5.4 for the two direct A2P systems, indicating that most spread comes from the audio stage rather than G2P.The oracle is hardest on Palestinian/Rural, where the shared G2P stage is weakest.Source: §B.3 · pp. 14–15
• Cascades do not beat direct A2P everywhere: Gemini is worse on Palestinian/Rural (22.4 against 21.0), and GPT-Audio's Lebanese cascade margin is nil (23.1 against 23.2).The comparison includes direct A2P, ASR-to-G2P cascades, and gold-transcript-to-G2P oracles.Source: §B.3 · pp. 14–15

B.4 Models

• The appendix provides model lists, links, and configuration details for the diacritization, G2P, ASR, and A2P experiments.These inventories are presented in Tables 15–18, respectively.Source: §B.4 · p. 15

B.5 LLM Generation Settings

• The prompted LLMs use documented decoding settings; text-input tasks request JSON, while A2P requests a single-line transliteration from audio, with output-token caps to limit runaway generation.Models are accessed through official APIs except JAIS, which is served locally with 4-bit quantization.Source: §B.5 · p. 15, Table 19 · p. 17

B.6 Example Outputs

• The appendix provides full-utterance examples for diacritization, G2P, ASR, and A2P, with one example per variety and gold references.The A2P examples compare outputs from GPT-Audio and Gemini 3.5 Flash; the G2P examples include Gemini 3.5 Flash and Levanti.Source: §B.6 · p. 15, Table 20 · p. 19, Table 22 · p. 19

D.1 Diacritization Prompt

• The diacritization prompt asks the model to infer likely spoken Levantine pronunciation from text alone while preserving the original Arabic letters and avoiding formal Fuṣḥā pronunciation unless natural.The model receives no audio and is instructed to return diacritized text in a JSON object.Source: §D.1 · p. 21
• Diacritization instructions use context to resolve ambiguity, reserve notes for important uncertainty, preserve Arabic letters despite dialectal consonant variation, and require valid JSON.The prompt says not to reject or alter a word solely because of dialectal pronunciation variation.Source: §D.1 · p. 23

D.2 Grapheme-to-Phoneme Prompt

• Grapheme-to-phoneme instructions request natural Levantine pronunciation in Latin script using DIN 31635 extended for Levantine sounds and dialectal realizations.The instructions distinguish symbols for consonant variants and vowels, including ē/ō for monophthongized diphthongs versus ī/ū for original vowels.Source: §D.2 · pp. 24–25

D.3 Audio-to-Phoneme (A2P) Prompt

• Audio-to-phoneme instructions require a single-line Latin-script transliteration of the heard dialectal speech, not a Modern Standard Arabic normalization.They specify DIN 31635 with Levantine conventions, including dialectal sounds, vowel symbols, clitics, and code-switched words in normal Latin spelling.Source: §D.3 · p. 27

D.4 Automatic Speech Recognition (ASR) Prompt

• ASR instructions require a single-line Arabic-script transcription of the exact dialectal speech, without diacritics, while retaining foreign code-switched words in Latin spelling.The output must contain no transliteration, JSON, labels, or commentary.Source: §D.4 · p. 27