Does Modern Standard Arabic (MSA) Dominate Arabic Dialects in LLMs? A Representation-Level Analysis

1 Introduction

• LLMs may understand dialectal Arabic but often default to MSA in generation, prompting a question about whether MSA also dominates internal representations.The paper frames this as a distinction between output behavior and internal processing.Source: §1 · p. 1
• Arabic varieties form a closely related continuum with shared vocabulary, morphology, and frequent interaction with MSA, making them a challenging test of language dominance.The paper contrasts this setting with clearly separated languages.Source: §1 · p. 1
• Adapting language-dominance probing to 25 dialects and MSA reveals no evidence of a dominant variety, but dense overlap and architecture-dependent layer-wise separability.The findings suggest MSA-biased generation and internal representational dominance are distinct phenomena.Source: §1 · p. 1
• The paper’s contributions include showing that MSA-biased generation does not imply MSA-dominated internal representations.It identifies dense dialectal overlap as a reason to analyze generation bias separately from representation geometry.Source: §1 · p. 1

2 Related Work

• Earlier logit-lens studies reported indications of language dominance in intermediate layers and hypothesized an English-centric representational space.The cited studies include Zhang et al. (2023), Wendler et al. (2024), and Zhong et al. (2025).Source: §2 · p. 2
• Belrose et al. (2025) showed that logit-lens approaches can mislead on modern architectures, while Shani and Basirat (2025) introduced GMM probing and suggested no single language dominates cross-lingual spaces.The present work adapts that framework to closely related Arabic varieties.Source: §2 · p. 2

Representational Challenges in Arabic Dialects

• Analyses of how LLMs process Arabic internally remain a significant gap, despite recent work on cross-lingual transfer and dialect-generation constraints.The cited work reports geographic-proximity effects in transfer and improved dialectal generation quality after decoupling dialectal subspaces from MSA during fine-tuning.Source: Representational Challenges in Arabic… · p. 2
• Whether MSA-biased output reflects true internal dominance or dense entanglement remains unexplored, motivating the paper’s layer-wise analysis.MSA’s prominence in posttraining datasets is noted as relevant context.Source: Representational Challenges in Arabic… · p. 2

3 Experimental Framework

• The adapted framework tests whether dialectal hidden states remain dialect-specific or systematically align with MSA or another Arabic variety.It transfers the language-dominance question from cross-lingual to cross-dialectal representations.Source: §3 · p. 2
• MADAR supplies parallel travel-domain sentences for 25 city-level dialects and MSA, enabling comparison of semantically equivalent sentences across varieties.BLOOM and mGPT use 400 sentences per variety, with default tokenizers and hidden states extracted across layers.Source: §3 · pp. 2–3
• Separability is measured by layer-wise NMI between dialect labels and cluster assignments, where lower NMI indicates less separability and more overlap.The analysis separately measures whether representations from one variety align with another variety’s embedding space.Source: §3 · p. 3
• The token-level likelihood ratio compares posterior probability for a token’s true dialect with each competing dialect using aligned parallel tokens.Values below one indicate stronger alignment with a competitor; the analysis pools token-competitor pairs by layer and reports the fraction below one.Source: §3 · p. 3
• The layer-wise dominance score is average posterior mass a variety attracts from tokens of other varieties; uniform-baseline values indicate no such pull.The baseline is 1/26 because the analysis covers 26 Arabic varieties.Source: §3 · p. 3
• Additional tests use Qwen2.5-7B, Meta-Llama-3.1-8B-Instruct, and ArabianGPT-08B-V2, contrasting multilingual with Arabic-native pretraining rather than exhaustively benchmarking Arabic LLMs.These models are evaluated with 100 parallel samples per variety; further robustness checks are in appendices.Source: §3 · p. 3

4 Results and Discussion

• Average NMI is around 0.1 for mGPT and 0.2 for BLOOM, but clear diagonals show dialect identity persists despite low separability.A supervised linear probe also predicts dialect labels above chance.Source: §4 · p. 3, Figure 2 · p. 4
• 1/|V| ≈0.038 is the near-uniform dominance baseline; Λ < 1 tokens stay around 30% across layers, without the distinct-language middle-layer spike.The probe is symmetric across dialect pairs and produces a middle-layer spike on distinct languages.Source: Figure 3 · p. 4, §4 · p. 4
• Geographically close pairs stand out, but overlap does not follow a simple lexical-similarity ranking; hidden states reflect lexical, regional, and model-specific factors.The parallel MADAR setup controls semantic variation, leaving surface-form, morphological, and lexical differences as much of the remaining signal.Source: §4 · pp. 3–4
• MSA-biased generation is not explained by internal MSA dominance: no dialect acts as an internal hub, and output preference should not be read as a hidden representational pathway.Under the probing framework, MSA behaves as one variety among many.Source: §4 · pp. 4–5
• Dialect separability changes at different depths across architectures; ArabianGPT-08B-V2 shows a sharp late-layer NMI collapse.Meta-Llama-3.1-8B-Instruct is relatively stable, while Qwen2.5-7B drops early and modestly recovers toward the final layer.Source: §4 · p. 5
• POS filtering and sentence-prefix removal preserve the main trends, suggesting overlap is not driven only by function words or sentence-initial artifacts.The authors conclude that generation bias and internal representation geometry must be analyzed separately.Source: §4 · p. 5

• No variety acts as a dominant internal reference point; dialectal information remains present but is not organized around one variety.The study adapts the language-dominance framework to 25 Arabic dialects and MSA.Source: §5 · p. 5
• Output preference does not necessarily reveal hidden-representation geometry, so generation bias should not be treated as direct evidence of internal routing.Source: §5 · p. 5
• Possible contributors to MSA-biased generation include data proportions, orthography, and decoding-time preferences, but disentangling them is beyond this work’s scope.Source: §5 · p. 5

• The experiments use 400 sentences per dialect for BLOOM and mGPT and 100 per dialect for additional MADAR models, whose data are restricted to the travel domain.The authors caution that the travel domain may not reflect the full range of dialectal variation in natural text.Source: §6 · p. 5
• The 400-sentence samples are moderate, so findings are exploratory rather than definitive; small writer pools may also modestly inflate separability estimates.Rule-based POS tagging in the ablations may introduce inaccuracies compared with fully trained Arabic NLP systems.Source: §6 · p. 5
• Model selection contrasts multilingual and Arabic-native pretraining rather than comprehensively surveying Arabic LLMs.Evaluating additional recent Arabic-centric models is identified as future work.Source: §6 · p. 5
• Positioning Arabic varieties more precisely on the dialect–language continuum would require separability curves for closely related language families.Source: §6 · p. 6

• The study uses existing datasets and pretrained models and involves neither human subjects nor newly collected user data.Source: §7 · p. 6
• Dialect labels are treated as dataset annotations for linguistic analysis, not as fixed social or national identities.Source: §7 · p. 6
• The results should not be interpreted as ranking dialects, speakers, or communities.Source: §7 · p. 6
• LLM assistants supported code development and grammatical or stylistic feedback; the authors manually verified the work and retain responsibility for it.Source: §7 · p. 6

Replication Setup and Results

• The setup follows Shani and Basirat (2025), using PUD treebanks with mGPT and BLOOM on the same language set to verify recovery of the original study’s trends.Source: A Replication Setup and Results · p. 7
• Meta-Llama-3.1-8B-Instruct maintains relatively stable dialect separability across middle and later layers.Source: Figure 4 · p. 7
• Qwen2.5-7B’s NMI drops sharply in early layers and partially recovers later.Source: Figure 5 · p. 7
• ArabianGPT-08B-V2 retains high dialect separability until later layers, where it collapses sharply.Source: Figure 6 · p. 7
• A minimum PCA dimensionality is enforced when retained variance would leave too few components, preventing unstable mixture modeling in highly anisotropic layers.The modification does not change the original framework’s logic.Source: A Replication Setup and Results · p. 7
• The replication broadly recovers the original trends: mGPT’s NMI drop appears between layers 3 and 14, while BLOOM’s curves are almost identical to the original.mGPT’s tail layers are roughly 0.1–0.3 NMI points higher; replicated confusion matrices and dominance scores also recover the main trends.Source: A Replication Setup and Results · pp. 7–8
• Token-level Λ statistics are broadly consistent across layers and models, with mGPT producing more very low Λ values than BLOOM.Source: A Replication Setup and Results · p. 8

B Additional NMI Results

• Across three additional models, strong dominance effects remain absent, while layer-wise separability ranges from relatively stable to sharper early- or late-layer collapses.The models are Meta-Llama-3.1-8B-Instruct, Qwen2.5-7B, and ArabianGPT-08B-V2.Source: B Additional NMI Results · p. 8

C Ablations and Validation Experiments

• Across validation experiments, Arabic dialectal representations remain overlapping without stable MSA or dialectal dominance.The tests examine token type, sentence position, and parameter choices.Source: C Ablations and Validation Experiments · p. 8

C.4 Supervised Linear Probe

• 0.15–0.25 test accuracy across layers for both models exceeds the 3.84% random-chance level, showing dialect information is encoded despite difficult unsupervised separation.Probe accuracy is higher in early and late layers and dips in middle layers where NMI collapse is strongest, consistent with an entangled rather than single-dominant-variety representation.Source: Figure 7 · p. 8, Figure 8 · p. 8, §C.4 · p. 9

C.1 POS Filtering

• Removing PREP, CONJ, PART, PRON, and PUNCT tokens leaves the no-dominance pattern unchanged, so shared function words do not primarily drive the observed overlap.The filtering removes 64,071 tokens, approximately 21.5% of the corpus.Source: §C.1 · p. 8

C.2 Sentence-Prefix Removal

• Across all prefix-removal settings, the overall absence of stable MSA or dialectal dominance remains unchanged, while removing larger prefixes slightly reduces the proportion of Λ < 1 tokens in intermediate layers.This suggests sentence-initial tokens contribute modestly to cross-dialect mixing.Source: §C.2 · p. 8

C.3 PCA Sensitivity Analysis

• Across sample sizes and PCA dimensionalities, NMI generally rises with more principal components, but stable dominance remains absent.The tested settings vary samples from 100 to 400 sentences per dialect and minimum PCA dimensionality from 0 to 300 components.Source: §C.3 · p. 9

D PCA Dimensionality and Experimental Parameters

• PCA projections retain 98% of explained variance, with a minimum dimensionality threshold used to prevent highly anisotropic layers from collapsing to very few components.The minimum threshold is adjusted by model and experiment; dialectal analysis uses 200 components for 400 samples and 100 for 100 samples in both mGPT and BLOOM.Source: D PCA Dimensionality and Experimental… · p. 9
• Tokenizer offset mappings are enabled to support subword averaging, with tensors, offsets, special tokens, and no truncation specified in the configuration.The procedure uses each model's tokenizer.Source: D PCA Dimensionality and Experimental… · p. 9
• The language-space construction configuration includes PCA variance, a minimum-dimension option and threshold, covariance regularization, and a seed.The listed settings are pca_variance=0.98, use_min_pca_dim=True, model-specific pca_min_dim, reg_covar=1e-6, and seed=0.Source: D PCA Dimensionality and Experimental… · p. 9
• For dialectal analysis, minimum PCA dimensionality is 200 components at 400 samples and 100 at 100 samples for both mGPT and BLOOM.Higher thresholds are used for dialects because their greater representational similarity requires more components for stable mixture modeling and interpretable confusion matrices.Source: D PCA Dimensionality and Experimental… · p. 9