PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Dialects Be Steered Like Languages? Sparse Neurons and Distributed Directions in Arabic LLMs

Kareem Elozeiri, Mervat Abassy, Omar Kallas, Fahim Dalvi, Preslav Nakov, Kentaro Inui, Nadir Durrani

arXiv:2607.03936v1cs.CL

TL;DR

Arabic LLMs often default to Modern Standard Arabic because dialectal data are scarce, raising questions about how dialects are represented and controlled. This paper probes sparse dialect-associated neurons and distributed activation directions at inference time, finding that vector steering provides more reliable dialect control while dialect information is both sparse and distributed.

  • Problem

    Arabic LLMs often default to MSA because dialectal data are scarce, while fuzzy dialect boundaries make dialect-specific representations difficult to disentangle.

  • Method

    The study compares inference-time neuron steering, which rescales selected dialect-associated neurons, with vector steering, which injects directions extracted from dialect–MSA activation contrasts.

  • Results

    Vector steering provides more reliable dialect control, while dialect information is neither fully localized nor entirely diffuse and sparse neurons capture only part of broader dialect directions.

  • Takeaways & Limitations

    Arabic dialect variation forms a structured, steerable dimension of LLM representation space, with distributed directions offering stronger control than localized neuron subsets.

  • Takeaways & Limitations

    Experiments cover only a limited set of Arabic dialects and two Arabic-centric LLMs, so results may not generalize to other dialects, architectures, or multilingual models.

Abstract

from arXiv · show

A key challenge in Arabic NLP is the scarcity of dialectal data relative to Modern Standard Arabic (MSA), causing LLMs to overproduce MSA and struggle with dialectally accurate generation. From an interpretability perspective, this raises a fundamental question: where and how are dialectal features encoded within model internals, and can these representations be leveraged to improve dialect generation without fine-tuning? This study investigates two complementary inference-time approaches that serve simultaneously as interpretability probes and control mechanisms. First, we conduct a neuron-level analysis, identifying sparse neuron populations that encode dialect-specific features and showing that amplifying or suppressing these neurons can steer model outputs toward target dialects. Second, motivated by the entanglement of dialectal features at the single-neuron level, we apply a vector-steering approach that extracts dialect-specific activation directions and injects them during inference. Together, these methods illuminate the geometry of dialectal knowledge in Arabic LLMs and offer a principled, interpretability-grounded framework for dialect control without requiring dialect-specific fine-tuning.

1 Introduction

Arabic LLMs often default to MSA because dialectal data is scarce, while lexical and syntactic overlap makes dialect features difficult to disentangle. This work therefore studies neuron-level and activation-vector interventions as complementary inference-time strategies for dialect control.

  • Motivation: Arabic LLMs often default to MSA or generate hybrid dialectal outputs because MSA dominates training data while dialectal text is scarce and inconsistently sourced.This limits applications requiring dialectal authenticity, including conversational agents and cultural content generation.
  • Challenge: Arabic dialects pose a difficult interpretability problem because they share substantial lexical and syntactic overlap, and their boundaries are fuzzy rather than discrete.These properties make dialectal features harder to disentangle at the neuron level.
  • Approach: The study investigates whether dialect-specific neurons can be manipulated to steer generation toward a desired dialect.This is the first of two complementary inference-time strategies examined for Arabic LLMs.
  • Approach: The study also extracts dialect-specific directions from activation space and injects them during inference as a broader representational control strategy.This addresses whether dialectal features are distributed across residual space in addition to being localized in neuron populations.
  • Findings: Dialectal information is both sparse and distributed: dialect-associated neurons concentrate in late generation-facing layers, while sparse neurons capture only a partial projection of broader dialect representations.MSA is clearly separated from spoken dialects, with substantial sharing among regional dialects.

2 Methodology

The methodology uses two fixed-weight, inference-time interventions for Arabic dialect control: sparse neuron steering and vector steering. These methods also function as causal probes of dialectal representations while enabling target-dialect generation without parameter updates.

  • Overview: Two complementary interventions steer Arabic dialect generation: sparse dialect-associated MLP neurons and dialect-specific directions extracted from contrastive dialect–MSA pairs.Both interventions operate at inference time with model weights fixed.
  • Neuron Steering: Dialect-associated neurons are selected using activation probability across dialects and low LAPE entropy, indicating selectivity for a small number of dialects.Selection requires top-n% activation probability for at least one dialect and bottom-m% LAPE across neurons.
  • Neuron Steering: During decoding, selected target-dialect neuron activations are amplified, while MSA and optionally competitor-dialect neurons are suppressed to bias generation toward the target dialect.The intervention uses α > 1 for target neurons and γ, γcomp < 1 for MSA and competitor neurons, without changing model parameters.
  • Vector Steering: Vector steering addresses distributed dialect features by operating on the full activation space rather than isolated neurons.It computes a dialect–MSA mean activation difference from parallel sentence pairs, using response-token hidden states at layer ℓ.
  • Vector Steering: A forward hook injects the extracted steering vector into layer-ℓ activations during generation, with scalar coefficient α controlling intervention strength and no retraining or parameter updates.The modified activations are passed to downstream layers to generate dialect-influenced tokens.

3 Experiments

Experiments evaluate neuron- and vector-based dialect steering across two Arabic-centric instruction-tuned models, multiple Arabic dialects, and complementary judge, benchmark, and human metrics. Results characterize dialect-neuron localization and overlap, parameter sensitivity, layer-dependent encoding, token-budget behavior, and steering-direction stability.

  • Experimental setup: Experiments use ALLaM-7B-Instruct-preview and Fanar-1-9B-Instruct, two Arabic-specific instruction-tuned models with 32 and 42 transformer layers, respectively.Using both models tests whether findings generalize across architectures, Arabic corpora, and instruction-tuning procedures.
  • Data and evaluation: Steering vectors and dialect-specific neurons are extracted from MADAR parallel data spanning Egyptian, Moroccan, Levantine, and Gulf Arabic, with 12,000 pairs for Cairo, Rabat, Beirut, and Doha and 2,000 each for Riyadh and Aleppo.Evaluation maps these cities to Egyptian, Moroccan, Syrian, and Saudi benchmark subsets.
  • Data and evaluation: Generation quality is measured with Gemini 2.5 Flash on dialect authenticity, coherence, Arabic fluency, and diagnostically reported MSA formality, using 1–5 scores.AL-QASIDA supplies ADI2 and macro-ADI2 as complementary automatic metrics, while human annotations use the same dimensions with hidden model identities.
  • Neuron analysis: 69.5% of ALLaM dialect-neuron records occur in layers 29–31, while 50.6% of Fanar records occur in layers 37–41, indicating late-layer localization with broader spread in Fanar.Average MSA–dialect neuron overlap is 0.09 in ALLaM and 0.14 in Fanar, versus dialect–dialect overlap of 0.40 and 0.55.
  • Neuron analysis: Target-dialect amplification is the main ablation-sensitive parameter, with best settings of α = 2.0 for ALLaM and α = 4.0 for Fanar; γ and γcomp are inconsistent.These ablations vary one parameter at a time on Egyptian and Moroccan Arabic across both models.
  • Vector steering: Fanar’s dialect steering peaks around layers 19–24, whereas ALLaM shows a different layer profile; ALLaM plateaus after 10–20 injected tokens, while Fanar requires sustained injection.Steering directions stabilize by 4k extraction examples at selected layers; Moroccan Arabic is more coefficient-sensitive than Egyptian Arabic, especially for ALLaM coherence.

4 Results

Results compare dialect steering when prompts are written in the target dialect versus MSA. Vector steering is effective across dialect targets, whereas neuron steering can reinforce but not induce dialectal generation from MSA prompts.

  • Evaluation settings: The evaluation uses mono-dialect prompts and MSA prompts to test both response matching and whether steering can override the prompt variety.All methods use deterministic decoding with up to 128 new tokens; vector directions are injected for the first 30 generated tokens.
  • Mono-dialect results: Vector steering matches or surpasses explicit prompting on judge averages for Fanar across all dialects and for ALLaM on Egyptian and Moroccan Arabic.Explicit prompting still leads on raw ADI2 and macro-ADI2, indicating that ADI2 does not always represent coherence or overall output quality.
  • MSA-prompt results: Neuron steering failed entirely on 300 MSA-prompt synthetic samples steered toward target dialects.Dialect authenticity collapsed to the minimum judge score and ADI2 reached zero.
  • MSA-prompt results: Vector steering produced dialectal signal across all targets despite MSA prompts, although performance was lower than in the mono-dialect setting and ALLaM was more responsive than Fanar.These results show that vector steering can induce dialectal output from MSA prompts, unlike sparse neuron interventions.

5 Analysis and Discussion

The analysis compares distributed dialect directions with the sparse subspace of LAPE-selected MLP neurons to explain differences between vector and neuron steering. Although the sparse neurons occupy less than 1% of MLP intermediate dimensions, they capture meaningful but incomplete dialect-direction projections, especially for Cairo and Rabat.

  • Comparing neuron and vector steering: Residual-subspace coverage measures how much each vector-steering dialect direction is captured by the down-projection subspace of LAPE-selected MLP neurons.The study also reports formal details and random-baseline tests in Appendices C.1 and C.2.
  • Comparing neuron and vector steering: LAPE-selected neurons occupy less than 1% of MLP intermediate dimensions yet capture a meaningful, incomplete projection of residual dialect directions, especially for Cairo and Rabat.Most of each residual direction remains outside the sparse neuron subspace.
  • Comparing neuron and vector steering: Higher residual-subspace coverage indicates stronger overlap between sparse neuron subspaces and distributed dialect directions.Figure 4 presents aggregate dialect coverage and mean coverage across normalized layer depth, with variation across dialects.

6 Related Work

Prior work shows that Arabic LLMs often default to MSA, while dialectal representations and language-control methods motivate neuron- and vector-based steering. This work extends those directions to fuzzy dialect boundaries within Arabic, modeling dialect identity geometrically and steering generation at inference time.

  • Arabic linguistic variation: Arabic LLMs often default to MSA, while dialect transfer varies geographically and suffers from negative interference and excessive MSA entanglement.Prior studies analyze Arabic model internals across varieties and motivate interventions that decouple dialect modeling from the dominant standard variety.
  • Neuron-level language control and activation steering: Neuron-level studies identify language-specific neurons whose activation, amplification, suppression, or overlap can steer outputs across related languages.These findings establish sparse neuron manipulation as a precedent for inference-time language control.
  • Activation steering: Activation-steering research extracts high-level concepts as linear directions and adds contrastive vectors to residual activations to shift model behavior.Related work also finds that mean differences between contrastive activations can provide effective steering directions.
  • Activation steering: Contrastive persona vectors can monitor and control undesirable behaviors, with response-token activations producing more effective steering directions than prompt-token activations.These semantic directions can also transfer across model variants and regulate emergent misalignment.
  • Contribution: Unlike cross-lingual transfer work, this study shows that Arabic dialect identity is geometrically structured and steerable despite fuzzy boundaries and high MSA overlap.Contrastive activation directions steer generation toward a target dialect at inference time, framing intralinguistic variation as an underexplored representation-space dimension.

7 Conclusion and Future Work

The study finds that Arabic dialect information is partly localized in late-layer neurons but also represented as a broader residual-space direction. Distributed vector steering provides more reliable dialect control than neuron steering, including when overriding MSA prompts, while future work targets broader dialectal, stylistic, and multimodal settings.

  • Conclusion: Dialect information is neither fully localized nor entirely diffuse: dialect-associated neurons concentrate in late layers but capture only part of the broader residual-space dialect direction.This conclusion motivates combining neuron-level and distributed representation analyses.
  • Conclusion: Vector steering provides more reliable dialect control across both models, including when overriding MSA prompts.The study compares neuron-based steering with distributed vector steering as inference-time control mechanisms.
  • Future Work: Future work should examine additional dialects, mixed-dialect settings, adaptive and compositional steering, and interactions with stylistic and sociolinguistic attributes in multilingual and multimodal models.These directions extend the analysis beyond the current dialect-control setting.

Limitations

The study’s findings may not generalize beyond the limited dialects and Arabic-centric LLMs examined. Dialectal-generation evaluation is also difficult because automatic metrics, LLM-as-a-judge scores, and human evaluations each have important limitations.

  • Generalizability: The experiments cover a limited set of Arabic dialects and two Arabic-centric LLMs, constraining generalization to other dialects, architectures, and multilingual models.The passage explicitly identifies these settings as limits on generalizability.
  • Evaluation: Automatic metrics and LLM-as-a-judge scores capture complementary performance aspects but neither fully measures dialect authenticity or sociolinguistic naturalness.Human evaluations are also described as inherently subjective.

Ethics and Broader Impact … B Vector Steering

The section frames dialect steering as a lower-cost approach to improving Arabic dialectal inclusivity while emphasizing misuse risks, sociolinguistic limitations, and parameter-ablation findings. Ablations indicate that target-dialect amplification is more influential than suppressing MSA or non-target dialect neurons, with model-dependent preferred settings.

  • Ethics and Broader Impact: Inference-time steering aims to reduce MSA bias and improve generation for underrepresented Arabic dialects without retraining or modifying model parameters.This may lower adaptation costs and improve accessibility for low-resource varieties.
  • Ethics and Broader Impact: Dialect steering could be misused for deceptive regional-style imitation, while fluid dialect boundaries make automatic authenticity evaluations incomplete.The methods are intended for controllable, interpretable generation research rather than impersonation or misinformation.
  • Ethics and Broader Impact: The study used trained Arabic annotators compensated at standard local hourly rates, with confidentiality agreements, limited personal-data collection, and license-consistent dataset release.The supplied passage also states that task instructions were clear and data collection was restricted to payment and administration needs.
  • A Neurons-Based Steering Ablations: Ablations independently varied target amplification α, MSA suppression γ, and competitor-dialect suppression γcomp while holding the other two parameters fixed.Experiments used ALLaM-7B-Instruct-preview and Fanar-1-9B-Instruct with Egyptian Arabic and Moroccan Arabic targets.
  • A.1 Target Dialect Coefficient Ablations: α = 2.0 provided ALLaM’s best overall trade-off across Egyptian Arabic and Moroccan Arabic, whereas α = 4.0 balanced Fanar’s dialect gains against quality degradation at α = 5.0.ALLaM responded non-monotonically, while Fanar’s ADI2 scores increased consistently but very large amplification reduced fluency and coherence.
  • A.2 MSA Neurons Suppression Ablations: MSA suppression had weaker, less consistent effects than target amplification and did not reliably improve dialect authenticity, fluency, or coherence.For ALLaM, larger γ generally preserved or improved performance; for Fanar, automatic scores typically exceeded the unsteered baseline without a consistently best γ.
  • A.3 Non-Target Dialects Neurons Ablations: Non-target suppression likewise showed limited, inconsistent benefits: automatic dialect scores stayed above the unsteered baseline, but judge scores showed no reliable quality or authenticity gains.The ablation fixed γ = 1.0 and used α = 2.0 for ALLaM or α = 4.0 for Fanar; smaller γcomp meant stronger suppression.
  • B Vector Steering: The ablation figures compare LLM-as-a-judge fluency, coherence, and dialect-authenticity scores with automatic ADI2 metrics against dashed unsteered baselines.Figures 5 and 6 cover target amplification and MSA suppression for Egyptian Arabic and Moroccan Arabic across both models.

B.1 Analysis of Extracted Dialect Representations … D.1 Setup

The analyses show that Arabic dialect representations preserve geographic structure, support inference-time steering, and are captured only partially by sparse neuron subsets. Steering directions are sample-efficient and vector steering can access distributed residual-space components beyond neuron steering, while evaluation uses deterministic LLM-as-a-judge scoring.

  • B.1 Analysis of Extracted Dialect Representations: Dialect representations preserve geographic structure across three Arabic LLMs, with nearby cities clustering and regional varieties occupying distinct regions in PCA space.PC1 explains 56.88% for ALLaM, 49.42% for Fanar, and 42.83% for Jais-2.
  • B.2 Coefficient Ablation: Higher steering coefficients increase dialect signal but can degrade coherence at later ALLaM layers, revealing a quality ceiling that ADI2 alone misses.For Egyptian Arabic, degradation begins at layers 19–21 under higher coefficients.
  • B.3 Token Budget Ablation: ALLaM commits to Egyptian and Moroccan targets after minimal intervention, whereas Fanar requires sustained injection to maintain the target register.ALLaM’s ADI2 plateaus around 0.51 for Egyptian Arabic and 0.31 for Moroccan Arabic after the first 20 tokens.
  • B.4 Sensitivity Analysis: The steering direction is robust to sample size, with even the smallest subset recovering the broad full-sample orientation.Increasing sample size mainly smooths the estimate rather than changing its orientation.
  • B.4 Sensitivity Analysis: Around 4k examples provide a stable steering-vector estimate effectively equivalent to the full 12k vector, while early layers remain more sampling-sensitive.At selected steering layers, the 4k, 6k, and 12k estimates form an almost collinear cluster.
  • C.1 Formalism: Neuron steering alters selected MLP coordinates, whereas vector steering injects residual-stream directions; their comparison uses selected neurons’ down-projection directions.Residual-subspace coverage measures how much of a vector-steering direction lies in the subspace reachable by selected neurons.
  • C.2 Significance Against Randomly Selected Neurons: Matched random-subspace comparisons show that raw coverage is insufficient: ALLaM has higher average coverage but uneven enrichment, while Fanar has lower coverage but more consistent alignment.Cairo and Beirut show clear ALLaM enrichment, whereas Doha, Rabat, and Riyadh are not reliably above random; Fanar’s tested dialects are significantly enriched.
  • C.2 Significance Against Randomly Selected Neurons: Cairo shows the clearest neuron–vector alignment, Riyadh is among the weakest, and sparse neurons explain only a minority of the full residual shift.This partial alignment helps explain why neuron steering is less effective than vector steering.

D.2 Detailed Results · E AL-QASIDA Evaluation Details · F Explicit Prompt Baseline

The section details how vector steering affects per-dimension dialect generation quality, describes the deterministic monolingual AL-QASIDA evaluation, and specifies the explicit-prompt baseline. Together, these details emphasize gains in dialect authenticity while identifying residual MSA formality as a limitation.

  • D.2 Detailed Results: The detailed results report per-dimension LLM-as-a-judge scores underlying the summary results in Section 4.The dimensions are fluency, coherence, dialect authenticity, and MSA formality.
  • D.2 Detailed Results: Vector steering mainly improves dialect authenticity and reduces MSA formality while preserving fluency and coherence in mono-dialect outputs.Table 6 evaluates Arabic fluency, coherence, dialect authenticity, and MSA formality, with lower scores preferred for MSA formality.
  • D.2 Detailed Results: In MSA-to-dialect outputs, fluency and coherence remain high, while dialect authenticity and residual MSA formality remain the main limitations.Table 7 reports these dimensions for vector steering with layer 21 and 30 steered tokens.
  • E AL-QASIDA Evaluation Details: AL-QASIDA assesses dialectal proficiency through monolingual generation, cross-lingual generation, and machine translation, but this study uses only monolingual generation.The monolingual task prompts in the target dialect and expects a response in kind.
  • E AL-QASIDA Evaluation Details: For each variety, 100 sentences are sampled from each corpus and wrapped in one of eight native-speaker-translated instruction templates.Template selection is described as part of the evaluation prompt construction.
  • E AL-QASIDA Evaluation Details: The AL-QASIDA prompt set is fully deterministic: template selection uses a seeded RNG, NADI-2023-TWT is excluded, and benchmark-seeded prompts are reused across experiments.NADI-2023-TWT is excluded because it is not publicly redistributable under X (Twitter) policy.
  • F Explicit Prompt Baseline: The explicit-prompt baseline prepends a target-dialect system message without few-shot examples or activation modifications, using deterministic decoding and a maximum of 128 new tokens.The user prompt remains unchanged, and TARGET DIALECT is replaced with the full dialect name, such as Egyptian Arabic or Moroccan Arabic.

G Human Evaluation Details

Human evaluation shows that vector steering generally improves dialect authenticity and reduces MSA formality while preserving fluency and coherence. Agreement is strongest for dialect-specific dimensions, supporting the LLM judge as a reliable proxy despite lower agreement on more subjective criteria.

  • LLM-as-a-judge agreement: LLM-to-human agreement is strongest for dialect authenticity and MSA formality, while fluency and coherence have lower exact agreement but above 74% within-one agreement.The within-one results indicate that disagreements on fluency and coherence are rarely larger than one point on the 1–5 scale.
  • LLM-as-a-judge agreement: The LLM judge is presented as a reliable proxy for human assessment, particularly for the main dialect-control dimensions.Fluency and coherence show weaker agreement because these dimensions are more subjective and less directly tied to the steering intervention.
  • Detailed human evaluation results: Vector steering consistently improves dialect authenticity and reduces MSA formality across nearly all settings, while largely preserving fluency and coherence.Neuron steering yields more modest and less consistent gains.
  • Inter-annotator agreement: κw = 0.760 and 86.1% within one point are reported for inter-annotator agreement on dialect authenticity, the strongest evaluated dimension.MSA formality also shows high agreement, with κw = 0.673 and 82.7% within one point.
  • LLM-as-a-Judge Prompt Template: The evaluation prompt requires an expert Arabic dialect evaluator to return only a JSON object with mandatory integer scores from 1 to 5 for four metrics.The required fields are dialect_authenticity, coherence, arabic_fluency, and msa_formality.
Loading 2607.03936v1…