PaperlayerView exact revision on arXiv

Source-linked AI summary

A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models

Amrit Gopinath, Raghul, Durairaj Thenmozhi

arXiv:2608.15102v1cs.CL

TL;DR

The paper asks whether bilingual MoE models organize expert routing around linguistic categories and how curriculum shapes that organization. It analyzes probe-token routing and finds category-sensitive specialization in intermediate layers, while curriculum makes specialization more language-balanced and stable across seeds.

  • Problem

    The study asks whether bilingual MoE models develop routing patterns corresponding to lexical, grammatical, and syntactic categories.

  • Method

    The authors analyze expert-routing distributions for lexical, grammatical, and syntactic probe tokens under staged bilingual exposure and a no-curriculum baseline.

  • Results

    Bilingual MoE routing shows category-dependent organization concentrated in intermediate layers, with curriculum producing a stable, language-balanced profile across seeds.

  • Takeaways & Limitations

    Curriculum does not uniformly increase specialization; its seed-robust effect is reducing single-language dominance and making crosslingual organization predictable.

  • Takeaways & Limitations

    The findings use one English–German language pair and one primary sparse architecture, so generalization to other languages, architectures, and curricula remains open.

Abstract

from arXiv · show

We investigate whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition. Inspired by the Declarative-Procedural framework, we analyze lexical, grammatical, and syntactic processing in a decoder-only English-German MoE Transformer trained under sequential language exposure. We construct a probe-based validation set and extract token-level routing distributions to quantify category-dependent specialisation using mutual information, routing entropy, and Jensen-Shannon distance. The curriculum-trained model exhibits a peak mutual information of 0.1148 at layer 5, indicating category-dependent differences in routing distributions across linguistic categories. Surprisingly, a no-curriculum baseline trained on mixed English-German data shows stronger aggregate specialisation, reaching a peak mutual information of 0.2599 at the same layer. These results suggest that interpretable linguistic organization emerges within MoE routing patterns even without sequential language exposure. A replication at a second training seed shows that the no-curriculum condition's specialisation concentrates on a single language whose identity is seed-dependent, whereas the curriculum consistently yields a stable, language-balanced routing profile; rather than uniformly increasing specialisation, staged bilingual exposure reduces single-language dominance. The official Github repository: https://github.com/Amrit828/DP-Theory-MOE-Interpretability-Research

1 Introduction

This section presents a routing-level framework for testing whether bilingual MoE models organize computation according to lexical, grammatical, and syntactic categories. It uses probe-token expert distributions and compares staged English–German exposure with a no-curriculum baseline, finding measurable, language-sensitive specialization concentrated in intermediate layers.

  • Motivation: MoE models route tokens across subsets of expert networks, enabling analysis of how linguistic tokens are distributed among experts.The routing mechanism may reveal organized expert allocation inside the model.
  • Research question: The study tests whether bilingual MoE routing corresponds to lexical knowledge, grammatical processing, and syntactic structure.These categories are operationalized through lexical, grammatical, and syntactic probes selected from sentences.
  • Experimental design: The model is trained with a staged English–German curriculum and compared against an unstructured no-curriculum setting.The curriculum begins with English and gradually transitions to German, while the baseline introduces both languages without gradual cumulative exposure.
  • Analysis framework: The analysis uses held-out bilingual probes and measures expert-probability distributions with mutual information, entropy, Jensen–Shannon divergence, and permutation testing.This moves beyond final-task behavior to examine internal routing at each MoE layer.
  • Findings: MoE routing contains measurable information about linguistic category membership, with different probe types routed differently and the clearest specialization in intermediate layers.Later layers route more diffusely and show reduced mutual information, while the overall structure is language-sensitive and quantifiable.

2 Related Work

The section frames the study through Declarative–Procedural theory, transformer linguistic structure, MoE routing, and bilingual curriculum learning. It motivates testing whether routing reflects linguistic categories and whether staged language exposure changes specialization.

  • Declarative–Procedural theory: Declarative–Procedural theory distinguishes stored word knowledge from rule-based grammar and compositional structure, serving here only as a linguistic lens rather than a model of human memory.The study applies the framework to interpret linguistic categories without equating MoE routers with human memory systems.
  • Declarative and procedural knowledge in LLMs: Rather than evaluating only model outputs, the study examines whether a sparse bilingual model allocates computation differently across lexical, grammatical, and syntactic probe tokens.This positions the work as an internal analysis of computation allocation in language models.
  • Transformer linguistic structure: Prior transformer research suggests that linguistic information varies by layer, motivating layer-wise analysis of whether middle routed layers show stronger category-sensitive expert allocation.Intermediate layers are described as often showing clearer syntactic and semantic structure than very early or very late layers.
  • Mixture-of-Experts routing: MoE routers direct tokens to subsets of experts, whose specialization can reflect token statistics, domains, routing design, and balancing constraints.This study extends that line of work to bilingual linguistic probe categories and curriculum effects on routing specialization.
  • Curriculum learning and bilingual exposure: Bilingual curriculum learning compares English-first-then-German exposure with mixed English–German training from the start to test whether language order affects routing structure.The comparison isolates staged exposure as a potential influence on how routing specialization develops across languages.

3 Routing Analysis Framework

The routing analysis tests whether expert-routing distributions depend on lexical, grammatical, and syntactic categories across four routed layers. It quantifies category association, expert-use concentration, and pairwise category separation using mutual information, entropy, and Jensen–Shannon distance.

  • Routing setup: The model routes tokens at layers L = {1, 3, 5, 7} through N = 8 experts, with categories C = {lexical, grammatical, syntactic}.For each probe token, the router produces a probability distribution over the eight experts.
  • Routing setup: Probe-token routing vectors are extracted during inference and analyzed independently at each routed layer.Probe tokens are parser-annotated in each sentence before routing statistics are computed.
  • Hypothesis testing: The analysis tests category-independent routing, H0: MI(E; C) = 0, against category-dependent routing, H1: MI(E; C) > 0.A random-routing control produces MI ≈0.002, demonstrating that the null is empirically realizable.
  • Statistical measures: Mutual information measures the association between expert probability mass and linguistic category, with higher values indicating stronger association.Robustness is assessed using a sentence-level permutation test at p < 0.01.
  • Statistical measures: Entropy measures whether expert usage is diffuse or concentrated, while Jensen–Shannon distance compares pairwise routing separations between linguistic categories.The reported Jensen–Shannon distance is the square root of JSD, making the three pairwise category separations directly comparable.

4 Methodology

The methodology builds category-specific bilingual probe datasets, trains a compact 8-layer MoE model under sequential and mixed-language exposure, and evaluates routing on a fixed minimal-pairs validation set.

  • Data construction: English data comes from FineWeb-Edu and German data from mC4, with Stanza assigning words to lexical, grammatical, or syntactic categories.The category taxonomy is provided in Appendix A.
  • Data construction: Each sentence-category pair receives one probe token, and sentences with multiple categories are duplicated across the corresponding category-specific datasets.
  • Limitations: Lexical probe extraction stopped before reaching the target size because of computational constraints, but analyses use a fixed held-out validation set.
  • Validation set: 12,000 validation samples form minimal pairs, with 6,000 samples per language and 2,000 per category per language.Each pair contains a valid S+ sentence and an invalid S− sentence.
  • Model and training: The model is a custom 8-layer MoE Transformer with feed-forward dimension 2048, embedding size 512, and 8 attention heads per layer.MoE layers occupy alternate transformer layers: 1, 3, 5, and 7.
  • Model and training: The forward curriculum trains on English first, then gradually introduces German, whereas the no-curriculum baseline mixes English and German from the start with fixed 80% English and 20% German proportions.Both conditions receive the same total number of training samples; the primary runs use top-k routing with k = 3 and a load-balancing coefficient of 0.01.

5 Results and Analysis

The analyses identify statistically significant, category-dependent expert routing across layers, with grammatical processing more concentrated and syntactic processing more distributed. Specialisation appears under both curricula, but no-curriculum training produces stronger pooled effects through seed-dependent single-language dominance, whereas staged exposure yields more balanced routing.

  • Mutual-information analysis: 0.1148 is the curriculum model’s peak mutual information at layer 5, indicating the strongest category-dependent routing association there.MI varies across layers, so routing specialisation is not uniform throughout the network.
  • Statistical validation: p < 0.001 across all routed layers after 1,000 sentence-level permutations supports category-dependent routing beyond shuffled-label explanations.Observed MI exceeded corresponding layer-wise null distributions.
  • Expert allocation and entropy: Grammatical routing is more concentrated, whereas syntactic routing is more distributed across experts, with grammatical probes showing the lowest and syntactic probes the highest entropy.Grammatical probes preferentially use E1 and E3, while syntactic probes more heavily use E3–E5, without single-expert collapse.
  • Curriculum comparison: 0.2599 is the no-curriculum model’s layer-5 MI peak, exceeding the forward curriculum peak and showing that specialisation develops without sequential exposure.The no-curriculum condition also reaches 0.1945 at layer 1.
  • Seed replication: DE layer-5 MI 0.213 vs. EN 0.078 at the second seed shows no-curriculum specialisation can switch languages, while staged exposure yields a stable, language-balanced profile.The replication reports higher pooled no-curriculum MI at every layer but seed-dependent language composition.

6 Discussion

Both training conditions develop category-sensitive routing, with the strongest routing-category relationship in intermediate layers, especially layer 5. Although pooled statistics favor the no-curriculum model, per-language analysis shows seed-dependent single-language dominance, while curriculum produces stable, language-balanced routing.

  • Routing specialisation: Category-sensitive routing emerges in both training conditions, so curriculum learning is not necessary for linguistic category membership to shape routing distributions.The discussion identifies how curriculum changes routing across languages and layers as the remaining question.
  • Routing specialisation: The strongest routing-category relationship occurs in intermediate layers, particularly layer 5.This pattern is consistent with prior work associating intermediate transformer layers with clearer linguistic information.
  • Cross-condition comparison: Pooled English-German statistics make the no-curriculum model appear stronger, but per-language analysis reveals a seed-dependent single-language phenomenon.The language benefiting from this pooled signal is determined by the random seed.
  • Cross-condition comparison: The forward curriculum yields a language-balanced profile that remains stable across seeds rather than simply boosting or suppressing specialisation.Sequential L1–L2 exposure therefore has a seed-robust distributional effect across languages.
  • Behavioural interpretation: Behavioural accuracy should be interpreted within the study design, which prioritizes comparable internal routing patterns over high benchmark accuracy.The small model and controlled minimal-pair validation set mainly establish that both models are functional enough for meaningful routing analysis.

7 Conclusion

Bilingual MoE routing develops measurable category-dependent organization across lexical, grammatical, and syntactic probes, strongest in intermediate routed layers. Across two seeds, curriculum learning reduces single-language dominance rather than uniformly increasing specialisation, while no-curriculum routing concentrates on a seed-dependent language.

  • Bilingual MoE routing develops category-dependent organization across lexical, grammatical, and syntactic probe families.
  • Routing specialisation is strongest in intermediate routed layers and remains robust on held-out validation data across two training seeds.
  • Curriculum learning does not uniformly increase specialisation; its seed-robust effect is reducing single-language dominance rather than favouring a specific language.
  • Without a curriculum, routing specialisation concentrates on a single language whose identity varies with the training seed.

Limitations … Computing Infrastructure and Budget

The paper’s limitations concern generalizability, probe validation, confounding, and causal interpretation, while the methods define operational linguistic probes and document the curriculum, model configuration, and computing setup.

  • Limitations: The findings use one English–German pair and primary sparse architecture, leaving generalization across languages, architectures, and curricula open.The evaluation set also lacks manual human validation, and matched-domain corpora are proposed for future work.
  • Limitations: The forward curriculum omits a reverse schedule because naive reversal would confound language-order effects with cumulative exposure differences.A fair reverse-curriculum comparison would preserve overall language proportions while reversing introduction order.
  • Limitations: Routing-category associations remain non-zero after language and coarse frequency controls, but interactions among category, token identity, frequency, morphology, and part of speech remain incompletely separated.A stricter frequency-balanced lexical control was not included because skewed curriculum-generated distributions made matched subsets difficult to construct.
  • Limitations: The study does not report broad architecture, language-pair, or hyperparameter sweeps, and lacks expert interventions needed for strong causal claims about individual experts.Future work specifically proposes expert masking or routing interventions.
  • A Probe Token Taxonomy: Probe families operationalize broad English, German, grammatical, and syntactic contrasts using parser-based labels rather than perfectly discrete theoretical classes.German lexical probes include exceptional forms, irregular plurals, and gender-marked determiners, while syntactic probes use dependency-relation labels.
  • German grammatical probes: The probe methodology includes German weak past-tense forms and dependency relations such as advcl, ccomp, xcomp, acl, and csubj.These labels test broad linguistically motivated contrasts under a consistent parser-based annotation scheme.
  • Category sampling: Lexical, grammatical, and syntactic sentences are sampled uniformly within each epoch and then randomly shuffled.This sampling procedure is described for category-balanced training batches.
  • Computing Infrastructure and Budget: The model has 8 layers, 512-dimensional embeddings, 8 attention heads, MoE layers 1, 3, 5, and 7, 8 experts, and top-k 3 routing.Training used AdamW, peak learning rate 3 × 10^-4, 500 warmup steps, batch size 8, gradient clipping 1.0, and 16 epochs on NVIDIA T4 and RTX A6000 GPUs; the model contains 86.4M parameters.

D Additional Routing Analysis Details … Parsing and Validation Tools

The analysis uses permutation testing to assess routing-category dependence and constructs validated bilingual probe examples through constrained generation, parser checks, and deterministic minimal-pair edits. Stanza validates English and German probe annotations and subtype constraints before analysis.

  • Permutation testing: 1,000 sentence-level category permutations per layer produce empirical p-values by measuring how often permuted I(R; C) reaches or exceeds Iobserved.This tests whether observed routing-category dependence exceeds expectations under category-independent routing.
  • E OpenAI Generation Details, Validation Parsing, and Prompt Examples: GPT-5.1 generated held-out validation sentences through the OpenAI Responses API under configurable model and request settings.The generation script used max_output_tokens=420, reasoning.effort="none", store=false, a 40-second timeout, and no retries.
  • Parsing and Validation Tools: Generated candidates were parsed with Stanza to verify unique probe occurrence, required subtype inventory, and subtype-specific linguistic constraints.Checks included finite past-tense verbs, plural nouns, auxiliaries, and subordinating conjunctions, depending on subtype.
  • E OpenAI Generation Details, Validation Parsing, and Prompt Examples: S− was deterministically derived from each validated S+ by editing or deleting only the probe, yielding controlled minimal-pair contrasts.The transformation depended on subtype, including irregular-to-regularized and regular-past-to-lemma changes in English.
  • E OpenAI Generation Details, Validation Parsing, and Prompt Examples: English irregular-verb prompts required exactly 10 sentences, 6–12 words each, with marked main-verb probes in correct simple-past forms.Templates prioritized listed lemmas such as become, awake, and draw while prohibiting auxiliaries, participles, and extra formatting.
  • E OpenAI Generation Details, Validation Parsing, and Prompt Examples: German subordinating-conjunction prompts required exactly 10 grammatical sentences using marked probes as true SCONJ elements in subordinate clauses.The template specified 6–12 words per sentence, listed conjunctions including weil and obwohl, and excluded coordinators, adverbs, and discourse markers.
  • Parsing and Validation Tools: English and German Stanza pipelines used tokenize, pos, lemma, and depparse to identify probes, verify subtype constraints, and validate held-out examples.These parser outputs supported both probe annotation and validation of generated data.

F Dataset Diversity Statistics

The diagnostic set was evaluated for dataset health and template collapse using diversity statistics. All reported probe sets maintained 100% sentence-pair uniqueness.

  • Diversity Statistics: 100% sentence-pair uniqueness was maintained across all reported probe sets.This statistic was provided to address concerns about dataset health and template collapse.

G Ablation Controls … G.3 Routing Specialisation Dynamics During Training

Ablation and supplementary analyses show that routing specialisation depends on adaptive routing, balancing strength, routing breadth, and training dynamics. Seed, frequency-binned, and epoch-wise analyses further indicate persistent category sensitivity, curriculum-related stability, and changing expert utilisation over training.

  • G Ablation Controls: Random routing collapses routing-category mutual information toward near-zero values, showing that specialisation does not arise from architectural sparsity alone.The condition uniformly samples expert assignments while preserving the underlying expert parameters.
  • G Ablation Controls: Frozen routing preserves non-trivial routing-category dependence, although specialisation is substantially weaker than in the fully trainable model.Token-representation changes can interact with fixed routing boundaries to produce expert differentiation.
  • G Ablation Controls: α = 0.005 markedly increases routing-category dependence but also produces increasingly skewed expert utilisation.The result indicates a trade-off between stronger expert partitioning and balanced expert participation.
  • G Ablation Controls: top-k = 2 produces a similar but slightly lower pooled routing-category dependence, whereas top-k = 4 yields substantially stronger association, especially for English probes.Broader routing coincides with stronger category-conditioned separation in these runs.
  • G.1 Seed Replication Detail: Across both seeds, no-curriculum routing has higher pooled layer-5 structure than forward curriculum, while the curriculum shows smaller cross-language asymmetry.The language dominating no-curriculum routing changes with seed, unlike the pooled ordering and curriculum asymmetry.
  • G.2 Frequency-Binned Lexical Analysis: Mutual information remains non-zero across all frequency bins, with the strongest routing-category association appearing for rare rather than most frequent lexical items.The lexical routing effect therefore cannot be explained solely by the highest-frequency lexical forms.
  • G.2 Frequency-Binned Lexical Analysis: Conditional mutual information remains clearly non-zero and permutation-significant at all routed layers in both conditions, indicating category-sensitive routing beyond language separation.The reported significance is p = 0.000999.
  • G.3 Routing Specialisation Dynamics During Training: Category-dependent routing emerges rapidly, remains layer-ordered with layer 5 highest, declines after epoch 8, and later stabilises as routing entropy increases.The entropy increase indicates progressively more distributed expert utilisation while category-dependent behaviour remains present.
Loading 2608.15102v1…