PaperlayerView exact revision on arXiv

Source-linked AI summary

Multilinguality in Hybrid Attention LLMs

Lucas Bandarkar, Junlin Hu, Chenyuan Yang, Mohsen Fayyaz, Nanyun Peng

arXiv:2609.35378v1cs.CLcs.AI

TL;DR

Hybrid attention combines full attention with recurrent mechanisms to address long-context costs, but its impact on multilingual processing has not previously been studied. This work analyzes multilingual efficiency and layer-wise representations, then compares attention-layer orderings through distillation. It finds representational reorganization around full-attention layers and faster learning for alternative orderings, motivating the authors’ theory that multilingual models should start with full attention.

  • Problem

    How hybrid attention organizes multilingual processing is understudied, despite multilinguality’s importance and its ability to reveal representations shared across languages.

    Source: §1 · p. 1

  • Method

    The authors analyze multilingual sequence efficiency and cross-lingual representations in hybrid models, then distill models using alternative attention-layer placements.

    Source: §1 · p. 2, §6.2 · p. 8

  • Results

    All alternative layer placements learn faster than standard periodic, reaching its final cross-entropy value with 40% of the tokens; hybrid models also show reorganization around full-attention layers.

    Source: §1 · p. 2, §6.3 · p. 9

  • Takeaways & Limitations

    The authors conclude that multilingual LLMs should use hybrid attention for long sequences and propose starting with a full-attention decoder layer.

    Source: §7 · p. 10

    A mechanistic explanation of the observed representational reorganization remains open.

    Source: §7 · p. 10

Abstract

from arXiv · show

In response to the growing demand for long sequences in agentic and reasoning use cases, many state-of-the-art LLMs combine multiple variants of attention to mitigate the quadratic complexity of traditional softmax attention. These hybrid attention LLMs aim to balance the strengths and limitations of full attention and alternatives based on recurrence. This work presents a first study of how hybrid attention impacts the multilinguality of LLMs. Beyond the impact on long sequences in poorly tokenized languages, our study is motivated by the possibility that the inductive biases of the recurrent state alter linguistic processing. Our interpretability analysis confirms this, showing that cross-lingual representations in hybrid models develop in patterns tied to the ordering of recurrent and full-attention layers. Across diverse models, we notably observe a pronounced spike in cross-lingual alignment around the first full-attention layer. These findings lead us to question the conventional ordering of attention layers. In distillation experiments on multilingual data, all alternative layer orderings outperform the standard throughout training, learning up to 2.5X faster. These stark, replicable results prompt our theory that multilingual models would benefit from starting with a full-attention layer rather than recurrent layers.

1 INTRODUCTION

Hybrid attention addresses the growing cost of long contexts, but its effects on multilingual processing remain understudied. The paper examines multilingual efficiency, layer-wise representations, and attention ordering, finding a reorganization around the first full-attention layer and better distillation results for alternative layer arrangements.

1 INTRODUCTION

  • Full softmax attention has O(T^2) computational cost as sequence length T grows, motivating hybrid models that combine it with recurrent mechanisms scaling linearly in T.Hybrid attention is increasingly used in state-of-the-art open-weight models for long-sequence workloads.Source: §1 · p. 1
  • Recurrent layers maintain compressed evolving states, whereas full attention accesses earlier token representations, giving the mechanisms distinct inductive biases.The paper notes that how hybrid models organize these processes remains understudied.Source: §1 · p. 1
  • The paper studies multilinguality because cross-lingual comparisons can reveal whether processing reflects surface form or representations shared across languages.The authors report that no prior work had studied recurrent attention and hybridization’s impact on multilingual processing.Source: §1 · p. 1
  • Hybrid attention offers large efficiency gains on multilingual inputs, while its retrieval shortcomings lessen as contexts grow.The study separates accommodating longer sequences from retrieving their contents.Source: §1 · p. 2
  • Hybrid models show abrupt representational changes around interleaved full-attention layers, with pronounced reorganization around the first such layer.Full-attention and SWA-hybrid models instead show smoother development of language-abstract representations.Source: §1 · p. 2
  • All four alternative layer orderings outperform the conventional periodic layout on multilingual distillation data, and all five tested layouts begin with full attention.The authors propose that multilingual models should start with full softmax attention.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Prior work develops linear-complexity recurrent alternatives and combines them with full attention in hybrid LLMs. Multilingual interpretability research studies shared representations, but the role of attention in multilingual processing remains a distinct concern.

2.1 RESEARCH IN ATTENTION ALTERNATIVES

  • Without softmax, self-attention can be reduced through matrix associativity to a linear-complexity recurrence, motivating recurrent mechanisms such as selective state-space models and the Delta Rule.Gated DeltaNet and Kimi Delta Attention maintain updatable associative key-value memory.Source: §2.1 · p. 2

2.2 HYBRID ATTENTION LLMS

  • Hybrid LLMs commonly interleave recurrent layers, which track state efficiently, with full-attention layers, which preserve precise content-based retrieval.The work uses “full” for traditional softmax attention, “recurrent” for linear-complexity recurrent alternatives, and treats SWA separately as sparse local attention.Source: §2.2 · p. 2

2.3 MULTILINGUALITY IN LLMS

  • Prior multilingual studies find language-shared representations in middle layers, while attention provides access to previous tokens and is linked to linguistic processing and multilingual performance.The cited findings suggest attention plays a disproportionate role in model multilinguality.Source: §2.3 · p. 3

3 MODELS

The study compares four hybrid-attention models with close non-hybrid counterparts, alongside partially studied additional models. The authors caution that imperfect pairwise comparability and benchmark-varying transfer advantages motivate analysis beyond simple evaluation.

3 MODELS

  • The study examines four diverse hybrid-model pairs, with three using periodic interleaving and Granite-4.0-H placing full-attention layers at layers 4, 14, 24, and 34.Table 1 lists the studied models and closest non-hybrid counterparts; larger Ling-2.6-flash and SWA-hybrid Tiny Aya are also studied partially.Source: §3 · p. 3, Table 1 · p. 3
  • The recurrent variants differ in state updates: Ring-linear uses input-independent decay, Qwen3.5 uses gated delta-rule updates, Granite uses Mamba-2 additive updates, and OLMo-Hybrid uses a close Gated DeltaNet variant.These differences complicate treating the pairs as identical apart from hybridization.Source: §3 · p. 3
  • The model pairs do not enable flawless comparisons, and relative cross-lingual-transfer advantages vary by benchmark.OLMo’s explicit exclusion of multilingual training data makes it least useful for these purposes; the authors therefore extend analysis beyond simple evaluation.Source: §3 · pp. 3–4

4 MULTILINGUAL TOKENIZATION

Poor tokenization can make multilingual contexts much longer, increasing the relevance of hybrid attention’s memory and runtime advantages. On OneRULER retrieval tasks, the hybrid deficit narrows as context length grows, though it can remain at longer contexts.

4 MULTILINGUAL TOKENIZATION

  • Poor tokenization can inflate multilingual inputs: Qwen3.5 produces sequences 2.6× longer in Yoruba than English, and English-centric OLMo can expand some non-Latin-script contexts 10×.The authors connect token inflation to the importance of long-sequence efficiency.Source: §4 · p. 4
  • At 128K, Qwen3.5 completed 25/40 language-task evaluations versus one for Qwen3; Granite’s 64K evaluations took 13.5 hours versus 61.5.On a single A6000 GPU with 48GB VRAM, hybrid models encountered substantially fewer out-of-memory failures.Source: §4 · p. 4
  • The hybrid retrieval deficit narrows as context length grows in English and non-English inputs, while hybrid attention’s performance cost is smallest at long contexts.Granite-H trails Granite substantially on non-English inputs at the shortest context; the gap narrows but remains at 64K. The Qwen pair shows a similar trend with a smaller difference.Source: Table 2 · p. 4, §4 · p. 4
  • OneRULER adjusts each language’s haystack so context length stays fixed, leaving non-English languages with less underlying content at the same length.The authors hypothesized this would reduce the retrieval gap, but instead found the hybrid deficit initially larger outside English.Source: §4 · p. 4

5 VISUALIZING REPRESENTATIONS

SoftCKA reveals that hybrid models’ cross-lingual representations change abruptly around full-attention layers, especially the first, unlike the smoother development in homogeneous models. The analysis uses softly matched token representations and block contributions to examine these patterns.

5.1 CROSS-LINGUAL ALIGNMENT METRIC: SOFTCKA

  • SoftCKA compares parallel-sentence hidden states through softly matched token similarities, normalizing cross-language similarity by within-language quantities.It evaluates layer inputs and post-attention residual-stream states; the authors find it less noisy than alternatives, while smoothness alone cannot prove metric faithfulness.Source: §5.1 · pp. 4–5

5.2 OBSERVATIONAL FINDINGS

  • Cross-lingual alignment is lower in Granite-H, Qwen3.5, and Ring-linear than their counterparts in most examined languages, but the authors do not claim lower alignment is worse.Granite-H is particularly less aligned in middle layers; the comparison is mixed for OLMo models.Source: §5.2 · p. 5, Figure 2 · p. 5
  • Across four model pairs, hybrid representations change abruptly around full-attention blocks and become more aligned before each block, unlike the smoother development in homogeneous models.Figure 3 visualizes the jagged hybrid curves against smoother non-hybrid curves.Source: §5.2 · p. 6, Figure 3 · p. 6
  • The largest cross-lingual spike occurs at the first full-attention layer across all hybrid models in four pairs and Ling-2.6-flash.Source: §5.2 · p. 6
  • Attention-block contribution alignment differs by model: it is high at the first full-attention block for Qwen3.5, OLMo-Hybrid, and Granite-H, but spikes negatively for Ring and Ling-2.6-flash.The authors call this inconsistent pattern an “event”; preceding MLP and attention blocks progressively prepare inputs for it.Source: §5.2 · p. 6, Figure 4 · p. 6
  • Unlike recurrent-attention hybrids, SWA-hybrid models show no comparable abrupt reorganization at full-attention layers and no noticeable pattern difference from full-attention models.The authors attribute this contrast to SWA’s inductive biases not differing enough to prompt the observed reorganization.Source: §5.2 · p. 7
  • Token attribution shows full-attention layers in hybrid models devote a larger proportion of attention to recent tokens than full-attention layers in non-hybrid models, consistently across languages.The authors note that attention-matrix attribution may not be entirely faithful and suggest these layers may handle local tasks.Source: §5.2 · p. 7
  • The authors hypothesize that a different layer ordering may better facilitate abstraction of multilingual inputs into language-independent representations.Source: §5.2 · p. 7

6 HYBRID DISTILLATION EXPERIMENTS

The experiments compare alternative full-attention layer placements under multilingual distillation, motivated by limited prior guidance on ordering. All tested alternatives that begin with full attention learn faster than standard periodic, with gains across languages and replication in two new conditions.

6.1 RELATED WORK ON FULL ATTENTION PLACEMENT

  • Prior work offers limited and mixed guidance on full-attention ordering, and multilingual considerations were unlikely to have informed those choices.Proposed rationales include regular exact access to earlier tokens, simpler KV-cache management, and positional-embedding considerations.Source: §6.1 · p. 7

6.2 EXPERIMENTAL SETUP

  • The study fixes full attention at 25% of layers and varies placement across standard periodic, reverse periodic, sandwich, interleaved sandwich, and endpoint spread.Standard periodic places full attention at the end of each four-layer block; all four alternatives use full attention in layer 0.Source: §6.2 · p. 8, Figure 5 · p. 8

6.3 RESULTS

  • All four alternative placements reached standard periodic’s final cross-entropy value with about 40% of training tokens, and the improvement replicated with resampled Qwen data and Granite.Reverse periodic reached the threshold at 34.5% of training for Qwen and 42.5% for Granite.Source: §6.3 · p. 9, Figure 6 · p. 9
  • Reverse periodic lowered perplexity in 23 of 26 languages with Qwen3-4B in both runs and 25 with Granite, while the largest gains were 3–13% in five languages.The median language improved by about 1%; the exceptions’ perplexity rose by less than 0.7%.Source: §6.3 · p. 9
  • Across alternatives with different placements, the shared feature is a full-attention first layer, while the largest differences occur for non-Latin scripts and may relate to detokenization.The authors note that their compute budget allowed replication of only reverse periodic; its improvement replicated in both new conditions.Source: §6.3 · pp. 9–10

7 CONCLUSION AND OPEN QUESTIONS

The paper’s first analysis of hybrid attention and multilinguality yields two developer-facing conclusions, while leaving the mechanism behind representational reorganization unresolved.

7 CONCLUSION AND OPEN QUESTIONS

  • The authors recommend hybrid attention for multilingual LLMs because it offers large efficiency gains with minimal performance cost on long multilingual sequences.They also suggest that multilingual learning improves when the first decoder layer uses full attention, but qualify this as a theory prompted by resource-limited experiments.Source: §7 · p. 10
  • The experiments cover distillation on 1B tokens, small LLMs, and one recurrent architecture, limiting the scope of the ordering conclusion.The authors nevertheless state that these conditions prompt the theory that multilingual models should start with full attention.Source: §7 · p. 10
  • The study identifies representational reorganization but leaves its mechanism unexplained, concluding only that block inductive biases matter and full-attention layers handle some multilingual-relevant task.The specific task assigned to full attention remains unspecified.Source: §7 · p. 10

A FURTHER DETAILS ABOUT MODELS

The appendix provides additional architectural details and flags positional embeddings as a possible influence on multilingual representation development.

A FURTHER DETAILS ABOUT MODELS

  • The appendix supplies additional architecture details, including which positional embeddings are used in full-attention layers.The caption identifies “Position” as the positional embeddings used in full-attention layers.Source: Table A.1 · p. 22
  • The appendix includes a table of architectural components.The supplied caption does not specify the components listed.Source: Table A.2 · p. 22
  • The models use differing positional-embedding formulas across attention blocks, which the authors say may affect multilingual representation progression but do not investigate.The authors describe this as outside the focus of the work.Source: A FURTHER DETAILS ABOUT MODELS · p. 22

B MULTILINGUAL TASK EVALUATIONS

The appendix reports inconclusive multilingual benchmark comparisons and documents the data, hyperparameter, distillation, and training procedures used in the experiments.

B MULTILINGUAL TASK EVALUATIONS

  • Benchmark comparisons across four hybrid/non-hybrid model pairs were inconclusive because results were noisy and inconsistent, and comparability was limited.The evaluations covered MGSM, MMLU ProX, Belebele, and Global-PIQA, with performance considered relative to English as a measure of cross-lingual transfer.Source: B MULTILINGUAL TASK EVALUATIONS · p. 22

C.1 DATA

  • The training recipe spans 27 languages from seven families, assigning English a fixed 30% of the 1B-token budget as an anchor and splitting the remaining 70% across three resource-based sampling tiers.Per-language shares across the tiers follow a 3.5:2:1 ratio.Source: §C.1 · p. 22
  • Validation and test sets use fixed document orders across runs, while the data seed changes training documents and language-interleaving order; replicate runs hold initialization fixed.The second runs differ in training sample, allowing the authors to estimate variability due to data rather than initialization.Source: §C.1 · pp. 23–24

C.2 HYPERPARAMETER

  • During distillation, the tied input embedding and language-model head are untied, with the head trained as a separate embedding copy and the untied student evaluated.The procedure follows a workaround suggested for smaller tied models; the head is not retied afterward.Source: §C.2 · p. 24

C.3 ALGORITHM OF DISTILLATION

  • The prescribed full-attention set determines layer ordering: other layers are converted to Gated DeltaNet, then the whole student is trained in one stage using forward DKL to a frozen teacher.The protocol omits hidden-state alignment and data-driven layer selection; each run makes one pass over 1B tokens packed into 1,024-token sequences.Source: §C.3 · p. 25, §C.4 · p. 26

C.4 TRAINING

  • Converted recurrent layers inherit teacher attention weights where possible, while retained attention layers, MLPs, norms, and embeddings are copied unchanged.The training curves show per-step DKL and its exponential moving average for the depicted runs; the supplied captions do not establish comparative outcomes.Source: Figure C.1 · p. 26, Figure C.2 · p. 26, §C.4 · p. 27

C.5 METRICS AND RESULTS

The appendix defines held-out evaluation metrics for distillation and details SoftCKA’s construction and interpretation, including its limits as a measure of cross-language similarity.

C.5 METRICS AND RESULTS

  • Final metrics use held-out test documents after 1B distillation tokens, while validation curves are evaluated every 10^7 training tokens; both LLM and DKL are averaged per language.Aggregate metrics average over 26 languages, including English and excluding Mon.Source: §C.5 · p. 27
  • The cross-entropy gap is the mean per-token student-minus-teacher LLM on held-out text, averaged over 26 languages, and corresponds to the geometric mean of student-to-teacher perplexity ratios.A gap of 0.05 means the student’s perplexity is about 5% above the teacher’s on average.Source: §C.5 · p. 27
  • Relative DKL compares each placement with the standard-periodic student from the same teacher at the same step; values below 1 indicate lower DKL than that baseline.The comparison uses the geometric mean of per-language DKL ratios.Source: §C.5 · p. 27
  • Mon is excluded from validation and test averages because its roughly 340 remaining training documents are repeated, creating a risk of reserved-document leakage.The corpus has only about 2,300 Mon documents, so 2,000 are reserved for validation and test.Source: §C.5 · p. 27

D SOFTCKA DETAILS

  • Corpus SoftCKA sums sentence-pair quantities before normalization rather than averaging sentence-level scores, so contributions depend on sequence lengths and centered kernel values.With comparable token counts and similar mean squared centered entries, contributions grow approximately quadratically with sequence length.Source: D SOFTCKA DETAILS · p. 31
  • SoftCKA compares centered within- and cross-language token-similarity kernels, but does not recover or verify semantic token correspondences.Its cross-language kernel compares every token in one sentence with every token in its translation without requiring equal lengths or token correspondences; the resulting entries are similarities, not alignment probabilities.Source: D SOFTCKA DETAILS · pp. 30–31

E SOFTCKA OF BLOCK DELTAS

The section examines SoftCKA of block-level residual-stream contributions alongside layer-entry hidden states, finding progressive changes before a pronounced event at the first full-attention layer and a large similarity drop afterward.

E SOFTCKA OF BLOCK DELTAS

  • The primary metric is SoftCKA on the hidden state entering each layer, attn_in.Source: E SOFTCKA OF BLOCK DELTAS · p. 31
  • The block-delta variant measures SoftCKA on each block’s residual-stream contribution, treating the block as a black box comparable across attention and MLP blocks.Source: E SOFTCKA OF BLOCK DELTAS · p. 31
  • Block-delta SoftCKA shows progressive changes in Qwen3.5 before a pronounced event at the first full-attention layer, followed by a massive similarity drop.The visualization also shows the sequential progress through attention and MLP blocks.Source: Figure E.1 · p. 31, E SOFTCKA OF BLOCK DELTAS · p. 31

F COMPARISON TO SWA HYBRIDS

In hybrid Qwen and Ring models, cross-lingual routing divergence begins decreasing only after the first full-attention layer, unlike in their non-hybrid counterparts. The section also reports no evidence that positional embeddings’ absence in SWA layers significantly impacts cross-lingual representations.

F COMPARISON TO SWA HYBRIDS

  • Figure F.1 plots SoftCKA for Tiny Aya Global, an SWA-hybrid model, with full-attention layers marked by red dotted lines.Source: Figure F.1 · p. 32
  • The authors find no evidence that omitting positional embeddings outside full attention significantly impacts cross-lingual representations.Source: F COMPARISON TO SWA HYBRIDS · p. 32

G MOE ROUTING ALIGNMENT

  • The routing-divergence metric compares sequence-level MoE routing across parallel-language sentences using Jensen-Shannon divergence; lower values indicate more similar routing.Router weights are mean-pooled across tokens, and corpus-wide averages are reported by layer.Source: G MOE ROUTING ALIGNMENT · p. 32
  • In Qwen3.5-35B-A3B and Ring-mini-linear-2.0, routing divergence decreases only after the first full-attention layer; in non-hybrid Qwen3-30B-A3B and Ring-mini-2.0, it decreases immediately.Source: G MOE ROUTING ALIGNMENT · p. 32, Figure G.1 · p. 32, Figure G.2 · p. 33

Related papers

Loading 2609.35378v1…