Hybrid Architectures for Language Models: Systematic Analysis and Design Insights

1 Introduction

• Transformer self-attention has quadratic sequence-length complexity and substantial inference memory costs, motivating more efficient primitives for long contexts.State-space models compress prior context into a finite-dimensional state; Mamba is described as competitive with Transformers on language modeling while accelerating training.Source: §1 · p. 2
• Inter-layer hybrids sequentially mix Transformer and Mamba blocks, whereas intra-layer hybrids fuse the primitives in parallel within individual layers.Intra-layer designs use head-wise or sequence-wise splits.Source: §1 · p. 2
• Hybrid architectures address a literature gap in systematic comparisons by evaluating inter-layer and intra-layer designs across quality, long-context retrieval, scaling, training, and inference.The study also examines Transformer, Mamba, and striped global/sliding-window attention models.Source: §1 · p. 2
• Both hybrid types outperform homogeneous architectures by up to 2.9% in few-shot accuracy, and hybrid architectures match or outperform Transformers on downstream tasks.The evaluation covers models up to 3B parameters and 32K context; hybrids also outperform widely adopted SWA models.Source: §1 · p. 2
• All hybrid architectures achieve robust, superior long-context retrieval, while baselines show poor in-context retrieval and length generalization.The paper connects these results to the computational characteristics of the primitives.Source: §1 · p. 2
• Hybrid models are compatible with MoE and have a compute-scaling slope between Transformer and Mamba.The paper also evaluates choices such as block ratios and primitive ordering.Source: §1 · p. 3
• Hybrid models achieve faster end-to-end training, lower peak memory, and higher inference throughput with smaller cache sizes by leveraging Mamba’s linear sequence complexity.The cited comparisons include Figures 1b–3b.Source: §1 · p. 3

2 Background

• Transformer blocks combine multi-head attention with a feed-forward network; attention heads capture diverse dependencies through learned projections of the input.Head outputs are concatenated and transformed, while grouped-query or multi-query attention shares key and value heads across query-head groups.Source: §2 · p. 3
• Mamba replaces attention with a state-space layer whose input-dependent parameters let it control context retention and input integration for each token.Mamba 2’s state-space duality reformulates SSM computation as matrix multiplications optimized for modern hardware; its blocks are followed by FFN layers, optionally replaced with MoE layers.Source: §2 · pp. 3–4
• At 1B parameters and 8K context, Transformer uses about 18% more FLOPs per sample than Mamba, while Mamba’s cache is 95% smaller.Mamba uses 2.5× more parameters per block (25M vs. 10M); cache sizes are 256 MiB for Mamba and 13.4 MiB for Transformer.Source: §2 · p. 4

3.1 Inter-layer Hybrid Model

• Inter-layer hybrids alternate attention and linear sequence-modeling layers; the study varies Transformer:Mamba block ratios and Transformer placement across ratios and model sizes.Source: §3.1 · pp. 4–5

3.2 Intra-layer Hybrid Model

• Intra-layer hybrids fuse softmax and linear attention in parallel, using separate head groups for Transformer and Mamba primitives.Source: §3.2 · p. 5
• The intra-layer study compares normalization, scaling, fusion operations, output projections, and value-state sharing at 350M and 1B scales.Source: §3.2 · p. 5
• It varies Transformer and Mamba dimension allocations, examining quality and the potential efficiency benefit of reducing Transformer dimensions under expert parallelism.Source: §3.2 · p. 5
• It also tests intra-hybrid-to-Mamba block ratios and intra-hybrid block placement to compare quality-efficiency trade-offs with inter-layer hybrids.Source: §3.2 · p. 5

• Under equal data and compute constraints, Table 2 reports better hybrid quality; Table 3 reports stable training and hybrid gains on summarization and QA.Source: Table 2 · p. 6, Table 3 · p. 6

4.1 Main Results on Quality

• Under matched FLOP budgets, hybrids improve few-shot accuracy by 2.9% and reduce NLL by 0.04 versus homogeneous models, with gains consistent across ratios and scales.The controlled comparisons include SWA; the authors also report that hybrid models outperform homogeneous architectures under the same data budget.Source: §4.1 · p. 6
• Hybrids consistently match or outperform Transformers downstream, while SWA and Mamba show retrieval-intensive summarization degradation despite competitive language-modeling performance.The latter models remain competitive on some QA benchmarks, but their overall downstream performance is inconsistent.Source: §4.1 · p. 7

4.2 Main Results on Efficiency

• Hybrid models achieve faster end-to-end training and a better inference throughput–quality frontier, with high throughput and sub-quadratic memory growth across context lengths.Intra-layer hybrids further outperform inter-layer hybrids even when executed sequentially; Mamba’s linear complexity and constant cache size support generation efficiency.Source: §4.2 · p. 7

4.3 Long-context Capability Evaluation

• With a 1:5 block ratio, hybrids extrapolate beyond the 8K-token pretraining context; inter- and intra-layer hybrids retain strong retrieval to about 1.5× pretraining length.Retrieval accuracy declines in the middle of extrapolated contexts, while NoPE is reported to further address Transformer length-generalization limitations.Source: §4.3 · p. 8

4.4 Scaling Analysis

• MoE integration reduces NLL by 0.08 and raises few-shot accuracy by 4 points across architectures; hybrid compute-optimal scaling lies between Transformer and Mamba, with intra-layer hybrids slightly more data-hungry.MoE remains compatible because hybridization is applied to attention while MoE is integrated into the FFN; Mamba favors larger models and less data, whereas Transformers favor a token-to-parameter ratio around 20.Source: §4.4 · p. 9

4.5 Ablation Studies for Inter-layer Hybridization

• For inter-layer hybrids, a 1:1 block ratio gives the best quality, while about 1:5 balances quality and efficiency; placing Transformer blocks early can cause substantial degradation.At a 1:12 ratio, middle placement performs best; the authors hypothesize early Transformer blocks’ uniform attention clashes with Mamba’s local inductive bias.Source: §4.5 · pp. 9–10

4.6 Ablation Studies for Intra-layer Hybridization

• The intra-layer design uses normalization and either subtractive or concatenation fusion; these variants outperform prior intra-hybrid designs and same-primitive differential architectures in quality and efficiency.Normalization addresses scale differences between modules, making additional scaling factors unnecessary.Source: §4.6 · p. 10
• For intra-layer hybrids, larger Transformer dimensions improve quality but reduce efficiency, while 1:1 dimension allocation balances the two; evenly scattered blocks outperform clustered or sandwich placement.More Transformer-containing blocks improve quality, but retaining more Mamba blocks remains a practical efficiency choice. Learned routing favors Mamba early and Transformer in the middle, with token-level specialization at local sub-word dependencies versus critical syntactic boundaries.Source: §4.6 · pp. 10–11

5 Distinctions from Prior Works

• Prior hybrid-model studies often present final architectures without fully disclosing the recipes or systematic processes behind their design choices.The rationale for those choices remains underexplored.Source: §5 · p. 11
• Evidence on inter-layer block ratios is sparse, while intra-layer design variants have not been exhaustively tested.Prior work often relies on verbal justifications or selective ablations.Source: §5 · p. 11
• A rigorous comparison of model quality and efficiency between inter-layer and intra-layer hybrids had been absent.Source: §5 · p. 11
• Systematic ablations provide design insights, identify choices for each hybridization strategy, and compare trade-offs to pinpoint robust configurations.Source: §5 · p. 11

6 Conclusion and Future Work

• Hybrid models consistently outperform homogeneous architectures across multiple quality metrics, with intra-layer hybrids showing the strongest results and faster training and inference than widely adopted sliding-window attention models.Source: §6 · p. 12
• The study is limited to 3B models pretrained on 60B tokens; validation with longer training, larger scales, and broader benchmarks remains a next step.The authors cite scaling analysis up to 3B parameters and prior work as grounds for optimism, not as completed validation.Source: §6 · p. 12
• Whether the design insights generalize to newer attention and linear-attention variants remains an open question; the experiments combine base Transformer and Mamba blocks.The authors note that their initialization protocol differs from the original Mamba 2 initialization for A.Source: §6 · p. 12
• Extending hybrid architectures beyond language, including to video and audio, is identified as a critical next step.Source: §6 · p. 12

A The Use of Large Language Models

• The authors used Llama 3.2 and Llama 4 to polish writing and assist with vibe coding for bug fixes and initial figure drafts.Source: A The Use of Large Language Models · p. 21

B Broader Impact Statement

• The paper says efficiency gains could reduce compute, memory needs, and latency, while also being exploited to generate disinformation at scale; it calls for rigorous safety alignment.Source: B Broader Impact Statement · p. 21

C Details for Computational and Memory Costs Comparison

• 95% smaller cache footprint for a 1B model at 8K context (256 MiB vs. 13.4 MiB); Mamba cache size is sequence-length-independent, while Transformer FLOPs scale quadratically with context length.For a 1B model, the Mamba attention component has about 2.5 times the parameters of a Transformer attention component (25M vs. 10M per block).Source: C Details for Computational and Memory… · p. 21

D Details for Intra-layer Hybrid Architectures

• The intra-layer design space varies output projection, scaling, normalization, fusion, and shared inputs; one proposed variant lets Mamba use attention’s value projection, hypothesized to promote denoising or synergistic fusion.The study also compares against homogeneous differential architectures with identical module configurations.Source: D Details for Intra-layer Hybrid… · p. 22

E Detailed Experimental Setup

• Models follow Llama 3.2 and Mamba 2 configurations across scales; sliding-window attention uses a 512-token window and 64-token attention sink.Table 8 summarizes the base Transformer and Mamba configurations used to construct the hybrid architectures.Source: E Detailed Experimental Setup · p. 23, Table 8 · p. 23
• Models are pretrained from scratch on 60B randomly sampled tokens from DCLM-Baseline using 8 H200 GPUs by default, with a 2M-token batch size.Experiments cover 100M, 350M, 1B, and 3B parameters.Source: E Detailed Experimental Setup · p. 23
• Evaluation covers DCLM-Baseline and PG19 language modeling, Needle-In-a-Haystack long-context retrieval, and few-shot accuracy on five benchmarks.The few-shot benchmarks are LAMBADA, HellaSwag, PIQA, ARC Easy and Challenge, and OpenBookQA.Source: E Detailed Experimental Setup · p. 23

F Expanded Main Results on Quality

• Pretraining comparisons cover 350M and 1B models under data and FLOPs constraints, evaluating SWA and hybrid block ratios from 1:1 to 1:12 and 0:1.The tables summarize configurations, validation losses, few-shot accuracy, training FLOPs, and cache sizes.Source: F Expanded Main Results on Quality · p. 24, Table 9 · p. 24, Table 10 · p. 25

G Expanded Results of Sliding Window Attention Ablation

• Middle placement of Transformer blocks proves most beneficial for overall SWA model performance, so the authors use it in all main experiments.The ablation uses an attention sink size of 64 and a sliding window size of 512.Source: G Expanded Results of Sliding Window… · p. 26, Table 11 · p. 26

H Expanded Results on Longer Context Pretraining

• At 2x and 4x extended context lengths, hybrid architectures significantly outperform their counterparts; Transformers degrade, while Mamba hybrids retain strong performance with substantially smaller cache size.Table 12 reports perplexity and few-shot accuracy for 1B models at 16K and 32K contexts.Source: H Expanded Results on Longer Context… · p. 26, Table 12 · p. 26

I Experimental Setup for Downstream Fine-tuning Tasks

• Downstream summarization and QA evaluation uses five 1B checkpoints pretrained with an identical 4e20 FLOPs budget and an 8K maximum sequence length, except SQuAD at 2K.Fine-tuning runs for 1,000 steps; evaluation samples 1,000 test instances per dataset except SQuAD, which uses its validation set.Source: I Experimental Setup for Downstream… · p. 27

J Expanded Main Results on Efficiency

• Hybrid architectures demonstrate exceptionally high throughput across settings measured at batch size 4, with a 512-token prompt and total sequence lengths from 2K to 32K.Table 13 compares throughput and latency across architectures and block ratios.Source: J Expanded Main Results on Efficiency · p. 27, Table 13 · p. 27

K Experimental Setup for Long-context Evaluation Tasks

• Long-context evaluation includes PG19 position-wise perplexity on 30 un-packed documents and a modified Needle-In-a-Haystack retrieval test across sequence lengths up to 16,320.For the retrieval test, a key fact is inserted at depths from 0% to 100%; accuracy requires generating the target 7-digit number.Source: K Experimental Setup for Long-context… · p. 28

L Experimental Setup for Integration of Mixture-of-Experts

• The MoE variant replaces every layer’s FFN with a token-choice routed MoE, using one shared expert and top-1 selection from eight experts while halving expert-FFN intermediate dimensions.It uses sigmoid-scaled router scores and auxiliary loss-free balancing, with a bias update coefficient of 1e-3.Source: L Experimental Setup for Integration… · p. 28

M Expanded Results of Scaling Laws Experiment

• The scaling experiments train four model sizes across five compute budgets, fit quadratic curves at each budget, and derive optimal compute-scaling lines.Figure 5 marks the best-quality parameter size at each budget and shows token-to-parameter ratios on the fitted optimal line.Source: M Expanded Results of Scaling Laws… · p. 29, Figure 5 · p. 29, Table 14 · p. 30

N.1 Insights for Block Ratios

• A 1:5 inter-layer block ratio strikes the reported balance: increasing Mamba blocks reduces FLOPs and cache requirements with marginal degradation in few-shot accuracy and perplexity.The trend holds at both 350M and 1B scales; 1:1 achieves the highest overall quality but has a substantial inference bottleneck from self-attention.Source: §N.1 · p. 32, Table 16 · p. 32

N.2 Insights for Positioning of Computational Primitives

• Transformer blocks placed toward the middle or later stages perform better across tested ratios and scales, while avoiding early-layer attention is reported as crucial for maximizing hybrid capacity.The authors describe this positioning trend as holding across both validation NLL and few-shot accuracy.Source: §N.2 · p. 32, Table 17 · p. 33

O Visualization of Attention Scores

• Transformer attention is highly uniform in initial layers, whereas Mamba focuses on local context with small effective receptive fields; early Transformer blocks may therefore negatively affect inter-layer hybrid performance.The initial-layer uniformity also appears in front-placed Transformer blocks in inter-layer hybrids.Source: O Visualization of Attention Scores · p. 34, Figure 6 · p. 34, Figure 7 · p. 34

P.1 Insights for Architectural Variants

• Prioritizing NLL and layer count amid few-shot accuracy variance led the authors to select Diff fusion for remaining experiments; global attention plus Mamba outperformed homogeneous and differential models.The ablations covered 350M and 1B models.Source: §P.1 · pp. 35–36
• Group normalization generally improves performance, and the authors suggest it stabilizes optimization against the modules’ >10× scale discrepancy.Source: §P.1 · p. 36
• Diff-T and Hymba’s Group normalization scaling schemes degrade performance.Source: §P.1 · p. 36
• Fusion choice has negligible performance impact; the authors hypothesize Mamba’s implicit attention scores can be negative, making specific fusion mechanisms unnecessary.Fusion interacts with value sharing and output projection configurations.Source: §P.1 · p. 36
• Sharing values causes only a minor performance drop but increases layer count under a fixed parameter budget; separate pre-fusion output projections are crucial for significant gains.Source: §P.1 · p. 36

P.2 Insights for Dimension Ratios

• A 1:1 Transformer-to-Mamba dimension allocation is the practical choice for co-optimizing efficiency and downstream performance, despite larger Transformer dimensions improving quality at cache and compute cost.The Transformer dimension refers to query/key dimensions, and the Mamba dimension to pre-expansion dimensions.Source: §P.2 · p. 37, Table 20 · p. 37

P.3 Insights for Block Ratios and Positioning of Primitives

• Scatter positioning—distributing intra-hybrid layers evenly across network depth—outperformed Cluster and Sandwich patterns in NLL and few-shot accuracy across block ratios.Source: §P.3 · p. 37, Table 21 · p. 37

• A 350M intra-layer hybrid model uses head-wise Transformer and Mamba modules, Group-normalizes their outputs, and combines them with weights α and 1 −α before output projection.The authors treat α as a quantitative measure of each module’s contribution to next-token prediction.Source: Q Analysis of Component Contributions… · p. 38
• Mamba receives higher average weight in earlier layers, while Transformer receives higher weight in layers 6 and 7; the coefficients are averaged across about 100 DCLM validation samples.The checkpoint was trained on approximately 4B tokens; Figure 8 reports weights for a 12-layer, 350M model.Source: Q Analysis of Component Contributions… · p. 38, Figure 8 · p. 38
• In sampled token-level visualizations, Mamba dominates overall in layer 0, while Transformer is used more at sentence boundaries and phrase starts.The authors associate those positions with ‘forking tokens’ for guiding reasoning.Source: Q Analysis of Component Contributions… · p. 38, Table 22 · p. 41
• At layer 6, global attention is activated for retrieving key context keywords and predicting numerical or complex information.Source: Q Analysis of Component Contributions… · p. 38
• At layer 7, Transformer is often activated at new syntactic units, while Mamba activates on tokenizer-split subwords.Source: Q Analysis of Component Contributions… · p. 38