PaperlayerView exact revision on arXiv

Source-linked AI summary

Olmo Hybrid: From Theory to Practice and Back

William Merrill, Yanhong Li, Tyler Romero, Anej Svete, Caia Costello, Pradeep Dasigi, Dirk Groeneveld, David Heineman, Bailey Kuehl, Nathan Lambert, Chuan Li, Kyle Lo, Saumya Malik, DJ Matusz, Benjamin Minixhofer, Jacob Morrison, Luca Soldaini, Finbarr Timbers, Pete Walsh, Noah A. Smith, Hannaneh Hajishirzi, Ashish Sabharwal

arXiv:2604.03444v4cs.LGcs.CL

TL;DR

The paper asks whether hybrid attention–recurrence models justify the cost and risk of replacing pure transformers. It combines theoretical expressivity results, controlled scaling studies, and a comparable Olmo Hybrid 7B model, finding better efficiency and evaluation performance while explaining how expressivity can improve scaling. The paper concludes that hybrid models offer a more expressive, scalable language-modeling architecture.

  • Problem

    There is no consensus on whether the potential benefits of hybrid models justify the cost and risk of scaling them instead of transformers.

  • Method

    The paper combines formal expressivity analysis, controlled transformer-versus-hybrid scaling studies, and a 7B Olmo Hybrid model using interleaved attention and Gated DeltaNet layers.

  • Results

    Olmo Hybrid matches Olmo 3 7B’s MMLU accuracy with 49% fewer training tokens and outperforms Olmo 3 across post-mid-training evaluation domains.

  • Takeaways & Limitations

    Hybrid models can extend expressivity beyond either attention or recurrence while retaining similar theoretical parallelizability and improving training-data efficiency.

  • Takeaways & Limitations

    The theoretical expressivity result for boolean formula evaluation is established for padded models, with broader analogous results remaining open.

Abstract

from arXiv · show

Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks (RNNs) and hybrid models that mix recurrence and attention. Yet there is no consensus on whether the potential benefits of these new architectures justify the risk and effort of scaling them up. To address this, we provide evidence for the advantages of hybrid models over pure transformers on several fronts. First, theoretically, we show that hybrid models do not merely inherit the expressivity of transformers and linear RNNs, but can express tasks beyond both, such as code execution. Putting this theory to practice, we train Olmo Hybrid, a 7B-parameter model largely comparable to Olmo 3 7B but with the sliding window layers replaced by Gated DeltaNet layers. We show that Olmo Hybrid outperforms Olmo 3 across standard pretraining and mid-training evaluations, demonstrating the benefit of hybrid models in a controlled, large-scale setting. We find that the hybrid model scales significantly more efficiently than the transformer, explaining its higher performance. However, its unclear why greater expressivity on specific formal problems should result in better scaling or superior performance on downstream tasks unrelated to those problems. To explain this apparent gap, we return to theory and argue why increased expressivity should translate to better scaling efficiency, completing the loop. Overall, our results suggest that hybrid models mixing attention and recurrent layers are a powerful extension to the language modeling paradigm: not merely to reduce memory during inference, but as a fundamental way to obtain more expressive models that scale better during pretraining.

1 Introduction

The paper asks whether hybrid architectures justify the cost and risk of replacing pure transformers. It combines theory, controlled scaling studies, and a large-scale Olmo Hybrid comparison to argue that hybrids are more expressive and efficient.

  • Hybrid models address complementary weaknesses: transformers struggle with sequential state tracking, while linear RNNs struggle with copying and recall.
  • Olmo Hybrid interleaves Gated DeltaNet and attention layers at a 3:1 ratio, replacing Olmo 3’s sliding-window-attention layers.
  • Olmo Hybrid matches Olmo 3 7B’s MMLU accuracy with 49% fewer training tokens and outperforms it across aggregated OlmoBaseEval domains after mid-training.
  • Theory shows hybrids can express formal code-evaluation tasks that neither transformers nor Gated DeltaNet can express alone.
  • The paper argues that hybrid models combine greater expressivity with transformer-like parallelizability, yielding better scaling efficiency and benchmark performance.

2 Olmo Hybrid Overview

Olmo Hybrid replaces most Olmo 3 sliding-window-attention layers with Gated DeltaNet while preserving close comparability. Across efficiency, benchmark, long-context, and compute-performance evaluations, it generally surpasses Olmo 3 and reaches the Pareto frontier among open-weight dense models.

  • Architecture: Olmo Hybrid uses three Gated DeltaNet layers followed by one multi-head attention layer, replacing 75% of Olmo 3’s layers.The model retains the overall Olmo 3 architecture while substituting sliding-window-attention layers with GDN layers.
  • Architecture: Gated DeltaNet adds state decay and a negative-eigenvalue extension, which the paper links to greater expressive power than standard linear attention or Mamba.
  • Training Overview: 35% fewer tokens reach the same Common Crawl loss, while 49% fewer tokens reach the same MMLU accuracy as Olmo 3 7B.The corresponding FLOP reductions are 35% and 49%, respectively.
  • Training Overview: Olmo Hybrid 7B has 7.0B parameters versus 6.8B for Olmo 3 and slightly outperformed Olmo 3 in early training throughput benchmarks.
  • Results: After mid-training, Olmo Hybrid outperforms Olmo 3 7B across every evaluation domain, while held-out results show gains on MMLU Pro and BBH but degradations on LBPP and DM Math.The reported held-out changes are +4.5% on MMLU Pro, +1.2% on BBH, -1.1% on LBPP, and -0.2% on DM Math.
  • Results: Olmo Hybrid 7B lies on the Pareto frontier of open-weight dense models for average OlmoBaseEval performance versus estimated training compute.The figure includes only models above 50% average OlmoBaseEval performance and uses C = 6ND to estimate training compute.

3 Expressive Power of Hybrid Models

Hybrid models combine the complementary strengths of transformers and linear RNNs while exceeding both in expressive power on composed tasks. The theory and synthetic evaluations show that hybrids can solve state-based recall and, with padding, capture NC1, while preserving parallelizability.

  • Complementary strengths: Transformers excel at recall, whereas GDN-based linear RNNs handle state tracking; hybrid models combine both capabilities.This complementarity motivates mixing attention and recurrence.
  • State-based recall: Hybrid models can express state-based recall, which neither transformers nor RNNs can express under standard complexity conjectures.The separation holds with one alternation between GDN and averaging-hard attention, in either order.
  • Padded expressivity: With polynomial padding, fixed-depth hybrid models can recognize every language in FO-uniform NC1.This strictly exceeds the TC0 class corresponding to padded transformers under the conjecture TC0 ≠ NC1.
  • Padded expressivity: A hybrid model can solve boolean formula evaluation with n^c padding, whereas no transformer or RNN can do so under TC0 ≠ NC1.The result extends to any NC1-complete problem, while the analogous unpadded formula-evaluation result remains open.
  • Empirical validation: Synthetic evaluations match the theory: hybrids remain robust on state-based recall while pure transformers and linear RNNs degrade as task difficulty grows.Transformers retain recall, linear RNNs retain state tracking, and hybrids remain strong on their composition.
  • Expressivity and parallelism: Hybrid models achieve greater expressivity while retaining theoretical parallelizability comparable to transformers.The paper frames this as progress beyond the usual expressivity–scalability tradeoff.

4 Scaling Behavior of Hybrid Models

Hybrid models achieve better scaling efficiency than transformers, especially with respect to data, while retaining comparable scaling exponents. The paper attributes this advantage to greater expressivity, which theory predicts can lower loss and improve resource efficiency without changing power-law exponents.

  • Empirical scaling results: Hybrid models are more data-efficient and compute-efficient than transformers across scales.Derived scaling laws provide a principled comparison across parameter counts, token budgets, and compute budgets.
  • Empirical scaling results: Chinchilla-style fits find a meaningfully lower data coefficient B for hybrids, while scaling exponents are statistically indistinguishable across architectures.The lower B indicates a fixed-factor reduction in data required to reach a target loss.
  • Empirical scaling results: The scaling-law predictions generalize beyond the fitting range, with 7B Olmo Hybrid prediction error of 0.28% at 5.5T training tokens.Olmo 3 7B also has 0.28% error, and the validation includes Olmo 3 32B.
  • Empirical scaling results: At 10^22 FLOPs, hybrid loss is 2.267 versus 2.308 for the transformer, a difference of -0.042 with a 95% CI of [-0.14, +0.06].Projected compute savings increase steadily with model size.
  • Empirical scaling results: 83.7 vs. 94.9 for B favors the hybrid over the transformer, with non-overlapping 95% confidence intervals.The fixed-exponent analysis identifies this as the most statistically robust finding.
  • Theory and interpretation: The expressivity-aware theory predicts that more expressive models learn more discrete tasks on fixed budgets, shifting loss curves downward without changing scaling exponents.The formal model can also improve irreducible loss and parameter efficiency under some assumptions.

5 Other Research Questions

The ablations identify GDN, interleaved placement, and a 3:1 linear-to-attention ratio as the strongest overall design choices. Across comparisons, the selected hybrid performs well against transformer and recurrent baselines, while post-training gains persist on knowledge tasks but not all reasoning benchmarks.

  • RNN Architecture: GDN-based hybrids outperform Mamba2-based models across scales and domains, with hybrid GDN achieving the best overall ablation results.At 760M and 1B parameters, hybrid GDN reaches 0.717 and 0.669 average BPB, respectively.
  • Layer Placement: Interleaving attention layers consistently outperforms concentrating them in the middle of the network.The authors associate this with broader access to global context and more alternations between layer types.
  • Attention Ratio: The 3:1 linear-to-attention ratio is the most consistent choice at larger scales, while 7:1 is competitive mainly at small scales.The 1:1 ratio does not justify its additional attention cost.
  • Open-Model Comparisons: Olmo Hybrid substantially outperforms pure RNN baselines across OlmoBaseEval task averages and remains competitive with larger-token hybrid models.It outperforms Falcon Mamba and xLSTM, while competing with Nemotron-H and Falcon H1 despite their larger token budgets.
  • Post-Training: Post-training preserves gains on knowledge tasks, but Olmo Hybrid still trails Olmo 3 on extended reasoning tasks such as AIME and Omega.The authors suggest adapting post-training data for the hybrid architecture as a possible improvement.
  • Post-Training: The post-training investigation does not assess whether switching to a hybrid architecture changes safety behavior.The authors identify safety evaluations as future work.

6 Conclusion

The conclusion presents Olmo Hybrid as evidence that combining attention with recurrence improves efficiency, downstream performance, and long-context ability relative to Olmo 3. It also notes implementation and comparison boundaries that qualify interpretation of these results.

  • Conclusion: Olmo Hybrid achieves better pretraining efficiency than Olmo 3, with corresponding gains on downstream tasks including long-context abilities.The paper frames this as evidence for hybrid architectures in a controlled, large-scale comparison.
  • Conclusion: Hybrid models can represent synthetic tasks that neither transformers nor RNNs alone can express, while theory links greater expressivity to improved training efficiency.The conclusion presents this theoretical link as an explanation consistent with the empirical scaling results.
  • Implementation: Olmo Hybrid’s throughput gap with Olmo 3 in eager mode was partly recovered by subsequent vLLM implementation improvements.This is reported as an implementation update rather than a new model-training result.
  • Comparison Boundary: The long-context recipe differs slightly from Olmo 3 7B, which could contribute to differences in post-training behavior.This limits how directly those post-training differences can be attributed to architecture alone.
  • Training Stability: Training stability was assessed with a gradient-norm spike score, and Olmo Hybrid showed a lower, flatter trajectory than Olmo 3.The authors treat this as preliminary evidence that the hybrid may tolerate large learning rates and noisy data better.

A.2 Mid-Training and Long Context Extension

The mid-training and long-context procedure adapts Olmo 3’s recipe while extending context with two positional strategies. Both work well, but DroPE performs better at the longest evaluated lengths and is selected for release.

  • Mid-Training: The adapted mid-training procedure uses Olmo 3 32B data, a doubled batch size, and two independent 100B-token runs.The runs use separate subsets of Dolma 3 Dolmino Mix before merging.
  • Long Context Extension: Long-context extension continues training on 100B tokens of Dolma 3 Longmino Mix and compares YaRN with DroPE.YaRN modifies RoPE frequencies, whereas DroPE removes RoPE during extension.
  • Long Context Extension: 85.0 vs. 76.9 on RULER 64k favors DroPE over YaRN at the longest reported evaluation length.Both strategies produce strong long-context results, but DroPE is adopted for the released checkpoint.
  • Long Context Extension: The authors attribute DroPE’s advantage partly to GDN layers carrying implicit positional information through recurrent structure.This reduces the attention layers’ dependence on explicit positional encodings such as RoPE.

A.3 Post-Training

The post-training section documents the configuration artifacts used for Olmo Hybrid 7B and the evaluation setup inherited from Olmo 3. The supplied passages emphasize configuration and evaluation metadata rather than performance findings.

  • Configuration: Table 10 summarizes Olmo Hybrid 7B training hyperparameters, including masked tokens in total-token counts.Think SFT totals 47.6B tokens after adding the tool-use upscale to the 45.4B Olmo 3 baseline.
  • Configuration: Table 11 records the configuration differences between Olmo 3 and Olmo Hybrid SFT training.The cited description highlights changes involving parallelism strategy and activation checkpointing.
  • Evaluation: Olmo Hybrid uses the same base and post-training evaluation suite described for Olmo 3.Tables 12 and 13 specify the base and post-training evaluation configurations.

B Proofs: Expressive Power of Hybrid Models

This section establishes the assumptions and proof strategy used to characterize the expressive limits of transformers and recurrent models. It develops the results through core lemmas, explicit hybrid constructions, and a padded characterization.

  • Assumptions: The analysis assumes FO-uniform complexity classes and logarithmic precision, using c log n bits for input sequences of length n.These assumptions define the computational setting for the theoretical results.
  • Assumptions: The transformer analysis distinguishes unique-hard-attention models, which select one position, from averaging-hard-attention models, which average all maximizers.Both attention variants are used in the theoretical treatment.
  • Proof strategy: The proofs first establish negative results through core lemmas, then provide explicit hybrid constructions and a complete padded characterization.This is the stated proof organization for the section.

B.1 Limitations of Transformers and RNNs

The section formalizes complementary limitations of fixed-depth transformers and log-precision RNNs, then shows that hybrid attention–GDN models overcome these limits on key formal problems. With polynomial padding, such hybrids reach FO-uniform NC1 and may extend to PNC1.

  • Transformer limitations: Fixed-depth transformers cannot solve NC1-hard problems such as the A5 word problem, state-based recall, and formula evaluation, assuming TC0 ≠ NC1.The result holds for AHATs and UHATs, even with polynomial padding.
  • RNN limitations: Log-precision RNNs, including DeltaNet, cannot solve recall, state-based recall, or Polish-notation formula evaluation when these require Ω(n) communication complexity.Their O(log n)-sized hidden state cannot transmit the required linear amount of prefix information.
  • Combined limitations: Under TC0 ≠ NC1 and logarithmic precision, both transformers and linear RNNs fail on state-based recall, the permuted A5 word problem, and formula evaluation.The two architectures fail for different reasons: sequential complexity for transformers and memory requirements for RNNs.
  • Hybrid separation: A hybrid with GDN having negative eigenvalues and averaging-hard attention solves state-based recall with one layer-type alternation, unlike either transformers or RNNs.The construction works in either layer order under the stated assumptions.
  • Padded expressivity: With polynomial padding, fixed-depth AHAT-plus-GDN hybrids recognize every language in FO-uniform NC1.The analysis further states that related results can extend to FO-uniform PNC1, including integer formula evaluation.

C.3 Training Details

The experiments use systematic hyperparameter sweeps, task-specific curricula, standardized run selection, and next-token prediction over Python-like execution traces. Additional ablations and logging conventions support comparisons across state-tracking, recall, and state-based recall tasks.

  • Training setup: The experiments use a batch size of 32, one gradient-accumulation step, and 250 warmup steps on a single H100 GPU.These fundamental training parameters are standardized across tasks.
  • Hyperparameter sweeps: Because synthetic-task performance is sensitive to optimization choices, the study systematically sweeps learning rates, schedulers, and random seeds.The state-based recall sweep covers four learning-rate/scheduler combinations and five seeds across three model types.
  • Curricula: State tracking uses a difficulty curriculum over n ∈ {8, 16, 32, 64}, while recall uses a fixed bit-array size of m = 128.State-based recall uses a separate hybrid threshold-and-budget curriculum.
  • Task construction: All tasks are framed as next-token prediction over Python-like execution traces, with state tracking exposing intermediate program-state assertions.Recall instead presents a bit-array definition followed by a direct query.
  • Ablations and reporting: The study also evaluates GDN ablations without negative eigenvalues and maps paper task names to their Weights & Biases labels and metrics.The ablations retain the same task definitions, metrics, and curricula as the main experiments.
  • Run selection: The reported run for each model and task is the one with maximum final accuracy at the hardest difficulty, with ties resolved using the next-longest setting.This criterion applies to the final numbers in the theoretical-task evaluation.

D.1.1 Additional Scaling Law Results

Additional scaling-law analyses compare hybrid, transformer, and linear-RNN architectures across parameter and data budgets. The results indicate a robust hybrid advantage in data efficiency, while fit diagnostics show that the scaling-law form describes all three architectures closely.

  • Scaling coefficients: The hybrid’s data-efficiency coefficient is B = 83.7 (CI [80.2, 87.1]), below the transformer’s 94.9 (CI [88.7, 102.0]) with non-overlapping intervals.The parameter coefficient A slightly favors the hybrid, but its confidence intervals overlap with the transformer’s.
  • Scaling coefficients: The fixed-exponent fit gives the transformer a lower irreducible loss E of 1.569 than the hybrid’s 1.597, reversing the unconstrained-fit ordering.The passage attributes the apparent unconstrained-fit hybrid advantage in E to a fitting artifact.
  • Compute-optimal predictions: Pure GDN improves predicted loss by approximately 0.02–0.05 across compute scales, but less than the hybrid because of weaker data efficiency.The compute-optimal allocations are derived from the fitted scaling laws.
  • Token projections: The scaling-law analysis reports projected token requirements and savings factors across a discrete set of model scales.These projections correspond to Figure 10(b) in the main text.
  • Fit quality: The power-law fits explain the observed data closely, with R2 values of 0.998 for Olmo 3 and 0.999 for both hybrid and linear RNN.Residuals remain within ±1% across model sizes from 60M to 1B parameters.

D.1.2 Ablation Results

The ablation and scaling analyses compare hybrid, transformer, and pure recurrent architectures across compute, domains, and token efficiency. Hybrid models show favorable scaling behavior and consistent advantages across Math, Code, and QA.

  • Scaling Trends of All Tested Architectures: Hybrid-model advantages appear across Math, Code, and QA in the per-domain scaling breakdown.The aggregate loss improvements are reported as consistent across domains.
  • Scaling Law Fitting: The scaling analysis uses seven model sizes and five Chinchilla multiples per size, based on post-decay checkpoints and averaged loss across 11 held-out corpora.The loss signal averages validation cross-entropy across diverse domains to reduce noise.
  • Scaling Law Fitting: Scaling-law fits are reliable, with mean absolute relative error below 1% across all model sizes.The reported R2 values are 0.9976 for Olmo 3, 0.9993 for Hybrid, and 0.9994 for Pure GDN.
  • Scaling Trends of All Tested Architectures: Olmo Hybrid reaches Olmo 3’s final performance using 19–58% fewer tokens across six downstream benchmarks.The savings depend on the benchmark and are reported in the extended downstream evaluation curves.
  • Architecture Ablations: Hybrid GDN models retain Olmo 3’s dimensions and layers while replacing a fraction of attention sub-layers with GDN sub-layers.The default transformer ratio is r=4, corresponding to a 3:1 linear-layer-to-attention ratio, with a final attention layer enforced.
  • Architecture Ablations: Pure GDN replaces all attention layers with GDN layers, while retaining the other hyperparameters of the hybrid variant.Pure GDN uses r=0 and does not force a final attention layer.

E.1 Scaling Law with Tasks Learned

The task-learning analysis models expected loss when tasks differ in whether the architecture can express them. It shows that the resulting loss is closely approximated by a power law and converges rapidly to that approximation.

  • Expressivity-Aware Extension: The analysis extends the original scaling law to expressivity-aware task learning, distinguishing tasks that are expressible from those that are not.The approximation is then used in the subsequent expressivity-aware results.
  • Task-Learning Scaling Law: The expected loss after learning tasks with varying expressibility is modeled using separate loss outcomes for expressible and inexpressible tasks.Expressible tasks occur with probability 1−ϵ and achieve a larger loss reduction than inexpressible tasks when ∆′<∆.
  • Task-Learning Scaling Law: The approximating power law bounds the true expected loss between adjacent terms, with relative error only O(1/n).The true loss rapidly converges to the power-law approximation as the number of learned tasks increases.

E.2 Parameter Scaling Law

The parameter-scaling analysis treats the number of learnable tasks as a function of model parameters and derives a power-law approximation for expected loss. It predicts that greater expressivity improves the loss curve under nontrivial cost or loss differences.

  • Parameter Scaling Law: Expected loss as a function of parameters is closely approximated by a power law.The approximation follows from modeling task costs and the expected number of tasks learnable with N parameters.
  • Parameter Scaling Law: The expected number of learnable tasks satisfies n_N≈N/c when task costs are independent with mean c.This relation supplies the parameter-to-task conversion used in the power-law derivation.
  • Expressivity and Parameter Scaling: The parameter cost of inexpressible tasks changes the scaling slope without changing irreducible loss.When inexpressible tasks cost more or reduce loss less, increasing expressivity decreases L(N) for every N.
  • Expressivity and Parameter Scaling: If C′≥C and either ∆′<∆ or C′>C, L(N) strictly decreases as the inexpressible-task probability ϵ decreases.The decrease is elementwise across all parameter counts.

E.3 Data Scaling Law

The data-scaling analysis extends the expressivity-aware model to token budgets and multiple task-learning phases. It concludes that increasing expressivity improves the expected loss curve when the stated loss or token-cost inequalities hold.

  • Data Scaling Law: Expected loss as a function of tokens is closely approximated by a power law under the phase-based task model.The derived expression is identified as the desired approximation L(D).
  • Data Scaling Law: The token-budget model separates task learning into phases where all tasks, only expressible tasks, or no tasks are learned.The phases correspond respectively to early learning, intermediate learning, and the regime beyond learnable tasks.
  • Expressivity and Data Scaling: Increasing expressivity improves L(D) elementwise when inexpressible tasks have smaller loss reductions or require more tokens.The result holds under ∆′<∆ or T′>T.
  • Expressivity and Data Scaling: For ∆′<∆, the loss increases pointwise as the inexpressible-task probability ϵ increases.Thus, the result is stated as a monotonic relationship between expressivity and token-scaling loss.
  • Expressivity and Data Scaling: In every non-trivial instantiation of the expressivity-aware quantization model, increasing expressivity improves L(D) for sufficiently large D.The power-law approximation is used for the expected loss at large token budgets.

E.4 Main Results

The section derives expressivity-aware parameter- and data-scaling laws, showing that greater expressivity improves scaling across budgets and can lower irreducible loss. It also identifies modeling assumptions and future extensions needed to connect the analysis more closely to practical training.

  • Expressivity-aware scaling laws: Theorem 4 derives parameter- and data-scaling laws by augmenting the quantization model of neural scaling laws with expressivity assumptions.The resulting losses are approximated by power laws in parameter count N and token budget D.
  • Main results: Expressivity strictly improves both parameter and data scaling under any nontrivial instantiation of the stated assumptions.Corollary 4.1 states that L(N) and L(D) decrease elementwise as ϵ decreases for all N and D.
  • Main results: Expressivity can shift irreducible loss precisely when inexpressible tasks have a smaller loss reduction than expressible tasks.This condition is formalized as ∆′ < ∆.
  • Future work: The analysis assumes learnable tasks are prioritized by frequency rather than parameter efficiency and leaves joint parameter-data scaling for future work.These extensions are proposed as ways to broaden the theoretical analysis.
  • Future work: The authors propose testing the derived scaling laws in controlled language-model settings to narrow the gap between the idealized quantization model and practical training.Such tests could examine settings where ϵ is known or tasks can be isolated in real data.
Loading 2604.03444v4…