Olmo Hybrid: From Theory to Practice and Back

• Hybrid models address complementary weaknesses: linear RNNs support state tracking and linear context scaling, while attention helps with copying and recall tasks that bounded recurrent state struggles to represent.Transformers retain training parallelism but scale quadratically with context length at inference.Source: §1 · p. 3
• Olmo Hybrid matches Olmo 3 7B on MMLU with 49% fewer training tokens and ultimately outperforms Olmo 3 across aggregated OlmoBaseEval domains, with a 14.1% RULER 64k improvement.The final checkpoint’s gains persist after mid-training, supporting hybrid models over pure transformers in a controlled comparison.Source: §1 · p. 3
• Because large-scale comparisons isolate the architecture change, the results support wider adoption of hybrids as models balancing expressivity, parallelism, benchmark performance, and long-context ability.Olmo Hybrid interleaves Gated DeltaNet and attention layers at a 3:1 ratio in place of Olmo 3’s sliding-window-attention layers.Source: §1 · p. 3
• Theory shows that hybrid models can represent and empirically learn code-evaluation problems that neither transformers nor Gated DeltaNet can express alone.This establishes expressivity beyond the sum of the two component architectures’ capabilities.Source: §1 · p. 3
• Controlled scaling studies show that hybrid models achieve better token efficiency than transformers, while a formal account explains how greater expressivity can improve data and compute efficiency for loss and downstream tasks.The explanation applies even when downstream tasks are unrelated to the newly expressible formal problems.Source: §1 · p. 3

2.1 Architecture

• The architecture replaces 75% of Olmo 3’s sliding-window attention layers with GDN layers, alternating three GDN layers with one multi-head attention layer.GDN uses negative eigenvalues, implemented by replacing β_t with 2β_t, and the model adjusts dimensions to preserve comparable parameter count and throughput.Source: §2.1 · p. 4

2.3 Olmo Hybrid Results

• Olmo Hybrid reaches Olmo 3’s Common Crawl loss with 35% fewer tokens and its MMLU accuracy with 49% fewer tokens during pretraining.These correspond to 35% and 49% fewer FLOPs, respectively.Source: Figure 1 · p. 5, §2.3 · pp. 5–6

2.2 Training Overview

• Training largely follows Olmo 3 while matching scale and throughput: Olmo Hybrid has 7.0B parameters versus Olmo 3’s 6.8B and uses two 100B-token mid-training subsets.Long-context extension uses 100B tokens of Dolma 3 Longmino Mix; the released model uses DroPE, while both DroPE and YaRN are evaluated.Source: §2.2 · pp. 4–5

2.3 Olmo Hybrid Results

• After mid-training, Olmo Hybrid outperforms Olmo 3 7B across every evaluation domain, with gains persisting after long-context extension.The final pretrained checkpoint gains on math (+4.3%), STEM MC (+3.4%), and non-STEM MC (+4.4%), while declining on code (-2.5%) and general QA (-2.3%); held-out gains include MMLU Pro (+4.5%) and BBH (+1.2%), alongside LBPP (-1.1%) and DM Math (-0.2%).Source: Table 2 · p. 6, §2.3 · p. 6
• Olmo Hybrid outperforms Olmo 3 7B at long RULER sequence lengths using both YaRN and DroPE, with particularly notable gains from DroPE.The released model uses DroPE, which removes RoPE after mid-training; the authors attribute long-context gains to the linear RNN layers’ recurrent structure.Source: Table 3 · p. 6, §2.3 · p. 6
• Olmo Hybrid 7B lies on the Pareto frontier of open-weight dense models for average OlmoBaseEval performance versus training compute.The comparison includes open-weight hybrid, RNN, and transformer base models of similar parameter count.Source: §2.3 · p. 6, Figure 2 · p. 7

3 Expressive Power of Hybrid Models

• The section motivates architectures that maximize expressivity while preserving trainability and parallelism, because unrestricted expressivity can undermine efficient scaling.Hybrid models are presented as occupying a better expressivity-parallelism frontier than transformers by combining complementary capabilities.Source: §3 · pp. 7–8

3.1 Background: Expressive Power of Transformers

• Transformers are limited to TC0-like parallelizable computations and cannot express hard sequential state tracking under TC0̸ = NC1.State tracking includes composing ordered updates, such as difficult chess and shell-game instances, which can be NC1-complete.Source: §3.1 · pp. 8–9

3.2 Background: Expressive Power of Linear RNNs

• Linear RNNs can express NC1-complete state tracking with suitable non-diagonal, time-dependent transitions, but bounded hidden state limits recall.For DeltaNet or GDN, negative eigenvalues enable swap-like dynamics needed for tasks such as parity and the A5 word problem.Source: §3.2 · pp. 9–10

3.3 Hybrid Models are More Than the Sum of Their Parts

• Hybrid models express state-based recall with one alternation between GDN and attention, whereas neither transformers nor RNNs can express it under TC0̸ = NC1.The task composes pointer state tracking with attention-based retrieval, and either layer order suffices.Source: §3.3 · pp. 10–11

3.4 Expressive Power of Padded Hybrid Models

• With polynomial padding, fixed-depth hybrids recognize every language in FO-uniform NC1, exceeding padded transformers’ TC0 expressivity under standard conjectures.This follows by combining attention-based first-order reductions with GDN transposition composition.Source: §3.4 · p. 12

3.5 Synthetic Evaluations

• State tracking remains near-perfect for GDN and hybrids but degrades rapidly for transformers as sequence length increases.Without negative eigenvalues, accuracy falls to 0.21484 and 0.22266 for the linear RNN at n = 64 and 128, and 0.36328 for the hybrid at n = 128.Source: Figure 7 · p. 12, §3.5 · p. 13
• Transformers and hybrids remain near-perfect on recall, while linear RNNs degrade as the stored bit-array grows.At m = 64 and 128, the negative-eigenvalue linear RNN reaches 0.83203 and 0.67578, respectively.Source: Figure 7 · p. 12, §3.5 · p. 13
• Hybrids with negative-eigenvalue GDN remain essentially perfect on state-based recall, while transformer and pure GDN performance degrades as difficulty increases.Removing negative eigenvalues drops the hybrid to 0.57031 at n = 128, supporting the theoretical compositional advantage.Source: §3.5 · p. 13

3.6 Discussion: Pushing the Expressivity-Parallelism Frontier

• Hybrid models express tasks that neither transformers nor linear RNNs can, despite retaining theoretical parallelizability comparable to transformers.They extract more expressivity from the same level of parallelism (Merrill et al., 2026).Source: §3.6 · pp. 13–14
• Hybrid architectures push language modeling beyond transformers by obtaining fundamentally greater expressivity while remaining similarly scalable.This reflects the principle of extracting more from the same resources.Source: §3.6 · p. 14

4 Scaling Behavior of Hybrid Models

• This theory explains improved hybrid scaling as a fixed-factor efficiency gain: expressive architectures can learn a larger proportion of discrete tasks, lowering loss without changing scaling exponents.The proposed mechanism aligns the theoretical prediction with the empirical lower data coefficient B and projected data savings.Source: §4 · p. 14, §4.1 · pp. 17–18

4.1 Scaling Laws for Hybrid Models

• Hybrid models achieve lower loss than transformers across scales, with greater data and compute efficiency supported by fitted scaling laws.The study compares transformers, pure GDN models, and hybrid GDN–transformer models using derived empirical scaling laws.Source: §4.1 · p. 14
• The hybrid’s data coefficient B is 83.7 versus 94.9 for transformers, with non-overlapping confidence intervals, while scaling exponents are statistically indistinguishable.The fixed-exponent fit sets α = β = 0.22 across architectures; the parameter coefficient A slightly favors the hybrid, but confidence intervals overlap.Source: §4.1 · pp. 16–17, Figure 9 · p. 17
• Projected token savings for hybrids grow from ∼1.3× at 1B parameters to ∼1.9× at 70B, while compute savings also increase with scale.At 7B and 30B, the projected token savings are 1.68× and 1.82×, respectively; at 70B they reach 1.89×.Source: §4.1 · p. 17, Figure 10 · p. 18

4.2 Theory: Increased Expressive Power Improves Scaling

• The expressivity-aware scaling theory models language modeling as learning many discrete tasks and predicts that greater expressivity improves scaling by making more tasks learnable within fixed parameter and token budgets.The framework allows inexpressible tasks either to yield higher residual loss or to require more parameters and relevant tokens, with expressible tasks requiring fewer resources.Source: §4.2 · pp. 18–20
• Greater expressivity strictly improves loss across all parameter and token budgets, yielding more parameter- and data-efficient scaling under the expressivity-aware quantization model.The effect can also reduce irreducible loss when inexpressible tasks achieve lower loss reduction, while requiring more parameters or tokens produces the strongest gains at smaller budgets.Source: §4.2 · pp. 20–21

4.3 Discussion: From Expressivity to Scaling

• The theory suggests that hybrid models’ greater expressivity explains their better data and parameter efficiency, while preserving the scaling-law exponent.Empirical scaling studies found better efficiency than the transformer baseline, consistent with improved pretraining efficiency for Olmo Hybrid versus Olmo 3.Source: §4.3 · p. 22
• Even tasks already expressible by an architecture can be learned faster when increased expressivity frees parameters from inefficiently learning other tasks.This mechanism is consistent with findings reported by Hu et al. (2025).Source: §4.3 · p. 22
• The theory is a conceptual explanation based on the simplified quantization model of scaling laws, not an analysis of actual language-model learning dynamics.The authors caution against treating its quantitative scaling coefficients and exponents as precise predictions and call for more realistic theory and empirical tests.Source: §4.3 · p. 22
• Expressivity may not be the only source of hybrid-model gains, because training stability and other Gated DeltaNet benefits could also improve scaling.GDN hybrids scale better than Mamba hybrids, but similar trends with and without GDN’s negative-eigenvalue extension leave the mechanism unresolved.Source: §4.3 · p. 22

5.1 RNN and Hybridization Architecture Choices

• At larger scales, the 3:1 interleaved GDN hybrid achieves the strongest overall ablation results, outperforming transformer and Mamba2 alternatives across evaluated domains.It achieves 0.717 BPB at 760M and 0.669 at 1B; GDN-based models consistently outperform Mamba2 across Math, Code, and QA.Source: §5.1 · pp. 24–25
• Interleaved attention consistently beats middle placement at comparable parameter counts, while all tested hybrid ratios outperform transformers at large scales.Interleaving distributes global-context access and increases layer-type alternations; 7:1 is competitive only at small scales, whereas 3:1 is the most consistent large-scale choice.Source: Figure 12 · p. 23, §5.1 · p. 24
• GDN gates help pure GDN consistently, whereas hybrid variants differ by at most ∼0.01 BPB and retain the gated negative-EV configuration for consistency.Positive-EV gated pure GDN is sometimes slightly better than negative-EV gated GDN, but differences are small and inconsistent; attention can partly compensate for removing the gate in hybrids.Source: §5.1 · p. 24

5.2 Olmo Hybrid vs. Other Open Models

• Olmo Hybrid substantially outperforms pure-RNN baselines and remains competitive with Nemotron-H and Falcon H1 despite receiving roughly half or fewer training tokens.Direct baseline comparisons are limited because models used different datasets and token budgets; Olmo Hybrid outperforms RecurrentGemma and matches or exceeds stronger-budget hybrids on some tasks.Source: §5.2 · pp. 25–26, Table 6 · p. 27

5.3 Post-Training Olmo Hybrid

• With required --enforce-eager decoding, Olmo Hybrid generates 1,425, 1,365, 1,197, and 654 tokens/sec/node at 4K, 8K, 16K, and 32K, respectively.Without the flag, it is competitive with or faster than Olmo 3 at most context lengths, but numerical stability requires the flag; FP32 GDN cache can recover similar scores when compilation is re-enabled.Source: §5.3 · pp. 27–29
• Post-training preserves Olmo Hybrid’s knowledge-task gains over Olmo 3, but it still trails on extended reasoning benchmarks such as AIME and Omega.The comparison covers Think SFT, Instruct SFT, and DPO checkpoints; adapting post-training data and decoding kernels may improve the remaining gaps.Source: Table 7 · p. 28, §5.3 · p. 29

6 Conclusion

• Olmo Hybrid achieves better pretraining efficiency than Olmo 3, improving downstream tasks including long-context abilities and supporting hybrid architectures over transformers in a controlled, large-scale setting.The paper links these empirical gains to theoretical analyses of expressivity and scaling efficiency.Source: §6 · p. 29
• Hybrid models can represent synthetic tasks that neither transformers nor RNNs in isolation can, and increased expressivity is theoretically predicted to improve training efficiency.The conclusion connects this expressivity advantage to the observed scaling behavior and performance gains.Source: §6 · p. 29
• The authors identify future work in post-training hybrid models, RNN architecture optimization, and deeper understanding of the connection between expressive power and scaling behavior.They also note that implementation improvements recovered much of the eager-mode throughput gap and that the long-context recipe differs slightly from Olmo 3 7B.Source: §6 · p. 29

A Training Olmo Hybrid

• The appendix provides implementation and development details for Olmo Hybrid, including the architecture, throughput calibration, training stability measure, and pretraining hardware.These details support the controlled comparison with Olmo 3 described in the conclusion.Source: A Training Olmo Hybrid · p. 39, §A.1 · p. 39

A.1 Pretraining

• The pretraining setup used Flash Linear Attention’s Gated DeltaNet implementation, a cosine learning-rate decay to 10% of maximum, and the improved Olmo 3 32B data mix.The training ran on 512 GPUs, migrating from H100s to B200s about halfway through pretraining.Source: §A.1 · p. 39
• Olmo Hybrid largely retains the Olmo 3 7B architecture while introducing Gated DeltaNet layers, small architectural and training changes, and a 3:1 hybridization ratio selected from early controlled experiments.The model uses proportionally sized GDN heads, with query/key size d = 96 and value size 2d = 192.Source: §A.1 · p. 39
• Removing two heads made Olmo Hybrid closely match Olmo 3 in parameter count and slightly exceed it in training throughput, motivating a 30-head configuration for the 7B run.Throughput was calibrated using configurations benchmarked on 128 H100s.Source: §A.1 · p. 39
• Olmo Hybrid shows a lower, flat gradient spike-score trajectory, whereas Olmo 3’s spike score is high and grows throughout training, providing preliminary evidence of greater stability.Spike score measures the percentage of values at least six standard deviations from a rolling average of the previous 128 values.Source: §A.1 · pp. 39–40, Figure 13 · p. 40

Mid-Training

• Mid-training doubles the batch size, runs two independent trainings on separate 100B-token Dolma 3 Dolmino Mix subsets, and merges the resulting checkpoints.The procedure adapts Olmo 3’s recipe using its 32B mid-training data mixture with additional light filtering (Olmo Team, 2025), motivated by learning-rate and batch-size insights (Merrill et al., 2025).Source: §A.2 · p. 40

Long Context Extension

• Long-context extension continues training on 100B tokens of Dolma 3 Longmino Mix while comparing YaRN with DroPE, which removes RoPE and relies on the causal mask and learned positional signals.YaRN interpolates low-frequency RoPE components, leaves high-frequency components unchanged, and applies an attention-temperature correction; DroPE removes the shorter-context RoPE constraint.Source: §A.2 · p. 40
• 85.0 vs. 76.9 on RULER 64k: DroPE outperforms YaRN at the longest evaluation length, so it is adopted for the released Olmo Hybrid checkpoint.Both positional strategies produce strong long-context results, but DroPE has a clear advantage at the longest lengths.Source: §A.2 · p. 40
• Ulysses-style context parallelism distributes sequences across devices and supports both attention and per-head GDN updates, with depthwise q, k, and v convolutions sharded by head.The implementation uses all-to-all communication to transpose sequence-parallel layouts into head-parallel layouts before each layer.Source: §A.2 · pp. 40–41

A.3 Post-Training

• Olmo Hybrid 7B post-training uses documented hyperparameters across its model variants.Table 10 summarizes the training hyperparameters.Source: Table 10 · p. 41
• Think SFT totals 47.6B tokens, combining the Olmo 3 baseline with a 3× tool-use upscale.The calculation is 45.4B×(1+2×2.47%) = 47.6B.Source: Table 10 · p. 41
• Olmo 3 and Olmo Hybrid use different SFT configurations, primarily in parallelism strategy and activation checkpointing.Table 11 details these configuration differences.Source: Table 11 · p. 41, §A.3 · p. 41
• SFT models were trained with OLMo-core, whereas the DPO model was trained with Open-Instruct.These are the specified training frameworks for the post-training stages.Source: §A.3 · p. 41

A.4 Evaluation Details

• Olmo Hybrid is evaluated with the same evaluation suite as OLMo 3 (Olmo Team, 2025).The appendix identifies the shared suite but does not report results here.Source: §A.4 · p. 41
• The evaluation details separate the base evaluation configuration from the post-training evaluation configuration.Tables 12 and 13 describe these respective configurations.Source: §A.4 · p. 41

• The analysis assumes FO-uniform complexity classes and logarithmic precision, using c log n bits for input sequences of length n.These assumptions define the formal setting for the expressive-power results.Source: B Proofs: Expressive Power of Hybrid… · p. 41
• The proofs consider unique-hard-attention transformers and averaging-hard-attention transformers, with the latter uniformly averaging over all maximizers and allowing masked pre-norm.Constructions stated for unique-hard attention also apply to averaging-hard attention, while the unique-hard formulation is retained for greater generality.Source: B Proofs: Expressive Power of Hybrid… · pp. 41–42
• The work first establishes the negative side of the main theorems via core lemmas, then gives explicit constructions for hybrid models, and finally presents a complete padded characterization.Source: B Proofs: Expressive Power of Hybrid… · p. 42

B.1 Limitations of Transformers and RNNs

• Assuming TC0 ≠ NC1, fixed-depth transformers cannot solve A5 word, state-based recall over five variables, or formula-evaluation problems, even with polynomial padding.These problems are NC1-hard under FO reductions, while the relevant transformer classes lie in TC0.Source: §B.1 · p. 43
• Log-precision RNNs, including DeltaNet, cannot solve recall, state-based recall, or Polish-notation formula evaluation because their O(log n)-bit state cannot transmit problems requiring Ω(n) communication complexity.With polynomial padding, precision remains logarithmic, so the memory restriction persists; under both assumptions, transformers and linear RNNs fail state-based recall, permuted A5 word, and formula evaluation.Source: §B.1 · pp. 43–44

B.2 Power of Hybrid Models

• A single alternation of GDN and averaging-hard attention solves state-based recall, which neither transformers nor RNNs can express under the stated assumptions.The construction supports unary pointers in either layer order; GDN followed by attention also supports binary pointers and UHAT blocks.Source: §B.2 · pp. 44–45

B.3 Padded Hybrid Models

• With polynomial padding, fixed-depth hybrid models with averaging-hard attention and GDN with negative eigenvalues recognize every FO-uniform NC1 language.The construction applies an FO reduction using attention and evaluates the resulting transposition problem with GDN.Source: §B.3 · p. 45
• Padded hybrid models extend to FO-uniform PNC1, including problems such as weighted-automata simulation and integer-formula evaluation, with PNC1 characterized as an exact expressivity bound modulo uniformity.GDN with negative eigenvalues can represent PNC1-complete problems, while nonuniform PNC1 is an upper bound.Source: §B.3 · p. 45

C Additional Details: Synthetic Evaluations

• The synthetic-evaluation appendix documents the plotted curves, architectures, hyperparameter search, curricula, and exact numerical values used for Section 3.5.These details cover the experimental setup underlying the synthetic comparisons.Source: C Additional Details: Synthetic… · p. 46

C.1 Tasks and Evaluation Protocol

• Synthetic evaluations generate code-like strings online for recall, state tracking, and state-based recall, measuring next-token accuracy on the final answer token over 256 fresh samples per difficulty.State-based recall uses m = n with n ∈ {4, 8, 16, 32, 64, 128}; recall and state tracking vary their respective array or update sizes over the same range.Source: §C.1 · p. 46

C.2 Model Architectures

• The synthetic study controls for over-parameterization by using compact models with shared width and depth that differ only in sequence-mixing layers: full attention, GDN, or hybrid.The shared and model-specific architecture hyperparameters are summarized in Table 14.Source: §C.2 · p. 46, Table 14 · p. 46

C.3 Training Details

• All experiments use bf16 AdamW training with batch size 32, one gradient-accumulation step, 250 warmup steps, and a single H100 GPU.Source: §C.3 · p. 46
• Because synthetic-task performance is sensitive to optimization choices, the study systematically sweeps learning rates, schedulers, and initialization-related settings.Source: §C.3 · p. 47
• The state-based recall sweep excludes unstable learning rate 10^-3 and evaluates 120 runs across four optimizer configurations, five seeds, and three model types for up to 200,000 steps.Source: §C.3 · p. 47
• Task-specific curricula control difficulty: State Tracking follows fixed milestones, State-Based Recall combines a 0.95 accuracy threshold with step-budget advancement, and Recall uses fixed m = 128.Source: §C.3 · p. 47
• The generated next-token traces expose evolving state through asserts, while State-Based Recall randomizes assert spacing and withholds asserts in 20% of strict examples to discourage pattern matching.Source: §C.3 · p. 47
• The study also evaluates linear-RNN and hybrid ablations whose Gated DeltaNet blocks disallow negative eigenvalues, using the main experiments’ task definitions, metrics, and curricula.Source: §C.3 · p. 47
• Reported Section 3.5 numbers use the run with highest final accuracy at the hardest difficulty, breaking ties by progressively checking the next-longest length.Source: §C.3 · p. 47
• The complete hyperparameter-sweep runs and plotted values are provided through Weights & Biases, with task-name mappings and selected-run references documented for reproducibility.Source: §C.3 · pp. 47–48

Scaling Coefficients

• The hybrid has a lower data-efficiency coefficient B than the transformer, while their scaling exponents are statistically indistinguishable and the fixed-exponent fit reverses the apparent irreducible-loss advantage.The unconstrained hybrid B is 83.7 (CI [80.2, 87.1]) versus 94.9 (CI [88.7, 102.0]) for the transformer; fixed-exponent E is 1.569 for the transformer versus 1.597 for the hybrid.Source: §D.1.1 · p. 49, Table 18 · p. 50

Compute-Optimal Predictions

• Across compute budgets from 10^18 to 10^23 FLOPs, pure GDN improves predicted loss modestly and consistently, but less than the hybrid because of its weaker data-efficiency coefficient B.The pure GDN loss improvement is approximately −0.02 to −0.05 across scales.Source: §D.1.1 · p. 49

Projected Token Savings

• The supplement also reports projected token requirements and savings factors at discrete model scales, corresponding to Figure 10(b) in the main text.These projections are provided in Table 20.Source: §D.1.1 · p. 49

Fit Quality

• The scaling-law fits are highly accurate, with R2 values of 0.998 for Olmo 3 and 0.999 for both the hybrid and linear RNN, and residuals within ±1% across 60M–1B parameters.Residuals are scattered around zero, and mean absolute relative error remains consistent across model sizes.Source: §D.1.1 · p. 50

D.2 Details on the Scaling Law Fits

• Scaling laws use seven model sizes, five Chinchilla multiples, post-decay checkpoints, and averaged validation losses from 11 held-out domains; fitted diagnostics achieve R2 = 0.9976 for Transformer, 0.9993 for Hybrid, and 0.9994 for Pure GDN.The Chinchilla form is fitted with Huber loss over log-residuals, multi-start optimization, and 95% bootstrap confidence intervals from 1,000 resamples.Source: §D.2 · pp. 50, 52

D.1.2 Ablation Results

• Hybrid architectures show favorable scaling across average evaluation and Math, Code, and QA, with aggregate loss improvements consistent across domains.The 7:1 configuration also scales favorably, likely because it uses fewer transformer layers and therefore fewer FLOPs.Source: §D.1.2 · p. 50, Figure 16 · p. 54

D.3 Architectural Details

• Hybrid GDN models retain Olmo 3 dimensions and interleave GDN with attention, using a default 3:1 linear-to-attention ratio and forcing the final layer to be attention.Pure GDN replaces all attention layers, while Hybrid Mamba2 uses the same interleaving scheme with Mamba2 non-attention layers.Source: §D.3 · p. 55

D.4 Parameter Count and FLOP Computations

• Parameter and FLOP counts are computed analytically from architecture specifications, with non-embedding parameters excluding the embedding and language-model head contributions.The accounting separately specifies attention, GDN, and Mamba2 layer components, including projections, convolutions, recurrences, MLPs, and normalization terms.Source: §D.4 · pp. 55, 58

D.4.1 FLOP Counting

• Training FLOPs use hardware-efficient chunkwise algorithms rather than inference-time recurrence, including Gated DeltaNet’s Extended WY representation and explicit intra- and inter-chunk overheads.The GDN chunk size is 256; Mamba2 instead uses a sequential chunkwise algorithm with sequence-independent state passing.Source: §D.4.1 · pp. 60–61
• The sequential algorithm’s inter-chunk cost includes unidirectional state passing for state updates and output computation.These operations are represented as state update S + V K⊤ and output computation QS⊤.Source: §D.4.1 · p. 61
• Scaling-law compute is estimated as C = 3FD, where F is analytically computed training FLOPs per token and D is the token count.F is computed using the parallel formulation and the factor 3 accounts for forward and backward passes.Source: §D.4.1 · p. 61

E Proofs: Increased Expressive Power Improves Scaling

• The quantization model ranks tasks learnable with D samples by frequency and greedily allocates samples and parameters until the parameter budget is exceeded.A task is learnable when the dataset contains enough relevant tokens for it, and the learner allocates resources in frequency order.Source: E Proofs: Increased Expressive Power… · p. 61
• Expressivity is modeled by task-specific expressibility: expressible tasks achieve larger loss reductions and require no more parameters or relevant tokens than inexpressible tasks.Expressible tasks use C parameters and T relevant tokens, whereas inexpressible tasks require C′ ≥ C and T′ ≥ T, with loss improvement Δ′ ≤ Δ.Source: E Proofs: Increased Expressive Power… · pp. 61–62

• The generalized scaling law defines expected loss after learning n tasks from independent learned-task and unlearned-task losses with means a and b.This extends the original derivation from Michaud et al. (2023).Source: §E.1 · p. 62
• The true loss is sandwiched between consecutive power-law approximations, with relative error O(1/n), so it rapidly converges to the approximation used thereafter.The bounds are L_n+1 ≤ ˜L_n ≤ L_n.Source: §E.1 · pp. 62–63
• With expressibility probability 1 − ϵ, tasks achieve loss L_0 − ∆ when learned and loss L_0 − ∆′ otherwise, yielding a scaling law that interpolates between these outcomes.The section assumes ∆′ < ∆ when comparing expressibility levels.Source: §E.1 · p. 63
• If a′ > a, the resulting loss curve satisfies L′_n > L_n for every n, establishing that the associated loss reduction persists throughout scaling.The proof shows the difference is positive because f(n) lies in (0, 1).Source: §E.1 · p. 63

Parameter scaling law

• Expected loss ˜L(N) is closely approximated by a power law L(N) = a + cα · (b − a) / α · ζ(α + 1) · N^−α.The approximation assumes independent task parameter costs with mean c.Source: §E.2 · p. 64
• The expected number of learnable tasks scales as n_N ≈ N/c, producing the power-law loss approximation when substituted into the task-loss expression.This connects parameter count to the number of tasks learned under the model’s assumptions.Source: §E.2 · p. 64

Expressivity awareness

• Increasing expressivity decreases L(N) for every N when inexpressible tasks either reduce loss less or cost more, while their cost changes the slope rather than irreducible loss.With C′ = C, c_α = C^α; with C′ = 0, c_α = (1 − ϵ)^α C^α.Source: §E.2 · p. 64
• If C′ ≥ C and either Δ′ < Δ or C′ > C, L(N) strictly decreases elementwise for all N as the inexpressibility probability ϵ decreases.Conversely, the loss increases elementwise with ϵ under the stated assumptions.Source: §E.2 · p. 64

Data scaling law

• The expected loss ˜L(D) is closely approximated by a power law whose irreducible loss matches the task scaling law.The scaling coefficient differs from the expressivity-unaware case by a factor of Bϵ/B0.Source: §E.3 · pp. 65–66
• The model partitions tasks into three token-budget phases: all tasks learned, only expressible tasks learned, and no tasks learned.Their expected losses are respectively L0 −∆ for expressible tasks and L0 −∆′ for inexpressible tasks, L0 −(1 −ϵ)∆, and L0.Source: §E.3 · p. 66
• If ∆′ < ∆, or equivalently T′ > T, then L(D) strictly decreases as ϵ decreases for every D.Thus, increasing expressivity improves the approximated expected-loss curve whenever inexpressible tasks yield smaller gains or require more relevant tokens.Source: §E.3 · pp. 66–67
• Overall, increasing expressivity improves L(D) in every non-trivial expressivity-aware quantization model for sufficiently large D.The conclusion also covers the case ∆′ = ∆ when T′ > T and ∆ > 0.Source: §E.3 · p. 67

• Theorem 4 states that loss as a function of parameters N and token budget D is closely approximated by power laws under Assumptions 1–3.The result combines parameter-scaling and data-scaling analyses.Source: §E.4 · p. 68
• For any nontrivial expressivity improvement, both parameter- and data-scaling losses strictly decrease elementwise as ϵ decreases.The stated conditions are either ∆′ < ∆, or C′ > C and T′ > T.Source: §4.2 · p. 20
• Expressivity shifts irreducible loss if and only if ∆′ < ∆.Source: §E.4 · p. 68
• The analysis leaves several extensions open, including parameter-efficiency task ordering, joint parameter–data scaling, and empirical tests against real language models.These directions aim to bridge the idealized quantization model and practical training.Source: §E.4 · p. 68