PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers

Zeyuan Allen-Zhu

arXiv:2512.17351v2cs.CL

TL;DR

Academic-scale noise and failed multi-hop reasoning make reliable architectural comparison difficult. The paper introduces a controlled synthetic playground to isolate capabilities and discovers Canon layers, which improve reasoning and knowledge-related capabilities across architectures.

  • Problem

    Academic-scale pretraining produces noisy architectural comparisons, with failed multi-hop reasoning obscuring differences between models.

    Source: §1 · p. 2

  • Method

    The paper decomposes intelligence into atomic capabilities and evaluates architectures using controlled synthetic pretraining tasks.

    Source: §1 · p. 2

  • Results

    Canon layers enhance reasoning depth and breadth, knowledge capabilities, and structural reasoning across diverse architectures.

    Source: §9 · p. 30

  • Takeaways & Limitations

    Canon layers provide a lightweight architectural component that can strengthen weaker models and improve diverse sequence architectures.

    Source: §9 · p. 30

    The reported findings are primarily at academic scale, while larger-scale validation remains future work.

    Source: §1 · p. 5

Abstract

from arXiv · show

Understanding architectural differences in language models is challenging, especially at academic-scale pretraining (e.g., 1.3B parameters, 100B tokens), where results are often dominated by noise and randomness. To overcome this, we introduce controlled synthetic pretraining tasks that isolate and evaluate core model capabilities. Within this framework, we discover CANON LAYERS: lightweight architectural components -- named after the musical term "canon" -- that promote horizontal information flow across neighboring tokens. Canon layers compute weighted sums of nearby token representations and integrate seamlessly into Transformers, linear attention, state-space models, or any sequence architecture. We present 12 key results. This includes how Canon layers enhance reasoning depth (e.g., by $2\times$), reasoning breadth, knowledge manipulation, etc. They lift weak architectures like NoPE to match RoPE, and linear attention to rival SOTA linear models like Mamba2/GDN -- validated both through synthetic tasks and real-world academic-scale pretraining. This synthetic playground offers an economical, principled path to isolate core model capabilities often obscured at academic scales. Equipped with infinite high-quality data, it may even PREDICT how future architectures will behave as training pipelines improve -- e.g., through better data curation or RL-based post-training -- unlocking deeper reasoning and hierarchical inference.

1 Introduction

The paper addresses unreliable architecture comparisons by decomposing intelligence into controllable synthetic tasks, then introduces Canon layers to improve horizontal information flow across sequence architectures. Across synthetic and academic-scale pretraining, Canon layers enhance reasoning and knowledge capabilities while exposing persistent limits of linear models and small-scale training.

Synthetic playground

  • Synthetic tasks isolate reasoning depth, breadth, knowledge capacity, manipulation, and hierarchical language structure, enabling economical, controlled architecture comparisons with infinite high-quality data.They reduce confounding from mixed skills, random variance, grokking, and inadequate difficulty distributions, while making advanced skills emerge reliably at small scale.Source: §1 · pp. 2–3

Canon layers

  • Canon layers add lightweight horizontal residual links across neighboring tokens, with trainable kernel-size-4 one-dimensional convolutions insertable before or inside attention and MLP sublayers.The design integrates across Transformers, linear attention, and state-space models with minimal computational overhead.Source: §1 · pp. 3–5

Transformer results

  • 200–400% reasoning-depth gains, 30% reasoning-breadth gains, and 30% knowledge-manipulation-length gains demonstrate Canon layers’ broad synthetic benefits with minimal overhead.Canon layers cumulatively help across sublayer positions, improve training efficiency and stability, and recover some knowledge capacity in gated MLP and MoE architectures.Source: §1 · p. 4

Linear-model results

  • Canon layers consistently improve GLA, Mamba2, and GDN on reasoning, knowledge, and structural tasks, while GLA reaches 4-hop reasoning and can surpass Mamba2.Mamba2’s conv1d supplies partial Canon-like behavior, GDN benefits least, and simple GLA+Canon captures much of Mamba2/GDN performance.Source: §1 · p. 4

Transformer–linear comparison

  • ∼40% higher knowledge capacity favors linear models, whereas Transformers achieve 2–4× greater reasoning depth and stronger structural reasoning under full-Canon comparisons.Linear models’ shallow reasoning reflects cumulative compression and retrieval errors rather than insufficient recurrent-state capacity, motivating Transformer–linear hybrids.Source: §1 · p. 4

Academic-scale validation

  • At 1.3B parameters and 100B tokens, Canon raises NoPE to RoPE-level performance and GLA to Mamba2/GDN-level performance, but all models still fail 2-hop reasoning.Linear models remain behind full Transformers on retrieval-heavy tasks; removing conv1d reduces Mamba2 to GLA level, while less RoPE improves long-context generalization with Canon.Source: §1 · pp. 2, 4

2 Synthetic Tasks for Decomposing Intelligence

The paper introduces controlled synthetic tasks that isolate distinct language-model capabilities while minimizing confounds. Five tasks target internal reasoning depth and breadth, knowledge capacity and manipulation, and hierarchical language structure through complementary evaluation designs.

2 Synthetic Tasks for Decomposing Intelligence

  • The task suite prioritizes internal mental reasoning without CoT [78], dense 4096-token contexts over unstable length generalization [83], and universally relevant skills aligned with real-world applications.Long-context optimization, such as 100k-token inputs, may reduce performance on standard 4096-token lengths, while long inputs are typically summarized into shorter windows before reasoning.Source: §2 · pp. 5–6

2.1 Our First Set of Five Synthetic Pretrain Tasks

  • The five tasks—Depo, Brevo, Capo, Mano, and Lano—cover non-overlapping skills and token-level, generative, and distributional accuracy dimensions.The suite is presented as a starting point for deriving meaningful architectural insights.Source: §2.1 · pp. 6, 8
  • Depo tests internal reasoning depth through k-hop traversal over directed permutations, while Brevo tests reasoning breadth through recursive DAG traversal requiring global topological preprocessing.Both tasks remove intermediate CoT steps; Brevo requires outputting all vertices reachable from a query in topological order.Source: §2.1 · pp. 6–7
  • Capo measures factual storage capacity in bits per parameter using paraphrased synthetic biographies [8], whereas Mano tests retrieval and hierarchical manipulation of parameterized knowledge through multi-step modular arithmetic.Capo uses an undertrained regime with each biography exposed 100 times; Mano requires entirely internal computation without CoT.Source: §2.1 · p. 7
  • Lano evaluates hierarchical language reasoning by resolving globally ambiguous, recursively structured CFG sequences rather than explicit key-value relationships.Its CFGs require dynamic programming with worst-case O(n3) complexity, and extended datasets increase sequence lengths to 200–1000 tokens.Source: §2.1 · pp. 7–8

3 Initial Comparison on Well-Known Base Architectures

The study compares standardized, isolated base architectures under identical training conditions using synthetic tasks, revealing distinct strengths across knowledge and reasoning. These results are informative but limited to fixed-budget skill-acquisition speed and motivate later architectural improvements.

3 Initial Comparison on Well-Known Base Architectures

  • Isolated base architectures are compared to expose intrinsic trade-offs that hybrid models can obscure, particularly when memory strength and structured-reasoning performance diverge.The study omits hybrids to preserve clarity within standard 4096-token windows, where local precision is prioritized.Source: §3 · p. 8

3.1 Initial Comparison Results

  • GLA performs weakest overall, Mamba2 excels on knowledge, and GDN improves reasoning, occasionally surpassing Llama(RoPE) on Brevo while RoPE remains strongest on other reasoning tasks.Figure 4 reports Mamba2(mlp) excelling on Capo and Mano, GDN surpassing Llama(RoPE) on Brevo, and RoPE leading Depo+Lano.Source: Figure 4 · p. 9, §3.1 · p. 10

3 Initial Comparison on Well-Known Base Architectures

  • All models use standardized sizes, identical training settings, fixed seeds, and the same data sequences to ensure fair architectural comparisons.Llama, GLA, Mamba2, and GDN are evaluated at comparable parameter counts across four configurations, with broader scaling for Capo.Source: §3 · pp. 9–10

3.1 Initial Comparison Results

  • 3×4 mini scaling laws reduce randomness from grokking and staged k-hop learning by testing tasks across three data scales and four model sizes.Hard-task outcomes can vary with seeds and learning rates, so the scaling design yields more stable and interpretable comparisons.Source: §3.1 · p. 10
  • Synthetic tasks expose architectural strengths and weaknesses more clearly than noisy real-world experiments, enabling clean evaluations of scalability and core capabilities.The passage contrasts synthetic differences such as 90% versus 5% with real-world differences around 2%.Source: §3.1 · p. 10
  • A failure at a given difficulty reflects limited-budget skill-acquisition speed rather than an architecture’s absolute inability to learn the skill.All models receive the same training steps, data, shuffling, and learning-rate sweep, with best accuracy reported.Source: §3.1 · p. 10
  • Synthetic tasks may predict future architecture performance under improved data curation or RL-based post-training, because they isolate core skills obscured in ordinary corpora.The paper notes that 100B-token pretraining currently fails even simple 2-hop reasoning (Result 12), motivating tasks such as Depo.Source: §3.1 · p. 10

4 Canon Layers: Enhancing Horizontal Information Flow

Canon layers are lightweight local-mixing components that enhance horizontal information flow across neighboring tokens and integrate across Transformer, linear-attention, and state-space architectures. Controlled evaluations show they improve reasoning depth, strengthen NoPE, and can reduce reliance on RoPE.

Canon layers: general form

  • Canon layers aggregate nearby hidden states within a fixed window, providing lightweight horizontal information flow instead of global attention or recurrent compression.The typical window size is 4 tokens, and the mechanism is designed to mix local context that attention or state-space compression handles inefficiently.Source: §4 · p. 11
  • A Canon layer computes element-wise weighted sums of the current token and the previous three hidden states, using padding zeros at sequence boundaries.The weights are position-specific vectors, and the name refers to sequential melodies played with fixed temporal delays.Source: §4 · p. 11

Flexible Integration

  • Canon integrates before or inside attention and MLP blocks, supports partial or full A/B/C/D combinations, and extends to linear-attention and state-space models.In Mamba2, Canon appears at A and B positions, while Mamba2(mlp) supports the complete Canon-ABCD configuration.Source: §4 · p. 11

Canon layers: Implementation variants

  • The paper implements Canon efficiently as a causal convolution with kernel size 4, optionally adds residual connections, and reports minimal parameter growth despite broad architectural applicability.A simple fixed- or random-weight version already improves performance, while dynamic input-dependent variants are left unstudied; full Canon-ABCD requires only minimal code changes.Source: §4 · pp. 12–13

Results

  • Canon layers enhance RoPE by improving reasoning depth 2–4×, make NoPE perform on par with RoPE-based models, and allow reduced RoPE usage while retaining strong performance.RoPE + Canon with RoPE enabled for 1/4 dimensions outperforms both RoPE + Canon and NoPE + Canon, according to the reported length-generalization result.Source: Figure 7 · p. 13, §4 · p. 13

5 When Transformer Meets Canon

This section examines how Canon layers interact with Transformer components and architectures, including attention, MLPs, and mixture-of-experts designs. It assesses their effects on reasoning, knowledge acquisition, language-task performance, and model capacity alongside variations such as gated MLPs, ReLU2, and NoPE.

5.1 RoPE with Canon Layers

  • Canon layers improve reasoning depth, breadth, knowledge capacity, manipulation length, and hierarchical language-structure reasoning, achieving over 50% at Depo k = 8 and high accuracy at k = 16.They improve Brevo reasoning breadth by roughly 30%, raise Capo effective capacity by 10–15% in the controlled 100-exposure pretraining regime, and extend Mano manipulable length by 30%.Source: §5.1 · pp. 14–15

5.2 Why Canon Helps: Hierarchical Feature Learning

  • Canon’s 2–4× reasoning-depth gain comes from faster hierarchical learning rather than substantially greater representational power.Canon provides local residual-like links across the time horizon, enabling diagonal backward feature correction through layers and tokens; with unlimited training, vanilla RoPE can eventually reach comparable accuracy but requires more steps.Source: §5.2 · pp. 15–17, Figure 10 · p. 17

5.3 NoPE with Canon Layers

  • NoPE+Canon matches or surpasses RoPE+Canon on Depo and significantly outperforms ALiBi and H-Alibi, lifting NoPE from near-zero performance to competitive levels.NoPE+Canon remains weaker on long-sequence hierarchical structure learning in Lano but stays competitive with Mamba2/GDN.Source: §5.3 · pp. 17–18
  • Reduced RoPE with Canon often outperforms full RoPE or NoPE, supporting better length generalization with less positional encoding.RoPE on only 1/4 dimensions outperforms the compared full-RoPE and NoPE+Canon variants, while half-head, half-dimension RoPE is often preferable to full RoPE.Source: §5.3 · pp. 17–18

5.4 Ablation Studies With Canon Layers

  • Canon-A/B/C/D provide cumulative, modular gains; residual links are essential for stable training, while added nonlinear activations provide no measurable benefit.Canon components can be inserted independently of attention or MLP modifications, and each contributes meaningfully when stacked.Source: §5.4 · p. 18, Figure 11 · p. 19

5.5 MLP and Mixture-of-Experts

  • Gated MLPs improve expressiveness and parameter efficiency but reduce knowledge capacity by about 30% in limited-exposure training because of slower convergence.This motivates Canon-based mitigation of capacity losses associated with slower-training architectures.Source: §5.5 · p. 20
  • Gated MLPs retain a slight advantage over standard MLPs on reasoning-heavy tasks, particularly for knowledge manipulation and more modestly for reasoning breadth.The comparison is based on Figure 25 and identifies gated MLP as the better tradeoff than replacing it with a standard MLP.Source: §5.5 · p. 20
  • Result 5 shows gated MLPs slightly outperform standard MLPs, especially on Mano, while Canon layers partially recover gated MLPs’ reduced knowledge capacity.The experiments report noticeable knowledge-manipulation gains and smaller reasoning-breadth gains for gated MLPs, with Canon layers recovering about half of their lost capacity.Source: §5.5 · pp. 20–21
  • ReLU2 slightly improves standard MLPs but degrades gated MLPs, and both effects are negligible compared with Canon-layer gains.This synthetic result is consistent with recent findings [82] that ReLU2 does not necessarily make standard MLPs outperform gated MLPs.Source: §5.5 · p. 21
  • Canon-ABC substantially improves MoE knowledge acquisition and bit-per-parameter capacity, recovering at least half of the capacity loss caused by MoE.Canon-D conflicts with MoE’s independent token-wise expert dispatch, so the experiments use Canon-ABC alone; dense models acquire 10× more knowledge than a 32-expert transformer in the same 100-exposure regime.Source: §5.5 · pp. 20–21

6 When Linear Models Meet Canon

Section 6 evaluates Canon layers across GLA, Mamba2, and GDN, using insertion points before or inside linear-attention and MLP sublayers. Across these architectures, Canon restores localized horizontal information flow and generally improves reasoning-related performance, though gains vary by model.

6 When Linear Models Meet Canon

  • The architectures share linear-attention-or-state-space blocks followed by MLPs, exposing four Canon insertion points before or inside each sublayer.These placements support comparisons of horizontal information flow independently of the underlying architecture.Source: §6 · pp. 21–22, §6.1 · p. 22

6.1 When Linear Attention Meets Canon

  • Canon-AbCD substantially improves original GLA, raising reasoning depth from 1-hop to 4-hop, doubling reasoning breadth, and more than doubling knowledge manipulation length.It brings GLA on par with or beyond Mamba2, significantly outperforming it on Brevo, and adds gains over GLA conv1d.Source: §6.1 · p. 22

6.2 When Mamba Meets Canon

  • Mamba2’s internal non-linear conv1d, a partial Canon-B layer, contributes more to performance than the SSM itself, while full Canon-AbCD further improves results, notably on Mano and Lano.Removing the conv1d drops Mamba2 to GLA-level performance; Canon layers also remain effective outside the SSM block.Source: §6.2 · p. 23

6.3 When Gated DeltaNet Meets Canon

  • GDN depends less on its internal conv1d than GLA or Mamba2, yet replacing it with Canon-AbCD still yields marginal improvements across benchmarks.Removing conv1d destroys Mano and Lano performance but largely leaves Depo and Brevo unaffected; Canon remains effective outside the GDN layer.Source: §6.3 · pp. 24–25

7 Final Comparisons and Lessons for Architecture Design

Under controlled comparisons with Canon layers, linear models show stronger associative-memory capacity but substantially weaker deep reasoning than Transformers. The results attribute this gap to compression and retrieval dynamics rather than recurrent-memory capacity, motivating hybrid architectures and simpler Canon-based designs.

7.1 Summary on Linear Models vs. Canon Layers

  • Canon layers are universal, robust, and often sufficient: Canon-ACD matches internal conv1d, never hurts through residual stabilization, and GLA+Canon-AbCD nearly matches Mamba2 and GDN.This suggests some modern innovations may replicate Canon-like horizontal mixing, although such replication can improve efficiency and reduce GPU memory.Source: §7.1 · p. 26

7.2 Summary on Transformer vs. Linear Models

  • Reasoning depth favors Transformers: RoPE and NoPE reach about 4× deeper reasoning than Mamba2, GLA, and GDN, while linear models gain ~40% Capo knowledge capacity.Linear models remain about 2× weaker on Depo1 (8-hop vs. 4-hop) and up to 4× weaker on Depo2 (16-hop vs. 4-hop).Source: §7.2 · pp. 26–27
  • Linear models’ reasoning bottleneck is compressed-memory dynamics: encoding and retrieval errors compound across hops despite ample recurrent capacity, making Canon-equipped Transformer-linear hybrids practical.Mamba2 has 128d per-layer state capacity, while GLA and GDN have 64d–144d; hybrids combine sliding-window attention for deep reasoning with linear components for long-context compression.Source: §7.2 · p. 27

8 Real-Life Experiments

At academic scale, 1.3B-parameter models trained on 100B tokens exhibit substantial seed noise and limited resolution, but several architectural patterns remain consistent. Canon improves weaker architectures, while linear models lag Transformers on retrieval and multi-hop reasoning remains difficult.

Key observations across both suites

  • Academic-scale pretraining cannot reliably distinguish finer architectural differences because benchmark scores vary substantially across random seeds, reaching 4% on LAMBADA, 3% on BoolQ, and 9% on FDA.Only differences beyond the observed task-specific noise thresholds are statistically meaningful.Source: §8 · pp. 28, 30

Result 12

  • Linear models underperform full Transformers on short-context retrieval, including FDA, SWDE, and 1-hop-L with L ≤500, even with Canon.The results attribute this weakness to inefficient compression and retrieval rather than memory size.Source: §8 · pp. 28–30
  • Canon elevates NoPE to RoPE-level performance and GLA to Mamba2/GDN-level performance, while convolution removal downgrades Mamba2 to GLA but barely affects GDN.These comparisons are reported as consistent findings under academic-scale pretraining.Source: §8 · pp. 28, 30

Multi-hop reasoning tasks

  • All models fail 2-hop reasoning, scoring only 30–36% even at L = 0, whereas 1-hop tasks separate architectures and favor Transformers at L ≤500.The 2-hop result exposes a limitation of academic-scale pretraining rather than a reliable architectural advantage.Source: §8 · pp. 29–30

Result 12

  • Reducing or removing RoPE improves long-context generalization, while RoPE, RoPE+Canon, and NoPE+Canon perform comparably at this scale.Transformers benefit particularly in 4k-token junk passages when RoPE is reduced or removed.Source: §8 · pp. 28–30

9 Conclusion and Future Direction · Appendix · A Details on Synthetic Pretraining Tasks

The paper presents a controlled synthetic playground that enables economical, principled architectural comparison and identifies Canon layers as broadly useful components for improving reasoning, knowledge, and structural capabilities. It concludes with validation limits, reproducibility plans, and future directions spanning Canon design, architecture evaluation, task enrichment, interpretability, and new architecture discovery.

9 Conclusion and Future Direction

  • Canon layers improve reasoning depth and breadth, knowledge capacity and manipulation, and structural reasoning across diverse architectures in a controlled synthetic playground.The playground addresses noise and failed multi-hop reasoning in academic-scale pretraining by decomposing intelligence into atomic tasks.Source: §9 · p. 30
  • Canon layers revive weaker NoPE and GLA models to match or surpass RoPE and Mamba2, while exposing linear models’ depth limits as compression and retrieval inefficiencies.They also reduce reliance on RoPE for improved length generalization and may serve as a minimal, broadly applicable architectural primitive.Source: §9 · p. 30
  • Industrial-scale validation remains necessary because the academic-scale real-world experiments only align with synthetic findings; the authors plan to open-source the playground and evaluation suite.The release is intended to support rigorous and reproducible architecture research.Source: §9 · p. 30
  • Future Canon work should test dynamic adaptive convolutions with hidden-state-conditioned weights to determine whether gating benefits justify added computational overhead.The paper currently focuses on simple linear convolutional Canon layers with kernel size 4 for simplicity and efficient CUDA kernels.Source: §9 · p. 30
  • Systematic evaluation should identify optimal selective and cross-layer Canon configurations, including fused operations such as h′_ℓ+1 = h_ℓ+1 + Canon(h_ℓ).The paper briefly explored applying Canon selectively, such as in early layers, and connecting across layers for efficiency.Source: §9 · p. 30
  • Controlled comparisons of emergent architectures could reveal statistically significant improvements without confusing Canon gains with inherent features such as Mamba2’s built-in conv1d.The authors selected one representative per architecture family to maintain consistent Canon inclusion.Source: §9 · p. 30
  • The five synthetic tasks are only a starting point; additional atomic tasks could isolate further architectural capabilities and sharpen characterization of model strengths and weaknesses.The framework also motivates new architecture designs by pinpointing weaknesses such as linear models’ reasoning-depth limits and compression inefficiencies.Source: §9 · pp. 30–31

Appendix

  • The appendix documents reproducible protocols, evaluations, architectures, ablations, and variant comparisons across five synthetic tasks, real-life experiments, and 100B-token pretraining.The synthetic-data code is intended for release, while experiments used combinations of A100, H100, and H200 GPUs with bf16 mixed precision.Source: Appendix · p. 32, A Details on Synthetic Pretraining Tasks · p. 32

A.1 Details on Task Depo: Mental Reasoning Depth · A.2 Details on Task Brevo: Mental Reasoning Breadth · A.3 Details on Task Capo: Knowledge Capacity

The paper defines three controlled synthetic tasks that isolate reasoning depth, reasoning breadth, and knowledge capacity. Depo and Brevo use structured graph traversal, while Capo measures reliably stored knowledge in bits per parameter under a controlled biography-pretraining setup.

A.1 Details on Task Depo: Mental Reasoning Depth

  • Depo measures mental reasoning depth by hiding randomly shuffled permutation edges and requiring each query’s k-th successor, with evaluation at maximum and intermediate depths.Problem sizes range from n = 3 to N, and each instance includes up to 10 queries; evaluation fixes n = N and tests k = K and k = K/2.Source: §A.1 · pp. 32–33
  • Depo’s two variants separately test ordinary node encoding and ambiguity from multi-token names, with Depo2 using 5–7-token names from a vocabulary of 4.Depo1 uses 1–2-token names and vocabulary size 50, whereas Depo2 introduces longer, ambiguous names to challenge disambiguation.Source: §A.1 · pp. 32–33
  • Its 1/√(N+n) sampling distribution emphasizes simpler cases early, supporting gradual difficulty while avoiding extra curriculum-related noise in architectural comparisons.The authors note that synthetic data cannot fully replicate real-world distributions but is intended to model an ideal future training regime with improved data or RL-based post-training.Source: §A.1 · p. 33

A.2 Details on Task Brevo: Mental Reasoning Breadth

  • Brevo measures mental reasoning breadth by requiring all vertices recursively reachable from a DAG query to be generated in a valid topological order.The task uses bounded-degree DAGs and queries from the last quarter of vertices to preserve meaningful dependency structure and avoid degenerate cases.Source: §A.2 · p. 34
  • Brevo evaluation accepts any complete valid topological ordering, but rejects non-reachable, duplicate, missing, or invalid tokens without partial credit.The protocol evaluates only the largest graph size, n = N, and validates the generated answer sequence against reachability and ordering requirements.Source: §A.2 · p. 34
  • Comparing Brevo1 with Brevo2 shows that ambiguity from multi-token vertex names does not noticeably affect architectural comparisons.Brevo1 uses unique single-token names, whereas Brevo2 uses 2–4-token names from a vocabulary of 4.Source: §A.2 · pp. 34–35

A.3 Details on Task Capo: Knowledge Capacity

  • Capo borrows the bioS(N) synthetic pretraining task [8], which uses biographies of randomly generated individuals with six attributes and represents each exposure through randomly selected English sentences.Source: §A.3 · p. 35
  • Capo measures knowledge capacity as reliably stored bits normalized by parameter count, using a Pareto frontier across datasets of 50K–2M individuals and models of 1M–500M parameters.Partial attribute recall contributes proportionally to the bit calculation, enabling fine-grained assessment of stored knowledge.Source: §A.3 · p. 36

A.4 Details on Task Mano: Knowledge Manipulation · A.5 Details on Task Lano: Hierarchical Language Structure

Mano tests multi-step internal manipulation of implicitly learned arithmetic knowledge, while Lano tests hierarchical structural reasoning over probabilistic context-free languages. Together, the tasks isolate reasoning that cannot be solved through surface memorization or local sequence patterns.

A.4 Details on Task Mano: Knowledge Manipulation

  • Mano evaluates hierarchical knowledge manipulation through multi-step arithmetic computation without explicit intermediate cues or Chain-of-Thought reasoning.The task tests internal computation rather than memorization of surface patterns.Source: §A.4 · p. 36
  • Mano represents prefix-form arithmetic expressions using addition, subtraction, and multiplication, with all computations performed modulo 23.Operands are integers from 0 to 22, and expressions are unambiguous because prefix notation removes precedence ambiguity.Source: §A.4 · pp. 36–37
  • The Mano factual base comprises three 23×23 arithmetic tables that models must learn implicitly, while special tokens structure expressions, lengths, and answers.Operands are encoded as single tokens, and the tables correspond to addition, subtraction, and multiplication.Source: §A.4 · p. 37
  • Mano uses modular arithmetic and a restricted operator set to keep knowledge manageable while preserving computational depth for hierarchical manipulation.Models are pretrained at maximum expression lengths L = 16, L = 13, and L = 10, with loss applied across all tokens.Source: §A.4 · p. 37

A.5 Details on Task Lano: Hierarchical Language Structure

  • Lano evaluates long-range structural planning over probabilistic context-free languages, requiring global reasoning and dynamic programming rather than local pattern resolution.During pretraining, models must infer structural relationships without access to the underlying grammar rules.Source: §A.5 · p. 38
  • Lano pretraining predicts next tokens from CFG-generated sequences, and evaluation grants accuracy only to fully valid generated sentences.Generation uses a single <bos> prompt, temperature 1, and beam width 1; CFG parsing and KL divergence provide consistent evaluation comparisons.Source: §A.5 · pp. 38–39
  • Lano’s cfg3k and cfg3j datasets extend cfg3f [8] by one grammar level, with cfg3k doubling sequence length and increasing parsing complexity eightfold.cfg3j reduces the number of rules and provides intermediate difficulty, while both retain probabilistic generation.Source: §A.5 · p. 38

B Details on Other + Real-Life Experiments

This section describes supplementary copying, synthetic retrieval, Babilong, and real-world pretraining experiments. It also reports that simple 2-hop retrieval remained difficult and that 100B-token, 1.3B-parameter pretraining was too noisy to expose subtle architectural differences.

Full Copy

  • Copying the first t = 1, 2, 4 tokens produced results nearly identical to the full 500-token copying experiment.The full-copy task used random permutations of 500 tokens followed by an identical copy, with loss computed only on answer tokens.Source: B Details on Other + Real-Life… · p. 39, Figure 19 · p. 39

Task 1-hop-L and 2-hop-L

  • The 1-hop-L task tested retrieval of randomly generated birth years embedded at up to five positions in random Wikipedia documents, using a natural-English prompt.Its design closely replicated needle-in-a-haystack evaluation [30] while replacing abstract numbers with birth-year statements.Source: B Details on Other + Real-Life… · p. 39
  • Even with L = 0, models largely failed the simplest natural 2-hop in-context birth-year retrieval task.The task linked three birth-year facts to three equivalence statements embedded in random Wikipedia-like documents, with an instructional statement provided.Source: B Details on Other + Real-Life… · pp. 39–40

Babilong

SlimPajama and FineWeb-Edu 100B

  • SlimPajama and FineWeb-Edu supplied over 100B-token-scale data for 1.3B-model pretraining, with matched data order and content across architectures.Runs processed approximately 100.2B tokens, and Llama(RoPE) additionally used eight random seeds to measure variance.Source: B Details on Other + Real-Life… · p. 40

Beyond 100B-1.3B

  • 100B-token pretraining of 1.3B-parameter models was too noisy to reveal subtle architectural gaps, motivating 1–8B models trained on 1–2T tokens in follow-up work.Source: B Details on Other + Real-Life… · p. 40

C Details on Architectures Used … E.1 Insufficiencies of Real-Life Pretraining at Academic Scale

The paper specifies controlled implementations and parameter-matched real-life experiments across Transformer, state-space, and linear-attention architectures, while showing that academic-scale pretraining results vary substantially across random seeds. Canon layers use a residual convolutional design whose initialization and residual link usually improve stability without hurting performance.

C Details on Architectures Used

  • The architecture suite includes Llama RoPE and NoPE variants, GPT2-style MLP comparisons, ALiBi and H-Alibi, Mamba, GLA, GDN, and ReLU2 activation variants.Llama uses gated MLPs and RoPE across hidden dimensions, GPT2 comparisons replace gated with standard MLPs, and Mamba1 was excluded from main results because Mamba2 consistently outperformed it.Source: C Details on Architectures Used · pp. 40–41

C.1 Real-Life Experiments

  • The real-life setup uses Llama2 tokenization with vocabulary size 32,000 and disables embedding-output weight tying for consistency with prior work.Synthetic pretraining generally uses no tokenizers, while Task Capo uses GPT2Tokenizer and tied weights to reduce capacity loss in small models.Source: §C.1 · p. 42, C Details on Architectures Used · p. 42
  • Real-life experiments increase width and depth for Llama, GLA, Mamba2, Mamba2(mlp), and GDN to yield approximately 1.35B parameters per model.Llama, GLA, Mamba2(mlp), and GDN use 24 layers with hidden size d = 2048, while Mamba2 uses 48 layers with hidden size d = 2048.Source: §C.1 · p. 42

C.2 Canon Implementations

  • Canon layers are implemented as causal Conv1D modules applied after normalization and before nonlinearities, with types A, B, C, and D distinguished by placement and residual design.The paper also defines Canon-b as the original non-residual convolution inside GLA, GDN, and Mamba2, and cst-Canon as an untrained fixed-weight variant.Source: §C.2 · pp. 42–43
  • Canon’s default initialization combined with its residual link usually improves performance stability without ever hurting it.Canon layers use PyTorch’s default O(1)-scale initialization, unlike the 0.02 initialization used in many GLA/GDN and other linear-model convolutions.Source: §C.2 · p. 43

D Extensions of Figures 8 and 16

  • The paper extends earlier synthetic-task figures with training curves for RoPE models across deposition tasks, graph sizes, depths, and reasoning widths.Figure 20 varies depth and maximum graph size for Depo2(K = 16) and Depo1(K = 8 or 4), while Figure 21 varies k and maximum n for Depo1 and Depo2.Source: Figure 20 · p. 44, Figure 21 · p. 45

E.1 Insufficiencies of Real-Life Pretraining at Academic Scale

  • Academic-scale real-life pretraining shows substantial benchmark variance across random seeds, even when data order is fixed and only model initialization changes.The experiments use 1.3B-parameter models trained on 100B tokens; HellaSwag and wiki-ppl are relatively stable, but perplexity alone is unreliable for judging capability.Source: Figure 22 · p. 46, §E.1 · p. 46

E.2 Complete Real-Life Experiments

Complete real-life experiments show that Transformers outperform linear models on multi-hop reasoning, while reducing RoPE improves long-context accuracy. Many other architectural differences remain statistically insignificant or buried in noise.

E.2 Complete Real-Life Experiments

  • Reducing RoPE improves length generalization, whereas standard versus gated MLPs and SiLU versus ReLU2 produce differences buried in noise.GPT2(RoPE) and GPT2(RoPE,R2) comparisons leave the key conclusions unchanged.Source: Figure 23 · p. 47
  • Transformers consistently outperform linear models on Babilong + S-NIAH multi-hop reasoning, while reduced RoPE variants improve long-context accuracy.S-NIAH can make linear models appear accurate because it is too easy, but they fail short-context 1-hop retrieval.Source: Figure 24 · p. 48

F More Synthetic Experiments

Additional synthetic experiments show that simple Canon variants are effective across model families, while reduced RoPE usage can outperform both full RoPE and NoPE. They also clarify how MLP type and ReLU2 affect performance.

Constant Canon and GPT2

  • A constant Canon averaging the previous three tokens already strongly outperforms vanilla Llama and also performs well in GPT2 models.The constant implementation is random and non-trained, while GPT2 uses a standard MLP.Source: Figure 25 · p. 49

MLP activation variants

  • Gated MLP outperforms standard MLP with silu, ReLU2 slightly benefits standard MLP, and ReLU2 harms gated MLP performance.These comparisons are reported for both model settings shown in the figure.Source: Figure 26 · p. 49

RoPE configurations

  • Reducing RoPE usage to one-quarter outperforms both full RoPE and NoPE with Canon layers, while one reduced-RoPE variant performs slightly best overall.Canon layers eliminate the need for extensive RoPE usage; among reduced-RoPE variants, the configuration labeled RoPEˇ “ achieves slightly better overall performance.Source: F More Synthetic Experiments · p. 50, Figure 27 · p. 50

G Complete Ablation Studies … G.5 GDN family

The complete ablation studies provide supplementary KL-divergence evaluations for Task Mano and detailed architectural comparisons across Llama, Mamba2, GLA, and GDN families. The figures examine Canon placement, residual connections, convolutional alternatives, feature maps, and initialization choices.

G Complete Ablation Studies

  • The appendix reports full ablation results, including KL-divergence evaluations for Task Mano omitted from the main text for clarity.These experiments provide additional technical insight and completeness for readers seeking deeper results.Source: G Complete Ablation Studies · p. 50

G.1 Llama(RoPE) family

  • Llama(RoPE) ablations compare multiple Canon layer configurations, including Canon applied only to the first layer.The study also compares Canon-B(no-res) with Primer [60] and finds Canon layers are not tied to Attention, while multi-point Canon(res) is safe and more effective.Source: Figure 28 · p. 51, Figure 29 · p. 51

G.2 Llama(NoPE) family

  • Llama(NoPE) ablations evaluate the original model, varied Canon placements, and Canon applied only to the first layer.These configurations provide technical detail on how Canon variants behave in the NoPE family.Source: Figure 30 · p. 52

G.3 Mamba2 family

  • Mamba2 ablations vary Canon layers, Canon residuals, original nonlinear conv1d, and mimetic initialization across model variants.Additional comparisons include removing conv1d and testing Canon-AB, Canon-Ab, and Canon-ACD configurations with and without residuals.Source: Figure 31 · p. 53, Figure 32 · p. 54, §G.3 · p. 54

G.4 GLA family

  • GLA ablations compare Canon and conv1d layers, residual versus non-residual designs, and identity versus nonlinear feature maps.The nonlinear feature map is specified as ϕ(x) = elu(x) + 1.Source: Figure 34 · p. 55, Figure 35 · p. 55

G.5 GDN family

  • GDN ablations compare no conv1d, the original conv1d model, and Canon-ABCD, Canon-AbCD, and Canon-ACD variants with residual choices.The study uses 12L768D GDN models and explicitly varies residual versus non-residual Canon designs.Source: Figure 36 · p. 56, Figure 37 · p. 57

Related papers

Loading 2512.17351v2…