PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers

Zeyuan Allen-Zhu

arXiv:2512.17351v2cs.CL

TL;DR

Architectural comparisons are difficult because academic-scale pretraining is noisy and complex skills may not emerge reliably. The paper introduces controlled synthetic tasks and Canon layers to isolate capabilities and improve horizontal information flow, finding stronger reasoning and broader architectural performance gains while identifying efficiency and scale boundaries.

  • Problem

    Academic-scale architectural comparisons are unreliable because noise, randomness, mixed skills, and failed multi-hop emergence obscure meaningful capability differences.

  • Method

    The paper decomposes intelligence into controlled synthetic tasks and introduces Canon layers that add weighted horizontal information flow across neighboring token representations.

  • Results

    Canon layers improve deep reasoning and other capabilities across architectures; Transformers with Canon achieve near-perfect 1- and 2-hop accuracy quickly, whereas linear models struggle with compression and retrieval errors.

  • Takeaways & Limitations

    Synthetic benchmarks provide a cost-effective way to expose architectural strengths and optimize lightweight information-flow components across diverse sequence models.

  • Takeaways & Limitations

    With unlimited training data and steps, vanilla RoPE can eventually reach comparable Depo2(K=16) accuracy; Canon primarily improves training efficiency.

Abstract

from arXiv · show

Understanding architectural differences in language models is challenging, especially at academic-scale pretraining (e.g., 1.3B parameters, 100B tokens), where results are often dominated by noise and randomness. To overcome this, we introduce controlled synthetic pretraining tasks that isolate and evaluate core model capabilities. Within this framework, we discover CANON LAYERS: lightweight architectural components -- named after the musical term "canon" -- that promote horizontal information flow across neighboring tokens. Canon layers compute weighted sums of nearby token representations and integrate seamlessly into Transformers, linear attention, state-space models, or any sequence architecture. We present 12 key results. This includes how Canon layers enhance reasoning depth (e.g., by $2\times$), reasoning breadth, knowledge manipulation, etc. They lift weak architectures like NoPE to match RoPE, and linear attention to rival SOTA linear models like Mamba2/GDN -- validated both through synthetic tasks and real-world academic-scale pretraining. This synthetic playground offers an economical, principled path to isolate core model capabilities often obscured at academic scales. Equipped with infinite high-quality data, it may even PREDICT how future architectures will behave as training pipelines improve -- e.g., through better data curation or RL-based post-training -- unlocking deeper reasoning and hierarchical inference.

1 Introduction

The paper addresses unreliable architectural comparisons by decomposing intelligence into controlled synthetic tasks, then uses them to study Canon layers that improve information flow and capabilities across architectures.

  • Motivation and approach: Synthetic tasks isolate capabilities such as reasoning depth and breadth, enabling economical, controlled architectural comparisons where real-world pretraining is noisy, random, and skills-mixed.They target failures from benchmark variance, emergence thresholds, and unstable data difficulty.
  • Canon layers: Canon layers add horizontal information flow across neighboring tokens and can be inserted before or inside attention and MLP sublayers.The design uses weighted neighboring representations and can be integrated across sequence architectures.
  • Main results: 200–400% reasoning-depth gains, 30% reasoning-breadth gains, and 30% knowledge-manipulation-length gains are reported with minimal computational overhead.The paper attributes these improvements to enhanced hierarchical learning dynamics.
  • Main results: Canon layers revive NoPE models, often matching or surpassing RoPE(+Canon), while reducing or removing RoPE improves length generalization.They also outperform positional fixes such as ALiBi and H-Alibi in the reported comparisons.
  • Main results: Across GLA, Mamba2, and GDN, Canon layers consistently improve reasoning, knowledge manipulation, and structural capabilities, though gains vary by architecture.For GLA, Canon raises reasoning depth from 1 to 4 hops and doubles reasoning breadth and knowledge length; Mamba2’s conv1d accounts for much of its gain.
  • Academic-scale validation: At 1.3B parameters and 100B tokens, Canon improves NoPE and GLA to match RoPE and Mamba2/GDN, but linear models still lag Transformers on retrieval-heavy tasks and all models fail 2-hop reasoning.These real-world results mirror the synthetic trends while demonstrating limited resolution at academic scale.

2 Synthetic Tasks for Decomposing Intelligence

The paper decomposes intelligence into five synthetic tasks that isolate distinct capabilities, enabling controlled architectural comparisons with reduced confounds. The tasks span reasoning depth and breadth, knowledge capacity and manipulation, and hierarchical language structure.

  • Design Criteria: Synthetic tasks target non-shallow, internally reasoned, broadly relevant capabilities while avoiding reliance on Chain-of-Thought and unstable length generalization.The design criteria prioritize architectural skill over shallow copying, external tools, or context-length specialization.
  • Task Design: Five synthetic tasks isolate reasoning depth, reasoning breadth, knowledge capacity, knowledge manipulation, and hierarchical language structure.They are named Depo, Brevo, Capo, Mano, and Lano.
  • Reasoning Tasks: Depo evaluates k-hop traversal for reasoning depth, while Brevo requires globally processing a dependency graph before generating its topological outputs.Brevo therefore tests simultaneous dependency processing rather than simple local recall.
  • Knowledge Tasks: Capo measures factual storage capacity in bits per parameter, whereas Mano tests internal retrieval and hierarchical manipulation of stored knowledge through multi-step modular arithmetic.Mano excludes intermediate Chain-of-Thought steps and varies expression difficulty through maximum length L.
  • Coverage: The five-task pool covers non-overlapping skills and token-level, generative, and distributional accuracy dimensions, providing a starting point for architectural insight.The authors characterize the pool as extensible rather than exhaustive.

3 Initial Comparison on Well-Known Base Architectures

The initial comparison uses representative Transformer, linear-attention, and state-space architectures under standardized training and model-size settings. Results show distinct capability profiles, but the comparison is incomplete because some architectures lack a critical shared component and hybrids are excluded.

  • Architectures: The study compares quadratic-time Transformers, linear-time GLA, and recurrent or state-space Mamba2 and GDN as representative architecture families.Hybrid architectures are omitted to preserve clarity and expose isolated module trade-offs.
  • Initial Results: GLA performs weakest overall; Mamba2 excels on knowledge, GDN improves reasoning, and RoPE leads on some depth and structural-reasoning tasks.GDN surpasses Llama(RoPE) on Brevo but not on all reasoning tasks.
  • Experimental Setup: Models use standardized parameter configurations, identical training settings, and fixed seeds so architectures train on the same data sequences.This controls model size, optimization conditions, and data order across comparisons.
  • Comparison Validity: The initial comparison is partially unfair because Mamba2 and GDN include internal conv1d layers while GLA and Transformers do not.The paper later adds Canon layers to all models to make the comparison scientifically meaningful.
  • Interpretation: Synthetic tasks can expose large capability differences more clearly than noisy real-world experiments, but fixed-budget failures indicate differences in skill-acquisition speed rather than absolute inability.The authors also note that future data curation or RL-based post-training may change how these capabilities appear.

4 Canon Layers: Enhancing Horizontal Information Flow

Canon layers add lightweight horizontal information flow by mixing nearby token representations, complementing attention and integrating across Transformer, linear-attention, and state-space architectures.

  • 1-layer RoPE + Canon achieves 100% token-copying accuracy at d = 16, matching 2-layer RoPE while 1-layer RoPE requires d ≥128.The experiment copies tokens across sequences of length 500.
  • Canon layers aggregate nearby hidden states within a fixed window, providing lightweight local mixing unlike global attention or recurrent compression.The general form uses element-wise weighted sums of the current token and the previous three positions.
  • Canon layers integrate at four Transformer locations—before attention, inside attention, before the MLP, and within the MLP—and can be combined as Canon-ABCD.They also integrate into linear-attention and state-space models, including Mamba2 variants.
  • Canon generalizes earlier uniform token mixing through learned, position-specific short-window mixing without value or projection matrices.This design emphasizes modularity and efficiency relative to heavier convolutional modules used for feature extraction.
  • Canon transforms NoPE into a strong performer on par with RoPE, while reduced RoPE plus Canon can outperform both RoPE and NoPE with Canon.The reported configuration enables RoPE for one-quarter of dimensions and supports length generalization.
  • Canon layers enhance horizontal information flow even with simple or random-weight implementations, while adding fewer than 0.45% of GPT2-small’s parameters.The paper implements them as causal convolutions with kernel size 4 and explicit residual connections.

5 When Transformer Meets Canon

Canon layers improve reasoning, knowledge, and hierarchical-language performance across tasks, with the largest gains arising from faster hierarchical feature learning rather than added representational power.

  • 5.1 Main Results: 30% broader reasoning and 30% longer manipulable knowledge sequences show Canon’s gains across breadth and knowledge manipulation.Canon matches vanilla RoPE on harder Brevo and Mano settings while preserving performance on simpler settings.
  • 5.1 Main Results: 10–15% higher effective knowledge capacity results from faster, more stable training in the controlled Capo regime.Canon offers an alternative to replacing gated MLPs with standard MLPs, which sacrifices reasoning capability.
  • 5.2 Why Canon Helps: Hierarchical Feature Learning: 2–4× reasoning-depth gains arise because Canon layers accelerate hierarchical feature learning, not merely local representation.Canon makes multi-token segments legible at shallow layers and supports progressive learning of deeper functions.
  • 5.2 Why Canon Helps: Hierarchical Feature Learning: Using Canon in only the first block recovers little of the full gain, indicating that its benefits must be applied throughout the network.The contrast is strongest on the deepest Depo task and for NoPE models.
  • 5.3 NoPE with Canon Layers: Canon layers substantially improve NoPE, bringing it to competitive levels and sometimes beyond RoPE+Canon on reasoning depth.NoPE+Canon also matches or surpasses RoPE+Canon on real-world pretraining and outperforms ALiBi and H-Alibi fixes.
  • 5.4 Ablation Studies With Canon Layers: Canon components contribute cumulatively, while residual connections stabilize and accelerate training independently of attention or MLP modifications.Canon-ACD, Canon-ABC, and Canon-AC outperform Canon-B(no-res), and removing residual links makes training slower and less stable.

6 When Linear Models Meet Canon

Canon layers add localized horizontal information flow to linear architectures, substantially improving reasoning and knowledge-related capabilities. Their benefits are broadly architecture-independent and often remain effective outside the recurrent or state-space sublayer.

  • GLA: Canon layers raise GLA from a weak baseline to a strong competitor, reaching or surpassing Mamba2 and improving reasoning depth, breadth, and manipulation length.Performance improves from 1-hop to 4-hop reasoning, doubles reasoning breadth, and more than doubles knowledge manipulation length.
  • Ablations: Ablations show that Canon placement and residualness are task-dependent, while combining positions generally performs best and simple residual configurations remain recommended.GLA favors non-residual variants for Mano but residual variants for Lano and Brevo1; Canon-AbCD(res) is recommended for GLA and Mamba2.
  • Mamba2: Mamba2’s internal conv1d contributes more to performance than its SSM formulation, while full Canon-AbCD further improves results, especially on Mano and Lano.Removing conv1d reduces Mamba2 to GLA-level performance on synthetic and real-world datasets.
  • Cross-architecture findings: Canon layers remain effective outside recurrent or state-space blocks, showing that horizontal information flow is not intrinsic to the attention or SSM layer.Canon-ACD surpasses Mamba2’s conv1d on Depo2 and Lano, while Canon-ACD performs on par with GDN+conv1d.
  • GDN: GDN depends less on its internal conv1d than GLA or Mamba2, yet Canon remains useful outside the GDN layer and matches its conv1d performance.Removing GDN’s conv1d destroys Mano and Lano while leaving Depo and Brevo largely unaffected.

7 Final Comparisons and Lessons for Architecture Design

Controlled comparisons show that Canon layers expose common strengths and weaknesses across linear architectures. Linear models retain advantages in knowledge capacity but struggle with deep reasoning because compressed-memory encoding and retrieval accumulate errors.

  • Architecture design: Canon-ACD matches internal conv1d across linear architectures, never hurts performance, and suggests that much of recent linear-model progress may reproduce Canon-like horizontal mixing.The authors caution that complex modules may learn simpler Canon-like functions, even when their representational forms are more elaborate.
  • Controlled comparison: Full Canon layers reveal a consistent reasoning-depth ranking in which RoPE and NoPE outperform Mamba2, GLA, and GDN by roughly 4× in the reported comparison.The controlled comparison adds Canon layers across architectures, addressing the earlier confound that some models already contained conv1d components.
  • Final comparison: Linear models achieve about 40% higher Capo knowledge capacity than full Transformers, while remaining substantially weaker on reasoning depth.The reported reasoning gap is about 2× on Depo1 and up to 4× on Depo2 under identical training conditions.
  • Reasoning depth: Linear models’ deep-reasoning weakness reflects accumulated compression and retrieval errors rather than insufficient recurrent memory.Mamba2 has enough state capacity to store the full input sequence, and it performs well on 1-hop tasks, but errors compound across reasoning hops.
  • Architecture design: Hybrid models combining Transformers and linear layers with Canon mitigate linear models’ deep-reasoning limitations, motivating sliding-window attention plus compressed long-context components.The proposed direction combines attention for deep reasoning with linear or state-space mechanisms for long-context compression.

8 Real-Life Experiments

Academic-scale real-world pretraining produces noisy, low-resolution architectural comparisons and widespread failure on multi-hop reasoning. Still, Canon consistently improves weaker variants and narrows gaps between architectures.

  • Multi-hop reasoning: All models achieve only 30–36% on 2-hop-L even at zero junk length, showing that academic-scale pretraining does not reliably produce multi-hop reasoning.The result persists even within 100 tokens, indicating a limitation of this training regime rather than only long-context noise.
  • Overall findings: At 1.3B parameters and 100B tokens, most architectural differences are statistically unresolved, although several consistent patterns remain.Seed variation reaches 4% on LAMBADA, 3% on BoolQ, 1–3% elsewhere, and up to 9% on generative FDA.
  • Retrieval and generation: Linear models underperform full Transformers on short-context retrieval and generative tasks, even when equipped with Canon.The pattern appears on FDA, SWDE, and 1-hop-L with contexts of at most 500 tokens.
  • Overall findings: Canon elevates NoPE to RoPE-level performance and GLA to Mamba2/GDN-level performance, while removing Mamba2’s conv1d reduces it to GLA-level performance.GDN is comparatively insensitive to Canon, and most linear+Canon variants cluster together at this scale.
  • Position encoding: Reducing or removing RoPE improves long-context generalization, particularly in 4k-token junk passages.The reported improvements occur for NoPE and reduced-RoPE Transformers in the long-context experiments.

9 Conclusion and Future Direction

The paper presents synthetic benchmarking and Canon layers as a cost-effective framework for isolating architectural capabilities and guiding design. It also emphasizes that larger-scale validation, richer tasks, and broader analyses remain necessary.

  • Conclusion: The synthetic playground decomposes intelligence into atomic tasks and uses Canon layers to improve reasoning, knowledge, and structural capabilities across architectures.The framework is intended to reveal architectural strengths and weaknesses that academic-scale real-world training obscures.
  • Conclusion: Canon layers revive weaker architectures, reduce reliance on positional encoding, and identify linear models’ depth bottleneck as compression and retrieval inefficiency rather than memory capacity.The authors characterize Canon as a potentially minimal, broadly applicable architectural primitive.
  • Limitations: Industrial-scale validation remains necessary because the reported real-world experiments are limited to academic-scale pretraining.The authors plan to release the playground and evaluation suite to support larger and more reproducible studies.
  • Future directions: Future work should test adaptive convolutions, finer-grained Canon configurations, additional architecture families, and synthetic tasks targeting capabilities beyond the current five.The paper specifically proposes state-dependent dynamic convolutions and systematic evaluation of selective or cross-layer Canon connections.
  • Limitations: The study omits interpretability and probing analyses from the main presentation, despite preliminary probing results for Depo.The omission is attributed to the paper’s focus on architectural comparison.

A.1 Details on Task Depo: Mental Reasoning Depth

Depo isolates mental reasoning depth by requiring models to traverse shuffled directed permutations and answer successor queries at controlled depths. Its variants and evaluation protocol vary name ambiguity, graph size, and reasoning depth to compare architectures under synthetic pretraining.

  • Training distribution: The task samples permutation sizes from 3 through N and uses a 1/√(N+n)-style distribution to emphasize easier cases early in training.The authors describe this curriculum as simple and effective for controlled architectural comparisons.
  • Task design: Depo requires computing each query node’s k-th successor in a randomly shuffled directed permutation.The shuffled edges conceal the cycle structure, preventing trivial recovery from sequence order.
  • Task variants: Depo1 uses short node names, whereas Depo2 uses ambiguous 5–7-token names from a four-token vocabulary.The second variant tests disambiguation in addition to multi-step traversal.
  • Training protocol: The framework uses online generation, label masking, left-aligned 2048-token windows, and fixed-seed training to reduce cost and comparison variability.Loss is computed only on answer-related tokens, while left alignment avoids truncating the first instance.
  • Evaluation: Evaluation tests both maximum depth k = K and intermediate depth k = K/2 with fixed permutation size n = N.Accuracy is computed across all answer tokens in online-generated 2048-token windows.

A.4 Details on Task Mano: Knowledge Manipulation

Mano tests hierarchical knowledge manipulation by requiring multi-step modular-arithmetic computation without explicit intermediate cues. The task learns arithmetic tables implicitly and varies expression depth while using exact single-token answers.

  • Task purpose: Mano evaluates internal multi-step manipulation of stored knowledge without relying on explicit chain-of-thought cues.It is designed to distinguish hierarchical computation from simple memorization.
  • Task format: Expressions use prefix notation with addition, subtraction, and multiplication computed modulo 23.The format removes ambiguity from parentheses and operator precedence.
  • Knowledge representation: Models implicitly learn three 23×23 arithmetic tables during pretraining and output exact modular results for generated expressions.Operands are single tokens, and special tokens encode sequence structure and requested length.
  • Design rationale: Modular arithmetic keeps the knowledge base manageable while preserving diverse intermediate and final results for depth-focused computation.The restricted operation set simplifies task design without eliminating hierarchical structure.
  • Training setup: Training varies maximum expression length across L = 16, 13, and 10 while applying loss to every sequence token.Full-sequence loss is used because hierarchical manipulation requires attention across the complete expression.

B Details on Other + Real-Life Experiments

The paper supplements its synthetic playground with copying and real-life retrieval experiments, including minimal one-hop and two-hop birth-year tasks. These experiments emphasize that simple multi-hop retrieval can remain difficult at academic scale, while larger-scale pretraining is needed to resolve subtle architectural gaps.

  • Copying task: The copying experiment measures exact reproduction of a 500-token permutation and separately tests copying the first t = 1, 2, 4 tokens.The partial-copy results are reported as nearly identical to the full-copy results.
  • Real-life experiments: Even the simplest natural 2-hop retrieval task largely fails at L = 0, despite directly stating birth-year facts and equivalences.The task embeds three direct facts and three bijective equivalence statements in context.
  • Pretraining setup: Real-world pretraining uses SlimPajama and FineWeb-Edu 100B-token datasets, with architectures trained for 510,000 steps on 4096-token windows.The setup uses a shared optimizer configuration and architecture-specific learning-rate ranges.
  • Controlled comparison: Shared random seeds keep data order and content identical across architectures, while additional Llama(RoPE) seeds quantify variance.This design aims to isolate architectural effects from data and initialization differences.
  • Scale boundary: Academic-scale pretraining with 100B tokens and 1.3B-parameter models is too noisy to expose subtle gaps such as Llama versus Llama+Canon.The paper reports larger 1–8B-model experiments trained on 1–2T tokens separately.

C Details on Architectures Used

The paper compares Transformer, state-space, and linear-attention architectures under parameter-matched configurations, including positional-encoding and convolutional variants. Implementation choices such as initialization and Canon residual links materially affect stability and are tested alongside the main architectures.

  • Transformer variants: Llama variants differ by positional encoding, including RoPE, NoPE, and reduced-dimensional RoPE configurations.The paper also compares GPT2-style models and ReLU2 activation variants.
  • State-space variants: Mamba2 alternates state-space and gated-MLP layers in its Mamba2(mlp) variant to match parameter counts with Llama-style blocks.The comparison uses an intermediate size of 2d for the gated MLP.
  • Convolutional components: Mamba1 is excluded from the main results because Mamba2 consistently outperformed it, while removing Mamba1’s convolution degraded performance to GLA-level.This makes the convolution a tested architectural component rather than a neutral implementation detail.
  • Model configurations: Real-life models are configured near 1.35B parameters, using 24-layer Llama, GLA, Mamba2(mlp), and GDN models alongside a 48-layer Mamba2.All listed configurations use hidden size d = 2048, with architecture-specific head counts.
  • Initialization and Canon: Default initialization combined with Canon’s residual link usually yields a stable performance improvement without observed regressions.The paper contrasts this behavior with implementations whose weights use O(1) rather than 0.02-scale initialization.

E.2 Complete Real-Life Experiments

The complete real-life experiments compare architectures on multi-hop reasoning and find reduced RoPE beneficial, while many architectural differences remain statistically insignificant or noisy.

  • Reducing RoPE improves length generalization, while standard versus gated MLPs and SiLU versus ReLU2 remain buried in noise.
  • Reduced RoPE improves long-context accuracy, while many architectural comparisons remain statistically insignificant.
  • Linear models consistently underperform Transformers, including in short contexts without junk.

F More Synthetic Experiments

These synthetic experiments examine Canon implementations, MLP variants, and RoPE configurations, showing that simple Canon layers help and reduced RoPE can outperform both full RoPE and NoPE.

  • A fixed random average of the past three tokens already outperforms vanilla Llama, and Canon layers also perform strongly with GPT2 standard MLPs.
  • Gated MLPs outperform standard SiLU MLPs, while ReLU2 slightly helps standard MLPs but hurts gated MLPs.
  • The experiments compare Transformer+Canon across full, partial, and absent RoPE configurations.
  • Reducing RoPE to one-quarter is preferable, outperforming both full RoPE and NoPE; among reduced variants, RoPEˇ “ performs slightly best overall.

G Complete Ablation Studies

The complete ablations examine Canon placement, residual connections, convolution alternatives, and feature maps across Llama, Mamba2, GLA, and GDN variants.

  • Canon variants are evaluated across Llama(RoPE), Llama(NoPE), Mamba2, GLA, and GDN families with different layer placements and residual settings.
  • Canon-B(no-res) is compared with multi-point Canon residual variants, highlighting that Canon layers are not tied to attention and that multiple residual placements are effective.
  • The Mamba2 ablations vary Canon layers, Canon residuals, original nonlinear conv1d, and mimetic initialization.
  • GLA ablations compare Canon and conv1d layers, residual choices, and identity versus nonlinear feature maps.
  • GDN ablations compare no-conv1d and original-conv1d models with several Canon placement and residual variants.
Loading 2512.17351v2…