PaperlayerView exact revision on arXiv

Source-linked AI summary

The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain

Adrian Kosowski, Przemysław Uznański, Jan Chorowski, Zuzanna Stamirowska, Michał Bartoszkiewicz

arXiv:2509.26507v1cs.NEcs.AIcs.LGstat.ML

TL;DR

Reasoning across long timescales remains challenging, and the brain’s microscale reasoning dynamics are not understood while Transformer function is described at the vector level. BDH connects Transformer and brain models; the paper presents it as experimentally validated at scale against GPT2-like Transformers.

  • Problem

    The brain’s reasoning function is not understood at the neuronal-dynamics level, while Transformer function is interpreted at the vector level rather than through particle dynamics or uniform distributed computation.

    Source: §1.1 · p. 4

  • Method

    BDH is a GPU-efficient reasoning-model architecture that links Transformer and brain models, with synaptic potentiation and excitatory and inhibitory circuits represented through spiking dynamics.

    Source: §1 · p. 3, §1.2 · p. 6, §2.5 · p. 16

  • Results

    BDH is presented as experimentally validated at scale in direct comparison with state-of-the-art GPT2-like Transformers.

    Source: §1.2 · p. 6

  • Takeaways & Limitations

    Sparse BDH activation vectors imply that specific synapses are potentiated rarely during inference, a property useful for interpretability and noise reduction in Linear Attention state.

    Source: §6.4 · p. 38

    The kernel was chosen because it worked well and could be trained on GPU; finding optimal kernels under different criteria remains an open foundational problem.

    Source: §2.4 · p. 16

Abstract

from arXiv · show

The relationship between computing systems and the brain has served as motivation for pioneering theoreticians since John von Neumann and Alan Turing. Uniform, scale-free biological networks, such as the brain, have powerful properties, including generalizing over time, which is the main barrier for Machine Learning on the path to Universal Reasoning Models. We introduce `Dragon Hatchling' (BDH), a new Large Language Model architecture based on a scale-free biologically inspired network of \$n\$ locally-interacting neuron particles. BDH couples strong theoretical foundations and inherent interpretability without sacrificing Transformer-like performance. BDH is a practical, performant state-of-the-art attention-based state space sequence learning architecture. In addition to being a graph model, BDH admits a GPU-friendly formulation. It exhibits Transformer-like scaling laws: empirically BDH rivals GPT2 performance on language and translation tasks, at the same number of parameters (10M to 1B), for the same training data. BDH can be represented as a brain model. The working memory of BDH during inference entirely relies on synaptic plasticity with Hebbian learning using spiking neurons. We confirm empirically that specific, individual synapses strengthen connection whenever BDH hears or reasons about a specific concept while processing language inputs. The neuron interaction network of BDH is a graph of high modularity with heavy-tailed degree distribution. The BDH model is biologically plausible, explaining one possible mechanism which human neurons could use to achieve speech. BDH is designed for interpretability. Activation vectors of BDH are sparse and positive. We demonstrate monosemanticity in BDH on language tasks. Interpretability of state, which goes beyond interpretability of neurons and model parameters, is an inherent feature of the BDH architecture.

1 Introduction

This section presents BDH and BDH-GPU as a bridge between Transformer computation and brain-like neuron interactions, framing scale-free dynamics, Hebbian state updates, and graph-based inference as foundations for interpretable reasoning systems.

1 Introduction

  • The paper identifies long-horizon reasoning as a generalization challenge: Transformers do not systematically generalize chain-of-thought reasoning to scenarios longer than those seen during training, motivating a direct Transformer–brain link.It notes that generic Turing-machine simulation of the brain could require billions of chain-of-thought tokens for a single brain reasoning step.Source: §1 · p. 3

1.1 Motivation

  • The motivation is to replace coarse Transformer-level accounts with micro-foundations for inference and a uniform model of temporal behavior, using scale-free limits to make long-run behavior extrapolatable from shorter tests.The paper presents this shift as moving from approximate interpretability toward Axiomatic AI, which aims to explain how micro-foundations shape later behavior.Source: §1.1 · pp. 4–5

1.2 Intuition of results: combining modus ponens reasoning with Hebbian learning

  • BDH combines modus ponens inference with heuristic rule reweighting: fixed connections act as parameters, while dynamically updated connections form Hebbian fast-weight state; BDH-GPU is its GPU-efficient special case.The paper describes the system as local graph dynamics, with synapses mediating communication between neurons.Source: §1.2 · p. 6

1.3 Contribution of this work

  • BDH represents inference as local particle interactions on a graph, with parameters encoded in graph structure and weights and model state encoded by edge reweighting; its local kernel can be emulated by spiking neurons with Hebbian learning.For efficient training, BDH-GPU restricts communication to a mean-field formulation while preserving equivalent inference behavior.Source: §1.3 · p. 7
  • BDH-GPU uses positive, sparse activation vectors in a high-dimensional neuron space; its graph interpretation links attention-based inference to state updates on individual neuron-pair synapses and explains emergent high-modularity communication structure.The introduction also states that BDH-GPU has (3 + o(1))nd parameters and is formally equivalent in inference behavior to a BDH model of the same particle count, up to Layer Norm placement.Source: §1.3 · pp. 7–8
  • BDH-GPU follows optimized GPT Transformer scaling laws on next-token tasks from 10M to 1B parameters, including language and translation.The comparison is at the same parameter scales and training data, as described in the paper context.Source: §1.3 · p. 8
  • A trained BDH-GPU’s parameter matrices directly reveal an emergent network reflecting its graph dynamics.This is presented as evidence of emergent graph structure.Source: §1.3 · p. 8
  • About 5% of BDH-GPU state-space y-vector activations are positive and sparse, with sparsity reflecting activity for a given token.The passage attributes these observed effects to design choices including ReLU thresholds, high-dimensional linear attention, and model dimensions with comparable model-to-state ratio; L1 regularization was disabled.Source: §1.3 · p. 8
  • Across multiple prompts, BDH-GPU in-context attention state localizes consistently on the same synapses, enabling basic feature interpretation from an individual synapse’s state.The interpretation concerns a feature associated with that synapse.Source: §1.3 · p. 8
  • BDH-GPU scales uniformly in one dimension, n, allowing concatenated models within the same architecture and study of reasoning-model limit behavior.The paper also frames BDH as a way to study lifelong inference from split-second dynamics through minutes of thought and language, but does not explain longer-timescale transfer to long-term memory.Source: §1.3 · p. 8
  • BDH and BDH-GPU fuse particle-based behavior with logic-programming behavior, bridging Transformer tensor operations and brain-model neuron interactions.Figure 1 describes attention and feed-forward computation at the tensor level for Transformers and through local graph dynamics for brain models, while BDH-GPU admits both representations.Source: §1.3 · p. 8, Figure 1 · p. 9

1.4 Notation

  • Inference is represented by a changing state σ(t) driven by architecture A and fixed model M; M is also called a program, while nonnegative matrices and vectors define weighted graphs and linear dynamics.For stochastic matrices, the graph transition is a Markov chain that preserves the vector’s L1 norm.Source: §1.4 · p. 9

2 BDH: a language model architecture given by local distributed graph dynamics

This section presents BDH as a local graph system, describing how node computations and pulse-based neighbor communication use context-sensitive attention and evolving synaptic state. It connects these dynamics to brain-like mechanisms for language and reasoning while noting limits and open questions about the kernel and learning timescales.

2.1 Formalism for local graph-based language models

  • BDH uses an edge-reweighting kernel: local rules update node state or communicate among two neighboring nodes and their connecting edge.The formalism assigns non-negative state variables to nodes and edges; the graph’s trainable communication structure acts as a rule-based program. The distributed-system interpretation is limited to non-negative graph weights.Source: §2.1 · pp. 10–11

2.2 Definition of BDH as a local edge-reweighting process (equations of reasoning)

  • BDH’s local graph protocol is equivalent to its state-space dynamics, using rapid neuron pulses and synaptic state variables on graph edges.Inference schedules rules in round-robin order, with tokens ingested and results read every 4L rounds; edge state σ is distinguished across layers to balance interpretability and system size.Source: §2.2 · p. 12, Table 1 · p. 13

2.3 Interpretation of attention as a micro-inductive bias of reasoning

  • BDH attention is a context-derived bias for considering an implication, not a truth value, and weights how strongly activation can propagate from neuron i to neuron j.Past context contributes synaptic state σ(i, j); current activation at i is multiplied by that state to determine the weighted inference to j.Source: §2.3 · p. 14

2.4 Interpretation of BDH as an oscillator network toy-model

  • The oscillator-network toy model represents BDH’s synapses as elastic connectors whose state changes more slowly than pulse-like node activations.Connector topology follows Gs, while activation x and y appears and vanishes on a faster timescale.Source: §2.4 · p. 15
  • BDH’s toy model combines fast node pulses and slower connector tension, with thresholded graph communication and attention state on connections that relaxes over time.Node pulses propagate through prods and wires, while their activations alter connector tension.Source: §2.4 · pp. 15–16
  • The toy model propagates state changes across 3-hop neighborhoods, while sparse pulses can make synaptic-state updates infrequent and produce heavy power-law-like intervals between updates.The update intervals may have different distribution parameters for different connection pairs.Source: §2.4 · pp. 15–16
  • 10^2–10^5 times greater cost- and time-effectiveness made the chosen BDH-GPU kernel practical relative to kernels trainable only on CPU, but finding optimal kernels remains open.The paper identifies kernel minimality, training rate per token, and closeness to brain function as possible optimization criteria.Source: §2.4 · p. 16

2.5 Expressing BDH using brain models

  • A distributed system that efficiently emulates BDH’s local kernels would be expressive enough for language inference and reasoning to at least BDH’s level.The paper frames this as a corollary about brain models capable of emulating those kernels.Source: §2.5 · p. 16
  • BDH’s local rules can be expressed through positive-state neuron activation, Hebbian learning, and excitatory and inhibitory circuits with thresholding.The rules include local integrate-and-fire thresholding, neuron competition, synaptic potentiation following endpoint activations, and later signal transmission with synaptic strength decrease.Source: §2.5 · p. 16
  • BDH’s Hebbian learning with neural circuits is plausibly needed and sufficient for brain-scale reasoning, including language with attention and thought over minutes.The paper grounds this claim in BDH’s reported Transformer-level performance, but presents it as a plausibility claim about the brain.Source: §2.5 · p. 17

3 BDH-GPU: a tensor-friendly version of the BDH architecture

This section presents BDH-GPU as a tensor-based, GPU-compatible implementation of BDH and explains how its thresholded matrices define the neuron communication graph and relate to BDH graph models.

3 BDH-GPU: a tensor-friendly version of the BDH architecture

  • BDH-GPU provides a GPU-compatible, PyTorch-implementable version of BDH that can be trained by backpropagation and is reported to match GPT-based LLM performance.The paper describes it as a tensor-friendly variant of BDH.Source: §3 · p. 17

3.1 Notation for BDH-GPU

  • BDH-GPU(n, d) scales with n particles, with d measuring parameters per neuron and assumed to satisfy n ≫ d > C log n for sufficiently large C.In the tensor representation, d-dimensional vectors act as fuzzy addresses in a virtual memory space of size n.Source: §3.1 · p. 17

3.2 Definition of BDH-GPU as a state-space system

  • BDH-GPU is defined as a state-space system that processes language tokens one per time step, using learned token encoders and decoders and trainable matrices E, Dx, and Dy.Its scalable parameter count is 3nd, and its state-space form supports a Transformer-like token-parallel training framework.Source: §3.2 · p. 18

3.3 Interpretation of BDH-GPU as a local interacting particle system

  • BDH-Normfree particles interact through a shared mean-field message, with O(dL) state parameters per particle and O(d) parameters in its representation.The architecture scales uniformly in n in chunks of k particles, where k is the largest block size in U; transformations between length-n vectors pass through an intermediate representation of dimension at most d.Source: §3.3 · p. 20

3.4 Expressing BDH-GPU using BDH: preserving parameter and state size

  • BDH-Normfree and BDH are formally equivalent under a special BDH parameter choice, up to LayerNorm placement, and BDH is not less expressive than its tensor-based counterpart.The paper states that BDH and BDH-GPU have equivalent communication and computation patterns for a special parameter choice.Source: §3.4 · p. 21

3.4.1 Expressing matrices Dx, Dy, E as graphs Gx, Gy

  • BDH graph mechanisms can represent the tensor model’s matrix-pair operations with O(nd) parameters, and are strictly more expressive at the same asymptotic parameter count.A graph interaction can be implemented through two linear-dynamics steps on a sparse circuit with O(m) parameters.Source: §3.4.1 · pp. 21–22

3.4.2 Expressing BDH-GPU attention on graphs: sparsification and trainability of Gs

  • BDH can express BDH-Normfree attention with a graph Gs of O(nd) edges, and has at least its expressiveness with the same O(nd) parameter count and O(nd) state size per layer.Unlike BDH-GPU, BDH can make Gs trainable; although the direct correspondence uses a dense graph, a sparse Gs suffices for the attention operation after state compression.Source: §3.4.2 · p. 22
  • After thresholding, BDH-GPU matrices DxE, DyE, and σ can be interpreted as n-node directed graphs, defining the BDH neuron-neuron communication graph via Eq. (9).The visualizations show σ and Gx := DxE as graph representations.Source: §3.4.2 · pp. 22–23

4 Implementation and scaling laws

BDH-GPU is presented with implementation guidance and compared with Transformer-like and other sequence architectures. Across the tested tasks, it retains Transformer-like advantages and generally compares favorably at matched parameter scales.

4 Implementation and scaling laws

  • An Appendix E code framework accompanies BDH-GPU, along with hyperparameter guidance and empirical comparisons with other language-model architectures.The section describes an empirical study of BDH-GPU models and comparisons to Transformers and other architectures.Source: §4 · p. 23

4.1 Implementation characteristics of BDH-GPU

  • BDH-GPU has three main n×d parameter matrices, scales primarily through neuronal dimension n, and shares its weights across layers.The parameter count is precisely (3+o(1))nd.Source: §4.1 · p. 23
  • Each layer has an independent, persistent n×d state for linear attention, and typical activations are nonnegative with only about 5% of entries nonzero.The state has an associative-memory interpretation, and the active fraction corresponds to the state space read and updated per token.Source: §4.1 · pp. 23–24

4.2 Comparison of BDH-GPU to GPT2-like Transformers

  • BDH-GPU uses fewer parameter matrices, scales mainly in n, applies linear attention in the neuronal dimension, and has no hard context-length bound.Its activations are positive after ReLU gates, and portions of its state and parameter matrices are attributable to individual neurons.Source: §4.2 · p. 24
  • BDH-GPU’ matches the GPT Transformer at every evaluated translation model size, while BDH-GPU generally compares favorably at matched scales across tested language and translation tasks.The reported comparison covers models tested up to 1B parameters.Source: §4.2 · p. 24, Figure 7 · p. 25

4.3 Comparison of BDH-GPU to other sequence processing architectures

  • BDH-GPU can prepare keys and queries so attention captures an LSH-like similarity with many buckets, and it is amenable to soft-rolling or hierarchical-attention extensions.The paper notes that the LSH function class differs from the one used in Reformer; some intuitions may carry over.Source: §4.3 · p. 25

5 Analysis: emergence of modularity and scale-free structure

The section examines how BDH’s local graph dynamics and ReLU-lowrank signal propagation relate to modular, scale-free organization, analyzing trained BDH-GPU matrices for signal-bearing positive tails, modularity, and heavy-tailed degree structure while noting uncertainty in isolating inhibitory structure in one encoder submatrix.

5.1 Background: modularity and scale-free property of systems

  • Modular graph organization allows nodes to bridge multiple communities and lets community scales and relationships evolve as their importance or connections change.For scale-free information-propagation dynamics, power-law degree distribution is described as a usual necessary, but not sufficient, condition under uniformity assumptions.Source: §5.1 · pp. 26–27

5.2 BDH-GPU feed-forward network with the ‘ReLU-lowrank’ block

  • ReLU-lowrank is designed to reduce noise and represent affinity functions on sparse positive vectors, with positive-orthant inputs and outputs.The passage identifies combination with Linear Attention blocks as a suitable use.Source: §5.2 · p. 27

5.3 ReLU-lowrank as a signal propagation dynamics

  • For a directed out-degree-r random walk, ReLU-lowrank approximates propagation from any node basis vector with L1 error O(ε) using d = O(r^3 log n/ε).The claim holds for any ε > 0 and some matrices D and E.Source: §5.3 · p. 28

5.4 Modularity in BDH-GPU signal propagation

  • Output neurons use a local F-score threshold on positive hidden-layer activity to activate when the signal matches their community.This thresholding is presented as a mechanism for propagating positive affinity functions.Source: §5.4 · p. 30
  • ReLU-lowrank represents in-cluster spreading for constant-density graphs with arbitrarily small positive modularity when d/log n = ω(1).This includes k-cluster stochastic block models.Source: §5.4 · p. 30
  • Thresholded ReLU-lowrank can be locally supermodular: two inputs may jointly activate common neighbors even when either input alone does not.The text contrasts this increasing-returns behavior with submodularity in general, while noting fDE is subadditive.Source: §5.4 · p. 31

5.5 Empirical findings: parameter distribution in ReLU-lowrank matrix products

  • The subsection motivates a heavy-tailed entry prior for G = DE because it is expected to reflect neuron-communication clustering, while low-rank representation introduces Gaussian noise.It frames training D and E as approximating a high-rank operation on a class of input vectors.Source: §5.5 · p. 31
  • The right-tail signal was confirmed even in 10M to 100M-parameter models; across five pretrained models, 12 of 16 parameter submatrices matched the predicted distribution.The matching pattern comprised three of four encoder heads and all decoder heads in each model.Source: §5.5 · pp. 32–33
  • In representative G*, modularity peaks at β2 ≈1.0, remains high as edges are added unlike random baselines, and also characterizes the inhibitory signal.The modularity remains nearly constant beyond 2^20 non-zero entries, while the random graph and random low-rank baselines quickly drop to zero.Source: Figure 9 · p. 33, §5.5 · p. 33
  • At β = 1.2, G* had a heavy-tailed, power-law-like degree distribution with more concentrated out-degree, and its visualization showed core-periphery structure.The graph contained 46,820 edges over 8,192 neuron nodes.Source: §5.5 · pp. 33–34, Figure 10 · p. 34
  • Positive modularity emerged during training in BDH-GPU parameter matrices DxE and DyE, plausibly from cluster-aware information propagation supported by the ReLU-lowrank mechanism.The authors describe this as a plausible link to the network’s inference function.Source: §5.5 · p. 34
  • One encoder submatrix per four-head model lacked a heavy positive tail and appeared inhibitory, but negative signal could not be separated convincingly from bias.The authors suggest this possible inhibitory circuit may help explain why adding a small number of heads improves model quality.Source: §5.5 · p. 34

6 Analysis: linear attention, sparse positive activation, and monosemanticity

The analysis characterizes BDH-GPU’s linear attention and positive activation space, then examines graph structure, concept-selective synapses, and activation sparsity. Its findings include attention capacity that scales with neuron dimension and sparse activity that varies with input predictability.

6.1 Macro-expressiveness of attention in BDH-GPU

  • BDH-GPU uses positive keys and queries in dimension n, and linear attention can separate positive keys that are close in L1 distance.The paper also describes an LSH bucketing function whose shared-bucket affinity is expressible through positive-orthant Linear Attention.Source: §6.1 · pp. 35–37
  • Linear Attention in dimension n can approximate attention affinities for up to t = ˜O(n) key-value pairs under stated assumptions; without them, it reaches at least t = ˜Ω(√n) for all but a negligible fraction of inputs.Keys and queries must be suitably prepared; the claim assumes comparable value norms, weakly correlated historical keys, and uniformity of expressed affinity.Source: §6.1 · pp. 35–36

6.2 Micro-interpretation of attention in BDH-GPU

  • The state matrix σ yields a thresholded neuron relationship graph whose in- and out-degree histograms show a scale-free structure.The analysis removes negative entries introduced by RoPE before applying a small positive threshold.Source: §6.2 · p. 37, Figure 11 · p. 38

6.3 Empirical findings: monosemantic synapses

  • Currency- and country-related synapses activate around their concepts in French and English, including across different words for the same concept.For generated currency-related versus politics-only sentences, currency-synapse values differed significantly (U = 2368, Uopt = 2500, p < 10−14; rank-biserial correlation 0.86).Source: §6.3 · pp. 37–38

6.4 Empirical findings: sparse neuron activations

  • In a synthetic repeated-word task, higher-layer activity becomes quieter after warm-up and fact introduction, with the largest memorization–repetition activity difference among slow-acting neurons.Layer 2 has 4.0%−7.5% non-zero entries during memorization and approximately 2.5% during repetition.Source: §6.4 · p. 38, Figure 14 · p. 40
  • Sparse activations make concept-linked synapse potentiation rare, which the paper identifies as useful for interpretability and reducing noise in Linear Attention state.The authors also say this opens opportunities for simplified and compressed state and gradient-backpropagation DAG representations.Source: §6.4 · p. 38

7 Playing with the Hatchling

The experiments test direct composition of BDH-GPU models and training without backpropagation through time. Direct merging preserves some translation capabilities but mixes output languages, while removing backpropagation retains some language modeling ability but loses cross-language concept matching.

7.1 Model merging: concatenating two models

  • The merge concatenates parameter tensors with an n dimension and averages other parameters, combining English–French and English–Portuguese models cloned from an English–Spanish base.The resulting model has 49152 neurons and 38M parameters; evaluation covers all three language pairs after each stage.Source: §7.1 · p. 41
  • The authors interpret direct weight composition as feasible when the latent space promotes concept disentangling, while noting the merged model’s language-mixing degradation.They frame the architecture as allowing models to be viewed as composable computer programs with emergent properties.Source: §7.1 · p. 41
  • The merged model translates Spanish, French, and Portuguese into English but mixes those languages when translating from English; a small amount of joint training restores their proficiency.The reported merged-model results are presented without subsequent tuning, although the authors verified improvement after training on all language pairs.Source: Table 2 · p. 41, §7.1 · p. 41

7.2 Training without backpropagation through time

  • Without backpropagation through time, the model retains some language-modeling ability but loses the ability to match concepts across languages during translation.English loss rises from approximately 0.65 for an unconditional English model trained with backpropagation to 0.75−1.05 for translation-model variants trained without it.Source: §7.2 · p. 42
  • The no-backpropagation-through-time results are preliminary, and the authors did not optimize BDH-GPU specifically for that training regime.They describe the architecture as a starting point for further investigation in this direction.Source: §7.2 · p. 42

8 Conclusions

The conclusion presents BDH-GPU as a state-space architecture with engineering opportunities in scaling, model-state interpretation, and model composition. It also proposes a brain-science hypothesis about reasoning dynamics while identifying deeper grounding in neuroscience and experimental validation as next steps.

8.1 Takeaways for model engineering

  • BDH-GPU scales model size through one large neuron dimension and offers localized data placement, state interpretation at neuron pairs, and potential direct composition of model weights.The authors also describe sparsity measures and neuron statistics as tools for evaluating model health and performance.Source: §8.1 · p. 43

8.2 Implications for brain science

  • The authors hypothesize that brain modularity, scale-free structure, synaptic plasticity, and Hebbian learning may arise from reasoning itself rather than longer-term training dynamics.They describe their artificial attention implementation as consistent with observed brain effects, but say deeper grounding in brain science and possible experimental validation remain future steps.Source: §8.2 · pp. 43–44

8.3 Societal impact

  • The paper frames controlling autonomous AI reasoning systems across timescales from seconds to years as a central problem for future work.This is presented as a motivation for principled understanding of reasoning in machine learning.Source: §8.3 · p. 44

A Connection between generalization of reasoning and computational expressiveness

The section frames reasoning generalization as matching model-family behavior on a task, then relates that match to computational simulation. It proposes the brain as a distributed-computing reference for language-model generalization.

A Connection between generalization of reasoning and computational expressiveness

  • Generalization is defined over bounded-length prompts and outputs, with models expected to produce evaluated outputs after some number of reasoning steps.The prompt set P comprises valid token sequences over alphabet Ω.Source: A Connection between generalization of… · p. 53
  • For Chain-of-Thought architectures, the section treats each model as a trainable probabilistic program and each architecture as a computational machine.Under this framing, matching task behavior connects reasoning generalization to program simulation.Source: A Connection between generalization of… · p. 53
  • A2 has sufficient expressiveness to efficiently simulate A1 programs on task P when an A2 model family generalizes on P in the same way as A1 models.The criterion is nearly identical output distributions within the given running-time bounds.Source: A Connection between generalization of… · p. 53
  • If A1 consists of human agents, A2 generalizes reasoning in the same way when trained models reproduce sampled humans’ reasoning outcomes.The section consequently proposes describing language-model generalization by reference to the human brain as a distributed computing architecture.Source: A Connection between generalization of… · p. 53

B.1 Language translation task

The experiments use Europarl as an interleaved, decoder-only language-modeling and translation task, with recurrent state carried across temporally related minibatches. The setup is intended to exercise long-context use and state retention as well as next-token prediction.

B.1 Language translation task

  • Models jointly learn language modeling and translation from interleaved, sentence-aligned Europarl source and target streams using decoder-only training on raw UTF8 data.The experiments use truncated backpropagation through time and carry recurrent state across minibatches: BDH-GPU uses a ρ matrix, while TransformerXL uses a FIFO buffer of recent KV-cache entries.Source: §B.1 · p. 53
  • The task promotes long-context capabilities because subsequent sentences are related and can support modeling the source language.Next-token prediction is also presented as representative of LLM training, with decoder-only models sufficient for the task.Source: §B.1 · p. 53
  • Table 4 lists GPTXL baseline hyperparameters; the baseline follows GPT2 and carries past KV-cache entries in a FIFO buffer.The table covers baselines used in scaling experiments.Source: Table 4 · p. 54
  • Temporally related minibatches make the previous minibatch’s final state a natural initialization for the next, promoting state carryover.The minibatches continue the prior data and contain related sentences.Source: §B.1 · pp. 53–54
  • Translation is framed as language modeling coupled with fuzzy copying, requiring capabilities such as inductive heads.The example sequence shows source sentences followed by translations, with later source sentences continuing the same larger document.Source: §B.1 · p. 54, Figure 16 · p. 54

B.2 BDH Scaling Experimental Details

The scaling experiments compare BDH-GPU variants with a GPT2-like GPTXL baseline under a shared Europarl training regime. The appendix identifies the hyperparameter tables for both model families and for model-merging experiments.

B.2 BDH Scaling Experimental Details

  • All scaling models use the same regime: En-PL and En-Cs Europarl data, 1.2B raw-UTF8 tokens, and minibatches 2048 tokens long.Training uses multi-GPU AdamW, warmup followed by linear learning-rate decay, adaptive gradient clipping, weight decay, and truncated backpropagation through time.Source: §B.2 · p. 54
  • GPTXL is a GPT2-like Transformer baseline with KV-cache carried across minibatches, ALiBi positional biases, and dropout selected by a small sweep at each model size.Table 4 lists its hyperparameters for scaling experiments.Source: §B.2 · p. 54
  • BDH-GPU’ adds xLSTM-like gating and merges next-token predictions from all layers, while sharing architectural hyperparameters with BDH-GPU.Table 5 gives BDH-GPU hyperparameters; Table 6 covers architecture and training details for model-merging experiments.Source: §B.2 · p. 54, Table 5 · p. 55, Table 6 · p. 55

B.3 BDH Monosemantic Synapse Experiment Details

The monosemantic-synapse exploration uses a BDH model trained on several Europarl language pairs with recurrent state carried across 2048-character sequences.

B.3 BDH Monosemantic Synapse Experiment Details

  • The exploration model has d = 256, n = 49152, four attention heads, and eight layers.It is trained on about one epoch of En-Es, En-Pt, and En-Fr Europarl data, totaling 1.9B tokens.Source: §B.3 · p. 55
  • Training carries the recurrent attention state across sequences of 2048 characters using truncated backpropagation through time.Training uses Distributed Data Parallel with AdamW, a 10^-3 learning rate, 1000 warm-up steps, linear decay to 10^-4, adaptive gradient clipping, and weight decay 0.1.Source: §B.3 · p. 55
  • The model uses minimal dropout of 0.01 for the synapse exploration.Source: §B.3 · p. 55

B.4 BDH Merging Experiment Details

BDH models were trained on Europarl for about two passes, using truncated backpropagation through time and recurrent-state carryover. Training used 2048-character sequences and AdamW with a decaying learning rate.

B.4 BDH Merging Experiment Details

  • Models trained on Europarl for about two passes, with truncated backpropagation through time and recurrent-state carryover across 2048-character sequences.Training used distributed data parallelism.Source: §B.4 · p. 55
  • AdamW training used a 10^-3 learning rate after 1000 warmup steps, followed by linear decay to 10^-4.Weight decay was 0.1, adaptive gradient clipping was used, and dropout was 0.01.Source: §B.4 · p. 55
  • The models used the architecture hyperparameters listed in Table 6.The passage does not provide the table’s hyperparameter values.Source: §B.4 · p. 55

C Omitted formal claims and proofs

The appendix formalizes when BDH-GPU linear attention approximates general attention and how sparse graph attention can represent BDH-Normfree attention. Its proof constructions use random key preparation or linear value preparation, depending on the claim.

C.2 Formal statement of Claim 7 (linear attention)

  • For C-non-adversarial keys, Claim 8 bounds the attention horizon by t < δn/((C + 1) log n) for a high-probability linear-attention approximation after suitable random key preparation.C counts the permitted exceptions to independence; adversarial inputs use C = t − 1.Source: §C.2 · p. 56
  • The proof targets a dimension-n linear-attention estimate of general attention for t < δn/log² n.It uses a Johnson–Lindenstrauss key transform to preserve scalar products with high probability.Source: §C.2 · p. 56
  • The extreme cases C = 0 and C = t − 1 clarify the time ranges over which linear attention can express general attention.The claim’s bounds depend on how many key vectors may fail the non-adversarial condition.Source: §C.2 · p. 56

C.4 Formal statement of Claim 4

  • Claim 9 represents BDH-Normfree attention using a sparse graph Gs with O(nd) edges and a sparse value-preparation function A.The representation concerns sequences of keys and values in the claim’s stated domain.Source: §C.4 · p. 57
  • The sparse attention state corresponds to the BDH-Normfree attention state after values are prepared with A.The appendix frames the construction as replacing the attention-state operator with a sparser one.Source: §C.4 · p. 57
  • Stochastic sparsification could approximate the low-rank attention operator, with value-dependent sampling plausibly reflecting the power-law element distributions observed in σ.The authors instead use a technical construction that establishes strict equality with linear value preparation.Source: §C.4 · p. 57
  • Under that reduction, two successive sparse-state BDH layers suffice to express one BDH-Normfree layer.The proof embeds the encoder matrix’s connection structure into Gs.Source: §C.4 · p. 57

C.1 Proof of Observation 1

The proof establishes equivalence between the protocol description and the matrix dynamics by matching their variables at each layer and time step.

C.1 Proof of Observation 1

  • At layer l and time t, protocol variables X, Y, and σ correspond to coordinates of x, y, and the attention-state matrix σ in the linear-algebraic dynamics.The vectors use the previous round’s inputs, while σ uses its state from the previous token.Source: §C.1 · p. 55
  • The auxiliary variable A represents the intermediate vector σt−1,l xt,l used to compute yt,l from xt,l.This connects the protocol’s auxiliary step to the matrix computation.Source: §C.1 · p. 55
  • The state parameter u corresponds to an entry of U, assumed diagonal for simplicity, as in ALiBi.The auxiliary node variables Xe, Xi, Ye, and Yi handle thresholding of the inhibitory circuit.Source: §C.1 · p. 55

C.3 Proof of Claim 3

The proof realizes matrix interactions with neuron-synapse graphs, while noting that the circuit construction can impose high synaptic-layer degree. The construction does not alter the analysis of neuron-neuron graph structure.

C.3 Proof of Claim 3

  • The construction builds neuron-synapse graphs He and Hi so their two-step neuron-level graphs Ge and Gi satisfy Ge − Gi = DE.The proof verifies this by decomposing the matrix into excitatory and inhibitory components.Source: §C.3 · pp. 56–57
  • The technical circuit gives synaptic-layer nodes high degree, but subdividing them can reduce degree at the cost of proportionally increasing circuit edges.The passage gives subdivision into a² nodes as an example, reducing degree Θ(a)-times while increasing edges Θ(a)-times.Source: §C.3 · p. 57
  • Branching circuits can reduce internal degrees while retaining O(nd) edges, even at constant degree.The considered token-propagation dynamics remain linear.Source: §C.3 · p. 57
  • The technical circuit construction does not affect the neuron-neuron graph analysis, whose degree distribution plausibly remains heavy-tailed and power-law-like.The passage relates this structure to models examined empirically in Section 5.5.Source: §C.3 · p. 57

D Desirable properties of a local graph dynamics for language models

The section develops first-principles criteria for attention-based distributed language models, arguing that language processing calls for interacting particles and flexible graph structure. It connects attention to pairwise particle interactions and expects language-model graphs to be scale-free and multi-scale.

D Desirable properties of a local graph dynamics for language models

  • The proposed criteria favor learned dynamics that solve a computational problem efficiently, while ruling out systems whose inference can be systematically approximated more easily than running them.The latter criterion is termed computational irreducibility.Source: D Desirable properties of a local… · pp. 58–59
  • Attention retrieval couples a query at the current time with an earlier key-value entry, so BDH’s dynamics cannot be represented by independent-walker Dyson correlations alone.With intermediate key-value state, particles can be viewed as taking l-step walks; directly on input tokens, the description requires at least 2l-point correlations.Source: D Desirable properties of a local… · pp. 59–60
  • Depth L should represent syntax-tree levels rather than leaf count, motivating multi-point token correlations and excluding L-layer systems that compress away most long-term correlations.The passage relates this expectation to the ability to build a syntax tree in one quick parallel scan.Source: D Desirable properties of a local… · p. 60
  • Attention-based state-space models can express nonlinear dynamics through pairwise particle interactions that alter correlation strength, the particle-level counterpart of attention.The authors describe these pair interactions as sufficient for the nonlinear rulesets needed by BDH.Source: D Desirable properties of a local… · p. 60
  • Language-model graph topology need not follow an externally imposed spatial grid; the authors expect multi-scale, expander-like shortcuts and scale-free structure.The passage contrasts this expected structure with two-dimensional cellular automata.Source: D Desirable properties of a local… · p. 61

E BDH-GPU PyTorch code listing

  • The BDH-GPU code listing implements a fixed context window, while unbounded context is technically supported through a state-space kernel for Linear Attention with suitable truncated-backpropagation adaptation.The listing extends the state dynamics with heads and allows flexibility in layer-normalization and residual-connection placement.Source: E BDH-GPU PyTorch code listing · p. 61

Related papers

Loading 2509.26507v1…