Source-linked AI summary
The Von-Neumann State-Space Transformer for neural decoding
Morteza Sarafyazd
TL;DR
Neural decoding seeks accurate behavioral read-out from limited data and small models, but standard Transformers apply the same feed-forward operator to every token. VN-SST instead synthesizes token-specific operators from a low-rank instruction bank steered by carried low-dimensional state, matching or beating a modern Transformer across three motor-cortex codecs and improving with longer context.
Problem
Neural decoding needs accurate behavioral read-out from limited data and small models, while standard Transformer feed-forward blocks use the same operator for every token.
Method
VN-SST uses a low-rank instruction bank and carried low-dimensional state to synthesize a token-specific feed-forward operator instead of blending fixed experts’ outputs.
Results
VN-SST matches or beats a modern Transformer across three motor-cortex decoding codecs, leads most strongly on the scarcest benchmark, and turns longer context into rising accuracy.
Takeaways & Limitations
With scarce data and small models, programmable operators steered by low-dimensional dynamics provide a useful inductive bias for neural decoding.
Takeaways & Limitations
The evaluation fixes VN-SST’s window, state size, memory dimensions, read-out dimension, and instruction-bank hyperparameters.
Abstract
from arXiv · showhide
Cortical computation is strikingly low-dimensional: a handful of latent variables, carried in a neural population's activity, steer the higher-dimensional responses of individual neurons. Our aim is sample efficiency-models that decode well from limited data and at small parameter budgets. In a standard Transformer layer, the feed-forward block applies the same operator to every token. We suggest a von-Neumann inspired hypothesis of efficient computation as an alternative for neural decoding: a controller decodes an instruction and then executes a token-specific operator; the usual realization-a soft mixture of experts-only blends their outputs, not operators. We introduce a von-Neumann State-Space Transformer (VN-SST), a memory-augmented Transformer whose feed-forward block is a low-rank instruction bank: a shared base operator plus a small set of learned low-rank instructions, from which a per-token code synthesizes the weight matrix actually used at that token. The code is read from a low- dimensional projection of a carried state-space memory, so a slow latent trajectory acts as an instruction pointer-mirroring how low-dimensional dynamics may route cortical computation. On three motor-cortex neural-decoding benchmarks, VN-SST is far more data-efficient than a modern Transformer, each jointly predicting spikes and decoding behavior. This model wins by a wide margin on the scarcest benchmark, leads on the other two, and turns longer context into rising rather than falling accuracy. We evaluated that the network compresses a large instruction bank to a few bits per token, so program capacity acts as a control channel, not an accuracy lever. The same model is also more parameter-efficient on two small text benchmarks used for language modeling (LLMs), suggesting a generic mechanism.
1 Introduction
The paper proposes VN-SST, a memory-augmented Transformer whose controller synthesizes token-specific feed-forward operators from low-rank instructions. It draws on low-dimensional, slowly varying neural dynamics to improve sample-efficient neural decoding with small models.
- Motivation: Cortical population activity is organized by a handful of latent variables whose slow trajectories steer higher-dimensional responses, motivating sample-efficient neural decoding.The stated goal is accurate behavioral read-out from limited data and small models.
- Von-Neumann computation: Unlike a standard Transformer’s fixed FFN or soft-MoE’s blended outputs, VN-SST constructs a token-specific operator by synthesizing weights from low-rank instructions.The instruction bank adds learned low-rank components to a shared base operator.
- Instruction generation: A slowly varying selective-SSM memory supplies a low-dimensional instruction pointer that generates each token’s operator rather than reading instructions directly from raw tokens.The carried latent state drives higher-complexity, token-specific computation.
- Model and evaluation: VN-SST combines local attention, a selective SSM, fast-weight memory, and a controller in a drop-in Transformer for motor-cortex decoding comparisons.Its feed-forward operator is synthesized per token from the low-rank instruction bank driven by carried low-dimensional state.
2 Models
The baseline is a stateless decoder-only Transformer whose shared feed-forward weights apply the same operator at every token. VN-SST retains the layer skeleton but adds persistent memories and a low-rank, state-conditioned programmable feed-forward operator.
- Baseline Transformer: The baseline combines RMSNorm, RoPE, causal multi-head self-attention, and SwiGLU feed-forward blocks in identical layers.Its three SwiGLU weight matrices are shared across positions, so every token uses the same operator.
- VN-SST architecture: VN-SST replaces plain attention with three memory pathways and fixed SwiGLU with a programmable feed-forward operator.The design follows a stored-program architecture separating control, execution, and persistent state.
- VN-SST architecture: The three pathways provide local causal attention, slow selective-SSM dynamics, and fast-weight associative memory carried across segments.Local attention costs O(Tw), while persistent state enables dependencies longer than the window width w.
- Programmable compute: A low-dimensional projection of the carried SSM state serves as an instruction pointer that helps select the per-token operator.The slow latent trajectory implements the model’s fetch-decode-execute loop.
- Programmable compute: VN-SST synthesizes each token’s feed-forward projections from shared base maps plus code-weighted low-rank instruction banks without materializing token-specific weights.The same code gates both the concatenated gate/up projection and the down projection.
3 Benchmarks and Setup
The benchmarks cast three motor-cortex recordings as joint neural-sequence-generation and behavioral-decoding tasks from a shared hidden state. Experiments compare four-layer Transformer and VN-SST width ladders at matched non-embedding parameter budgets, using controlled data and context sweeps.
- Joint neural codec: Three Neural Latents Benchmark motor-cortex recordings are converted into dual-objective codecs that autoregressively continue binned population spikes and decode behavior from one shared hidden state.The codec uses a shared hidden state for both neural sequence generation and behavioral decoding.
- Setup: Both architectures use 4 layers and scale only width within the same ∼64–270K non-embedding parameter window.Fixed depth removes a shape confound; VN-SST is narrower because its carried state and instruction bank add fixed per-width overhead.
- Training and evaluation: ≈25% of the full training budget is used for the smallest Section 4.2 budget, while data- and context-scaling sweeps use the largest available budget.The smallest budget is ∼2/14/7K bins for MC RTT/MC Maze/Area2, and neural sweeps report means over 3 seeds.
- Training and evaluation: Context-length sweeps hold optimizer steps fixed across lengths, scaling epochs with L so context length is not confounded with training budget.Both models use Ada…
4 Results
Across matched parameter, data, and context analyses, VN-SST is at least as accurate as the Transformer on all neural codecs, with its largest advantages under scarce data and longer context. Its program capacity is not an accuracy lever, while the same mechanism is more parameter-efficient on two text corpora.
- Evaluation metric: Validation behavioral-decoding R2, rather than single-trial spike prediction near its noise floor, separates VN-SST from the modern Transformer.The comparison uses matched budgets and focuses on behavioral decoding because spike prediction is near the noise floor for both architectures.
- Data scaling: At the smallest data budget, VN-SST reaches decode R2 0.68 versus 0.60 on Area2 Bump and 0.30 versus 0.19 on MC RTT.The advantage narrows as data grows; at full recording, MC RTT is 0.61 versus 0.52, MC Maze is 0.85 versus 0.85, and Area2 is 0.79 versus 0.78.
- Program capacity: Across K ∈{1,...,32}, decode R2 is essentially flat and non-monotonic, while a K=32 bank compresses to only a few effective instructions.The empirical code uses approximately 2.55 bits on MC RTT, 3.20 on Area2, and 2.89 on MC Maze; the broader range is ≈2.5–3.2 bits and ≈3.4–6.5 effective operators.
- Language modeling: On tiny-Shakespeare and WikiText-2, the unchanged instruction-bank model is more parameter-efficient than the Transformer across the tested parameter ladder.The comparison uses byte-level BPE text corpora, two layers, and approximately 0.13–1.7M non-embedding parameters.
5 Conclusion
VN-SST replaces a fixed Transformer feed-forward operator with a low-rank, state-conditioned instruction bank whose per-token code acts as an instruction pointer. On three motor-cortex decoding codecs, it improves data and context efficiency, motivating programmable operators steered by low-dimensional dynamics when data and models are scarce.
- 5 Conclusion: VN-SST uses a shared base operator plus K rank-r instructions to synthesize a per-token operator from a code read from carried low-dimensional state.The slow latent trajectory organizing cortical population activity acts as an instruction pointer, making the feed-forward block programmable.
- 5 Conclusion: VN-SST leads by a wide margin on the scarcest codec, is at least as accurate at every data budget across all three codecs, and uniquely benefits from longer context.These results identify small data and small models as the setting where the mechanism earns its keep.
- 5 Conclusion: The results suggest that scarce-data, scarce-model settings benefit from programmable operators steered by low-dimensional dynamics rather than simply more parameters.The paper identifies reusable contributions in token-programmable feed-forward computation and measuring how much program a trained model uses.