Source-linked AI summary
Anatomy of Associative Recall in Fixed-State Recurrences: A Matched-State Decomposition, an Interference Wall, and a Curriculum That Breaks It
Julian Boesch, Andrew Wee
TL;DR
Whole-architecture comparisons cannot isolate which ingredients account for fixed-state recurrences’ associative-recall gap. The paper decomposes masked recall at matched state and tests a distance curriculum. The curriculum lifts retrieval from 0.021 to 1.000, while results also show a length boundary and no measurable bidirectional advantage over causal training.
Problem
Whole-architecture comparisons confound recurrence ingredients, leaving unclear which design choices account for fixed-state sequence mixers’ associative-recall gap.
Source: §1 · p. 1Method
The paper toggles convolution, transition structure, and decay in matched-state cells and tests distance curricula on retrieval tasks.
Source: §1 · pp. 1–2Results
A distance curriculum lifts retrieval from 0.021 to 1.000 with the architecture unchanged, while lock-in rises from 1/10 to 7/10 versus dense supervision alone.
Source: §1 · p. 2Takeaways & Limitations
After arming and curriculum, hybridization remains relevant for loads beyond state capacity, lengths beyond the curriculum boundary, and query-conditioned regimes.
Source: §10 · p. 12Converted models solving their trained token band read chance on held-out token bands, so transfer is limited to binding circuits over trained symbols.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Fixed-state recurrences--linear attention and state-space models--are reported to lag behind attention on associative recall, but whole-architecture comparisons cannot say which ingredient is responsible. We decompose masked multi-query recall at a fixed state budget along three single-knob axes: a short causal convolution, the transition structure (rank-1 delta rule vs. diagonal), and decay. The convolution dominates (~+0.5 recall in both families under matched training): comparisons that pit convolution-free cells against a convolution-equipped Mamba measure the missing convolution, not the recurrence. The rank-1 transition beats its diagonal ablation by +0.19/+0.32 at 16/32 pairs, but the margin shrinks to +0.03 once both cells carry the convolution, and a state-matched Mamba-2 ties the unarmed rank-1 cell: no class claim survives. Cells that solve 32-pair recall degrade gracefully with load yet fall to chance retrieving 4 pairs from a distractor haystack--flat across lengths and transitions. Interference under sparse supervision, not capacity: a distance curriculum takes the unchanged architecture from 0.021 to 1.000. Training is a lock-in lottery--a seed either locks in or does not--and the curriculum is the lever. Lock-in rises from 1/10 to 7/10 (p=0.02); dense supervision adds nothing; at L=256 a shaped ramp reopens a boundary the uniform curriculum cannot (4/5 vs. 0/9); and at L=512, where the ramp collapses (0/6), gating it on measured accuracy locks in 6/6 (p=0.001). Bidirectional denoiser cells, reading the query before the haystack, show no measurable advantage over causal training (ten seeds), and collision-key retrieval needs two layers. Arming for recall is free on an S_5 state-tracking guardrail--the armed cell is significantly better at every depth (p<=0.0044). These replace "recurrent models are bad at recall" with a measured decomposition and two cheap interventions.
1. Introduction
Whole-architecture comparisons confound convolution, transition structure, and decay, motivating a matched-state decomposition of associative recall. The resulting evidence identifies convolution and training coverage as major levers while limiting claims about recurrence classes.
1. Introduction
- At fixed state, convolution transfers across cell families (+0.47 delta-rule, +0.44 diagonal at K=32), while the rank-1 margin shrinks to +0.034 with convolution and no class advantage survives against Mamba-2.The matched-state Mamba-2 ties the unarmed rank-1 cell (0.592 vs. 0.653, p=0.88).Source: §1 · p. 2, §2 · p. 3
- Recall degrades gracefully with load, but distractor-haystack retrieval reaches chance at the shortest tested length, flat across lengths and transitions.This signature distinguishes the observed wall from capacity and decay accounts; a two-layer attention control solves the task at 1.0.Source: §1 · p. 2, §2 · p. 3
- A distance curriculum lifts retrieval from 0.021 to 1.000 with the architecture unchanged, while lock-in rises from 1/10 to 7/10 under curriculum versus dense supervision alone.At L=256, a shaped ramp locks in 4/5 seeds versus 0/9 for the uniform curriculum; at L=512, a success-gated ramp locks in 6/6 versus 0/6 for the time-based ramp.Source: §1 · p. 2
2. Related Work
- Bidirectional denoisers have no significant directional advantage in the ten-seed collision replication, and any solving circuit requires two layers.The related-work connection is native query-first reading; the measurable advantage over matched causal training remains unsupported.Source: §1 · p. 2, §2 · p. 3
1. Introduction
2. Related Work
- The paper frames its contribution as a controlled decomposition of existing ingredients, including a large, family-transferable convolution effect and a learning-rate sensitivity measurement.Prior intervention evidence on convolution is qualified by a learning-rate confound; this factorial toggles ingredients at fixed state in both cell families.Source: §1 · p. 2, §2 · p. 3
3. Cells, Tasks, and Protocol
The experiments compare recurrence configurations in one harness under matched state and use synthetic masked recall, haystack, collision-key, and state-tracking tasks. A hardened protocol uses rebinding controls and reports lock-in rates when seed outcomes are bimodal.
3.1. The cell zoo
- The cell zoo holds embeddings, readout, and training loops fixed while varying transitions, decay, convolution, and directionality across recurrence configurations.The paper calls each tested configuration an arm; the short causal depthwise convolution adds parameters but no recurrent state.Source: §3.1 · pp. 3–4
3.2. State matching
- The factorial matches rank-1 cells and the Mamba-2 reference at 1,024 recurrent state elements per head group and approximately 29k parameters.At ample state (d=128), every cell solves every setting, so differences are studied in the constrained-state regime.Source: §3.2 · p. 4
3.3. Tasks and floors
- Masked MQAR uses random per-sequence values and K∈{8,16,32}; haystack retrieval uses four pairs followed by distractors, while collision-key distractors reuse table keys with fresh values.The S5 guardrail uses uniformly drawn S5 generators and parallel masked answers to avoid answer-prior and prefix-leakage confounds.Source: §3.3 · p. 4
3.4. Statistical protocol
- Headline ablations use ten seeds, permutation tests, and a rebinding control; bimodal outcomes are reported as lock-in rates with Fisher exact tests.Exploratory sweeps use three seeds, and task floors are reported with results.Source: §3.4 · p. 4
4. A Matched-State Decomposition of Recall
At matched state, the short convolution is the dominant recall lever, while the rank-1 transition’s gain depends on whether convolution is present; the ablation does not support a class-wide rank-1 advantage.
4.1. The factorial
- The factorial compares recall at constrained state by toggling convolution, transition structure, and decay one knob at a time.The table describes the hardest setting, K=32, with 1,024 state elements; its K=32 statistics supersede the earlier n=3 results.Source: §4.1 · p. 5, Table 1 · p. 6
4.2. Reading the factorial: three separable levers
- The convolution gap depends on tuning: per-arm tuning narrows it in the delta family, while the diagonal cell retains a +0.42 within-cell gap.At tuned learning rates, the delta-family gap is about +0.13 (0.999 vs. 0.870); the armed transition margin falls to about 0.002 at ceiling.Source: §4.2 · p. 5
- At K=32, the decay comparison is not significant: deltanet scores 0.561 and gated deltanet 0.518 (p=0.86; n=20).The reported seed outcomes are bimodal, and the S5 lock-in comparison does not establish a state-tracking benefit from decay (p=0.65).Source: §4.2 · p. 5, Figure 1 · p. 6
4.3. The hardened transition ablation, with controls
- The rank-1 transition beats its diagonal ablation by +0.191 at K=16 and +0.319 at K=32, but by only +0.034 at K=32 when both have convolution.The with-convolution comparison is 0.992 vs. 0.958; at K=16, all armed arms are at ceiling.Source: §4.3 · p. 5, §4.2 · p. 5, Figure 1 · p. 6
- A state- and parameter-matched Mamba-2 ties the rank-1 cell at K=16 and beats it at K=32, so these results do not establish a rank-1 class advantage.At K=32, Mamba-2 scores 0.849 vs. 0.702 for the rank-1 cell (p=0.0036).Source: §4.3 · p. 5
4.2. Reading the factorial: three separable levers
- The short convolution is the dominant, family-transferable lever in the K=32 single-knob comparison.Figure 1 also shows that the transition margin is smaller with convolution, while the decay comparison is not significant at n=20.Source: Figure 1 · p. 6
5. Capacity Curves vs. the Interference Wall
Load produces graceful degradation, but retrieval from a distractor haystack is already at chance at the shortest tested length and remains flat across lengths and transitions. The contrast supports an interference-under-sparse-supervision diagnosis rather than a capacity limit.
5. Capacity Curves vs. the Interference Wall
- Cells that store 32 bindings at ≥0.99 fall to near chance retrieving 4 bindings with distractors, with means of 0.017–0.019 across L ∈{64, 128, 256}.The failure is present at the shortest tested length and remains flat through four times that length.Source: §5 · p. 7
- The delta rule, selective diagonal, and RWKV-7 all fail at chance across tested lengths, while the two-layer attention control solves every trial at 1.000.The transition comparison uses n=5 per recurrent cell; attention is exact in 15/15 trials.Source: §5 · p. 7
- The authors diagnose interference under sparse supervision: the end-only masked-token signal gives the write gate little gradient, allowing distractor writes to overwrite bindings.The proposed account differs from a decay explanation because a training-signal change can fix the failure without changing the architecture.Source: §5 · p. 7
6. The Wall Is a Training-Coverage Gap
An unchanged two-layer recurrence moves from chance to perfect retrieval under a distance curriculum, supporting a training-coverage explanation for the haystack wall. Across longer horizons, curriculum shape and accuracy-gated scheduling determine whether seeds lock in; converted-model transfer remains bounded to trained token bands.
6. The Wall Is a Training-Coverage Gap
- The curriculum samples table-to-query gaps uniformly from zero to a batch maximum, while evaluation uses the maximum gap; dense supervision instead supervises all queried pairs.The task and architecture remain unchanged between training conditions.Source: §6 · p. 7
- 0.021 to 1.000: an unchanged two-layer recurrence moves from chance to perfect retrieval under a distance curriculum, an existence proof for this task at this length.The haystack task is 4-pair retrieval across distractors; recurrent cells otherwise sit at chance across tested lengths.Source: §6 · p. 7, Table 4 · p. 8, Table 5 · p. 9
- Converted models that solve their trained token band read chance on held-out token bands, so the reported transfer is a binding circuit over trained symbols.This boundary is reported in the companion conversion work.Source: §6 · pp. 7–8
- 7/10 vs. 1/10: the distance curriculum raises lock-in over dense supervision alone, while adding dense supervision to the curriculum does not help.The comparison is a lock-in-rate shift, not a deterministic outcome for every seed.Source: §6 · p. 8, Table 5 · p. 9
- The attention reference's curriculum ceiling depends on position generalization: absolute positions collapse when layout shifts, while rotary positions solve the fixed-layout wall and learn under shifting layouts.The absolute-position baseline collapses in 7/7 trials; rotary attention locks in on the shifting curriculum in 6/10 lexical and 7/10 collision trials.Source: §6 · p. 8
- At L=256, a shaped ramp locks in 4/5 seeds versus 0/9 pooled uniform trials; at L=512, a success-gated ramp locks in 6/6 versus 0/6 for the time-based ramp.The L=512 gate advances the gap only while measured accuracy stays above 0.6.Source: §6 · pp. 8–9
7. Bidirectional Denoisers Read Twice for Free
Bidirectional denoisers natively read the query before the haystack, but collision-key tests find no measurable recall advantage over matched causal training. Solving the collision task requires two layers, and curriculum shape stabilizes lock-in in both directions.
7. Bidirectional Denoisers Read Twice for Free
- At ten seeds, bidirectional cells lock in on 3/10 seeds versus 1/10 for causal cells, but the mean difference is not significant (p=0.63).Under the shaped curriculum both directions reach 9/10 lock-in, with no significant directional difference (p=0.76).Source: §7 · p. 10
- Collision-key retrieval is solvable in both directions: bidirectional cells reach 1.000 and causal cells reach 0.844 in their best seeds.The causal result is consistent with, but does not isolate, a depth-as-novelty-gate hypothesis.Source: §7 · p. 10
- Both arms collapse to chance at one layer; solving the task requires two layers to route query-first information to the answer position.The masked answer is first in the backward scan, so a second layer's forward stream must route information from earlier positions.Source: §7 · p. 10
- The ten-seed comparison does not establish a directional advantage, and the observed lock-in lottery makes larger samples necessary for differential claims.The cited 3/10 versus 1/10 comparison has p≈0.58 by Fisher exact; the convolution-free arm has not locked in under either tested recipe.Source: §7 · p. 10
8. Guardrail: Arming Is Free on State Tracking
On the S5 running-product guardrail, arming for recall does not reduce state-tracking performance: the armed cell wins at every tested depth. The comparison concerns learnability at fixed depth, not circuit complexity.
8. Guardrail: Arming Is Free on State Tracking
- The guardrail compares armed Gated DeltaNet with Gated DeltaNet on the S5 running-product task.The question is whether adding a local finite-window convolution costs performance on state tracking.Source: §8 · p. 10
- The armed cell wins significantly at all four tested composition depths.The reported result is scoped to learnability at fixed depth.Source: §8 · p. 10
- Accuracy falls steeply with composition depth for both arms, and the results do not address asymptotic separations.The authors explicitly frame the experiment as a fixed-depth learnability comparison.Source: §8 · p. 10
9. Limitations and Threats to Validity
The pure-PyTorch Mamba-2 reference under-trains relative to the official fused kernel, softening absolute margins against real Mamba. Within-harness comparisons remain internally valid, and a smaller margin survives a paired learning-rate sweep.
9. Limitations and Threats to Validity
- 0.592 versus 0.884: the pure-PyTorch Mamba-2 reference trails the official fused Mamba-2 at matched state.The pure-PyTorch reference reaches 0.641 at three seeds, compared with 0.884 for the official fused kernel at the same matched state.Source: §9 · p. 10
- Absolute margins against a real Mamba are softer because the reference comparator under-trains the official kernel.The within-harness factorial remains internally valid because all cells use one harness and budget.Source: §9 · p. 10
- At each arm's best learning rate, the margin is 1.000 vs. 0.955 (p=0.016), while the shared-default comparison reproduces 0.967 vs. 0.867.Most of the cross-session gap was attributable to the official arm's learning rate.Source: §9 · p. 10
2. Toy scale, single task family. d ∈{32, 128}, one
The reported comparisons find no established bidirectional advantage in collision-key retrieval, while arming does not trade away S5 state tracking. The paper limits these small-scale, constrained-state findings and its attention comparisons to the settings actually tested.
2. Toy scale, single task family. d ∈{32, 128}, one
- In the hardened collision-key discriminator (L=64, dense-4 + curriculum, 2 layers, n=10 seeds), the bidirectional–causal mean difference is +0.082 (permutation p=0.63); no directional difference is established at this power.Per-seed accuracies were bimodal: most seeds locked out, while some solved outright.Source: Table 6 · p. 11
- The armed cell is significantly better at every probed S5 depth, so arming for recall does not trade away state tracking.The S5 guardrail is internally comparable; an armed RWKV-7 cell also beats its unarmed counterpart at both reported depths.Source: Table 7 · p. 11
- The decomposition’s convolution, transition, and decay magnitudes apply to constrained state; at ample state, the knobs stop mattering as everything saturates.The paper frames these results as mechanism isolation, not deployment guidance.Source: §2 · p. 11
- For the delta family, tuning narrows the armed–unarmed gap to ≈+0.13, while the diagonal cell reaches only 0.172 at its best learning rate.Sweeps span one decade at 5 seeds; the official kernel’s tuned optimum is 0.955.Source: §2 · p. 11
- The one-layer attention control collapses at every K, so the paper reports attention ceilings only in two-layer regimes and makes no attention-superiority claims beyond those settings.Cross-family magnitude comparisons carry a parameterization caveat, and the interpretation excludes rank-1-versus-diagonal class claims and curriculum extrapolation beyond L=64.Source: §2 · p. 11
10. Discussion
The discussion reframes recurrent recall failures as a combination of architectural and training factors, with some apparent retrieval barriers dissolving after cheap interventions. It identifies remaining boundaries and treats transfer beyond the measured settings as unresolved.
10. Discussion
- The discussion decomposes recurrent recall into a missing convolution, a within-cell transition effect, a scalar-gate-specific decay tax, sparse-supervision interference, and query-first directionality.It concludes that none of these findings means recurrence cannot bind.Source: §10 · p. 11
- Adding convolution and using a distance curriculum dissolve two classic triggers for adding attention, while capacity, lengths beyond the curriculum boundary, and unanticipated query-conditioned regimes remain hybrid territory.The discussion calls a hybrid decision after arming and curriculum a smaller, different decision.Source: §10 · p. 12
- For recurrent diffusion LMs, bidirectional denoisers have the Just-Read-Twice property natively, with two layers minimum and a short convolution in the backward stream.Whether the query-first mechanism carries to scale is untested.Source: §10 · p. 12
- At L=512, gating the ramp on measured competence locks in 6/6, but whether some length defeats the gated schedule remains open.The discussion states the broader claim—that fixed-state retrieval walls are training-coverage gaps—as a conjecture worth falsifying.Source: §10 · p. 12
Impact Statement
- The paper presents controlled, small-scale measurements to advance understanding of sequence-model architectures; downstream benefits depend on whether findings transfer to scale.It identifies no societal consequences specific to this work beyond those generic to improving machine learning methods.Source: Impact Statement · p. 12
A. The Commodity-Hardware Harness
The appendix describes a commodity-hardware harness for running the experiments and lists commands for reproducing the factorial, retrieval, curriculum, and guardrail probes.
A. The Commodity-Hardware Harness
- Every arm has a pure-PyTorch reference implementation, and the Mamba-2 comparator runs without Triton or mamba ssm.Setting kernel size to 1 neutralizes Mamba-2’s convolution for the toggle.Source: A. The Commodity-Hardware Harness · p. 15
- An atomic-free two-pass backward enables the RWKV-7-style scan on Pascal (sm 61) at ≈26× pure-PyTorch reference throughput.Headline experiments ran on a single GTX 1070.Source: A. The Commodity-Hardware Harness · p. 15
- Runs are seed-pinned and resumable, and each task includes floor calculators plus a rebinding-control self-test.The listed reproduction commands cover state tracking, MQAR, the matched factorial, the haystack wall, curriculum and collision-key probes, and the S5 guardrail.Source: A. The Commodity-Hardware Harness · p. 15, B. Reproduction Commands · p. 15
B. Reproduction Commands
- The reproduction commands include separate invocations for the haystack-wall and curriculum/collision-key probes and for the S5 guardrail.The supplied command list also includes probes for state tracking, MQAR, the factorial, and the hardened transition ablation.Source: B. Reproduction Commands · p. 15
C. Floors and Controls, Stated
The paper reports task-specific floors for interpreting recall and control measurements.
C. Floors and Controls, Stated
- For MQAR, the value-marginal floor is 1/26 ≈0.038; for haystack/collision it is 1/54 ≈0.019; for S5 it is 1/120 ≈0.008.The S5 value is exact because its generator is uniform over the full group.Source: C. Floors and Controls, Stated · p. 15
- The strongest no-binding strategy is ≈0.068 for MQAR and ≈0.26 for collision; the rebinding-control table floor is 1/K.Source: C. Floors and Controls, Stated · p. 15
- No headline number is within 10× of its floor except where explicitly marked as “chance”; the S5 probe masks all product slots simultaneously and uses single-token answers.This makes prefix-completion leakage structurally impossible in the S5 probe.Source: C. Floors and Controls, Stated · p. 15