DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

1 INTRODUCTION

• Serial nonlinear TTT can underperform fixed-base parallel TTT, despite the expectation that state-dependent updates incorporate new information better.That expectation is supported in linear memories, where DeltaNet uses residual corrections based on the current memory.Source: §1 · p. 1
• A linear memory achieves lower reconstruction error than a two-layer MLP within one epoch, while the MLP’s advantage emerges with additional epochs.The authors argue that greater expressivity alone is insufficient when the inner-update budget is usually one epoch.Source: §1 · p. 1
• DeltaTTT assigns each layer a local reconstruction target and state-dependent delta-rule update, preserving nonlinear composition while enabling chunkwise parallel computation.Both TTT and DeltaTTT use a nonlinear readout after the writes.Source: §1 · pp. 1–2, Figure 1 · p. 2
• DeltaTTT improves language modeling and retrieval over recurrent baselines on both DeltaNet and LaCT backbones.The reported models have 0.5B parameters and are trained on 92.5B tokens.Source: §1 · p. 2

2 BACKGROUND

• Test-Time Training (TTT) compresses key–value associations into an inner network’s parameters through online optimization, with queries retrieving from the resulting memory (Sun et al., 2024).For a fixed-size inner network and one sequence pass, total read and update cost scales linearly with sequence length.Source: §2 · p. 2
• Serial TTT evaluates each chunk’s gradient using the memory updated by earlier chunks, whereas fixed-base parallel TTT evaluates gradients at the shared initialization.The fixed-base gradients can be computed independently across chunks and accumulated with a causal prefix sum; both variants read before writing the current chunk.Source: §2 · pp. 2–3
• With a linear memory and one-token updates, serial gradient descent gives DeltaNet’s delta-rule update; fixed-base evaluation instead corresponds to additive linear attention.The state-dependent update is affine in the preceding memory, enabling chunkwise parallel evaluation while preserving tokenwise corrections (Yang et al., 2024).Source: §2 · p. 3

3 MOTIVATION: EFFECTIVE OPTIMIZATION IN A SINGLE PASS

• 28.25 versus 37.87 single-needle accuracy: serial nonlinear TTT trails fixed-base parallel TTT.This reverses the expected advantage of updates that account for the current memory state.Source: §3 · p. 3
• After one epoch, linear memory reaches lower reconstruction error than the MLP; at small chunk sizes, the tuned MLP reaches lower error by epoch 16.The experiment fits linear and two-layer MLP memories to the same fixed K/V sequence; existing methods typically use one pass because extra inner-learning epochs incur substantial overhead.Source: §3 · p. 3, Figure 2 · p. 4
• At a fixed replay budget, smaller chunks allow more update steps and can lower reconstruction error, but the trend is not strictly monotonic.In serial nonlinear TTT, smaller chunks require more sequential updates and offer less parallelism per step; linear delta updates preserve tokenwise corrections while enabling chunkwise parallel evaluation.Source: §3 · pp. 3–4

4.1 LAYERWISE MEMORY OPTIMIZATION

• Layerwise learning gives the first layer a key-to-value prediction target and the second a value-reconstruction target from the intermediate representation.The second layer provides another learned reconstruction stage that can compensate for imperfections in the first layer's prediction.Source: §4.1 · p. 4
• Each layer's objective is linear regression in its own matrix, so a gradient step yields a delta-rule update.Source: §4.1 · p. 5
• DeltaTTT combines a nonlinear query readout with layerwise delta-rule updates that enable chunkwise parallel computation.Each layer's tokenwise updates can reuse DeltaNet's chunkwise scan algorithm.Source: §4 · p. 4, §4.1 · p. 5
• DeltaTTT-pre uses the previous first-layer state to form the intermediate representation, allowing the two current-token writes to proceed concurrently.The authors adopt Pre as the default because its ablation performance is comparable to Post and it has greater potential concurrency.Source: §4.1 · p. 5
• The update-schedule comparison distinguishes methods by the fast-weight state used for writes, state-evolution granularity, and query-readout state.DeltaTTT combines state-dependent tokenwise updates with nonlinear memory and chunkwise parallel computation.Source: Table 2 · p. 5, §4.1 · p. 5

5.1 EXPERIMENTAL SETUP

• The experiments train approximately 0.54B-parameter models for 92.5B tokens and evaluate recurrent configurations with a 512-token sliding-window attention setting.The setup includes DeltaNet- and LaCT-based DeltaTTT models, recurrent baselines, and full attention as an additional reference.Source: §5.1 · pp. 5–6, Table 3 · p. 6

5.2 MAIN RESULTS

• DeltaTTT gains 12.60 points in single-needle retrieval over parallel LaCT and improves training loss and retrieval across both backbones.Improvements over DeltaNet and Gated DeltaNet on the DeltaNet backbone support applicability across backbone designs.Source: Table 3 · p. 6, §5.2 · p. 6

5.4 TRAINING EFFICIENCY

• At 32K training length, DeltaTTT is 37% faster than full attention and uses 31% less peak memory.It is also 65% faster than serial LaCT and 28% faster than parallel LaCT, with comparable or lower peak memory usage.Source: Table 4 · p. 7, §5.4 · p. 7

5.3 CONTEXT USE AND POSITION-DEPENDENT LOSS

• DeltaTTT has the lowest average loss among evaluated LaCT-backbone recurrent models and narrows the gap to full attention, which retains the lowest overall loss.Its advantage over parallel LaCT grows at later positions; losses are averaged in 1K-token bins across 43 held-out documents.Source: Figure 3 · p. 6, §5.3 · p. 7

5.5 ABLATIONS

• Pre and Post achieve similar training losses and comparable single-needle accuracy; Pre enables concurrent inference updates by removing the second-layer write address's dependence on the current first-layer update.Source: §5.5 · p. 7
• The full model has the lowest training loss and highest retrieval accuracy; disabling second-layer updates reduces retrieval, while retaining a static second layer performs worse than the single-layer model.Removing the learnable initial memory further degrades single-layer retrieval.Source: §5.5 · p. 7, Table 5 · p. 7

5.6 MECHANISM ANALYSIS

• With both initial matrices zeroed, learned B0 still develops a pronounced diagonal structure, consistent with a prior for refinement.The intended division of labor is for A to produce a value-aligned intermediate representation and B to adjust it toward the value.Source: Figure 4 · p. 8, §5.6 · p. 8
• On the same 32K validation sequence, DeltaTTT reduces reconstruction MSE by 81.88%, versus 70.88% for zero-initialized DeltaNet and 76.44% for learned-initialization DeltaNet.Each checkpoint is evaluated using its self-generated K/V pairs.Source: Table 6 · p. 8, §5.6 · p. 8

TTT and nonlinear fast weights

• TTT uses inner learners for self-supervised sequence-context updates; related work explores chunking, momentum, forgetting, hierarchical memory, and alternative adaptation objectives.The cited work also studies deeper fast-weight networks, KV fitting, and TTT adaptations in vision.Source: §6 · pp. 8–9

Delta-rule memories and online optimization

• Delta-rule work develops recurrent associative writes and chunkwise evaluation, while DeltaTTT assigns explicit value-reconstruction targets to each layer and derives the second-layer write address from the first layer's prediction.The local objectives replace joint inner-loop optimization while a nonlinear intermediate representation couples the memory stages.Source: §6 · p. 9

Layerwise and local learning

• Layerwise-learning methods use local objectives to ease optimization or relax update dependencies; DeltaTTT applies local learning to context-dependent online memory states while training the outer language model end to end.Source: §6 · p. 9

• Greater memory expressivity alone does not guarantee effective learning within a single pass.Source: §7 · p. 9
• DeltaTTT decomposes nonlinear memory optimization into local linear reconstruction objectives with layerwise delta-rule updates, preserving nonlinear composition and enabling chunkwise parallel evaluation.Source: §7 · p. 9
• Experiments on two backbones demonstrate improvements in language modeling and associative retrieval.Source: §7 · p. 9

8 LIMITATIONS

• Current DeltaTTT is slower and uses more memory than DeltaNet.The authors attribute potential efficiency gains to future kernel implementation improvements.Source: §8 · p. 9
• Experiments cover only two-layer MLP memories with a 1× hidden expansion ratio and relatively small models under a bounded training-token budget.Wider or deeper memories and larger-scale training remain to be assessed.Source: §8 · pp. 9–10

A CHUNKWISE SCAN OF DELTATTT

• DeltaTTT-pre’s chunkwise algorithm uses a first scan to update A and form representations, then a second scan to update B and produce outputs.The algorithm returns the concatenated output chunks and updated memories A and B.Source: A CHUNKWISE SCAN OF DELTATTT · p. 13

B FIXED-KV RECONSTRUCTION EXPERIMENTS

• The experiment fits memories to fixed K/V caches from a frozen 0.5B full-attention model trained on approximately 10B tokens.It uses eight 32K sequences across all 24 layers, with four sequences for learning-rate selection and four for reporting.Source: B FIXED-KV RECONSTRUCTION EXPERIMENTS · p. 13
• The comparison uses a zero-initialized linear memory and two-layer SiLU MLPs with hidden widths of 1×, 2×, and 4×, optimized by ordered SGD replay.Chunk sizes range from 1 to 512 in powers of two; each chunk produces one update under key-normalized squared loss.Source: B FIXED-KV RECONSTRUCTION EXPERIMENTS · p. 13
• Learning rates are tuned separately by model and chunk size, and reconstruction MSE is averaged over complete sequences and reporting layers.Equal numbers of passes do not imply equal computational cost because smaller chunks require more updates.Source: B FIXED-KV RECONSTRUCTION EXPERIMENTS · p. 14

C MAIN-TABLE EXPERIMENTAL DETAILS

• All recurrent models use SWA with a 512-token window.Table 7 summarizes the shared training configuration.Source: C MAIN-TABLE EXPERIMENTAL DETAILS · p. 14, Table 7 · p. 14
• DeltaTTT replaces the recurrent memory module in both backbones while retaining backbone-specific feature processing and output integration.DeltaNet uses RoPE only in the SWA branch and directly adds normalized memory output; LaCT also applies RoPE to memory Q/K and uses input-dependent output modulation.Source: C MAIN-TABLE EXPERIMENTAL DETAILS · p. 14
• The main-table DeltaTTT models use Pre updates, 16 memory heads of width 64, shared memory/SWA Q/K/V projections, and summed branch outputs.Initial states use A0 ∼N(0, 0.022) and B0 = LR + 0.5I, with learned rank-32 factors L and R.Source: C MAIN-TABLE EXPERIMENTAL DETAILS · p. 14

D INITIAL-STATE ABLATION

• Different learnable initializations yield similar training loss and single-needle retrieval accuracy on the DeltaNet backbone.Table 8 reports single-needle accuracy as an average percentage.Source: D INITIAL-STATE ABLATION · p. 14, Table 8 · p. 14
• The reported similarity covers both training loss and single-needle retrieval accuracy.The supplied passage gives no numerical values for these comparisons.Source: D INITIAL-STATE ABLATION · p. 14
• The ablation compares DeltaTTT initial-state choices on the DeltaNet backbone.The table’s Single metric is average single-needle accuracy (%).Source: Table 8 · p. 14