PaperlayerView exact revision on arXiv

Source-linked AI summary

TraceRelay: Attention-Aligned Recurrence over Rolling Traces

Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

arXiv:2610.11743v1cs.LG

TL;DR

Attention can access multiple past representations, but incurs quadratic attention arithmetic and a context-growing key–value cache; longer inputs also do not guarantee effective use of history [1]–[3]. TraceRelay organizes recurrence around attention with delayed writes, phase transport, and a left reader over the augmented stream. At the longest evaluated lengths, wider centers perform better on Dyck and narrower centers on Most-Freq, motivating task-specific study rather than a general rule.

  • Problem

    Attention offers direct access to past representations but has quadratic arithmetic and a context-growing key–value cache, and accepting longer inputs does not guarantee effective history use [1]–[3].

    Source: §1 · p. 1

  • Method

    TraceRelay forms right-context-conditioned delayed increments, transports them through phase recurrence, and exposes the rolling representations to left-window attention.

    Source: §1 · p. 2, §2.1 · p. 2, §2.3 · p. 3

  • Results

    At the longest evaluated lengths, mean OOD performance favors wider centers on Dyck and narrower centers on Most-Freq.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The contrasting results motivate further study of task-specific representational bandwidth without establishing a general rule across tasks or configurations.

    Source: §5 · p. 10

    The study does not measure language-model quality, agent performance, or compute-matched long-context scaling, and bounded phase values do not guarantee reliable retention at arbitrary horizons.

    Source: §5 · p. 10

Abstract

from arXiv · show

We present TraceRelay, an attention-aligned recurrent architecture that distributes persistent representations over a rolling sequence of low-dimensional traces. Local right looking attention forms increments from lower-layer representations; delivery is delayed until all attended inputs are in the causal past. A fixed additive phase recurrence accumulates the delayed increments, and left-looking attention reads the resulting residual augmented stream. A stride-wise prefix sum supports parallel prefill and bounded-buffer continuation. We study 36 small-model runs on Equal Repeats, bounded Dyck closing-type prediction, and causal Most-Freq generation, using three seeds per setting. At trained length 256, Equal Repeats models with recurrent phase inheritance reach 98.81-99.69% accuracy versus 50.73-51.63% for separately trained variants without inheritance, despite the latter receiving more updates. Accuracy drops sharply at lengths beyond the training range. At the longest evaluated lengths, models with more dimensions in the middle layer's recurrent traces perform better on Dyck (76.34% versus 55.49% close accuracy at length 4096), whereas models with fewer trace dimensions perform better on five-symbol Most-Freq (70.74% versus 55.60% exact generation at length 1024). These contrasting cases motivate further study of how the size of recurrent representations should be chosen for different tasks, without establishing a general rule across tasks or model configurations.

1 Introduction

Attention offers access to multiple past representations but carries quadratic attention arithmetic and a context-growing key–value cache, while longer inputs do not guarantee effective use of history [1]–[3]. TraceRelay organizes recurrence around attention, using delayed writes and phase accumulation before a left reader; the study evaluates this design and its long-narrow capacity hypothesis.

1 Introduction

  • Full attention incurs quadratic attention arithmetic and a key–value cache that grows with context, while longer inputs do not guarantee effective use of history [1]–[3].The paper frames these as distinct concerns: accessible history is not necessarily useful persistent information.Source: §1 · p. 1
  • Recurrence and attention can each create difficulties: larger sliding windows may weaken long-term memory learning, while very small windows can harm tasks benefiting from local context.Mamba and Mamba-2 demonstrate input-dependent state updates and efficient parallel formulations of structured recurrence [4] [5].Source: §1 · p. 1
  • TraceRelay asks whether attention can learn what to write and retrieve while recurrence uses a simple temporal transition.It forms candidate updates from short right context, delivers them after a causal delay, accumulates them as phase increments, and exposes the result to left-window attention.Source: §1 · pp. 1–2
  • The proposed long-narrow hypothesis pairs a longer read window with a narrower projected channel to expose intermediate recurrent representations while limiting direct-access capacity.H-Net motivates non-uniform capacity allocation [7]; TraceRelay instead preserves the temporal grid while varying trace width and read horizon.Source: §1 · p. 2
  • The study contributes a delayed-write, phase-accumulation, left-read computation graph, a causal and parallel execution account, and a controlled evaluation protocol.Its empirical questions concern inheritance, bounded structured state, and compact aggregates for autoregressive generation.Source: §1 · p. 2

2 Attention-aligned rolling recurrence

TraceRelay uses right-looking attention to form delayed increments, transports them through interleaved phase traces, and lets left-looking attention read the augmented stream. The design preserves causal computation and supports parallel prefill, while its long-narrow capacity rationale remains a task-dependent hypothesis.

2.1 Notation

  • L and R count preceding and following positions visible to Left and Right attention; each window includes its center, and causality requires W > R.The trace width is d; both attention outputs have width d, while learned maps project phase features and Left output into the residual stream.Source: §2.1 · p. 2

2.2 Delayed right-looking writes

  • Right attention forms a candidate update from the lower-layer stream, then delivers it at position i + W after its attended inputs are in the causal past.The reference graph shows the stride-wise phase scan and readout before the residual merge and Left attention; Right does not read this layer’s phase.Source: §2.2 · pp. 2–3, Figure 1 · p. 3

2.3 Fixed phase accumulation, learned increments

  • The phase recurrence accumulates learned, input-dependent increments across W interleaved trace lineages, with W separating successive states of one lineage.The transport law has no learned decay or dense state-transition matrix.Source: §2.3 · p. 3

2.4 Left attention reads a recurrently augmented stream

  • Left attention reads a merge of lower-layer features and phase readout, rather than a phase-only bank.Until a lineage receives a valid delayed update, its readout is zero; the rolling collection acts as distributed working state, and information read in one layer can condition a writer in the next.Source: §2.4 · p. 4

2.5 Causality and parallel execution

  • All increments can be computed from the complete lower-layer stream before a stride-wise cumulative-sum phase scan, enabling parallel prefill and saved-phase-tail continuation.The latest lower-layer position used by an arriving update is earlier than its arrival, and Left attention reads only positions at or before the current position, yielding a causal decoder.Source: §2.5 · p. 4

2.6 Why long and narrow?

  • The long-narrow design is a hypothesis: longer read access may expose more trace generations, while narrower projected channels may limit direct-access capacity.The experiments vary center width with fixed Left attention windows and do not independently establish a benefit from a longer center window or that narrowing is necessary.Source: §2.6 · p. 4

3 Controlled evaluation

The controlled evaluation compares phase inheritance and recurrent-trace widths across three sequence tasks using a shared reference core and three seeds per setting. Task-specific protocols and ID-based checkpoint selection define the scope of the comparisons.

Controlled evaluation

  • All three tasks use a three-layer, four-head reference core with trace widths [64, dc, 64] for dc ∈ {16, 32, 64}, while Left and Right attention windows and relay stride remain fixed.The Left and Right windows see up to 15 and 7 preceding or following positions, respectively, and the stride is 8.Source: §3 · pp. 4–5

Equal Repeats

  • Equal Repeats compares separately trained, geometry-matched models that differ only in whether the previous phase is inherited.Inputs use the historical semantics of Delétang et al. [9], with uniform training lengths from 32 through 256.Source: §3 · p. 5

Bounded Dyck

  • Bounded Dyck evaluates closing-type prediction for k = 8, m = 10 on training lengths from 32 through 256, without supplying stack metadata.The diagnostic reports accuracy by input length, matching-opener distance, and depth; it does not measure full-language likelihood or unbounded-stack recognition.Source: §3 · p. 5

Most-Freq

  • Most-Freq tests five-symbol rankings with separated counts and greedy generation requiring the full ranking followed by EOS.The four-count minimum gap means this sweep does not exercise the tie rule; evaluation feeds back only the model's generated symbols.Source: §3 · p. 5

Selection and reporting

  • Checkpoints are selected by unweighted ID-validation means, and all 36 planned runs are retained with three-seed means and sample standard deviations.Runs stop after two consecutive validation means reach 99% or at the 20,000-update cap; OOD results do not determine selection or stopping, and training exposure is unequal.Source: §3 · pp. 5–6

4 Results

The results show a strong in-range Equal Repeats advantage from phase inheritance that largely disappears out of distribution, alongside opposite width orderings on long-length Dyck and Most-Freq evaluations. The reported comparisons are based on ID-selected checkpoints and three training seeds.

Evaluation protocol

  • The common comparison uses T = 256 for ID and T = 1024 for OOD, with task-specific metrics and sample standard deviations across three seeds.Error bars are not confidence intervals over independent evaluation datasets.Source: §4 · p. 6, Table 1 · p. 9

Equal Repeats

  • At T = 256, inherited Equal Repeats reaches 98.81%, 99.24%, and 99.69% at widths 64, 32, and 16, versus 51.03%, 51.63%, and 50.73% without inheritance; the separation largely disappears OOD.The models without inheritance use 20,000 updates, while inherited models stop after 4,250–10,250 updates.Source: Figure 2 · p. 6, §4 · p. 6

Bounded Dyck

  • At T = 4096, 77.98% of evaluated Dyck closes have matching distance 1–8, while width 64 reaches 27.70 ± 6.14% at distance 513–1024 on 1,639 closes.The next distance bin contains only 43 closes, no example has distance above 2048, and distance-slice ordering is not uniformly monotone in width.Source: §4 · p. 7, Figure 3 · p. 7
  • At T = 4096, Dyck close accuracy is 76.34 ± 1.81% for width 64, 63.47 ± 8.36% for width 32, and 55.49 ± 10.49% for width 16.The wider center has the highest mean length-OOD accuracy in this sweep.Source: §4 · pp. 6–7, Figure 3 · p. 7

Most-Freq

  • At T = 1024, Most-Freq exact ranking-plus-EOS accuracy is 55.60 ± 12.82% for width 64, 65.40 ± 2.39% for width 32, and 70.74 ± 4.95% for width 16.Top-1 accuracy remains 86.82–90.76%, higher than full-output exact accuracy; narrower centers have higher OOD means in this sweep.Source: §4 · pp. 7–8, Figure 4 · p. 8

Optimization and summary

  • Most-Freq width comparisons are matched-protocol comparisons, not equal-compute experiments, because width changes parameter count and early stopping changes token exposure.Width 64 stops after a mean 2,750 updates, versus 4,000 for width 32 and 3,667 for width 16.Source: §4 · p. 8

5 Discussion and scope

TraceRelay’s contribution is an attention-organized recurrence graph, not a new primitive; its width sweep shows task-specific preferences without establishing a general rule. The study does not test language-model quality, agent performance, or compute-matched long-context scaling.

5 Discussion and scope

  • TraceRelay combines right-context-conditioned delayed writes, phase transport, and a left reader, so its contribution is at the dependency-graph level rather than a new primitive.The paper distinguishes this arrangement from prior attention-and-memory combinations, including block-level states, segment memory tokens, feedback, and prefiller-generated targets.Source: §5 · p. 10
  • The fixed transport law leaves learned selection and content formation to attention and projections; related complex-valued state updates do not validate TraceRelay’s phase update.The discussion cites Mamba-2 and Gated DeltaNet for efficient structured recurrence, and Mamba-3 as independent evidence for complex-valued state updates.Source: §5 · p. 10
  • At the longest evaluated lengths, wider centers perform better on Dyck and narrower centers on Most-Freq, without establishing a general width rule.The contrast is reported as a case study of task-specific bandwidth preferences in bounded nested-state prediction and compact frequency aggregation.Source: §5 · p. 10
  • The study does not measure language-model quality, agent performance, or compute-matched long-context scaling, and bounded phase values do not guarantee retention at arbitrary horizons.Width changes parameter count and optimization, while early-stopped runs can have different training exposures.Source: §5 · p. 10

6 Conclusion

TraceRelay uses delayed writes, phase transport, and left-looking attention to organize recurrence around attention. Its results show task-dependent width preferences, but do not establish reliable length extrapolation, a universal optimal width, or an efficiency advantage.

6 Conclusion

  • At the longest evaluated lengths, mean OOD performance favors wider centers on Dyck and narrower centers on Most-Freq.The contrast motivates further study of task-specific representational bandwidth allocation, not a universal rule.Source: §6 · p. 10
  • TraceRelay’s right-looking writer prepares causally delayed increments, a phase scan transports them, and a left-looking reader accesses rolling representations.The design preserves the token grid and supports parallel prefill and bounded inference buffers without an explicit slot-commit subsystem.Source: §6 · p. 10
  • Equal Repeats shows a strong contribution from recurrent inheritance within the training range but does not establish reliable length extrapolation.The conclusion identifies separate variation of read horizon, writer width, and training exposure as a next step for testing the long-narrow hypothesis.Source: §6 · p. 10

A Executed task protocols and diagnostics

The appendix records the executed task protocols, sampling conditions, diagnostic metrics, and training exposure. Comparisons are bounded by the specified samplers, validation-based stopping, and unequal training exposure.

A Executed task protocols and diagnostics

  • All tasks sweep d_c ∈{64, 32, 16} with three seeds per variant and width; two consecutive ID validation means ≥99% trigger stopping, while OOD results never control training.Validation is performed every 250 updates, and Table 5 records stopping and selected steps and token exposure.Source: Table 2 · p. 13
  • The task adapters share fixed core geometry, AdamW, a 100-update linear warm-up, constant learning rate, and batch size 64, while pilot window and symbol-count choices were exploratory.Evaluation examples match across seeds and variants at each length; reported uncertainty is sample SD across trained seeds.Source: A Executed task protocols and… · p. 13
  • Equal Repeats uses inclusive run-length sampling, assigns all-equal examples to class 0, and has a 33.34% majority baseline in the tested evaluation sets.The phase-inheritance comparison pairs evaluation fingerprints and uses fresh initialization; variants have identical parameter counts within each width.Source: A Executed task protocols and… · pp. 13–14
  • Bounded Dyck sampling begins and ends with an empty stack, caps depth at m, and does not require reaching m; at T=4096, diagnostic counts are targets from a shared 2,000-sequence set.Empty bins are unavailable rather than zero-accuracy observations, and longer examples may change opener-distance and depth distributions.Source: A Executed task protocols and… · p. 14, Table 3 · p. 14
  • Most-Freq’s separated-count sampler is not uniform over count vectors, and greedy exact generation requires the complete ranking and termination.Table 4 reports Top-2, set exact, and termination separately; set exact ignores rank, while termination records predicted EOS within the decode limit.Source: A Executed task protocols and… · p. 14, Table 4 · p. 15
  • Training histories show unsmoothed logged validation points, while width and early stopping can change parameter count and token exposure; these are not equal-compute comparisons.Table 5 records actual steps, tokens, and wall time; wall time includes periodic validation and is not an isolated kernel-throughput measure.Source: A Executed task protocols and… · p. 16, Figure 5 · p. 16, Table 5 · p. 17

Related papers

Loading 2610.11743v1…