PaperlayerView exact revision on arXiv

Source-linked AI summary

Shared Phase and Retention Control for Efficient Adaptive Spectral Recurrence

Wentao Wang, Hengyu Zhong, Yunhan Jiang, Jialiang An, Meng Lu

arXiv:2609.39082v2cs.LG

TL;DR

Fixed-state recurrent models offer constant-memory inference, but fixed transitions do not adapt how stored information decays or rotates, and per-mode dynamic controls scale with memory size. SPARC uses two shared input-dependent signals to adapt retention and phase across spectral modes while preserving their baseline dynamics. It reports stronger benchmark outcomes and lower recurrent-mixer latency in the evaluated settings.

  • Problem

    Fixed transitions cannot generally adapt how stored information evolves, while independent dynamic controls for every mode make control cost grow with memory size.

    Source: §1 · p. 2

  • Method

    SPARC uses two shared input-dependent signals to jointly modulate retention and phase while preserving mode-specific baseline timescales and frequencies.

    Source: §1 · pp. 2–3

  • Results

    SPARC improves Walker-P return by 9.09% and FordA accuracy by 1.36% over the strongest competing means, and reduces fixed-token recurrent-mixer latency by 18.2%–34.2%.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The results support shared retention and phase control as an efficient mechanism for adapting recurrent memory.

    Source: §7 · p. 10

    Shared controls limit independent adaptation across modes, and evaluation is limited to control and sequence classification at moderate model sizes.

    Source: §6 · p. 9

Abstract

from arXiv · show

As new evidence arrives, a sequence model must update what it remembers and how memory influences predictions. While Transformers incur computation and cache costs scaling with context length, fixed-state recurrent models offer constant-memory inference. However, linear and spectral recurrences traditionally rely on static transitions, failing to dynamically revise how stored representations decay or rotate. While recent selective architectures introduce input-dependent transitions, they assign independent controls to every memory mode, coupling control cost to state capacity. We show that high-dimensional spectral memory does not require high-dimensional control, and introduce Shared Phase and Retention Control for Efficient Adaptive Spectral Recurrence (SPARC). SPARC employs just two input-dependent scalar signals to coordinate memory retention and phase rotation across heterogeneous complex modes, while preserving mode-specific baseline timescales and frequencies. Its diagonal affine recurrence supports parallel associative scans for sequence-level BPTT as well as exact structured Real-Time Recurrent Learning (RTRL) for online credit assignment. Across partially observable continuous control, POPGym, and sequence classification, SPARC achieves a 9.09% relative return improvement on Walker-P and a 1.36% relative accuracy gain on FordA over second-best methods. On an NVIDIA Blackwell GPU, our implementation reduces recurrent-mixer training latency by 18.2%-34.2% in fixed-token workloads and accelerates scans by 3.1x-4.7x over an optimized RG-LRU baseline. These results show that two shared control signals can efficiently govern adaptive spectral memory across online and full-sequence settings. Code is available at https://github.com/Botwwt/sparc.

1 INTRODUCTION

Fixed-state recurrence offers an efficient alternative to growing attention costs, but fixed transitions cannot generally adapt how stored information evolves. SPARC uses two shared input-dependent controls to adapt retention and phase while preserving mode-specific spectral properties, and supports both parallel sequence training and exact structured online learning.

Motivation

  • Fixed-state recurrence avoids history-length-dependent inference-state storage, while attention computation and KV-cache costs grow with context length.Dense global attention computation is quadratic in sequence length for fixed model width, whereas the autoregressive KV cache grows linearly.Source: §1 · p. 1
  • Input-dependent writing alone cannot generally change how differences between historical states evolve, motivating adaptive transition control of retention and phase.In complex spectral memory, transition magnitude controls retention and phase controls rotation.Source: §1 · p. 2

Shared retention–phase control

  • SPARC shares two input-dependent signals across heterogeneous spectral modes while retaining each mode’s learned decay, frequency, input projection, and write response.The shared signals coordinate retention and phase, and also regulate writing through a modal gate and retention-dependent normalization.Source: §1 · pp. 2–3

Exact structured RTRL

  • Its diagonal affine recurrence supports associative scans for full-sequence BPTT and exact structured RTRL for recurrent-layer sensitivities at fixed parameters.The RTRL updates exploit the diagonal state Jacobian, mode-specific parameter dependencies, and low-dimensional shared controllers.Source: §1 · pp. 2–3

Efficient GPU implementations

  • 18.2%–34.2% lower recurrent-mixer training latency and 3.1–4.7× faster scans are reported against an optimized RG-LRU baseline on one Blackwell GPU.The latency comparison uses fixed-token workloads.Source: §1 · p. 3

Evaluation across tasks and training regimes

  • 9.09% higher Walker-P return and 1.36% higher FordA accuracy than the second-best methods are reported for SPARC across the evaluated tasks.The evaluation includes partially observable continuous control, POPGym, and sequence classification.Source: §1 · p. 3

2 RELATED WORK

Prior work develops structured spectral recurrences, input-dependent dynamics, and methods for recurrent credit assignment. SPARC combines shared retention and phase controls with scan-compatible recurrence and exact structured RTRL.

2.1 SPECTRAL RECURRENCE AND PARALLEL COMPUTATION

  • SPARC preserves scan-compatible spectral recurrence while reducing dynamic-control overhead through two shared signals.With controls disabled, linear content, and matched write parameters, it recovers recurrent forms of LRU and linear RTU.Source: §2.1 · p. 3

2.2 INPUT-DEPENDENT RECURRENT DYNAMICS

  • Unlike per-mode or otherwise input-dependent transition parameterizations, SPARC shares two signals across modes while preserving learned baseline decay and frequency.The paper contrasts this control design with RG-LRU gates, GateLoop complex transitions, and Mamba selective dynamics.Source: §2.2 · p. 3

2.3 ONLINE LEARNING FOR RECURRENT MODELS

  • SPARC’s diagonal state Jacobian, mode-local parameters, and shared controllers permit exact structured RTRL without temporal truncation or stochastic approximation.The claim applies at fixed parameters and supplied layer inputs.Source: §2.3 · p. 4

3 METHOD

SPARC adapts a diagonal spectral recurrence with two shared input-dependent controls while retaining mode-specific dynamics. Its affine form supports parallel scans and exact structured online sensitivities, with GPU kernels designed to reduce recurrent computation and storage overhead.

3 METHOD

  • SPARC combines shared input-dependent retention and phase controls with mode-specific decay rates and frequencies in a diagonal spectral recurrence.The recurrence retains the affine form needed for parallel sequence computation and exact structured RTRL.Source: §3 · p. 4

3.1 SPECTRAL RECURRENCE PRELIMINARIES

  • In a fixed spectrum, input-only writing can add content but cannot generally change how differences between historical states are transformed.This distinction motivates adapting the transition itself to alter stored information’s evolution.Source: §3.1 · p. 5

3.2 SHARED ADAPTIVE SPECTRAL DYNAMICS

  • Two shared scalar controls rescale modal decay and adjust rotation, while each transition remains contractive and each mode keeps its learned decay and frequency.The phase adjustment is reduced for persistent modes by the factor 1 − ρ_j,t.Source: §3.2 · p. 5
  • The state update remains affine in the previous state, and sharing controls preserves distinct modal decays, frequencies, input projections, and write responses.With zero-initialized controllers, SPARC recovers the fixed spectrum; under stated write conditions it has the recurrent form of LRU and recovers the linear RTU core’s decay-dependent write scale.Source: §3.2 · p. 5

3.3 PARALLEL BPTT AND STRUCTURED RTRL

  • Associative composition of input-dependent transition and write coefficients enables a work-efficient prefix scan with O(TH) work and O(log T) temporal depth.Backpropagation through coefficient generation and the scan gives full-sequence BPTT without temporal truncation; online use propagates recurrent sensitivities one step at a time.Source: §3.3 · p. 5
  • Exact recurrent-layer derivatives at fixed parameters follow by propagating historical sensitivity and adding current transition and write effects.Mode-local parameters affect only their own modes, while shared controllers reuse instantaneous input Jacobians; readout state gradients contract with the traces to yield recurrent-parameter gradients.Source: §3.3 · p. 6

3.4 GPU-EFFICIENT IMPLEMENTATION

  • Shared control uses 2(D + 1) controller parameters and O(D) projection work per step, versus 2H(D + 1) and O(HD) for independent dense per-mode controls.Dense content projections keep the full cell and structured RTRL at O(HD + H) work; the GPU implementation uses chunked scans, parallel replay, fused operations, and two B × T control arrays instead of a B × T × H complex transition tensor.Source: §3.3 · p. 6, §3.4 · p. 6

4 EXPERIMENTS

The experiments evaluate SPARC across partially observable control, POPGym, and sequence classification, using online sensitivities for reinforcement learning and full-sequence BPTT for classification. Results include leading returns and accuracies on several tasks, while ablations show that phase and write components affect performance.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers masked-observation control, six POPGym tasks, three UCR datasets, permuted MNIST, and sequential grayscale CIFAR-10.All benchmark tasks use three training seeds; classification inputs are scalar sequences without two-dimensional spatial modules.Source: §4 · p. 6, §4.1 · p. 6
  • Reinforcement-learning methods use PPO and stored sensitivities or length-4 TBPTT, while classification uses a four-layer width-64 LRU residual backbone and full-sequence BPTT.State budgets are matched at 384 coordinates for continuous control and 128 for POPGym; classification uses 64 complex modes per SPARC layer.Source: §4.1 · pp. 6–7

4.2 PARTIALLY OBSERVABLE REINFORCEMENT LEARNING

  • SPARC achieves 994.8 mean return on Walker-P, 9.09% above GateLoop’s 911.9, and 4507.8 on Ant-P, 0.70% above LRU.SPARC ranks second on Cheetah-P with a return of 2563.8.Source: §4.2 · p. 7
  • SPARC leads CountRecall, RepeatFirst, and Noisy Pendulum; Noisy Pendulum return rises from GRU’s 0.311 to 0.461.CountRecall improves by 0.050 over RTU, and RepeatFirst reaches 0.829 versus GRU’s 0.811.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3 SEQUENCE MODELING UNDER FULL-SEQUENCE BPTT

  • Under full-sequence BPTT, SPARC leads FordA and permuted MNIST and ties the highest displayed StarLightCurves mean; FordA rises from 95.38% to 96.68%.The FordA increase is a relative gain of 1.36%.Source: §4.3 · p. 7, Table 2 · p. 8

4.4 ABLATION STUDIES

  • Removing adaptive phase lowers Walker-P return from 994.79 to 837.70; removing the phase clock lowers RepeatFirst from 0.829 to 0.288 and sequential CIFAR-10 accuracy from 59.92% to 49.03%.These ablations isolate phase adaptation and the phase clock as distinct components.Source: §4.4 · p. 8
  • Neutralizing the write gate lowers RepeatFirst return to 0.663 while FordA remains at 96.68%; linear content lowers RepeatFirst to 0.123.Gate variation and content saturation are stronger on RepeatFirst than on FordA.Source: §4.4 · p. 8

5 COMPUTATIONAL EFFICIENCY

On a single Blackwell GPU, SPARC reduces scan and recurrent-mixer training times versus an optimized RG-LRU baseline, but runtime gains depend on batch size.

5 COMPUTATIONAL EFFICIENCY

  • The comparison uses matched workloads, BF16 inputs and outputs, and FP32 recurrent accumulation; timings exclude the surrounding model and optimizer.Scan timings follow coefficient generation, while mixer timings include coefficient generation and forward–backward passes.Source: §5 · p. 9
  • At batch size 8 and width 1024, SPARC scans are 3.1–4.7× faster for sequence lengths 2048–16384; at 8192 tokens, mixer training latency falls 18.2%–34.2%.Training-latency reductions are 18.2%–21.0% at width 2048 and 32.1%–34.2% at width 2560.Source: §5 · p. 9
  • SPARC is 42.2% and 21.4% slower at batch sizes 1 and 2, but its runtime ratio is approximately 0.6 at batch sizes 8 and above.Peak allocated memory is roughly 13%–15% lower for most tested batch sizes.Source: §5 · p. 9

6 LIMITATIONS AND FUTURE WORK

Shared control constrains mode-specific adaptation, and the evaluation leaves larger-scale language-model settings untested.

6 LIMITATIONS AND FUTURE WORK

  • Because shared control coordinates changes across modes, individual memory modes cannot adapt independently.The passage identifies this as a limitation of shared control.Source: §6 · p. 9
  • Ablations show task-dependent gains and variation across training seeds, motivating further study of control grouping, initialization, and training stability.Source: §6 · p. 9
  • The evaluation covers control and sequence classification at moderate model sizes; large language models still need assessment of quality, long-context behavior, and training efficiency at scale.Source: §6 · p. 9

7 CONCLUSION

SPARC uses two shared input-dependent signals to adapt high-dimensional spectral memory while preserving mode-specific timescales and frequencies. It supports both parallel full-sequence training and exact recurrent-layer sensitivities, with reported benchmark and GPU efficiency gains.

7 CONCLUSION

  • SPARC’s two shared signals adapt a high-dimensional state while preserving distinct modal timescales and frequencies.Source: §7 · pp. 9–10
  • The affine recurrence supports parallel full-sequence BPTT and exact recurrent-layer sensitivities at fixed parameters.Source: §7 · p. 10
  • SPARC improves Walker-P return by 9.09% and FordA accuracy by 1.36% over the strongest competing means, while reducing fixed-token mixer latency by 18.2%–34.2%.Source: §7 · p. 10

AI USE STATEMENT

The authors disclose generative-AI assistance with implementation and manuscript polishing, and describe technical choices for adaptive transition control, structured recurrent sensitivities, and SPARC’s control and write pathways.

AI USE STATEMENT

  • Generative AI assisted research execution and ideation, primarily in software implementation, debugging, technical discussions, and manuscript polishing; the authors report verifying generated code and text.They state that AI was not used to produce core scientific claims and that the authors take responsibility for the paper.Source: AI USE STATEMENT · p. 10

A WHY WRITING CANNOT GENERALLY REPLACE TRANSITION CONTROL

  • An input-only write can match an adaptive transition across reachable histories only when the transition difference vanishes on their state-difference span.The derivation subtracts the update equations for two reachable states, canceling the input-only writes; if the span is the full state space, the transitions must be equal.Source: A WHY WRITING CANNOT GENERALLY REPLACE… · p. 12

B.5 LOSS CONTRACTION AND TRACE STORAGE

  • Exact structured RTRL stores 4HD + 7H trace entries and uses O(HD+H) update work per step; shared control does not reduce storage to two traces.Each mode retains its own accumulated response to controller parameters.Source: §B.5 · p. 14

B.6 EXACTNESS SCOPE AND STORED-TRACE PPO

  • At fixed recurrent parameters and supplied inputs, Eq. (11) gives full recurrent-layer temporal sensitivities, but omitting upstream history leaves encoder and stacked-layer gradients inexact.Stored rollout traces reused by PPO retain collection-time semantics rather than becoming exact sensitivities under updated parameters.Source: §B.6 · pp. 14–15

B.7 CONTROL-PATH COMPLEXITY

  • For input width D and H modes, shared controls use 2(D + 1) parameters and O(D) projection work, versus 2H(D + 1) and O(HD) for per-mode controls.The complete cell remains O(HD + H) per step because modal coefficient generation and dense content projection remain mode-dependent.Source: §B.7 · p. 15

C COMPONENT ANALYSIS AND CONTROLLED DIAGNOSTICS

  • The component analysis defines interventions, with retrained comparisons reported in Appendix E and checkpoint diagnostics in Appendix F.Matched retrained variants use task protocols, initialization sources, and state sizes; diagnostics measure how trained cells use the components.Source: C COMPONENT ANALYSIS AND CONTROLLED… · p. 15

C.1 CONTROL RANGES AND RETENTION-COUPLED PHASE

  • Retention scaling ranges from just above 1/16 to just below 16 times baseline; phase correction is under a quarter turn and is reduced for persistent modes.The phase-clock factor protects nearly persistent modes from large observation-driven corrections.Source: §C.1 · pp. 15–16

C.2 CONTENT, WRITE GATE, NORMALIZATION, AND GAIN

  • Tanh content bounds projected inputs but can weaken gradients at saturation; mode-wise write gates share the controls, while retention-dependent normalization adjusts write strength.The independently learned reference write gain is separate from the decay parameter.Source: §C.2 · p. 16

C.3 PROGRAMMATICALLY GENERATED DIAGNOSTIC TASKS

On two generated diagnostic tasks, SPARC achieves zero event-order error and the lowest reported dynamical-tracking NMSE. The surrounding sections describe the task protocols, component interventions, and benchmark implementation settings.

C.3 PROGRAMMATICALLY GENERATED DIAGNOSTIC TASKS

  • Zero event-order error for SPARC and RG-LRU compares with 13.7% for RTU; dynamical tracking yields NMSE 0.096 for SPARC, 0.358 for RTU, and 0.382 for RG-LRU.Event-order sequences contain three relevant events among distractors; tracking predicts latent trajectories under event-dependent contraction and rotation.Source: §C.3 · pp. 16–17
  • The generated-task evaluation uses 384 training and 512 evaluation sequences of length 48, with 128 real recurrent-state coordinates and task-specific update counts.All methods share generated data, and every evaluation sequence contributes to each reported score.Source: §C.3 · pp. 16–17

C.4 MATCHED COMPONENT-REMOVAL DEFINITIONS

  • Matched component interventions retain other uses of shared signals where specified, and retrained variants use matched task protocols, initialization sources, and state sizes.Checkpoint diagnostics assess component use separately from retrained task-performance comparisons.Source: §C.4 · p. 17

D.1 SEEDS AND REPORTING CONVENTION

  • Benchmark and ablation results average three training seeds; task-specific seed sets match across methods, while state budgets and learning rates are controlled within tasks.The methods retain their native recurrent equations, gating, normalization, and initialization within a common task wrapper.Source: §D.1 · pp. 17–18

D.2 REINFORCEMENT-LEARNING ARCHITECTURE AND OPTIMIZATION

  • RL uses width-64 encoder and policy/value heads; recurrent sensitivities are stored for most methods, while GRU and Mamba-3 use TBPTT of length 4.POPGym uses Easy configurations; task observations retain native noise, and recurrent states and eligibility traces reset at episode boundaries.Source: §D.2 · p. 18

D.3 SPECTRAL AND CONTROLLER INITIALIZATION

  • Controller vectors and write-gate coefficients initialize to zero, setting both controls to zero and the modal write gate to 1/2.For classification, the squared spectral radius is sampled uniformly from [0.92, 0.9992] and phase uniformly from [0, 2π).Source: §D.3 · p. 19

D.4 FULL-SEQUENCE ARCHITECTURE AND OPTIMIZATION

  • Classification uses four unidirectional residual sequence layers of width 64, with 64 complex SPARC modes per layer and sequence-mean pooling.Training runs 20,000 full-sequence updates with batch size 32; recurrent parameters use Adam, while remaining parameters use AdamW.Source: §D.4 · p. 19

D.5 CLASSIFICATION DATA AND PREPROCESSING

The section specifies classification preprocessing and score aggregation, then points to additional POPGym means and continuous-control learning curves.

D.5 CLASSIFICATION DATA AND PREPROCESSING

  • Classification uses internal validation from a split of the official training partition, with preprocessing fitted only on training-fit examples and no data augmentation.UCR uses a global z-score transformation; pixel normalization is likewise fit on training-fit examples, and splits and pixel permutation stay fixed across runs.Source: §D.5 · p. 19

D.6 SCORE AGGREGATION AND CURVE CONSTRUCTION

  • RL benchmark tables use the three-seed mean of final-window scores, while classification uses selected internal-validation accuracy.Each rollout receives equal weight in the final-window average; learning curves apply a 100-rollout moving average within each run before averaging seeds.Source: §D.6 · p. 20

D.7 ADDITIONAL BENCHMARK RESULTS

  • Table 6 reports complete POPGym means, while Figure 5 shows continuous-control learning curves aggregated across three smoothed seed trajectories.Figure 5 shading denotes one sample standard deviation; curves retain the original environment-step and return scales.Source: §D.7 · p. 20, §D.6 · p. 20

E COMPONENT ABLATIONS

The ablations find the full SPARC design strongest in aggregate, while showing task-dependent effects from adaptive phase, retention, phase-clock, and write-path components.

E COMPONENT ABLATIONS

  • Nine retrained variants test transition and write components across 15 main tasks plus CartPole, retaining task-specific backbones and three training seeds.Transition ablations keep adaptive writing active, and all measured variants use shared control; the unshared-control performance cost was not measured.Source: E COMPONENT ABLATIONS · p. 20, §E.10 · p. 28

E.1 ADAPTIVE PHASE

  • Removing adaptive phase lowers Walker-P return from 994.79 to 837.70, while removing adaptive retention lowers Ant-P return from 4507.80 to 3930.02.Adaptive-phase removal also lowers RepeatFirst and FordA; phase-only transitions lower mean return on all four continuous-control tasks, though some other task means improve.Source: §E.1 · p. 21, §E.2 · p. 22

E.3 STATIC TRANSITION WITH ADAPTIVE WRITING

  • With adaptive writing but a static transition, SPARC scores 4507.80 versus 3794.64 on Ant-P and 0.829 versus 0.534 on RepeatFirst.The static-transition variant nevertheless improves CountRecall and Noisy Pendulum, so the advantage is not uniform across tasks.Source: §E.3 · p. 23

E.4 RETENTION-COUPLED PHASE CLOCK

  • Removing retention-coupled phase scaling lowers RepeatFirst return from 0.829 to 0.288, while a static phase clock reaches 59.68% on sequential CIFAR-10 versus SPARC’s 59.92%.The static clock substantially exceeds the uncoupled variant’s 49.03% on sequential CIFAR-10; the additional benefit of following current retention is less consistent.Source: §E.4 · p. 23, §E.5 · pp. 23, 25

E.6 MODAL WRITE GATE

  • Neutralizing the modal write gate lowers RepeatFirst return from 0.829 to 0.663, while freezing reference gain lowers Walker-P return from 994.79 to 856.69.Removing retention-dependent write normalization lowers Ant-P from 4507.80 to 2983.48, with substantial seed variation.Source: §E.6 · p. 25, §E.7 · p. 25, §E.8 · p. 27

E.9 CONTENT NONLINEARITY AND SATURATION

  • Replacing tanh content with the identity lowers RepeatFirst return from 0.829 to 0.123, while improving permuted MNIST accuracy from 96.87% to 97.07%.SPARC’s RepeatFirst content activations are highly saturated, unlike FordA’s, but the diagnostics alone do not explain run-to-run variation.Source: §E.9 · p. 27

E.10 AGGREGATE COMPARISON AND CONTROL DIMENSIONALITY

  • SPARC has the lowest mean rank, 1.9375, across 16 tasks, and the full design leads this ablation panel despite some variants improving individual tasks.The retention-only variant is closest at 2.0625; ranks use equal task weighting, and per-task differences and seed spreads remain relevant.Source: §E.10 · pp. 27–28

F.2 GATE ACTIVITY AND CONTENT SATURATION

  • Checkpoint diagnostics show RepeatFirst’s gate increases recorded write energy by 1.816, while its clock reduces phase correction to 3.29% of the uncoupled value.The diagnostics use trained checkpoints and recorded inputs; fixed-input interventions produce a mean policy KL of 1.7050 on RepeatFirst versus 0.0014 on Walker-P.Source: §F.2 · p. 29, §F.4 · p. 29, §F.5 · p. 30

F.6 DEPTH AND TRAINING EVOLUTION

The section reports task- and layer-dependent control activity, outlines the chunked GPU scan and replay, and specifies efficiency benchmark conditions and results.

F.6 DEPTH AND TRAINING EVOLUTION

  • Across classification layers, FordA content saturation falls from 9.02% to 0.75%, while UWave normalized-gate temporal standard deviation rises from 0.0119 to 0.0315.During RepeatFirst training, saturation rises from 54.152% to 95.625%, gate standard deviation from 0.331 to 0.735, and state RMS from 2.666 to 22.638.Source: §F.6 · p. 30

G GPU IMPLEMENTATION DETAILS

  • The GPU scan composes chunk updates, prefix-scans chunk summaries, and replays local states; backward computation uses reverse summaries, a reverse scan, and local replay.The online path fuses control evaluation, transition and write reconstruction, state update, and output emission; recurrent accumulation and adjoints use FP32.Source: G GPU IMPLEMENTATION DETAILS · p. 31

G.1 EFFICIENCY BENCHMARK PROTOCOL

  • For the batch sweep at T = 2048 and dmix = 2048, SPARC’s latency ratio versus matched-configuration RG-LRU is approximately 0.6 at batch sizes 8 and above, while peak allocated-memory ratios are approximately 0.85–0.87 for most batch sizes.Source: §G.1 · p. 32

G.2 PERFORMANCE COMPARED WITH A NAIVE IMPLEMENTATION

The accelerated SPARC implementation substantially outpaces its naive sequential implementation in the tested full-sequence workloads and agrees closely with an independently differentiated reference.

G.2 PERFORMANCE COMPARED WITH A NAIVE IMPLEMENTATION

  • Combined forward–backward speedups over the naive implementation range from 137.13× to 869.62× across fixed-token widths, with batch-sweep combined speedup reaching 230.69×.For width 2048, backward speedups span 194.81–1061.91× across sequence lengths; the batch sweep’s combined speedup remains 45.68× at batch size 32.Source: §G.2 · p. 32, Table 19 · p. 32, Table 20 · p. 33

G.3 NUMERICAL AGREEMENT WITH THE NAIVE REFERENCE

  • Forward outputs agree to a maximum absolute error of 5.66 × 10^-7, and all reported normalized output and gradient errors fall between 1.27 × 10^-7 and 1.43 × 10^-6.The comparison uses a length-64 chunked scan and an independently differentiated sequential reference.Source: §G.3 · p. 33, Table 21 · p. 33

Related papers

Loading 2609.39082v2…