PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Weight Attention for Continual Learning

Yifan Zhang, Steve Ta, Jasper Zhang, Jichen Feng, Shuzhen Li, Yongxin Zhang, Yifeng Liu, Huizhuo Yuan, Mengdi Wang, Quanquan Gu, Andrew Chi-Chih Yao

arXiv:2608.27763v1cs.LGcs.CLstat.ML

TL;DR

Long-context recurrent models need fast-memory updates that respect what was available at prediction time, but their temporal alignment and local objectives are often unclear. This paper derives prefix-aligned normalized updates for regression and inner-product objectives, finding competitive language modeling and improved arithmetic length extrapolation.

  • Problem

    Recurrent fast-memory models require temporally aligned local objectives for causal continual learning, but their update rules often leave this alignment underspecified.

    Source: §1 · p. 1, C. Falcon-3: sliding-window mini-batch · p. 2

  • Method

    The paper formulates recurrent sequence modeling as online continual learning and derives normalized Falcon updates for prefix-aligned regression and inner-product fast-memory objectives.

    Source: §7 · p. 28

  • Results

    Representative variants remain competitive in language modeling, while the best inner-product variants improve arithmetic length extrapolation.

    Source: §5.1 · p. 26, §7 · p. 28

  • Takeaways & Limitations

    Explicit temporal alignment separates fast-memory learning objectives from plasticity, forgetting, and bounded rehearsal in recurrent sequence models.

    Source: §7 · p. 28

Abstract

from arXiv · show

Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step $t$ is the prefix-aligned pair $(\mathbf{x}_t,\mathbf{y}_t)=(φ(\mathbf{k}_{t-1}),\mathbf{v}_t)$. The common same-step association $(φ(\mathbf{k}_t),\mathbf{v}_t)$ remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.

1 Introduction

The introduction frames long-context modeling as both an efficiency challenge and a continual-learning problem: recurrent fast-memory systems replace growing KV caches with fixed-size states whose update rules must respect causal temporal alignment. It motivates Falcon’s local-learning perspective alongside recurrent alternatives that offer linear training and constant per-step inference.

Motivation

  • Transformers capture global dependencies but incur O(N^2) attention and growing KV-cache costs on long sequences.Both the attention matrix and KV-cache memory traffic become major bottlenecks as sequence length N grows.Source: §1 · p. 1
  • Long-context modeling requires online binding of new evidence without catastrophic interference, making temporal alignment essential for causal fast-memory updates [3] [23] [32] [38].Transformers use a growing external KV cache, whereas SSMs and fast-weight models compress memory into a fixed-size recurrent state whose update acts as a local learning rule.Source: §1 · p. 1

Recurrent alternatives

  • Linear Attention [20], Fast Weight Programmers and Delta Networks [30], RWKV [26], and Mamba [12] provide recurrent alternatives with O(N) training and O(1) per-step inference.Their state-update equations are often presented architecturally, motivating analysis of the underlying local objective.Source: §1 · p. 1

Falcon-1

  • Falcon-1 uses scalar plasticity: one scalar η_t is shared across all d_v value channels, so all columns are updated equally.This identifies the scalar-update mechanism described for Falcon-1.Source: §1 · p. 2

B. Falcon-2: per-channel plasticity · C. Falcon-3: sliding-window mini-batch

The paper aligns fast-memory writes with prefix prediction and derives normalized Falcon updates spanning per-channel plasticity and sliding-window mini-batches. These regression and inner-product variants preserve explicit plasticity, forgetting, normalization, and chunk-parallel training semantics.

C. Falcon-3: sliding-window mini-batch

  • Falcon-3 derives a normalized first-order sliding-window mini-batch regression write for the aligned fast-memory objective.The supplied passages identify the windowed mini-batch step and place Falcon-3 within the normalized regression family.Source: C. Falcon-3: sliding-window mini-batch · pp. 2–3
  • The updated state is used in all three update cases, including the sliding-window mini-batch step.The figure compares Falcon-1, Falcon-2, and Falcon-3 as fast-weight updates on a fixed-size matrix state.Source: C. Falcon-3: sliding-window mini-batch · p. 2, Figure 1 · p. 2
  • Prefix prediction uses (φ(k_t−1), v_t), whereas same-step pairing remains causal but trains a different fast-memory objective.The state predicts the newly revealed target from the prefix feature available when that target was predicted.Source: C. Falcon-3: sliding-window mini-batch · pp. 2–3
  • The resulting recurrences support recurrent, masked-parallel, and chunk-parallel implementations compatible with SSD-style training.The paper evaluates representative regression and inner-product variants in language modeling and arithmetic extrapolation.Source: C. Falcon-3: sliding-window mini-batch · p. 3

B. Falcon-2: per-channel plasticity

  • Falcon-2 extends normalized regression writes with per-column gains, while β controls plasticity, λ_t controls shrinkage, and η matches local smoothness.This provides the per-channel plasticity mechanism within the Falcon regression family.Source: C. Falcon-3: sliding-window mini-batch · p. 3
  • Falcon-1A, Falcon-2A, and Falcon-3A are inner-product counterparts retaining the same plasticity and forgetting semantics.Falcon-2A is specifically the per-column inner-product analogue of Falcon-2.Source: C. Falcon-3: sliding-window mini-batch · p. 3

C. Falcon-3: sliding-window mini-batch

  • In the inner-product variants, the denominator normalizes write magnitude rather than curvature.This distinguishes their normalization interpretation from the regression family.Source: C. Falcon-3: sliding-window mini-batch · p. 3

2 Background

This background connects selective state-space models, linear attention, and Delta Networks through recurrent state compression and online value-retrieval updates. It also establishes the paper’s shifted read-after-write alignment and normalized versus unnormalized memory formulations.

2.1 State Space Models

  • Selective SSMs make state transitions input-dependent; Mamba-2’s SSD duality links recurrent SSMs to causal linear attention with chunk-parallel training and constant-state inference [12] [9].The transition parameters B, C, and Δ depend on the current input, while Mamba-2 simplifies A to scalar or diagonal structure.Source: §2.1 · p. 3

2.2 Linear Attention

  • Linear attention replaces softmax with kernel features, compressing context into a recurrent matrix state and normalizer instead of incurring standard attention’s O(N^2) complexity [20].The normalized read uses the denominator state z_t; shifted indexing writes v_t under the previous feature φ(k_{t−1}) for next-latent prediction.Source: §2.2 · pp. 3–4
  • Unnormalized linear attention drops the denominator and normalizer, using explicit decay or gain control for memory scale; its numerator update is gradient descent on an inner-product objective.With zero decay the write is additive rank-one Hebbian learning, while positive decay adds scalar shrinkage; Mamba-2 relates this recurrence to a class of SSMs [9].Source: §2.2 · p. 4

2.3 Delta Networks

  • Delta Networks interpret sequence modeling as online learning of a value-retrieval function, updating fast weights by an error-driven gradient step on instantaneous squared error [30].The update performs targeted rank-one shrinkage or editing along the current key direction, becoming an orthogonal projection only when η_t = 1/∥k_t∥^2.Source: §2.3 · p. 4

A. Recurrent Form (causal scan)

The recurrent causal-scan form maintains a fixed-size matrix state while processing key-value streams sequentially. It supports O(N) time and O(1) inference state, with equivalent masked-parallel and chunk-parallel views.

Recurrent form

Masked-parallel form

  • Unrolling from S_0 = 0 yields an equivalent causal masked-attention computation that is fully parallel over the sequence dimension.This masked form costs O(N^2) time.Source: Figure 2 · p. 5

Chunk-parallel form

  • Chunk-parallel computation interpolates between recurrent scanning and masked attention by combining intra-chunk parallel attention with fixed-size state propagation across chunks.The sequence is divided into M chunks of size C, with state S^(k) passed between chunks.Source: Figure 2 · p. 5

3 Autoregressive Next-Latent Prediction … C. Falcon-3 / Falcon-3A: Sliding Window

The paper frames fast-weight updates as causal online learning under next-latent alignment, where each target is paired with the preceding prefix feature. Falcon derives normalized regression and inner-product updates, extending scalar and per-column rules to sliding-window dynamics with explicit forgetting and stability controls.

3 Autoregressive Next-Latent Prediction

  • Under read-after-write semantics, the causal training pair is (ϕ(k_t−1), v_t), whereas same-step pairing defines a different fast-memory objective.The initial feature sentinel is x_1 := 0, with η_1 := 0 when shrinkage is explicit.Source: §3 · p. 6
  • The recurrent state is optimized online as a ridge-regression predictor, using Online Gradient Descent rather than offline cumulative-loss minimization.The local fast-memory objective is differentiated through during training but is not an additional supervised loss beyond autoregressive likelihood.Source: §3 · p. 6

3.1 Online Gradient Descent Update

  • Normalized gradient steps yield NLMS-style updates that are numerically stable and reduce to classical NLMS when λ_t = 0 and ε = 0.The smoothness scale is L_t = ∥x_t∥^2 + λ_t; positive ε stabilizes the denominator and only decreases the step size.Source: §3.1 · p. 7

3.2 Analysis

  • The normalized update guarantees per-step descent of the instantaneous regularized objective, but not monotonic decrease of cumulative online loss or outer autoregressive loss.The stabilized parametrization uses η = β/(L+ε), while zero-gradient degenerate cases use a no-op update.Source: §3.2 · p. 7

3.3 Delta Networks as Regression

  • The regression view recovers Delta Networks under shifted assignments, while replacing squared error with a negative inner-product objective produces additive Linear-Attention-style writes.The next-latent assignment yields the one-step-shifted variant used by Falcon-3A.Source: §3.3 · pp. 7–8

4 Falcon: Fast Weight Attention

  • Falcon derives normalized squared-error and negative-inner-product families with explicit forgetting controls and scalar, per-column, and sliding-window dynamics.The suffix “A” denotes inner-product objectives, while indices 1, 2, and 3 denote scalar, per-column, and sliding-window rules.Source: §4 · pp. 8–9

B. Falcon-1 / Falcon-2: Next-Latent Prediction

  • Falcon-1 and Falcon-2 use next-latent prediction with scalar and per-channel learning rates, respectively, extending NLMS normalization across value columns.Falcon-2 applies independently within each head and is computationally tractable through a vectorized dual form.Source: B. Falcon-1 / Falcon-2: Next-Latent… · p. 9

C. Falcon-3 / Falcon-3A: Sliding Window

  • Falcon-3 and Falcon-3A apply sliding-window mini-batch regression or inner-product updates over causal prefix-feature/value pairs, using a window of nominal size B.The realized active-window size is B_t := |I_t| ≤ B.Source: Figure 3 · p. 9

C. Chunk-wise Parallel Form (per-chunk WY/Gram) · A. Recurrent Form (per-channel rates)

The section presents recurrent and chunk-parallel realizations of Falcon fast-weight updates, while specifying normalization, timing, feature-sign, and boundary conditions that determine their exact behavior. It also clarifies when sliding-window states support exact continuation and how their effective memory horizon differs from strict locality.

C. Chunk-wise Parallel Form (per-chunk WY/Gram)

  • Figure 4 gives equivalent recurrent, masked-parallel, and chunk-parallel views of the Falcon-1 rank-one kernel, with positive-decay renormalization reducing λt > 0 to the no-ridge WY form.The recurrent view maintains a fixed-size matrix state and applies a rank-one edit before writing the new value.Source: Figure 4 · p. 12, C. Chunk-wise Parallel Form (per-chunk… · p. 12
  • Chunk-parallel Falcon-2 uses a per-chunk Gram/WY representation, with shared key-Gram and channel-dependent triangular computations applied independently within each head.The representation separates injected-value and projected-history paths within each chunk.Source: C. Chunk-wise Parallel Form (per-chunk… · p. 12

A. Recurrent Form (per-channel rates)

  • The recurrent form performs an O(N)-time causal scan with O(1) sequential state and is equivalent to a per-channel WY representation of the no-ridge rank-one kernel.This provides the sequential counterpart to the chunk-parallel formulation.Source: A. Recurrent Form (per-channel rates) · p. 13

C.1 Scaled Fast Weight Recurrences

  • Normalized scaled recurrences preserve attention-like behavior under nonnegative feature maps only when γt ≥ 0; clamping enforces γt ≥ εγ > 0, while ηt ≡ 1 and λt ≡ 0 recover the additive normalized case.When the clamp activates, the shrinkage path uses the positive-decay surrogate with effective coefficient αt/ηt for ηt > 0.Source: §C.1 · p. 35

C.2 Signed-Feature Normalizers

  • Signed-feature normalized reads can still have zero or negative denominators, so denominator-free inner-product reads are the default unless an explicit sign-stable normalization is used.A positive additive constant alone does not guarantee a safe denominator for signed features.Source: §C.2 · p. 35

C.3 Step Size and Gating Conventions

  • Falcon’s ridge shrinkage uses αt := ηtλt and carry γt := 1 − αt, tying global decay to the local objective and normalized step-size parameterization.This distinguishes Falcon’s convention from a separately specified gated Delta carry.Source: §C.3 · p. 36

C.4 Relation to Test-Time Training and Titans

  • Falcon’s fast-weight update is a constrained single-step test-time-regression mechanism that enforces strict autoregressive causality through the prefix feature xt = ϕ(kt−1) after vt is revealed.Falcon-3 extends this perspective as a sliding-window specialization compatible with SSD-style chunk-parallel training.Source: §C.4 · p. 36

C.7 Sliding-Window Boundary State

  • Sliding-window Falcon-3 and Falcon-3A require a fixed-width tail of recent causal pairs for exact continuation; resetting or passing only the matrix state changes boundary updates.Away from boundaries, each token’s direct injection sums to η independently of B, but later carry factors can preserve its effect, so B controls an effective horizon rather than strict locality.Source: §C.7 · p. 39

B. Parallel Form (batched per-channel TriSolve) · B. Falcon-2A: per-channel inner-product write

This section gives a chunk-parallel, batched per-channel TriSolve implementation for Falcon-2A-style writes, with stable positive-decay handling and shared triangular factors. It also distinguishes the next-latent inner-product objective from standard Linear Attention writes and defines the Falcon-A family.

B. Parallel Form (batched per-channel TriSolve)

  • The chunk-parallel implementation partitions the sequence into M=L/C chunks and performs M batched solves with O(NC) time, versus O(N^2) time for full batched solves.Within each chunk, log-prefix decay factors provide numerically stable positive-decay renormalization before constructing the kernel terms.Source: B. Parallel Form (batched per-channel… · pp. 13–14
  • Falcon-2’s batched kernel solves one triangular residual system per value channel while sharing the write-feature direction and residual structure.The per-channel formulation lets each value channel follow its own plasticity trajectory; merged residual solves preserve the recurrence and asymptotic complexity.Source: Figure 5 · p. 13, B. Parallel Form (batched per-channel… · p. 13
  • The recurrence enforces nonnegative per-channel step sizes and ridge coefficients, clamps shrinkage to preserve a positive decay floor, and reduces to the no-ridge case when λt=0.When the clamp is inactive, the update uses ridge coefficient λt; otherwise it uses the effective coefficient αt,j/ηt,j when ηt,j>0.Source: B. Parallel Form (batched per-channel… · pp. 13–14

4.3 Inner Product Loss (Linear Attention and Mamba-2)

  • The inner-product objective aligns state predictions with targets, while λt=0 yields a purely additive update because the objective has no finite minimizer.The optional L2 penalty supplies the shrinkage term used by the recurrent update.Source: §4.3 · p. 14
  • Unlike standard Linear Attention’s unshifted write (ϕ(kt),vt), the next-latent framework uses the prefix-aligned pair (ϕ(kt−1),vt), producing a one-step-shifted write.The unshifted choice matches the usual Linear Attention write, whereas the shifted choice is used by the next-latent formulation.Source: §4.3 · p. 14
  • The suffix A denotes inner-product variants: Falcon-1A is scalar, Falcon-2A is per-column, and Falcon-3A is sliding-window, with ηt interpreted as an energy-normalized write gain.For this family, αt=ηtλt is derived rather than independently parameterized.Source: §4.3 · p. 14

B.1 Language Model Training Setup

  • Language-model runs use bfloat16 AdamW with tied embeddings, Pre-Norm RMSNorm, no biases or dropout, cosine-decayed learning rate 10^-3, and gradient clipping at 1.0.Each run uses a single 4-GPU node with NVIDIA H100 or H200 GPUs.Source: §B.1 · p. 35

C. Falcon-3A: sliding-window inner-product write

Falcon-3A develops a sliding-window, energy-normalized inner-product write over prefix-aligned causal pairs, with positive decay and window-size-stable normalization. Its recurrent formulation supports chunk-parallel masked-attention execution while characterizing descent and boundary cases.

C. Falcon-3A: sliding-window inner-product write

  • Falcon-3A’s recurrent update reads and writes a fixed-size state using shifted features x_t = ϕ(k_{t−1}), with decay-weighted masked and chunk-wise equivalents.The inner-product family writes targets directly rather than residuals; Falcon-2A uses per-column decay-weighted masks.Source: Figure 6 · p. 15, Figure 7 · p. 16, Figure 8 · p. 18

4.4 Mini-batch Update Rule for Regression (Falcon-3)

  • Falcon-3 averages a finite history window through one mini-batch squared-error gradient step, evaluating all residuals at the pre-update state.This reduces noise accumulation and is the sliding-window specialization of the internal-objective view, with strict next-latent alignment.Source: §4.4 · pp. 19, 21
  • Falcon-3 supports chunk-parallel training through a fixed-rank-B recurrence, while exact continuation across segments requires the final state and a tail of the last B−1 causal pairs.Falcon-3A additionally admits an explicit masked linear-attention form for vectorized sequence training.Source: §4.4 · pp. 19, 20, 23
  • Window-averaged Falcon-3 normalization prevents the injection and decay fraction from being systematically amplified as nominal window size B increases.With RMS-normalized write features, the denominator remains O(d_x) rather than O(Bd_x); the implementation uses positive-decay clamping as a surrogate when needed.Source: §4.4 · pp. 21–22

C. Falcon-3A: sliding-window inner-product write

  • Falcon-3A applies the sliding-window principle to the inner-product objective, averaging recent prefix-aligned writes while retaining dimensionless gains, shrinkage coefficients, and normalized step sizes.The window uses at most B pairs and skips the boundary update at t = 1.Source: §4.5 · p. 23

4.5 Mini-batch Update Rule for Inner Product Loss (Falcon-3A)

  • For λt > 0, window-energy normalization yields ηt < 2/λt and per-step descent for the unclamped inner-product update.If positive-decay clamping activates, the effective ridge coefficient is αt/ηt; when λt = 0, normalization stabilizes additive writes but does not guarantee a bounded objective.Source: §4.5 · p. 23
  • A window of size B = 1 reduces Falcon-3A to Falcon-1A, while λt ≡ 0 gives purely additive writes and the per-column analogue is Falcon-2A.Because the objective uses a window average, typical window-statistic scale and decay fraction do not systematically grow with B.Source: §4.5 · p. 24

4.6 Parallel (Attention) Form of Mini-batch Inner Product Update

  • The update uses γt = 1 − αt with αt = min(ηtλt, 1 − εγ), while ηt is normalized by window energy, ridge strength, and ε.The implementation sets η1 = 0, α1 = 0, and γ1 = 1, with log-space decay evaluation for numerical stability.Source: §4.6 · p. 24
  • Falcon-3A rewrites the recurrent sliding-window update as decayed boundary history plus an unnormalized dot-product attention matrix with a structured causal mask, enabling GPU-parallel training.The stacked form is O = Diag(δ)QS0 + (QX^T⊙M)V; with S0 = 0, only the masked-attention term remains.Source: §4.6 · p. 25

5 Experiments … C Implementation Details

The experiments show that aligned, normalized fast-weight updates remain competitive in language modeling and improve variable-length arithmetic extrapolation. The surrounding sections position these updates within efficient sequence modeling, online optimization, and continual-learning perspectives, with implementation details specifying causal boundaries and normalized updates.

5 Experiments

  • The evaluation trains 124M–130M-parameter models for approximately 49.2B tokens and reports teacher-forced perplexity plus zero-shot and one-shot downstream accuracy.Models use sequence length 1,024, global batch size 480, and 100,000 optimization steps; fast-weight blocks generally use QK-RMSNorm and short convolutions.Source: §5 · p. 25, §5.1 · p. 26

5.1 Language modeling experiments

  • On FineWeb-Edu, Falcon-1.3 reaches 17.10 perplexity, while Falcon-1A.2 and Falcon-1.3 lead listed downstream averages at 49.30 zero-shot and 49.54 one-shot, respectively.Gated DeltaNet is the strongest baseline at 17.32, and Falcon-1A.3 is the best evaluated inner-product variant at 17.40.Source: §5.1 · p. 26, Table 1 · p. 26, Table 2 · p. 27

5.2 Variable-length arithmetic addition tasks

  • Falcon-3A.3 achieves 87.2 mean accuracy on variable-digit addition, followed by Falcon-1A.3 at 85.9, outperforming the reported RetNet/LightningAttn and Transformer baselines.The task uses variable widths from 1 to 32 digits and evaluates teacher-forced extrapolation over 33–48 digits.Source: §5.2 · p. 27, Table 3 · p. 28

6 Related Work

  • The work reframes state-space and linear-attention updates as objective-induced online learning, deriving causally aligned NLMS-stabilized regression and sliding-window rules compatible with chunk-parallel training.This contrasts with additive or gated accumulation while connecting the method to fast weights, adaptive filtering, and implicit optimization.Source: §6 · p. 28

7 Conclusion

  • The conclusion presents Falcon-1/2/3 as regression variants and Falcon-1A/2A/3A as inner-product variants, separating alignment, plasticity, forgetting, and bounded rehearsal.The numbering denotes scalar, per-column, and sliding-window dynamics, while the A suffix denotes the inner-product objective.Source: §7 · p. 28

Appendix

  • The appendix specifies an online ridge update with boundary feature x_1 := 0, next-latent features x_t := ϕ(k_{t−1}), nonnegative ridge coefficients, and normalized gain β_t/(||x_t||^2 + λ_t + ε).The state initializes at S_0 = 0, and the gain is set to zero when the denominator vanishes.Source: Appendix · p. 34

A More on Background

  • The background notation defines x_t as the write feature, y_t as the target, S_t as the fast-weight state, and the residual as r_t := y_t − S_t^⊤x_t.Under next-latent alignment, the write feature is typically ϕ(k_{t−1}) and the target is typically v_t.Source: A More on Background · p. 34

C.6 Implementation Notes and Related Update Rules · D Falcon-3A Mask and Backward Pass

The implementation notes clarify decay, normalization, and numerical-stability conventions for Falcon updates, including chunk-local renormalization and sliding-window computation. They also position the normalized, causally aligned rule as complementary to gated Delta-style architectures.

C.6 Implementation Notes and Related Update Rules

  • The NLMS gain β_t lies in (0, 2), and values above 1 produce a stable sign flip along the current write-feature direction in the unclamped ε = 0 analysis [11].The notation distinguishes the dimensionless gain β_t from the resulting normalized step size η_t.Source: §C.6 · p. 38
  • Ridge regularization yields a scalar carry on the orthogonal subspace, while the write-feature direction has eigenvalue 1 − β_t when ε = 0.The full transition uses γ_t = 1 − η_tλ_t; positive-decay clamping can instead implement an effective coefficient eλ_t ≤ λ_t.Source: §C.6 · p. 38
  • Positive-decay chunk renormalization accumulates log γ_t within each chunk rather than forming fragile global products in mixed precision.Exact equivalence requires rescaling both the step size and write target; normalized readouts additionally require undoing the common rescaling or rescaling the stabilizer.Source: §C.6 · p. 38
  • Optional short causal convolution may improve associative recall [10] [24], but it is orthogonal to and unnecessary for the fast-memory update.The notation distinguishes feature-space and value-space identity matrices through I_dx and I_dv.Source: §C.6 · p. 38
  • Scale-coupled ridge uses sliding-window smoothness or energy statistics, with the current shrinkage multiplier detached while the step-size denominator remains live.This separates statistics used for shrinkage from the statistic used to normalize the update.Source: §C.6 · p. 39
  • Falcon-3’s sliding-regression denominator can use the exact top eigenvalue of a small active-window Gram matrix or a few power iterations.The active window has size B_t ≤ B, making the B_t × B_t computation tractable in the intended regime.Source: §C.6 · p. 39
  • The formulation complements RWKV-7 and Kimi Linear Attention [34] by supplying causal next-latent alignment, explicit one-step key shifting, and NLMS normalization.The normalization and alignment can replace local updates inside delta-style blocks independently of their gating or mixing parameterization.Source: §C.6 · p. 39

D.1 Mask Identities and Chunk-Local Evaluation … E.3 Chunk-Parallel Backward Pass

The appendices derive masked chunk-local evaluation, stable log-decay renormalization, differentiable backward rules, and WY-based chunk-parallel algorithms for DeltaNet. Together, these formulations preserve causal read-after-write semantics while enabling parallel forward and backward computation.

D.1 Mask Identities and Chunk-Local Evaluation

  • The mask combines a decayed-history term with masked attention, using step-dependent coefficients that can remain nonzero across all prior positions and need not decay monotonically with lag.Each coefficient aggregates up to B decayed write contributions because tokens are re-injected within the local window.Source: §D.1 · p. 39
  • Chunk-local evaluation resets log-prefix decay at boundaries and propagates only within-chunk differences plus chunk-exit decay, avoiding long-range underflow in reduced precision.The resulting implementation uses parallel per-chunk precomputation, a short inter-chunk recurrence, and parallel output materialization.Source: §D.1 · p. 40

D.2 Backward Pass

  • The backward pass differentiates through clamped decay, structured masks, local log-prefix decays, boundary propagation, and output materialization while respecting the hard-set boundary step.When clamping is active, the log-decay adjoint is gated by the unclamped-step indicator; the boundary parameters receive no gradient.Source: §D.2 · p. 41

E DeltaNet WY Parallelization

  • With ridge decay, DeltaNet’s transition is rescaled into an equivalent no-ridge rank-one recurrence, allowing the WY machinery to be reused unchanged.The rescaling uses cumulative scalar decay and transformed states, step sizes, and values; chunk kernels reset the log-prefix to prevent numerical overflow or underflow.Source: E DeltaNet WY Parallelization · p. 41

E.1 Affine WY Form

  • For a chunk, the affine no-ridge transition is a product of rank-one updates represented by a single WY transform with update matrix U and triangular mixing matrix T.The factorization uses u_t = sqrt(η_t) x_t and compresses the accumulated transition into chunk-local operators.Source: §E.1 · p. 42

E.2 Chunk-Parallel Forward Pass

  • The chunk-parallel forward algorithm has three phases: parallel intra-chunk operator construction, sequential boundary-state recurrence, and parallel causal output materialization.A single residual triangular solve replaces two shared solves per chunk without changing the recurrence.Source: §E.2 · pp. 42–43

E.3 Chunk-Parallel Backward Pass

  • The chunk-parallel backward algorithm reverses output materialization, inter-chunk recurrence, and local structure in parallel or reverse-sequential phases to produce gradients for queries, inputs, values, step sizes, and the initial state.Its core identities backpropagate through triangular solves and Gram matrices while preserving the causal diagonal mask.Source: §E.3 · p. 44
  • The backward implementation safely handles zero step sizes by masking their gradients instead of forming 0/0, including boundary cases where the shifted feature vanishes.Chunk gradients are reshaped back to the full sequence after returning gradients for all required inputs and state variables.Source: §E.3 · p. 45

F Falcon-2 Parallel Implementation … G.3 Forward and Backward Algorithms

This appendix develops chunk-parallel Falcon-2 with per-value-channel adaptive rates and introduces Falcon-1 as a shared-dynamics variant. It specifies normalized, decay-stable forward and backward algorithms, including gradient computation and boundary conditions for recurrent inputs and the initial state.

F Falcon-2 Parallel Implementation

  • Falcon-2 assigns each value channel its own learning-rate trajectory, but shared key geometry enables vectorized chunk recurrences and batched channel-specific triangular solves.The shared Gram matrix is computed once per chunk, while per-channel rates modulate the corresponding systems.Source: F Falcon-2 Parallel Implementation · p. 45, §F.1 · p. 46, §F.2 · p. 46

F.2 Chunk-Parallel Algorithm

  • Falcon-2’s chunk formulation uses causal read-after-write scores, shared Gram and score matrices, channel-wise step-size modulation, and a batched residual solve over value channels.The diagonal is included through tril(·, 0), and keys, queries, values, and rates use explicitly specified chunk layouts.Source: §F.2 · p. 46
  • The consolidated Falcon-2 forward kernel merges two residual paths into one batched triangular solve, reducing the dominant forward solve count from two to one per chunk.Its asymptotic complexity is unchanged, with O(dC^2), O(dvC^2), and O(d dv C) terms, while the rate dependence remains confined to C × C systems.Source: §F.2 · pp. 46–47

G Falcon with Shared Dynamics

  • Falcon-1 enforces a shared adaptive learning rate across value channels, reducing distinct C × C triangular systems per chunk from dv to 1 and using one multi-right-hand-side solve with O(dvC2) cost.The state update remains O(d dv C), while savings come from eliminating construction and factorization of dv distinct C × C systems.Source: G Falcon with Shared Dynamics · pp. 47–48

G.1 Scalar versus Vector Step Sizes

  • Falcon-1 derives a scalar NLMS step size from write-feature energy, making the transition matrix shared across columns while allowing stored values to differ.The shared scalar dynamics are applied within each attention head after head factorization, unlike Falcon-2’s per-channel rates.Source: §G.1 · p. 48
  • Positive-decay renormalization converts the decayed recurrence to a no-ridge rank-one WY kernel, with clamping and chunk-local log prefixes preventing full-sequence exponential overflow.The implementation computes log1p(−αt) in fp32, resets prefixes at chunk boundaries, and rescales outputs and states locally.Source: §G.1 · p. 48

G.3 Forward and Backward Algorithms

  • Falcon-1 forward and backward algorithms run the single-solve WY kernel on locally renormalized chunks, then backpropagate through NLMS normalization, decay prefixes, clamping, and rescaling.The backward pass reverses chunks, unscales outputs and states, invokes WY gradients, and propagates gradients through the normalized step-size and decay pathways.Source: §G.3 · pp. 49–50
  • The backward pass unscales step-size gradients, propagates gradients through local prefix sums, and differentiates the positive-decay renormalization while zeroing gradients for clamped decay values.It backpropagates through log γ_t = log1p(−α_t), α_t = min(η_tλ_t, 1 − εγ), and accumulates gradients for η_t and λ_t.Source: §G.3 · p. 50
  • The algorithm differentiates the normalized step size η_t = β_t/(∥x_t∥2 + λ_t + ε), accumulating gradients for β_t and the denominator term.The denominator is defined as d_t = ∥x_t∥2 + λ_t + ε, with derivatives propagated through β_t and d_t.Source: §G.3 · p. 50
  • Boundary constants bη_1 := εη and log γ_1 := 0 carry no gradient, and the routine returns gradients for Q, x, V, β, λ, and the initial recurrent state.The initial-state gradient is returned as ¯S_init = ¯S(1).Source: §G.3 · p. 51

G.4 Complexity Analysis … H.2 Matrix-Valued CDEs and Low-Rank Drivers

The appendix presents Falcon-3’s ParallelFlow implementation, showing how low-rank sliding-window recurrences admit associative chunk scans and structured triangular solves. It also reports that Falcon-1’s residual formulation reduces per-chunk system-construction overhead and improves GPU practicality.

G.4 Complexity Analysis

  • Falcon-1 removes per-chunk construction and factorization of dv distinct C × C systems, while retaining one GPU-friendly multi-RHS triangular solve.The residual formulation removes one forward TriSolve in Falcon-1 and Falcon-2; Falcon-1 additionally shares system construction across value channels, eliminating O(dvC2) construction overhead.Source: §G.4 · p. 51, Table 4 · p. 51

H Falcon-3 ParallelFlow Implementation

  • Falcon-3 converts the sliding regression window of size B into a rank-B affine recurrence implemented with ParallelFlow without dense transition matrices or token-by-token scans.ParallelFlow trains and evaluates the recurrence without materializing dense state-transition matrices or scanning sequentially over tokens.Source: H Falcon-3 ParallelFlow Implementation · p. 51
  • TensorInv uses a block-strict-causal mask for pre-update residuals, while inclusive block-causal output masking implements read-after-write within each chunk.Same-time rank components do not interact inside tensorInv, and each residual is evaluated at the pre-update state St−1.Source: H Falcon-3 ParallelFlow Implementation · p. 51

H.1 Affine Chunk Map

  • Falcon-3 chunk maps compose associatively, allowing Algorithm 3’s boundary updates to run either sequentially or as an associative scan.The affine chunk map provides the scan-compatible composition rule for Phase 2.Source: §H.1 · p. 51

H.2 Matrix-Valued CDEs and Low-Rank Drivers

  • ParallelFlow separates temporal dynamics from input computation by computing chunk-local propagators and injections in parallel, then linking chunks through a global associative scan.The matrix-valued CDE solution factors through a linear propagator, which supports this chunked computation.Source: §H.2 · p. 52
  • Rank-R drivers with a shared left factor yield a compact flow representation and reduce chunk computation to structured causal solves involving low-rank drivers.The low-rank structure enables efficient propagation, while tensorInv applies the structured inverse without explicit dense dx × dx chunk propagators.Source: §H.2 · p. 52
  • The compressed O(L2cR + d) complexity is misleading because dominant dimension-dependent terms remain in the structured solve and dx-, dv-dependent matrix multiplications.The appendix explicitly cautions that the displayed compressed asymptotic expression hides these dominant terms.Source: §H.2 · p. 52
  • TensorInv applies C−1 to two low-rank-driver right-hand sides, and the resulting solutions combine with the incoming boundary state to produce the chunk-exit state.Algorithm 3 also handles scalar decay γt through chunk-local renormalization.Source: §H.2 · p. 53

H.3 Mapping Falcon-3 to ParallelFlow

This section maps Falcon-3’s sliding-regression update to a fixed-rank low-rank affine recurrence, enabling exact chunk-parallel execution after zero-padding active windows. The resulting rank-B implementation avoids dense state propagation and is substantially cheaper than full O(L^2) attention for small windows.

The Falcon-3 Recurrence

  • The Falcon-3 recurrence uses a mini-batch window of size B with shifted write features x_j = ϕ(k_{j−1}), while chunked implementations handle scalar decay through chunk-local log-decay renormalization.The unkernelized case sets ϕ to the identity.Source: §H.3 · p. 53

Identification of Drivers

  • Active windows are zero-padded to width B, producing fixed-size write-feature and value blocks whose padded columns vanish identically.The active index set contains at most B entries, enumerated in increasing order before padding.Source: §H.3 · pp. 53–54
  • At t = 1, the update is a strict no-op, and positive-decay renormalization replaces η_t with η̃_t while applying a common rescaling to the step-t value block.The triangular solve uses a block-strict-causal mask over time-rank pairs, with residuals evaluated at the pre-update state S_{t−1}.Source: §H.3 · p. 54

Complexity

  • Falcon-3 maps exactly to a rank-B low-rank affine recurrence after zero-padding each active sliding window, enabling chunk-parallel scans without discrete approximation.The fixed rank is R = B, and the block-strict-causal mask preserves pre-update residual evaluation and prevents same-time padded-window interactions.Source: §H.3 · pp. 53–54
  • For small windows such as B = 4, the rank-B overhead remains far cheaper than full O(L2) attention while avoiding dense dx × dx state propagation.The mapping permits use of the tensorInv algorithm without approximation at the discrete level.Source: §H.3 · p. 54

Related papers

Loading 2608.27763v1…