PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuum Attention

Afshin Khadangi

arXiv:2610.20586v1cs.LGcs.AI

TL;DR

Discrete token-position attention motivates a continuous retrieval alternative. Continuum Attention integrates query-induced relevance over piecewise-linear trajectories of Transformer keys and values, and in controlled byte-level language modeling it achieves lower validation NLL than matched softmax at two context lengths.

  • Problem

    Discrete attention aggregates values at sampled token positions, motivating investigation of retrieval by integration along a continuous trajectory of adjacent memory vectors.

    Source: §1 · p. 2

  • Method

    Continuum Attention constructs piecewise-linear trajectories from Transformer keys and values and integrates a query-induced exponential relevance density over them.

    Source: §1 · pp. 2–3

  • Results

    0.0200 and 0.0480 nats per byte: CA achieves lower mean validation NLL than fixed-temperature softmax at context lengths 192 and 512, respectively.

    Source: §4 · p. 12

  • Takeaways & Limitations

    Controlled byte-level experiments show promising predictive benefits from continuous memory integration without additional trainable parameters.

    Source: §5 · p. 13

    Evidence is limited to approximately five-million-parameter Transformers trained on a byte-level TinyStories subset, leaving generality to larger models, other corpora, and longer contexts unassessed.

    Source: §4 · p. 12

Abstract

from arXiv · show

We introduce Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve. Integrating the density and interpolated values yields exact normalized attention from closed form exponential moments, including half token boundary contributions. The gradients follow analytically from moments of order at most two, enabling parallel GPU implementations while retaining quadratic sequence complexity and the standard Transformer decoder. Compared with ordinary softmax and focal attention, CA reduces mean validation negative log likelihood by 0.0200 and 0.0480 nats per byte, respectively, winning all paired comparisons. An additional disjoint validation slice yields a 0.0457 nats per byte advantage over the strongest selected softmax baseline, with a paired 95% confidence interval. These findings motivate continuous integration as a promising attention alternative while leaving larger scale generalization open.

1 Introduction

Discrete token-position aggregation motivates Continuum Attention (CA), which integrates over a continuous trajectory of Transformer keys and values. Its query-dependent exponential relevance density admits exact evaluation, and controlled byte-level language modeling shows lower validation NLL than matched softmax attention at two context lengths.

1 Introduction

  • Discrete attention aggregates values at sampled token positions, motivating retrieval by integration along a trajectory defined by adjacent memory vectors.Earlier work explored alternative attention distributions, adaptive receptive fields, efficient operators, and continuous-domain attention.Source: §1 · p. 2
  • CA constructs a continuous memory trajectory from Transformer keys and values, connecting consecutive vectors by piecewise linear interpolation and integrating query-induced exponential relevance.This causal construction directly uses Transformer representations rather than fitting a separate continuous value function.Source: §1 · p. 2
  • Half-unit constant extensions at the visible memory boundaries preserve the intended token-mass convention, including uniform weighting when relevance scores are equal.The piecewise-linear segments yield exponential integrals expressible through moments; moments through order one determine forward aggregation, while order two completes analytical score derivatives.Source: §1 · pp. 2–3
  • Each query controls effective retrieval locations within memory segments, with query-dependent adaptation remaining present when attention temperature is fixed.Independently learned projections define distinct trajectories within each layer and attention head.Source: §1 · p. 2
  • The operator supports parallel GPU evaluation, retains quadratic sequence complexity, and fits within a standard causal Transformer decoder.The paper also reports parallel GPU implementations and controlled comparisons across multiple training seeds and context lengths.Source: §1 · p. 3
  • On TinyStories byte-level modeling, CA reduced mean validation NLL versus matched softmax attention at fixed temperature 0.30 by 0.0200 and 0.0480 nats per byte at context lengths 192 and 512, respectively.Improvements occurred in all five paired seed comparisons at each length.Source: §1 · p. 3

2 Method

Continuum Attention replaces discrete lookup with an exact integral over a continuous memory trajectory, using query-dependent relevance and analytical segment contributions. Its construction retains the causal Transformer decoder while supporting analytical gradients and parallel evaluation.

2.1 Continuous Memory Representation

  • CA interpolates adjacent keys and values into a continuous memory trajectory, extending the visible endpoints by half a unit while preserving token order and causal visibility.Each interval uses linear interpolation between neighboring memory vectors.Source: §2.1 · p. 3

2.2 Query Adaptive Relevance Density

  • Each query induces an exponential relevance density over the continuous memory, so different queries can form different relevance profiles over the same segments.With fixed temperature τ, the score gain is γ = 1/τ.Source: §2.2 · p. 4

2.3 Exact Segment Integration

  • Closed-form exponential moments give each segment’s unnormalized mass and value contribution, making its integral exactly computable from endpoint scores and values.The normalized mean interpolation coordinate shifts retrieval toward the right or left endpoint as the score difference becomes positive or negative; equal scores select the midpoint.Source: §2.3 · pp. 4–5

2.4 Boundary Contributions and Normalization

  • Half-unit endpoint extensions contribute to the normalizer and value numerator, and equal visible key scores make the normalized operator exactly uniform causal attention.The two half-unit contributions yield o0 = v0 when i = 0.Source: §2.4 · p. 5

2.5 Analytical Differentiation

  • Analytical differentiation uses exponential moments of order at most two, with each endpoint score receiving contributions from incident segments and boundary terms where applicable.The second-order moment suffices because key scores and values are linear within each segment.Source: §2.5 · p. 6

2.6 Numerical Evaluation and Complexity

  • Independent segment integrals permit parallel GPU evaluation without quadrature, while computation remains O(T^2 dh) per head, matching dense softmax attention asymptotically.Series expansions near equal endpoint scores and stable exponential expressions address numerical precision.Source: §2.6 · p. 7

2.7 Integration into Causal Transformers

  • CA replaces attention within the Transformer blocks while retaining the surrounding decoder; fixed-gain CA-P adds no trainable parameters relative to matched softmax attention.CA-P and learned-gain CA-H both use query adaptation; the fixed-gain setup enables comparison under matched model size, temperature, and training budget.Source: §2.7 · p. 7

3 Experiments and Results

In five-seed byte-level language-modeling experiments, fixed-temperature Continuum Attention (CA) outperformed matched softmax attention on development validation at both context lengths. Results on disjoint validation text also favored CA at length 512, while throughput measurements showed a slowdown at that length.

3.1 Experimental Setup

  • Experiments used five seeds for byte-level TinyStories modeling with matched architectures and budgets; configurations varied attention type, temperature, and context length.The study trained eight configurations at two context lengths and evaluated final-step models without validation-based checkpoint selection.Source: §3 · p. 7, §3.1 · pp. 7–8

3.2 Development Validation Results

  • 0.020016 and 0.048045 nats per byte: fixed-temperature CA P beat matched softmax B0.30 at T = 192 and 512, respectively.The improvement occurred across all five paired seeds; the paired 95% confidence intervals were [0.013779, 0.026253] at T = 192 and [0.033504, 0.062587] at T = 512.Source: Table 1 · p. 9, §3.2 · p. 9
  • Temperature mattered: CA S, initialized at effective temperature 1, improved on correspondingly initialized softmax F, but fixed-temperature softmax (τ = 0.30) achieved lower NLL than S at both context lengths, motivating identical-scaling comparisons.Table 1 reports mean development validation NLL in nats per byte over five training seeds; lower values are better, with equal training-token budgets within each context.Source: §3.2 · p. 9, Table 1 · p. 9

3.3 Fixed and Learned Attention Gains

  • Learned-gain CA H beat matched softmax G by 0.012389 nats per byte at T = 192 and 0.072698 at T = 512, but the T = 192 interval included zero.Fixed-temperature CA P had small mean advantages over H at both lengths, but neither comparison established a clear advantage for learning the gain; P adds no parameters over matched softmax.Source: §3.2 · p. 9, §3.3 · pp. 9–10

3.4 Evaluation on Disjoint Validation Text

  • 0.045741 nats per byte: on disjoint validation text, fixed-temperature CA improved over B0.30 at T = 512, with a paired 95% confidence interval of [0.031634, 0.059848].All five seeds favored CA at T = 512; at T = 192 the exploratory mean improvement was 0.016368 nats per byte, with an interval including zero. The learned-gain variant improved by 0.046990 at T = 512.Source: §3.4 · p. 10, Table 3 · p. 11

3.5 Computational Performance

  • At T = 192, fixed-temperature CA had training throughput comparable to softmax; at T = 512, it was approximately 7.3% slower in training and 39.1% slower in forward throughput.These measurements characterize the tested Metal kernels and context lengths; the larger forward gap reflects the current implementation.Source: Table 4 · p. 11, §3.5 · p. 11

4 Discussion and Limitations

The discussion presents CA as query-adaptive integration over continuous memory segments, reports fixed-temperature gains over softmax alongside a throughput cost, and qualifies the evidence as limited to controlled small-model experiments.

Continuous memory retrieval

  • CA integrates query-dependent relevance across intervals between neighboring memory vectors, assigning adaptive retrieval positions while retaining normalized aggregation as a convex combination of original values.Source: §4 · p. 11

Attention temperature and architectural effects

  • Learned-gain CA beats the corresponding softmax configuration at effective temperature 1, but stronger temperature-adjusted softmax controls achieve better validation performance.Source: §4 · p. 12
  • At fixed temperature 0.30, CA lowers mean validation NLL by 0.0200 and 0.0480 nats per byte at context lengths 192 and 512, respectively, versus softmax.Both paired 95% confidence intervals exclude zero.Source: §4 · p. 12
  • Fixed-temperature CA performs comparably to learned-gain CA, preserving parameter count and temperature in its direct comparison with softmax.The authors suggest this indicates gains under suitable score scaling without additional learned temperature parameters.Source: §4 · p. 12

Relation to continuous attention

  • Building on continuous-attention mathematics, CA’s contribution is its causal Transformer memory construction, exact exponential-moment formulation, and autoregressive implementation.Further comparisons with alternative continuous-attention parameterizations could isolate the influence of its interpolation geometry.Source: §4 · p. 12

Computational considerations

  • At context length 512, fixed-temperature CA has approximately 7.3% lower training throughput and 39.1% lower forward throughput; exact integration adds arithmetic while retaining dense attention’s quadratic sequence complexity.At context length 192, training throughput is comparable to softmax.Source: §4 · p. 12

Limitations

  • The evidence is limited to roughly five-million-parameter Transformers on byte-level TinyStories, two context lengths, and five seeds, so gains on larger models and broader tasks remain unestablished.Source: §4 · pp. 12–13
  • The disjoint validation slice is exploratory, and the tested CA temperatures are limited; the usefulness of linear interpolation may depend on data, representations, and context length.Interpolated locations need not be independently meaningful memory states.Source: §4 · p. 12

5 Conclusion

In controlled byte-level language modeling, fixed-temperature CA outperformed matched softmax across five seeds at context lengths 192 and 512. The conclusion presents exact continuous-memory integration and its analytical, parallel implementation as the method, while noting that broader generality remains to be established.

Conclusion

  • CA exactly integrates over continuous memory trajectories constructed from Transformer keys and values using piecewise linear interpolation and exponential moments.Source: §5 · p. 13
  • Analytical gradients and parallel GPU implementations let CA operate in conventional Transformer decoders while preserving quadratic sequence complexity.Source: §5 · p. 13
  • Fixed-temperature CA outperformed matched softmax across five training seeds at context lengths 192 and 512 in controlled byte-level language modeling.Source: §5 · p. 13

Related papers

Loading 2610.20586v1…