PaperlayerView exact revision on arXiv

Source-linked AI summary

Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference

Siyuan Yan, Guo-Qing Jiang, Yuchen Zhang, Xiaoxing Ma, Ran Zhu, Chun Cao, Jingwei Xu

arXiv:2510.18413v1cs.CL

TL;DR

Long contexts increase the latency and memory burden of self-attention, while existing sparse methods can miss critical KV pairs. Adamas uses transformed, compressed query and key representations for token-level selection; it reports accuracy comparable to full attention at very small budgets, alongside substantial sparsity and speedups.

  • Problem

    Longer contexts make quadratic self-attention costly, while existing sparse approaches can struggle to recall critical KV pairs for each query.

    Source: §1 · p. 1

  • Method

    Adamas transforms and bucketizes queries and keys into 2-bit codes, then uses Manhattan-distance estimation to select top-k KV pairs for sparse attention.

    Source: §1 · p. 2, §3.2 · pp. 4–5

  • Results

    With a 64-token budget, Adamas achieves accuracy comparable to full attention and becomes nearly lossless at 128 tokens, with up to 8× higher sparsity than prior SOTA methods.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Adamas reports up to 4.4× self-attention and 1.5× end-to-end speedups on 32K-length sequences while maintaining attention quality under aggressive sparsity.

    Source: §6 · p. 10

    The stated no-information-loss property applies to applying the Hadamard transform to both Q and K, where HH⊤=I.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Large language models (LLMs) now support context windows of hundreds of thousands to millions of tokens, enabling applications such as long-document summarization, large-scale code synthesis, multi-document question answering and persistent multi-turn dialogue. However, such extended contexts exacerbate the quadratic cost of self-attention, leading to severe latency in autoregressive decoding. Existing sparse attention methods alleviate these costs but rely on heuristic patterns that struggle to recall critical key-value (KV) pairs for each query, resulting in accuracy degradation. We introduce Adamas, a lightweight yet highly accurate sparse attention mechanism designed for long-context inference. Adamas applies the Hadamard transform, bucketization and 2-bit compression to produce compact representations, and leverages Manhattan-distance estimation for efficient top-k selections. Experiments show that Adamas matches the accuracy of full attention with only a 64-token budget, achieves near-lossless performance at 128, and supports up to 8x higher sparsity than prior state-of-the-art (SOTA) methods while delivering up to 4.4x self-attention and 1.5x end-to-end speedups on 32K-length sequences. Remarkably, Adamas attains comparable or even lower perplexity than full attention, underscoring its effectiveness in maintaining accuracy under aggressive sparsity.

1 INTRODUCTION

Longer context windows support tasks requiring global coherence, but quadratic self-attention and KV-cache scaling increase latency and memory overhead. Adamas dynamically selects token-level KV pairs using compressed, transformed representations, reporting high sparsity with accuracy and speedups close to or better than dense attention.

1 INTRODUCTION

  • Longer contexts support global-coherence tasks, but quadratic self-attention and KV-cache scaling increase per-token latency and memory overhead.The cited applications include long-document summarization, code synthesis, multi-document question answering, and persistent multi-turn dialogue.Source: §1 · p. 1
  • Static sparse patterns can miss dynamic query–key interactions, while page-level selection is coarse and can retain redundant tokens, limiting sparsity.The paper contrasts these approaches with Adamas’s dynamic token-level KV selection.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • At 128 tokens, Adamas achieves accuracy comparable to full attention with up to 8× higher sparsity, plus up to 4.4× self-attention and 1.5× end-to-end speedups on 32K sequences.The reported perplexity is even lower than full attention, and prior SOTA methods are surpassed by more than 2×.Source: §1 · p. 2
  • Adamas combines Hadamard transforms, bucketization, 2-bit compression, and Manhattan-distance estimation to select top-k KV pairs for sparse attention.Its GPU kernels fuse bucketization and compression with a lightweight Manhattan-distance estimator.Source: §1 · p. 2
  • Ablations are reported to show that each component, including the Hadamard transform, bucketization, compression, and estimation, contributes to overall performance and efficiency.The introduction states that extensive ablation studies validate these component contributions.Source: §1 · p. 2

2 PRELIMINARIES

Sparse attention approximates query–key attention with selected keys to reduce computation and memory costs. The section also introduces the Hadamard transform and illustrates Adamas’s pipeline from transformed, compressed queries and keys to top-k selection and sparse attention.

2 PRELIMINARIES

  • Adamas’s overview transforms and compresses queries and keys, estimates Manhattan distances between them to select top-k KV pairs, then performs sparse attention with Q and the selected pairs.The overview presents the estimator as the basis for selection and uses the original query in the final sparse-attention step.Source: Figure 2 · p. 3
  • Sparse attention approximates query–key similarity using a carefully chosen subset of keys, reducing computation and memory costs while aiming to preserve model accuracy.Computing QK⊤ against all keys incurs quadratic complexity in sequence length.Source: §2 · p. 3
  • The Hadamard transform projects vectors to a Walsh-function basis, and its recursive construction enables matrix–vector computation in O(d log2 d) rather than O(d2).For dimensions that are not powers of two, the passage describes a factorization-based construction with complexity O(d(m + n)).Source: §2 · p. 3

3 METHOD

Adamas uses Hadamard-transformed, compressed query and key representations to select token-level candidates for sparse attention. Its design combines similarity preservation with compact storage and lightweight estimation.

3.1 MOTIVATIONS

  • The orthogonal Hadamard transform preserves query–key similarity in the transformed domain without information loss.Source: §3.1 · p. 4
  • By redistributing variance and suppressing outliers, the Hadamard transform smooths value distributions so bucketization can approximate similarity with minimal degradation.Source: §3.1 · p. 4

3.2 ADAMAS

  • Adamas selects token-level candidates by applying the Hadamard transform, bucketizing and 2-bit encoding queries and keys, then estimating similarity with Manhattan distance.Source: §3.2 · pp. 4–5
  • Each transformed element is mapped to one of four threshold-defined levels, encoded in 2 bits, and compressed keys are stored in the KV cache for similarity estimation.Packing eight elements into one 16-bit value increases cache size by only 1/16.Source: §3.2 · pp. 4–5
  • Adamas shows the smallest performance drop from full attention on LongBench while maintaining high sparsity.Source: Figure 3 · p. 5

4 EXPERIMENTS

Across LongBench, PG19, and passkey retrieval, Adamas preserves strong accuracy against full attention and sparse baselines; its kernels and end-to-end decoding also accelerate inference. Ablations identify Hadamard smoothing and 2-bit bucketization as design choices supporting accuracy and memory efficiency.

4.1 SETUP

  • Evaluation uses LongChat and Yarn models with PG19, passkey retrieval, six LongBench datasets, and training-free StreamingLLM and Quest baselines.Yarn is used for extremely long-context scenarios up to 100K tokens.Source: §4.1 · pp. 5–6

4.2 EFFICACY EVALUATION

  • Adamas closely tracks full-attention perplexity on PG19 across settings and achieves lower perplexity at larger token budgets.Figure 4 compares StreamingLLM, Quest, Adamas, and full attention under varying token budgets.Source: Figure 4 · p. 6, §4.2 · p. 7

4.3 EFFICIENCY EVALUATION

  • Adamas reaches up to 4.4× self-attention speedup over FlashInfer at a 256-token budget and 1.5× end-to-end speedup on 32K sequences.End-to-end latency increases only marginally with larger token budgets; its speedup is diluted because attention is 30%–40% of decoding time.Source: Figure 5 · p. 7, §4.3 · pp. 7–8, Table 2 · p. 8

4.2 EFFICACY EVALUATION

  • Across six LongBench datasets, Adamas surpasses prior SOTA, while passkey retrieval remains consistently accurate under constrained budgets.The supplied evaluation reports reliable accuracy across diverse long-context scenarios and short and long passkey contexts.Source: Table 1 · p. 6, §4.2 · p. 7

4.4 ABLATION STUDY

  • Without Hadamard, scores stay near zero at small budgets and below baseline even at 4096 tokens; smoothing suppresses outliers and limits bucketization loss.Source: Figure 6 · p. 8, §4.4 · p. 8
  • Adamas-3bit only slightly outperforms 2-bit, while 1-bit is comparable only above 256 tokens; 2-bit avoids higher storage costs.The authors select 2-bit bucketization as a memory-efficient choice without sacrificing accuracy.Source: Figure 6 · p. 8, §4.4 · pp. 8–9
  • L2 performs comparably overall, doing slightly better on single-document QA, while Manhattan has margins on other tasks; L2 favors coherent contexts and L1 noise robustness.Source: Figure 6 · p. 8, §4.4 · p. 9

5 RELATED WORK

Prior sparse-attention work selects tokens through fixed patterns, input-adaptive methods, or training-based mechanisms. These approaches trade off query-specific recall, granularity, deployment scope, and training requirements.

Static sparse methods

  • Static patterns such as sliding windows and attention sinks offer throughput but may discard query-critical tokens and hurt retrieval under constrained budgets (Tang et al., 2024).Source: §5 · p. 9

Dynamic KV selection methods

  • Dynamic methods adapt selection to queries, but Quest's page-level granularity can miss critical tokens, while MInference and FlexPrefill primarily target prefilling (Jiang et al., 2024; Lai et al., 2025).SeerAttention adds learnable gates and training parameters, complicating out-of-the-box inference deployment (Gao et al., 2024).Source: §5 · p. 9

Training-based sparse methods

6 CONCLUSION

Adamas combines compact token representations with Manhattan-distance estimation to select relevant KV pairs. It reports strong accuracy under tight token budgets alongside faster long-context decoding.

6 CONCLUSION

  • At 64 tokens, Adamas achieves accuracy comparable to full attention; at 128 tokens, it is nearly lossless and reaches up to 8× higher sparsity than prior SOTA methods.The conclusion also reports perplexity even lower than full attention.Source: §6 · p. 10
  • On 32K-length sequences, Adamas delivers up to 4.4× self-attention and 1.5× end-to-end speedups.The conclusion attributes its dynamic KV-pair selection to Hadamard transform, bucketization, 2-bit compression, and a Manhattan-distance estimator.Source: §6 · pp. 9–10

A.1 MATHEMATICAL NOTATIONS

Table 3 is a mathematical-notation reference, with symbols paired with descriptions.

A.1 MATHEMATICAL NOTATIONS

  • Table 3 is titled “Mathematical notations” and presents symbols with descriptions.The supplied passage does not identify individual symbols or their meanings.Source: Table 3 · p. 13

A.2 COMPUTATIONAL WORKLOAD

The appendix gives computational-workload and memory-access breakdowns for full attention, Quest, and Adamas, with corresponding tables.

A.2 COMPUTATIONAL WORKLOAD

  • The cited result reports full attention’s computational workload as 4(h + hkv + s)bh FLOPs, specifying this quantity directly as the computational-workload measure for full attention.Source: §A.2 · p. 13
  • Adamas’s computational workload is given as [4h + 4hkv + 2n + 2 + (4log2(k) d + 3)s + 4k] · bh FLOPs.The passage also introduces a Quest workload expression, but its formula is split across fragments.Source: §A.2 · p. 13

A.3 MEMORY ACCESS

  • The appendix separately details memory access for full attention, Quest, and Adamas, and identifies Table 5 as the memory-access table.The supplied memory-access expressions are fragmented.Source: §A.3 · p. 14, Table 5 · p. 14

B LONGBENCH EVALUATIONS

Table 6 provides LongBench evaluation data for StreamingLLM, Quest, and Adamas on LongChat-7b-v1.5-32k.

B LONGBENCH EVALUATIONS

  • Table 6 presents detailed data for Figure 3 for reference.The supplied passages do not state the values or outcome shown in Figure 3.Source: B LONGBENCH EVALUATIONS · p. 15
  • Table 6 covers LongBench evaluations of StreamingLLM, Quest, and Adamas on LongChat-7b-v1.5-32k.The caption establishes the evaluation scope but supplies no metric values or winner.Source: Table 6 · p. 15

C ABLATION STUDIES

The ablation results are evaluated on LongBench using LongChat-7b-v1.5-32k.

C ABLATION STUDIES

  • Table 7 reports ablation results on LongBench with LongChat-7b-v1.5-32k.The caption identifies the evaluation benchmark and model but gives no ablation outcomes.Source: Table 7 · p. 16
Loading 2510.18413v1…