PaperlayerView exact revision on arXiv

Source-linked AI summary

Sliding-window beats linear attention

Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron, Emy Gervais

arXiv:2608.28444v1cs.CLcs.LG

TL;DR

LLMs face growing memory and compute costs from quadratic attention, while linear-attention retrofits lacked a proper comparison with simpler baselines. The paper directly compares post-trained linear attention with training-free SWA with sinks and finds that SWA performs as well or better, especially on long-context tasks, at higher speed and lower memory cost.

  • Problem

    Quadratic attention creates growing memory and energy costs, while post-trained linear-attention methods had not been properly compared with SWA with sinks.

  • Method

    The paper directly evaluates training-free SWA with four attention sinks against post-trained linearized attention across multiple LLMs and short- and long-context benchmarks.

  • Results

    SWA with sinks performs as well as or better than most post-trained linear-attention methods, with especially large advantages on long-context tasks and lower memory cost.

  • Takeaways & Limitations

    At context length 256, SWA recovers 20% and 25% of baseline performance on S-NIAH-3 and BABILong, versus 2.2% and 5% for LoLCATs.

  • Takeaways & Limitations

    The study focuses on training-free SWA and does not evaluate hybrid models combining SWA or linear attention with full-attention layers.

Abstract

from arXiv · show

Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previous one. For each additional token, the keys and values must be stored in memory indefinitely, which is unsustainable. Several alternatives have been proposed to fix the quadratic scaling problem, one of which is retrofitting LLMs to use Linear Attention. This idea has attracted a lot of attention, given its promise to solve the quadratic scaling problem with state-of-the-art performance at low cost. However, this line of research has not been properly compared to simpler baselines. In this work, we show that Sliding Window Attention (SWA) with sinks performs as well or better than post-trained Linear Attention models. We observe this across multiple LLMs on various downstream tasks. For long-context reasoning tasks (Needle-in-a-Haystack and BABILong), SWA achieves massively higher performance (2 to 10 times higher than linear attention). SWA requires no post-training, is extremely fast, and requires low memory; therefore, making it an extremely cheap and reliable solution. To reduce inference memory cost, we strongly recommend switching to SWA instead of post-training linear models. Linear attention models may have shown some promise, but they likely require to be trained from scratch or extensive post-training in order to even match SWA.

1 INTRODUCTION

The paper examines whether simpler Sliding Window Attention with sinks can match or outperform post-trained linear attention while reducing inference costs. It reports stronger downstream performance for SWA, especially on long-context tasks, without post-training.

  • Motivation: Quadratic attention makes LLM memory and energy costs grow with context because every token adds keys and values to an expanding KV cache.The growing cache increases both memory use and compute cost as sequences lengthen.
  • Motivation: Linear attention reduces time and memory complexity from quadratic to linear by replacing the growing KV cache with a small fixed inference state.This addresses the scaling problem but does not eliminate the method’s practical drawbacks.
  • Motivation: Linear attention can have lower expressivity, difficult information-retention decisions, and expensive training on hardware optimized for softmax attention.These concerns motivate comparison with simpler alternatives.
  • Findings: The paper finds that SWA with attention sinks—attending to the k previous tokens and the first 4 tokens—outperforms most linearized models on downstream tasks.The comparison spans multiple pretrained LLMs and task types.
  • Evaluation: SWA with sinks is evaluated as a training-free alternative across models from 1.3B to 70B and short- and long-context benchmarks.The authors position this as a direct comparison that avoids post-training and specialized linear kernels.

2 BACKGROUND

The background contrasts self-attention’s quadratic scaling and growing KV cache with SWA and linear attention alternatives. It explains how sinks stabilize SWA and how linear attention achieves fixed-size inference state at the cost of difficult training and retention decisions.

  • Transformers and Self-Attention: Self-attention has complexity O(DL^2) and requires a KV cache whose memory and processing costs grow with sequence length.The cache stores previous keys and values so each new token can attend to the preceding sequence.
  • Sliding Window Attention: SWA attends only to the previous w tokens, while stacked layers expand the effective receptive field to lw after l layers.The local mask therefore produces a larger effective context across depth.
  • Sliding Window Attention: Attention sinks are the first few tokens that transformers use as repositories for unnecessary attention, so excluding them can cause catastrophic degradation.SWA avoids this failure by attending to the first s = 4 tokens alongside the sliding window.
  • Sliding Window Attention: Training-free SWA can receive additional post-training improvements, but this work focuses exclusively on SWA with sinks without training.Learnable attention sinks, in contrast, require additional post-training.
  • Linear Attention: Linear attention replaces the quadratic attention computation with a transformation that yields inference cost O(1) with respect to sequence length.Its state variables remain fixed in size and are updated over time instead of storing all prior keys and values.
  • Linear Attention: Linear attention remains challenging because it must continuously rewrite its state without forgetting or ignoring important information, while training and hardware support are costly.Retrofitting pretrained models with post-training can recover a large portion of baseline performance, including with 40M tokens in LoLCATs.

3 THE MISSING COMPARISON: SLIDING WINDOW VS LINEAR ATTENTION

Post-training linear attention has drawbacks, while its long-context behavior remains understudied. This work compares it directly with SWA with attention sinks, finding SWA performs equally or better on short-context tasks and gains especially on long-context tasks.

  • Post-training linear attention introduces drawbacks including lower expressivity, difficult retention decisions, and additional training requirements.
  • Most prior comparisons use regular sink-free SWA, an inadequate baseline because sink-free SWA can catastrophically fail after sink tokens leave the window.
  • SWA with attention sinks performs equally or better than post-trained linearized models on knowledge and reasoning short-context tasks.
  • SWA achieves especially large gains on long-context tasks, where post-trained attention models remain understudied.

4 EXPERIMENTS

The experiments compare pretrained models using SWA with linearized attention methods across general reasoning, long-context reasoning, speed, and memory. SWA generally matches or outperforms linearized models, while remaining training-free and efficient at inference.

  • General knowledge and reasoning: SWA obtains the best average downstream performance in 9 out of 11 cases.The exceptions are narrow: LoLCATs slightly exceeds SWA on Phi-1.5-1.3B, QRWKV6 matches the Qwen2.5-32B-Instruct baseline, and DiJiang exceeds SWA on one Llama2.0-7B MMLU result.
  • General knowledge and reasoning: SWA recovers 99.0% of average baseline performance and requires 0 tokens of training, while LoLCATs uses 40M tokens to recover 97.5%.For MMLU specifically, SWA recovers 93.2% of baseline performance.
  • Single Needle-in-a-Haystack: At 4K context, SWA recovers 17.2-23% of regular full-attention accuracy, compared with at most 5.8% for LoLCATs and 0.8% for Liger-GLA.These S-NIAH comparisons use Llama 3.1 8B across window sizes 128, 256, and 512.
  • BABILong: At 4K BABILong context, SWA recovers 25% of baseline performance, whereas LoLCATs recovers only 5%.LoLCATs performs slightly better at 0K and 1K, but SWA is higher at 2K and 4K.
  • Speed and memory: SWA is the fastest method, while its memory cost becomes constant after reaching the window size and is similar to or lower than alternatives below window size 512.Full Attention slows beyond 1K context and its memory cost grows linearly; SWA with window size 64 has the lowest memory cost among the listed SWA and linear methods.

5 LIMITATIONS / FUTURE WORK

The work evaluates training-free SWA while identifying several directions beyond its current scope. Related evidence also indicates strong training-free sliding-window baselines in video diffusion models.

  • Scope boundaries: The study focuses on training-free SWA, leaving post-training effects and scaling-law comparisons with linearizing methods for future work.It also excludes hybrid models combining SWA or linear attention with full-attention tokens or layers.
  • Future extensions: Future evaluations could extend to extremely large models, complex agentic tasks, multimodal LLMs, and video generation with multidimensional windows.
  • Related evidence: Training-free sliding-window attention is reported as a strong baseline for video diffusion models.Sliding Tile Attention recovered 97% of baseline VBench while running 3.53 times faster on HunyuanVideo with the same sampling steps.

6 CONCLUSION

SWA with attention sinks outperforms most post-trained linear-attention methods while requiring no post-training, higher decoding speed, and lower memory cost. Its advantage is especially large on long-context reasoning tasks, while short-context performance remains comparable.

  • Conclusion: SWA with attention sinks achieves better performance than most post-trained linear-attention methods without post-training, at higher decoding speed and lower memory cost.
  • Short-context reasoning: 99%: SWA recovers the average baseline performance on short-context reasoning tasks, matching more expensive state-of-the-art linear-attention post-training methods.
  • Long-context reasoning: At context length 256, SWA recovers 20% on S-NIAH-3 and 25% on BABILong, versus LoLCATs’ 2.2% and 5%.
  • Practical implication: The authors strongly recommend SWA with attention sinks for best performance at a fixed small memory cost.

A.1 LIST OF MODELS AND METRICS

The evaluation spans pretrained models from 1.3B to 70B parameters and considers a broad set of language and reasoning metrics.

  • Models: The benchmark includes pretrained architectures ranging from Phi-1.5-1.3B to Llama 3.1-70B.Models include Phi, Mistral, Llama, Qwen, and QwQ variants.
  • Metrics: The listed evaluation metrics include MMLU, ARC-C, and ARC-E.

A.2 ADDITIONAL RESULTS ON NEWLY LINEARIZED MODELS

Additional comparisons examine train-free sliding-window baselines against LoLCATs-style linear variants across newer architectures and attention mechanisms.

  • Training setup: Teacher and SWA(64,4) serve as train-free baselines, while linear variants use LoLCATs-style two-stage distillation on approximately 0.1B cleaned-Alpaca tokens.
  • Compared methods and models: The comparison covers GLA, Gated DeltaNet, and QRWKV6 on Qwen-3, Phi-4-mini-reasoning, and Phi-4-reasoning-plus.

A.3 BABILONG FULL RESULTS

Table 6 reports BABILong performance across increasing context lengths from 0K to 4K, focusing on long-context reasoning.

  • Table 6 compares BABILong performance across context lengths ranging from 0K to 4K.
  • The table evaluates long-context reasoning performance on the BABILong benchmark.
  • The accompanying figures compare attention types using decoding throughput and memory cost across context lengths from 128 to 256K.
  • The accompanying FLOPs figure compares computational requirements for different attention types across context lengths from 128 to 256K.
Loading 2608.28444v1…