PaperlayerView exact revision on arXiv

Source-linked AI summary

SWAA: Sliding Window Attention Adaptation for Efficient and Quality Preserving Long Context Processing

Yijiong Yu, Jiale Liu, Qingyun Wu, Huazheng Wang, Ji Pei

arXiv:2512.10411v5cs.CLcs.AI

TL;DR

Long-context inference with Transformer LLMs is costly under quadratic self-attention, while SWA can lose quality when applied to FA-pretrained models. SWAA combines attention-pattern choices, sink-token preservation, and lightweight fine-tuning; experiments report synergistic recovery and configurations spanning speed and quality priorities.

  • Problem

    Quadratic self-attention makes long-context inference inefficient, and applying SWA to FA-pretrained models faces training–inference mismatch and restricted access to distant tokens.

    Source: §1 · p. 1

  • Method

    SWAA is a plug-and-play toolkit that adapts FA-pretrained models to SWA by combining FA Decode, interleaved FA/SWA layers, sink-token preservation, and lightweight fine-tuning.

    Source: §1 · p. 2, §3.1.1 · p. 3, §3.2.2 · p. 5

  • Results

    Specific synergistic combinations recover long-context performance, including 30% speedup with nearly 100% accuracy retention in quality-first scenarios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments identify SWAA configurations for diverse performance-efficiency priorities across Qwen and Llama families.

    Source: §5 · p. 9

    Interleaving may be sensitive to layer selection in FA-pretrained models, and the paper reports no reliable method for finding the best selection.

    Source: §3.1.2 · p. 4

Abstract

from arXiv · show

The quadratic complexity of self attention in Transformer based LLMs renders long context inference prohibitively expensive. While Sliding Window Attention (SWA), the simplest sparse attention pattern, offers a linear complexity alternative, it suffers from catastrophic long context performance collapse, which stems from two fundamental factors: the training inference mismatch when naively applying SWA to models pretrained with Full Attention (FA), and the inherent structural inability to access distant information when applying SWA to every module at all times. To address these dual challenges, we propose Sliding Window Attention Adaptation (SWAA), a plug and play toolkit of recipes that adapts FA models to SWA without costly pretraining. SWAA systematically combines four core strategies to tackle these distinct issues: (1) Full Attention (FA) Decode and (2) Interleaving FA and SWA layers, which mitigate structural defects by selectively allowing access to distant information; alongside (3) preserving ``sink'' tokens and (4) lightweight fine tuning, which mitigate the training inference mismatch. Our experiments reveal that while isolated strategies are insufficient, specific synergistic combinations effectively recover long context performance. Despite varying computational overheads, our performance efficiency trade off analysis identifies optimal SWAA configurations for diverse scenarios, achieving 30% to 100% speedups for long context inference with acceptable quality retention. Our code, data and model weights are available at https://github.com/yuyijiong/sliding-window-attention-adaptation

1 Introduction

Sliding Window Attention (SWA) lowers attention complexity but can cause long-context performance collapse through training–inference mismatch and loss of direct access to distant tokens. SWAA combines four adaptation strategies, and experiments report that synergistic combinations can recover performance with different speed–quality trade-offs.

1 Introduction

  • SWA reduces attention complexity from quadratic to linear by restricting each token to a fixed-size local window, but naively applying it can cause long-context performance collapse.The stated causes are training–inference mismatch in FA-pretrained models and structural loss of direct pathways to distant tokens.Source: §1 · p. 1
  • SWAA is a plug-and-play toolkit that adapts FA-pretrained models without costly pretraining or modules beyond the standard Transformer architecture.Its four strategies are FA Decode, interleaving FA/SWA layers, preserving the first k sink tokens, and lightweight SWA-aware fine-tuning.Source: §1 · p. 2
  • FA Decode and interleaved FA/SWA layers address structural limits by restoring access to distant information during decoding or through FA layers.FA Decode uses SWA during prefill and full attention during generation; interleaving mixes FA and SWA across layers.Source: §1 · p. 2
  • Preserving initial sink tokens and lightweight SWA-aware fine-tuning address mismatch between SWA inference and FA pretraining.The paper frames fine-tuning as long-context supervised adaptation and sink-token preservation as keeping attention to the first k tokens.Source: §1 · p. 2
  • Specific combinations of SWAA strategies recover long-context performance, with nearly 100% speedup and 90% accuracy retention in efficiency-first scenarios or 30% speedup and nearly 100% accuracy retention in quality-first scenarios.Experiments on Qwen3 and Llama3.1 across several long-context benchmarks identify configurations for different performance-efficiency priorities.Source: §1 · p. 2
  • The paper presents its work as the first systematic study of combining adaptation methods for FA-pretrained LLMs and as a flexible toolkit for balancing performance and efficiency.The methods are implemented with Flash-Attention and vLLM for practical plug-and-play use.Source: §1 · p. 2

2 Related Works

Related work explores sparse and linear attention as alternatives to full attention, but many approaches require architectural changes or training tailored to the attention pattern. Post-training adaptation of existing FA models is also explored, with cited approaches carrying cost or generalization concerns.

2 Related Works

  • Quadratic self-attention complexity has motivated research into more efficient language-model architectures, including sparse and linear attention.Source: §2 · p. 3

2.1 Sparse Attention

  • Sparse-attention designs combine local attention with global tokens or FA layers, while other approaches modify architecture or select tokens semantically.Examples include Longformer, BigBird, RATTENTION, Gemma2, SWAT, and Native sparse attention.Source: §2.1 · p. 3
  • Most cited sparse-attention methods require pretraining with a specific pattern; DeepSeek sparse attention uses nearly 1T tokens for continued pretraining, while LightTransfer may generalize poorly across model families.The passage identifies LightTransfer as sharing SWAA's goal of adapting existing models without pretraining.Source: §2.1 · p. 3

2.2 Linear Attention

  • Linear-attention models and state-space models offer O(N) complexity, but the cited hybrids depart from standard Transformer architecture and must be trained from scratch.The passage also reports generally weaker performance than traditional Transformer-based LLMs and possible harm to math and reasoning ability.Source: §2.2 · p. 3

3 Method

SWAA combines temporal and spatial attention choices with sink-token preservation and fine-tuning to adapt FA-pretrained models to SWA. The methods selectively retain distant-context access and adapt weights, while layer selection and thinking-model data needs shape their use.

3 Method

  • SWAA frames its four adaptation methods as addressing structural limitations and training–inference mismatch.Source: §3 · p. 3

3.1.1 Full Attention (FA) Decode (Temporal Selection)

  • FA Decode uses SWA during prefill and restores full attention during autoregressive decoding, allowing generated tokens to access the full preceding context.The paper links this design to a “reading casually, thinking carefully” analogy and notes that CoT may complement it by extending generation and retrieval pathways.Source: §3.1.1 · pp. 3–4, Figure 1 · p. 4

3.1.2 Interleaving FA/SWA Layers (Spatial Selection)

  • Interleaving retains FA in some layers and applies SWA to others, allowing distant information to propagate through network depth while balancing pure-SWA efficiency.Layer choice may matter for FA-pretrained models; because no reliable selection method is established, experiments use fixed-interval selections.Source: §3.1.2 · p. 4

3.2.1 Keep First k Tokens

  • Keeping the first k sink tokens preserves attention to initial tokens alongside each token’s local window, maintaining attention-distribution and output stability.The implementation customizes the Flash-Attention-2 kernel to accelerate prefill with SWA without cumbersome KV-cache modifications.Source: §3.2.1 · pp. 4–5

3.2.2 Fine tuning

  • Fine-tuning updates model weights for sparse-attention patterns and can incorporate FA Decode, interleaving, or sink-token preservation during training.For thinking models, the original FA model generates answers that are filtered for correctness to construct training data and preserve CoT style.Source: §3.2.2 · p. 5

4 Experiment

The experiments systematically test SWAA recipe combinations for accuracy and efficiency across long-context settings. They find that combining methods that address different bottlenecks can restore accuracy, with deployment-specific trade-offs between quality and speed.

4 Experiment

  • The experiments compare combinations rather than activating every method at once, enabling ablations and cost-effectiveness comparisons across recipes.They use Qwen3-4B Thinking and Instruct as primary models, with Qwen3-30B-A3B and Llama3.1-8B evaluations for generality.Source: §4 · p. 5, §4.1.1 · p. 5

4.1.2 Evaluation Dataset

  • LongMemEval 24k is the main benchmark: 500 samples from 10 chat sessions each, with contexts averaging 24k tokens; LongBench-V2 and Ruler provide additional validation.LongBench-V2 is restricted to 311 samples under 128k tokens, and Ruler uses 500 128k Multi-Query samples.Source: §4.1.2 · p. 6

4.1.3 Training Details

  • SWA-aware fine-tuning uses LoRA on query, key, and value projections for one epoch; each Qwen3-4B configuration takes about 12 hours on an 8*H20 GPU server.The training data combines LongAlign with 6,000 Fusang-v1-long samples; full-parameter fine-tuning was avoided after preliminary overfitting and capability degradation.Source: §4.1.3 · p. 6

4.2.1 Accuracy Analysis

  • Single methods yield negligible accuracy gains over pure SWA, while pairing methods that address distinct bottlenecks significantly recovers accuracy.Structural methods paired with lightweight fine-tuning can match the full-attention baseline; for Ruler, recovery requires interleaving plus another method.Source: §4.2.1 · pp. 6, 8
  • FA decoding performs better with the thinking model than the instruct variant, while interleaving’s zero-shot accuracy depends on layer selection; odd-numbered FA layers outperform even-numbered ones for Qwen3-4B.The authors suggest that misaligning SWA with pretrained global-routing layers can disrupt semantic flow.Source: §4.2.1 · pp. 7–8
  • Combining FA decoding and interleaving layers also substantially improves accuracy, with structural changes implicitly easing the training-inference mismatch.Keep First stabilizes generation before fine-tuning, but its effect is weaker than SFT and becomes largely optional after SFT.Source: §4.2.1 · p. 8

4.2.2 Efficiency Analysis and Recommended Recipes

  • After SFT, enabling either FA decoding or Interleaving Layers increases inference speed by almost 100% while maintaining nearly 90% of original accuracy; combining them maintains nearly 100% and accelerates over 30%.Source: §4.2.2 · p. 9

5 Conclusion

The paper presents SWAA as a cost-effective way to adapt full-attention pretrained LLMs to SWA without training sparse-attention models from scratch. Across Qwen and Llama experiments, synergistic method combinations mitigate degradation, while optimized configurations balance inference efficiency and performance.

5 Conclusion

  • Experiments across Qwen and Llama show that optimized SWAA configurations can balance computational overhead and model performance to accelerate inference.The paper identifies synergistic combinations of auxiliary methods as a way to mitigate the degradation of naive SWA adaptation.Source: §5 · p. 9
  • SWAA adapts full-attention pretrained LLMs to SWA without costly pretraining from scratch.The conclusion frames it as a cost-effective alternative to training sparse-attention models from scratch.Source: §5 · p. 9
  • Naive SWA degradation can be mitigated by synergistic combinations of auxiliary methods.Source: §5 · p. 9

A SWA’s Benefits and Each Method’s Drawbacks

SWA reduces attention computation and KV-cache requirements, but methods that preserve long-context capability give back some of those benefits. Their costs differ across FA decoding, interleaving layers, and Keep First.

A SWA’s Benefits and Each Method’s Drawbacks

  • SWA reduces attention complexity to O(N · W), where W is the window size, and limits KV-cache needs while enabling cache reuse beyond traditional prefix caching.Source: A SWA’s Benefits and Each Method’s… · p. 12
  • FA decoding accelerates only prefilling, not decoding, and retains the full-context KV cache, so it does not reduce KV-cache memory.Chain-of-thought decoding further reduces its speedup ratio.Source: A SWA’s Benefits and Each Method’s… · p. 12
  • Interleaving layers has the greatest overhead because only some layers benefit from SWA; full-attention layers also retain KV-cache memory and negate SWA’s cache-reuse advantage.Source: A SWA’s Benefits and Each Method’s… · p. 13
  • Keep First adds little computation but complicates cache reuse across requests because positional encoding makes a token’s KV state depend on its position relative to the initial k tokens.The authors note that positional-encoding separation or offsetting may be needed.Source: A SWA’s Benefits and Each Method’s… · p. 13

B Some Problems of Other Long-context Benchmarks

The paper identifies limitations in commonly used long-context benchmarks that affect their suitability for evaluating general long-context capability. It nevertheless uses LongBench-V2 and Ruler to check whether conclusions generalize.

B Some Problems of Other Long-context Benchmarks

C Experiment Results of Other Models

The authors tested additional Qwen3-30B-A3B and Llama3.1-8B-Instruct models, finding accuracy trends mostly consistent with earlier conclusions across benchmarks.

C Experiment Results of Other Models

D Inference Efficiency

Efficiency varies across SWAA configurations: interleaving layers and FA Decode slow inference relative to pure SWA, while preserving initial tokens has negligible efficiency impact.

D Inference Efficiency

  • The reported TTFT, TPOT, and throughput measurements use vLLM on a single H100 and are reference values because speed depends on hardware, implementation, and workload.Source: D Inference Efficiency · p. 13
  • Interleaving Layers and FA Decode significantly slow inference compared with pure SWA.Source: D Inference Efficiency · p. 13
  • Keep First k Tokens has negligible impact on efficiency, while increasing the window from 2k to 4k decreases throughput by only 10%.A 4k window generally achieves higher accuracy in previous experiments, so the authors describe it as a more common practical choice.Source: D Inference Efficiency · p. 13
  • FA Decode can accelerate TPOT under vLLM-v1 request mixing, but offers almost no efficiency improvement when a single request spends over 95% of its time decoding.The latter condition is reported for a generation length of 2000.Source: D Inference Efficiency · p. 14

E Influence of Training Epochs

Training beyond one epoch yielded no improvement, so the authors selected a one-epoch training run.

E Influence of Training Epochs

F Results of LightTransfer

LightTransfer’s results vary across models, and the authors conclude that fine-grained layer selection needs further investigation before becoming part of SWAA recipes.

F Results of LightTransfer

  • LightTransfer selects SWA layers using a lazy ratio comparing attention to recent versus global tokens; the remaining layers retain full attention.The ratio is calculated from attention by tokens at the sequence end on a calibration dataset.Source: F Results of LightTransfer · p. 14
  • The reproduced LightTransfer setup uses LongAlign for lazy-layer calibration, with 64 last tokens and a 1024-token recent window.Source: F Results of LightTransfer · p. 14
  • LightTransfer’s outcomes differ by model: it is counterproductive for Qwen3-4B, adds nearly no improvement over fixed-interval selection for Qwen3-30B, and shows advantages only for Llama3.1-8B.Table 7 reports the LongMemEval results; for Qwen3-4B, allowing lazy layers to use FA scores higher than the original selection.Source: F Results of LightTransfer · p. 14, Table 7 · p. 16
  • The authors conclude that LightTransfer is not stable across models and that fine-grained layer selection needs further investigation before integration into SWAA recipes.Source: F Results of LightTransfer · p. 14

Related papers

Loading 2512.10411v5…