Source-linked AI summary
IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, Zhengxiao Du, Aohan Zeng, Jie Tang, Juanzi Li
TL;DR
Long-context inference is bottlenecked by the quadratic cost of independently running DSA indexers at every layer, despite highly similar top-k selections across consecutive layers. IndexCache reuses indices across layers, using retained Full layers and optimized layer-selection patterns, and removes 75% of indexer computations with negligible quality degradation while improving prefill and decode speed.
Problem
DSA reduces core attention cost, but its indexer still scores all preceding tokens independently at every layer, creating O(NL^2) cost despite cross-layer top-k redundancy.
Method
IndexCache retains Full layers with indexers and lets Shared layers reuse top-k indices from preceding Full layers, while training-free greedy search selects retained layers by minimizing calibration-set language-modeling loss.
Results
75% fewer indexer computations yields negligible quality degradation, up to 1.82× prefill speedup, and up to 1.48× decode speedup; searched patterns recover Long Avg to 49.9 at 1/4 retention.
Takeaways & Limitations
Cross-layer index reuse extends efficient sparse-attention inference by exploiting redundancy in token selection without requiring weight updates for the training-free variant.
Takeaways & Limitations
At 1/8 indexer retention, long-context performance declines substantially even with searched patterns, with Long Avg falling to 46.1.
Abstract
from arXiv · showhide
Long-context agentic workflows have emerged as a defining use case for large language models, making attention efficiency critical for both inference speed and serving cost. Sparse attention addresses this challenge effectively, and DeepSeek Sparse Attention (DSA) is a representative production-grade solution: a lightweight lightning indexer selects the top-k most relevant tokens per query, reducing core attention from $O(L^2)$ to $O(Lk)$. However, the indexer itself retains $O(L^2)$ complexity and must run independently at every layer, despite the fact that the resulting top-k selections are highly similar across consecutive layers. We present IndexCache, which exploits this cross-layer redundancy by partitioning layers into a small set of Full layers that run their own indexers and a majority of Shared layers that simply reuse the nearest Full layer's top-k indices. We propose two complementary approaches to determine and optimize this configuration. Training-free IndexCache applies a greedy search algorithm that selects which layers to retain indexers by directly minimizing language modeling loss on a calibration set, requiring no weight updates. Training-aware IndexCache introduces a multi-layer distillation loss that trains each retained indexer against the averaged attention distributions of all layers it serves, enabling even simple interleaved patterns to match full-indexer accuracy. Experimental results on a 30B DSA model show that IndexCache can remove 75% of indexer computations with negligible quality degradation, achieving up to 1.82$\times$ prefill speedup and 1.48$\times$ decode speedup compared to standard DSA. These positive results are further confirmed by our preliminary experiments on the production-scale GLM-5 model (Figure 1).
1 Introduction
Long-context inference is bottlenecked by quadratic attention and redundant per-layer DSA indexer computation. IndexCache reuses cross-layer token selections, retaining fewer indexers through training-free or training-aware configuration methods while preserving quality and improving speed.
- Motivation: O(L^2) indexer computation remains at every DSA layer, making it a significant long-context attention cost despite sparse core attention costing O(Lk).The indexer scores all preceding tokens independently at each layer, while core attention selects only the most relevant subset.
- Motivation: 70-100% adjacent-layer top-k overlap indicates that many DSA indexer computations are redundant.Layer-overlap heatmaps also reveal clusters with mutually high overlap.
- IndexCache: IndexCache eliminates up to 75% of indexer computations by retaining Full layers and reusing their top-k indices in Shared layers.Inference adds a conditional branch; Shared layers reuse indices from the nearest preceding Full layer.
- Optimization approaches: Training-free IndexCache uses greedy calibration-set LM-loss search to retain only 1/4 of indexers while matching original DSA downstream performance.It requires no weight updates, whereas naïve uniform interleaving can degrade quality.
- Optimization approaches: Training-aware IndexCache distills each retained indexer against the attention distributions of all layers it serves, allowing uniform interleaving to match the original design.The method optimizes model parameters for cross-layer sharing.
2 Preliminary
DSA reduces core attention through lightweight top-k token selection, but its indexer remains quadratic and runs independently at every layer. IndexCache is motivated by cross-layer stability, while the paper asks whether indexer outputs can be reused without a full-attention oracle.
- DeepSeek Sparse Attention: DSA separates attention into selection and computation, using a lightning indexer to choose top-k tokens before sparse core attention.The indexer scores preceding tokens and selects the highest-scoring positions.
- DeepSeek Sparse Attention: k=2048 ≪L reduces per-layer core attention from O(L^2) to O(Lk), while the lightweight indexer uses low-rank projections and FP8 arithmetic.The indexer is described as an order of magnitude cheaper per-FLOP than MLA.
- DeepSeek Sparse Attention: O(NL^2) total indexer cost remains because every layer independently scores all preceding tokens.This cost becomes a significant fraction of the attention budget at long context lengths.
- Cross-Layer Stability of Token Selection: Prior cross-layer reuse methods rely on full attention as an oracle, but DSA eliminates full attention and relies on its lightweight indexer instead.This creates the paper’s specific question about whether indexer outputs also exhibit cross-layer stability.
- Cross-Layer Stability of Token Selection: The paper investigates whether redundant indexer computations can be eliminated, how far reuse can go before quality degrades, and whether adaptation can close the resulting gap.These questions motivate the two IndexCache methods introduced later.
3 Method
IndexCache reduces DSA indexer computation by assigning layers either fresh indexers or reused top-k indices, then optimizes this sharing pattern through greedy search or multi-layer distillation.
- IndexCache design: IndexCache partitions layers into Full layers with retained indexers and Shared layers that inherit indices from the nearest preceding Full layer.Shared layers skip the indexer forward pass and reuse the cached index tensor through a conditional branch.
- IndexCache design: The first layer remains Full, while each Shared layer applies sparse core attention using inherited top-k indices.
- Training-free selection: Uniform interleaving can remove sensitive indexers, whereas searched patterns outperform uniform interleaving at the same retention ratio.Validation-loss curves distinguish easier early steps from critical later layers, revealing nonuniform indexer importance.
- Training-free selection: Training-free IndexCache uses calibration-set language-modeling loss to select which indexers can be converted from Full to Shared without weight updates.The search starts from an all-Full pattern and evaluates candidate flips incrementally.
- Training-free selection: Greedy search converts layers one at a time, committing the flip with the lowest resulting loss while keeping the first layer fixed as Full.Pipeline parallelism can reduce the search’s forward-pass cost by roughly P×.
- Training-aware distillation: Training-aware IndexCache distills each retained indexer against the attention distributions of the layers it serves, equivalent to distillation toward their averaged target.This objective encourages a jointly useful top-k set and gives the multi-layer loss the same gradient as a single averaged-target loss.
4 Experiments
Experiments evaluate IndexCache’s efficiency and quality across 30B DSA benchmarks, inference settings, retention ratios, and preliminary GLM-5 results. Training-free search preserves quality at 1/4 retention, while training-aware distillation enables uniform sharing patterns to match DSA.
- Evaluation setup: Experiments use a 30B DSA model, nine long-context and reasoning benchmarks, and inference measurements across 10K–200K-token contexts.The model is evaluated with retention ratios of 1/2 and 1/4 for end-to-end speed, while quality experiments also test 1/8 retention.
- End-to-end inference speedup: 1.82× prefill speedup at 200K tokens reduces latency from 19.5s to 10.7s with 1/4 indexer retention.This removes 75% of indexer computations, which dominate the prefill phase at long contexts.
- End-to-end inference speedup: 1.48× decode speedup at 200K tokens increases per-request throughput from 58 to 86 tok/s with 1/4 retention.Total decode throughput rises from 197 to 297 tok/s at 200K when the KV cache is fully saturated.
- GLM-5 results: At least 1.3× improvement in both prefill latency and decode throughput is observed for training-free IndexCache on GLM-5 beyond 100K contexts.These are preliminary results on the 744B-parameter production-scale model.
- Training-free quality: Greedy layer search recovers Long Avg to 49.9 at 1/4 retention, compared with 43.0 for uniform interleaving and 50.2 for original DSA.At 1/8 retention, searched patterns still reach only 46.1 Long Avg, indicating a non-negligible quality decline at extreme sparsity.
- Reasoning quality: Training-free IndexCache preserves reasoning quality across configurations except uniform interleaving at 1/8 retention.G&R Avg remains within 1 point of the 74.6 DSA baseline for the other configurations.
- Training-aware quality: Training-aware IndexCache keeps 1/4-retention Long Avg and G&R Avg within 0.4% of the DSA baseline using uniform interleaving.Removing cross-layer distillation drops Long Avg from 51.6 to 49.8 and AA-LCR from 49.8 to 44.0.
- Training-aware quality: Training removes the pattern sensitivity seen without retraining, allowing uniform interleaving to perform on par with the searched pattern.Retraining lets shared layers adapt to inherited indices and retained indexers generalize across the layers they serve.
5 Related Work
Prior work reduces sparse-attention redundancy by reusing top-k indices across layers, while IndexCache targets DSA's lightweight indexer and optimizes the sharing configuration systematically.
- Prior methods reuse top-k indices from periodic anchor layers to exploit representation consistency across adjacent layers.
- IndexCache shares DSA indexer outputs rather than full attention scores, making its oracle cheaper.
- IndexCache combines training-free greedy pattern search with training-aware multi-layer distillation to optimize cross-layer sharing.
- The principle may extend to dynamic token-selection methods such as MoBA and NSA.
6 Conclusion
IndexCache accelerates sparse attention by reusing token-selection indices across layers, eliminating substantial indexer computation while preserving performance in the reported setting.
- IndexCache partitions layers into Full layers with indexers and Shared layers that reuse inherited top-k indices.
- 75% of the O(NL2) total indexer cost is eliminated with a single conditional branch and no performance degradation.
- Cross-layer sharing extends beyond settings where full attention serves as the token-selection oracle.
A Cross-Layer Top-k Index Overlap
DSA layers show strong but structured top-k overlap, supporting cross-layer reuse while revealing why aggregate similarity alone cannot determine the best sharing pattern.
- Figure 4 maps pairwise top-k index overlap across all layer pairs and marks the greedy-searched 1/4-retention sharing blocks.
- Adjacent DSA layers exhibit top-k overlap ratios of 0.7-1.0, while layer groups form distinct high-overlap clusters.
- Overlap decreases more rapidly across block boundaries than within blocks, indicating uneven cross-layer token-selection stability.
- Greedy sharing blocks do not fully match natural overlap clusters because aggregate overlap ignores which tokens differ.
- Local attention-output similarity and top-k overlap cannot identify the optimal pattern reliably, so end-to-end evaluation is necessary.
B Searched Patterns
The appendix lists searched Full/Shared layer patterns for GLM-4.7-Flash 30B DSA and GLM-5 at three retention levels for the former and two for the latter.
- GLM-4.7-Flash 30B DSA: For GLM-4.7-Flash 30B DSA, searched patterns retain 1/2, 1/4, or 1/8 of layers as Full indexer layers.
- GLM-5: For GLM-5, searched patterns retain either 1/2 or 1/4 of layers as Full indexer layers.
C Similarity-based Pattern Search
The paper evaluates similarity-based layer-pattern search for IndexCache, then finds that optimizing local attention-output similarity does not preserve downstream quality as reliably as greedy LM-loss search.
- Similarity matrix: Similarity-based search builds a lower-triangular matrix measuring how well an earlier layer’s indexer approximates a later layer’s attention output.Each entry uses cosine similarity between the later layer’s original output and its output when reusing the earlier layer’s index.
- Dynamic programming: Dynamic programming selects a pattern with a target number of Full layers by maximizing summed similarities for Shared layers.Each Shared layer inherits indices from the most recent preceding Full layer, and the optimal pattern is recovered by backtracking.
- Results: Similarity-optimal patterns perform comparably to uniform interleaving and retain significant quality degradation relative to the original DSA model.The similarity-based search provides no meaningful downstream advantage over the naïve uniform baseline.
- Why similarity fails as a proxy: Local output similarity can miss critical-token errors whose effects accumulate through later layers and degrade final quality.Two layers may have nearly identical attention outputs while differing in tokens that become important during later reasoning steps.
- Results: Greedy LM-loss search substantially outperforms both uniform and similarity-optimal patterns, especially at aggressive retention ratios.Unlike local similarity, LM loss captures the end-to-end effect of sharing decisions and identifies critical layers whose indexers must be retained.
D Evaluation Setup
The evaluation fixes decoding parameters and uses specified context and output limits across long-context, general, and reasoning tasks.
- Evaluation parameters: All benchmarks use temperature 1.0, top-p = 0.95, and top-k = 40.
- Context limits: Long-context tasks use a 200K-token context window with 32K tokens reserved for output.
- Context limits: General and reasoning tasks allow a maximum output length of 64K tokens.
- Benchmark aggregation: MRCR v2 averages scores across 2-, 4-, and 8-needle settings, while GraphWalks averages Parent-type and BFS-type problems.