PaperlayerView exact revision on arXiv

Source-linked AI summary

TopK-Guided: Adaptive, Budget-Aware Activation Sparsity for Efficient LLM Inference

Mukund Agarwalla, Chih-Jen Lin

arXiv:2610.01763v1cs.AI

TL;DR

Threshold masking adapts sparsity to each token but does not tightly control realised sparsity, while TopK fixes sparsity per token and block-level budgets ignore differing block sensitivities. TopK-Guided combines bounded token-level adaptation with sensitivity-aware block allocation, improving perplexity and downstream accuracy over TEAL and WINA while preserving essentially the same projection compute as WINA.

  • Problem

    Threshold methods adapt sparsity to each token without tightly controlling realised sparsity, TopK methods use a fixed token budget, and both apply uniform budgets despite differences in block sensitivity.

    Source: §1 · p. 2

  • Method

    TopK-Guided clips each token’s threshold-based natural sparsity estimate to a bounded band around the target and allocates block budgets according to measured sensitivity.

    Source: §1 · p. 2, §2.1 · p. 3, §2.2 · p. 3

  • Results

    TopK-Guided improves perplexity and downstream accuracy over TEAL and WINA across both models and all tested sparsities, with the largest gains at high sparsity and no added cost over WINA.

    Source: §1 · p. 2, §4 · p. 4, §3 · p. 4

  • Takeaways & Limitations

    The method maintains essentially the same projection compute as WINA while improving on both WINA and threshold masking at matched compute.

    Source: §3 · p. 4

    Evaluation covers decoder-only models up to 8B parameters, so behavior at larger scale and on other architectures remains to be checked.

    Source: §4 · p. 4

Abstract

from arXiv · show

Activation sparsity speeds up large language model (LLM) inference by setting unimportant activations to zero so that the corresponding computations can be skipped. Existing training-free methods, however, make different trade-offs: threshold-based methods such as TEAL adapt the sparsity level to each token but do not tightly control the realised sparsity, while TopK-based methods such as WINA enforce a fixed sparsity level but use the same sparsity budget for every token. Both also apply the same budget across transformer blocks, despite large differences in block sensitivity. We introduce TopK-Guided, a training-free method that addresses both limitations by combining bounded token-level sparsity adaptation with sensitivity-aware block-level budget allocation. Across Llama-2 and Llama-3 models, TopK-Guided consistently improves perplexity and downstream accuracy over TEAL and WINA while preserving essentially the same sparsitydependent projection compute as WINA, with the largest gains at high sparsity. Ablations show that both components provide complementary improvements.

1 Introduction

Activation sparsity skips projection work by zeroing small activation entries, but existing methods trade token-level adaptation against precise sparsity control. TopK-Guided adds bounded token adaptation and sensitivity-aware block budgets, improving perplexity and downstream performance over TEAL and WINA, especially at high sparsity.

1 Introduction

  • Activation sparsity zeros small-magnitude activation entries so their corresponding projection columns can be skipped, unlike weight pruning, which permanently removes parameters.Source: §1 · p. 1
  • TEAL adapts sparsity to each token but its realised sparsity can drift from the target, whereas WINA enforces exact sparsity with the same budget for every token.Source: §1 · pp. 1–2
  • Both methods assign every transformer block the same budget even though early blocks are more sensitive, causing fragile blocks to be over-sparsified and robust blocks underused.Source: §1 · p. 2, Figure 1 · p. 2
  • TopK-Guided combines bounded per-token sparsity adaptation with sensitivity-based block budgets, giving sensitive blocks less sparsity and robust blocks more.Source: §1 · p. 2
  • Across Llama-2-7B and Llama-3-8B, TopK-Guided beats TEAL and WINA on perplexity and downstream tasks at matched compute, with the largest gains at high sparsity.Source: §1 · p. 2

2 Method

TopK-Guided changes WINA’s masking rule and block-level budget assignment while retaining its other components. It estimates each token’s natural sparsity, clips it near the target, and redistributes the global budget according to measured block sensitivity.

2 Method

  • The method builds on WINA, retaining its tensor rotation and greedy per-projection allocation while changing only the masking rule and block-level budget assignment.Source: §2 · p. 2

2.1 TopK-Guided Masking

  • The resulting per-token target preserves adaptation while guaranteeing realised sparsity within ±δ of the target; with δ = 0.01, deviation is at most one percentage point and mask construction is O(d).Source: §2.1 · p. 3
  • TopK-Guided estimates each token’s natural sparsity from the calibrated threshold and clips that estimate to a ±δ band around the target before applying TopK.Source: §2.1 · p. 3

2.2 Sensitivity-Aware Block Sparsity

  • The method assigns less sparsity to above-average-sensitivity blocks and reallocates the global budget toward robust blocks, with λ controlling redistribution and a [0.05, 0.95] clamp.Source: §2.2 · p. 3
  • A calibration pass measures each block’s output change at a probe sparsity, with larger sensitivity indicating a less robust block.Source: §2.2 · p. 3
  • After fixing δ = 0.01, probe sparsity and redistribution strength remain the method’s two free hyperparameters.Source: §2.2 · p. 3

3 Experiments

Across Llama-2-7B and Llama-3-8B, TopK-Guided improves on TEAL and WINA in perplexity and downstream accuracy, while retaining WINA-like projection compute. Its margins grow with sparsity.

Evaluation setup

  • Evaluation covers C4 perplexity on 500 validation samples and average zero-shot accuracy across eight tasks on both models.The target sparsities are 0.3, 0.5, and 0.7; hyperparameters are selected on a held-out dataset rather than the reported evaluation dataset.Source: §3 · pp. 3–4

Main results

  • Across the twelve comparisons in Table 1, TopK-Guided achieves the lowest C4 perplexity and highest average accuracy over eight tasks on both Llama-2-7B and Llama-3-8B.Source: §3 · p. 4, Table 1 · p. 4
  • The margin over both baselines is small at sparsity 0.3 and grows with sparsity, reaching its largest at 0.7.The authors note that masking and budget choices have more room to affect results at higher sparsity.Source: §3 · p. 4

Computational savings and quantization

  • At matched compute, TopK-Guided improves on WINA while preserving essentially the same projection-work reduction.Its realized per-token sparsity stays within ±δ of the target; the per-token mask adds only O(d) overhead, negligible beside the projection.Source: §3 · p. 4

4 Conclusion and Limitations

TopK-Guided makes two training-free changes to WINA and improves on both WINA and TEAL across the evaluated models and sparsities. The study is limited to decoder-only models up to 8B parameters and reports FLOP savings, not wall-clock speedup.

Conclusion

  • TopK-Guided combines bounded-adaptive token masking with sensitivity-based block budgets and consistently improves on WINA and TEAL without added cost over WINA.The gains hold across both evaluated models and all tested sparsities, are largest at more aggressive sparsity, and each contribution helps on its own.Source: §4 · p. 4

Limitations

  • The evaluation covers decoder-only models up to 8B parameters, leaving behavior at larger scale and on other architectures to be checked.The authors note that the method makes no architecture-specific assumptions.Source: §4 · p. 4
  • The paper reports FLOP-level savings rather than wall-clock speedup, leaving kernel-level implementation for future work.Source: §4 · p. 4

Future work

  • The authors identify combining TopK-Guided with rotation-based quantization as a next step toward faster low-precision inference.Source: §4 · p. 4

A Notation

The notation table organizes sparsity quantities and parameters by their scope and points to where each is defined.

Sparsity quantities

  • The paper distinguishes a global target, per-block budgets, and per-token targets as quantities operating at different levels.Source: A Notation · p. 7

Notation table

  • Because several sparsity quantities share the symbol s, the table records each symbol’s scope and meaning.Source: A Notation · p. 7
  • The table also gives the equation or section where each quantity is defined.Source: A Notation · p. 7

B Target Sparsity Vs Realised Sparsity

The section contrasts threshold and TopK control of realized sparsity, then explains how WINA’s rotation makes its per-element ranking account for joint-removal error. Orthogonalized weight columns remove the cross-terms that otherwise complicate that ranking.

B Target Sparsity Vs Realised Sparsity

  • At target sparsity 0.5, TopK-Guided stays within ±0.01 across three datasets, while TEAL’s realized sparsity drifts and WINA hits the target exactly.The comparison is on Llama-2-7B; the table reports target versus realized sparsity for the three methods, with perplexity for reference.Source: Table 3 · p. 7, B Target Sparsity Vs Realised Sparsity · p. 7

C Tensor Rotation: What, Why, and How

C.1 The Problem: The Per-Element Bound Ignores Cross-Terms

  • WINA’s per-element score omits cross-terms between dropped weight columns, which can make its ranking unreliable for joint removal.Aligned columns with same-sign activations can reinforce errors; learned weight matrices are not orthogonal.Source: §C.1 · p. 8

C.2 The Idea: Column-Orthogonalise the Weight Matrix

  • WINA right-multiplies each weight matrix by an SVD-derived orthogonal matrix, making the transformed columns exactly orthogonal.For the matrix whose SVD is taken, the cross-terms vanish and the per-element score recovers the true error; the propagation machinery must preserve the model’s output.Source: §C.2 · p. 8

C.3 The Implementation: Computational Invariance

SliceGPT’s rotations are absorbed into adjacent weights throughout the transformer, avoiding runtime rotation while preserving the model’s outputs.

C.3 The Implementation: Computational Invariance

  • Rotations can be absorbed into adjacent weight matrices throughout the network, closing the compensation chain without computing rotations at runtime.The preceding layer supplies the input transformation needed to preserve each rotated projection.Source: §C.3 · p. 8
  • RMSNorm’s rotation equivariance and the linearity of residual connections let both commute with the rotation Q.Source: §C.3 · p. 8
  • After offline preprocessing, the rotated model produces exactly the original model’s outputs while using the same inference FLOPs.The paper’s sensitivity estimation and TopK-Guided masking experiments all use rotated weights.Source: §C.3 · p. 9

D Block-wise Greedy Sparsity Allocation

The block-wise greedy procedure allocates sparsity among a block’s matrices according to their measured output-error costs, rather than assigning each matrix a uniform budget.

D Block-wise Greedy Sparsity Allocation

  • The sensitivity-aware block budgets feed the existing greedy per-projection allocation unchanged.The allocation exploits differences in how much sparsifying each matrix changes the block output.Source: D Block-wise Greedy Sparsity Allocation · p. 9
  • The procedure repeatedly tests a sparsity increment for each matrix and permanently assigns the next increment to the matrix with the least output reconstruction error.It records each resulting allocation against the block’s overall sparsity budget.Source: D Block-wise Greedy Sparsity Allocation · p. 9
  • Each matrix’s step size is inversely proportional to its parameter count, so one increment moves the compute-weighted block sparsity by roughly the same amount.The experiments use a base step α = 0.025.Source: D Block-wise Greedy Sparsity Allocation · p. 9
  • Across the seven real matrices, large MLP matrices tend to absorb sparsity first, while smaller, more sensitive attention projections are protected until later.Source: D Block-wise Greedy Sparsity Allocation · p. 9
  • P = 0.3 is reached with pA = 0.5 and pB = 0.25 in the toy block, after the algorithm switches from B to A.Table 4 illustrates the two-matrix trajectory with fA = 100, fB = 400, and α = 0.1.Source: Table 4 · p. 9, D Block-wise Greedy Sparsity Allocation · p. 9

E Hyperparameter Sweep

The WikiText-2 sweep finds that TopK-Guided performs robustly across tested hyperparameters, with a consistently strong region that avoids extensive per-setting search.

E Hyperparameter Sweep

  • The sweep tests all 16 combinations of ¯s and λ in {0.2, 0.4, 0.6, 0.8} at target sparsities 0.3, 0.5, and 0.7 on Llama-2-7B WikiText-2 perplexity.Table 5 lays out λ by column and ¯s by row; bold entries meet or beat WINA for that target sparsity.Source: E Hyperparameter Sweep · p. 10, Table 5 · p. 10
  • At s = 0.5, all 16 combinations beat TEAL; at s = 0.7, all 16 beat TEAL and 15 of 16 beat WINA.Across all three sparsity levels, ¯s ∈ [0.2, 0.6] and λ ∈ [0.2, 0.4] is consistently strong.Source: Table 5 · p. 10, E Hyperparameter Sweep · p. 11
  • At s = 0.3, all 16 combinations beat TEAL and 15 of 16 meet or beat WINA, with perplexity ranging only from 6.77 to 6.80.This low-sparsity result is essentially insensitive to parameter choice.Source: Table 5 · p. 10, E Hyperparameter Sweep · p. 10

F Ablation of our Contributions

The ablation finds that both contributions help, and their combination gives the best perplexity at every tested sparsity, with the largest joint gain at high sparsity.

F Ablation of our Contributions

G Per-Task Downstream Accuracy

This appendix provides per-task zero-shot accuracy breakdowns for Llama-2-7B and Llama-3-8B. It also states the work’s intended societal benefit and its assessment of direct negative impacts.

G Per-Task Downstream Accuracy

H Impact Statement

  • The work aims to reduce LLM inference cost, potentially lowering energy use and broadening deployment in resource-constrained settings; it identifies no direct negative societal impacts specific to the method.Source: H Impact Statement · p. 12

Related papers

Loading 2610.01763v1…