PaperlayerView exact revision on arXiv

Source-linked AI summary

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

Qitong Wang, Xinwei Niu, Mingluo Su, Shanwei Zhao, Shiai Zhu, Huan Wang

arXiv:2610.12327v1cs.LGcs.CL

TL;DR

Memory-bound LLM decoding makes inference latency a practical concern, while fixed-sequence calibration and sparse kernels focused on SpMM do not match key aspects of autoregressive decoding. SparseDecoding calibrates from decode-time activations and provides a specialized N:M SpMV kernel. It consistently outperforms conventional calibration approaches across multiple LLMs and sparsity patterns, with up to 1.48× end-to-end decoding speedup on A100 GPUs.

  • Problem

    Memory-bound decoding incurs latency, while fixed-sequence calibration differs from self-generated decoding inputs and common sparse kernels focus on SpMM rather than decoding's SpMV operations.

    Source: §1 · pp. 1–2

  • Method

    SparseDecoding builds calibration matrices from dense-model autoregressive decoding activations, excludes prefill, and uses an N:M SpMV kernel with bitmask indexing and fixed-step traversal.

    Source: §1 · pp. 2–3

  • Results

    SparseDecoding consistently outperforms conventional calibration approaches across multiple LLMs and sparsity patterns, achieving up to 1.48× end-to-end decoding speedup on NVIDIA A100 GPUs.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The experiments highlight the importance of decoding-aware calibration for training-free LLM pruning.

    Source: §5 · p. 9

Abstract

from arXiv · show

The memory-bound nature of the decoding stage of large language model (LLM) inference incurs significant latency. Layer-wise training-free network pruning approaches guided by the Hessian have been a prominent solution to this problem, as pruning reduces the number of nonzero parameters read from memory during decoding. Nevertheless, typical methods in this line compute the Hessian using pre-collected natural sequences, whereas the model is fed self-generated tokens during decoding, creating a distribution shift between the two sequences. The Hessian calculated on the natural sequence is different from that calculated on the generated sequence. We observe that this discrepancy causes the activation distribution during generation to deviate from that used for pruning, further hurting the pruned model performance. Moreover, most existing LLM pruning methods that bring actual speedup primarily target the sparse matrix-matrix (SpMM) multiplication, providing limited support for the sparse matrix-vector (SpMV) operations, which dominate decoding. To solve these problems, we introduce SparseDecoding, a principled decoding-aware pruning framework tailored for accurate and efficient LLM decoding. Specifically, at the algorithmic axis, SparseDecoding constructs calibration matrices from layer-wise activations collected during the dense-model autoregressive generation, excluding prefill, thereby aligning the pruning objective with the decoding activations. At the system axis, we develop an optimized N:M sparse matrix-vector kernel with bitmask indexing and fixed-step traversal. Substantial empirical results on representative LLMs (Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B / 32B) demonstrate that our method consistently outperforms standard fixed-text calibration on the long-form generation benchmarks while achieving up to 1.48x end-to-end wall-clock decoding speedup on A100 GPUs.

1 INTRODUCTION

LLM decoding is memory-bound, and fixed-sequence calibration can diverge from the self-generated inputs encountered during autoregressive decoding. SparseDecoding addresses the calibration and execution gaps with decode-aware calibration and a specialized sparse decoding kernel.

1 INTRODUCTION

  • Fixed-sequence teacher-forced calibration can diverge from decoding because early pruning errors may alter later predictions and contexts.That shift can cause subsequent activation distributions to deviate from those observed during calibration.Source: §1 · p. 1
  • All layers reach at least 90% of their eventual activation-discrepancy plateau within the first 12 decoding steps.The discrepancy then remains largely stable during subsequent decoding.Source: §1 · p. 2, Figure 1 · p. 2
  • Existing 2:4 Sparse Tensor Core kernels target SpMM, while decoding is dominated by SpMV and is not accelerated by those implementations.The figure reports 1.31–1.46× speedups during prefill but only 0.85–0.87× of dense performance during autoregressive decoding.Source: §1 · p. 2, Figure 1 · p. 2
  • SparseDecoding builds calibration matrices from dense-model autoregressive decoding activations and discards prefill activations.This decoding-aware calibration can be used with existing pruning methods such as SparseGPT.Source: §1 · pp. 2–3
  • Its customized N:M SpMV kernel uses bitmasks and fixed-step traversal to reduce metadata and indexing overhead.For the considered 50% N:M patterns with M ≤32, each aligned 32-bit mask word has 16 set bits, enabling uniform traversal steps.Source: §1 · pp. 2–3
  • SparseDecoding improves long-form generation quality over existing pruning baselines while achieving up to 1.48× end-to-end decoding speedup on A100 GPUs.The reported experiments span multiple LLMs.Source: §1 · p. 3

2 RELATED WORK

Related work covers pruning granularity, calibration-data choices, and GPU execution of sparse operations. SparseDecoding focuses on decode-time calibration activations and an N:M SpMV path for autoregressive decoding.

2.1 ONE-SHOT PRUNING

  • N:M semi-structured pruning retains N nonzero weights in each group of M, balancing hardware efficiency with flexibility in weight selection.Structured pruning preserves regular computation but is less flexible, while unstructured pruning offers flexibility with irregular-access and scheduling overheads.Source: §2.1 · p. 3

2.2 CALIBRATION DATA FOR PRUNING

  • Prior studies find calibration-data choice can affect pruning quality, while SparseDecoding collects task-conditioned decode-step activations and excludes prefill.Related methods also use self-generated or decode-time information, but SparseDecoding focuses on aligning the layer-wise reconstruction objective with actual decode-time activations.Source: §2.2 · p. 4

2.3 SPARSE GPU KERNELS AND EFFICIENT DECODING

  • Tensor-Core-oriented sparse matrix-matrix paths suit prefill or larger-batch operations, whereas autoregressive decoding is closer to GEMV/SpMV.SparseDecoding develops an N:M SpMV path using bitmask indexing and fixed-step traversal, reporting up to 1.48× end-to-end decoding speedup on A100 GPUs.Source: §2.3 · p. 4

3 PROPOSED METHOD: SPARSEDECODING

SparseDecoding adapts layer-wise post-training pruning to autoregressive decoding by calibrating on decode-time activations, and pairs this with a decoding-specialized N:M SpMV kernel. It retains the pruning solver while changing the activations that define its reconstruction objective.

3.1 PRELIMINARY

  • Layer-wise post-training pruning decomposes global compression into reconstruction problems that minimize changes to layer outputs on calibration activations under a sparsity constraint.Applying the resulting sparse weights sequentially yields the pruned network.Source: §3.1 · p. 4
  • OBS uses a second-order approximation to select the least salient weight for removal and update surviving weights to compensate, repeating until target sparsity is reached.SparseGPT fixes the pruning order to support efficient, stable maintenance of inverse-Hessian information.Source: §3.1 · pp. 4–5

3.2 PROPOSED METHOD: SPARSEDECODING

  • SparseDecoding uses autoregressive decode-time activations to define the pruning objective, while leaving the solver unchanged and requiring no additional training or new optimizer.It can be combined with OBS-based solvers such as SparseGPT.Source: §3 · p. 4, §3.2 · p. 5
  • Fixed-text calibration summarizes activations under supplied contexts, whereas decoding states depend on earlier generated tokens and can diverge from those fixed-text activations.The framework is presented as a layer-wise post-training pruning method that can be combined with OBS-based solvers such as SparseGPT.Source: §3 · p. 4, §3.2 · p. 5
  • For each target-task prompt, the dense model generates continuations; calibration retains prunable-layer activations from decoding steps, discards prefill activations, and forms the pruning objective from the retained activations.The reconstruction loss ranges over the generated decoding tokens across prompts.Source: §3.2 · p. 5

3.3 N:M SPMV KERNEL DESIGN

  • The N:M SpMV kernel uses bitmasks to identify retained weights and fixed-step traversal to process each row uniformly during decoding.At 50% sparsity, bitmask indexing uses 2 metadata bits per nonzero, reducing metadata size 16× versus int32 indices and 4× versus uint8 offsets; with M ≤32, each aligned 32-bit mask has 16 set bits.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments evaluate decoding-aware calibration on writing and code benchmarks, alongside decoding throughput and calibration ablations. SparseDecoding improves generation quality over fixed-text calibration and achieves practical N:M decoding speedups.

4.1 EXPERIMENTAL SETUP

  • Evaluations cover four instruction-tuned models from Llama and Qwen families, using WritingBench for writing and ClassEval for code generation.WritingBench generations use a maximum length of 16,000 tokens; ClassEval reports class-level Pass@1.Source: §4.1 · pp. 6–7
  • C4 and SparseDecoding comparisons on Qwen3 use 50% unstructured and 2:4 sparsity, with both configurations using 1M calibration tokens and the same SparseGPT backend; SparseDecoding counts decode-step tokens only.C4 uses fixed C4 text, whereas SparseDecoding collects layer inputs from dense-model decode steps.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS: WRITINGBENCH; 4.3 MAIN RESULTS: CLASSEVAL

  • SparseDecoding improves WritingBench overall across all models and sparsities, and leads across all eight ClassEval model-sparsity pairs by 4 to 15 points.For ClassEval under 2:4 sparsity, Pass@1 rises from 9.0 to 24.0% on Qwen3-32B and from 0.0 to 4.0% on Llama-3.1-8B.Source: Table 1 · p. 7, §4.3 · p. 8, Table 2 · p. 8

4.4 EMPIRICAL DECODING SPEEDUP

  • Every evaluated N:M pattern decodes faster than dense execution, with speedups ranging from 1.34 to 1.48× on A100 GPUs.The patterns retain 50% of the weights; measurements use context length 512 and GPT-Fast.Source: §4.4 · p. 8, Table 3 · p. 9

4.5 ABLATION STUDY

  • Self-generated calibration outperforms cross-model calibration for Qwen3-14B, and SparseDecoding remains ahead of C4 when the pruning backend changes to Wanda.Self-generated versus cross-model WritingBench scores are 4.18 vs. 4.13 under 2:4 and 5.35 vs. 5.23 under 50% unstructured sparsity.Source: §4.5 · p. 9

5 CONCLUSION

SparseDecoding aligns pruning calibration with autoregressive decoding and pairs it with an N:M SpMV kernel. Across multiple LLMs and sparsity patterns, it improves generation quality and achieves up to 1.48× end-to-end decoding speedup on A100 GPUs.

  • SparseDecoding is a decoding-aware pruning framework that uses autoregressive outputs to align pruning calibration with downstream decoding behavior.The framework is training-free.Source: §5 · p. 9
  • SparseDecoding consistently outperforms conventional calibration approaches across multiple LLMs and sparsity patterns.The conclusion describes improved generation quality without additional training.Source: §5 · p. 9
  • The N:M SpMV kernel achieves up to 1.48× end-to-end decoding speedup on NVIDIA A100 GPUs.Source: §5 · p. 9

A ANALYSIS OF OUR METHOD

This section characterizes the discrepancy between calibration and decode-time reconstruction objectives. It also empirically compares that discrepancy for autoregressive and fixed-text calibration.

A.1 THEORETICAL ANALYSIS

The theoretical analysis defines calibration discrepancy relative to the decode-time objective and relates it to worst-case relative reconstruction-loss discrepancy. It also states that, under the theorem’s condition, AR calibration has a smaller discrepancy than fixed-text calibration.

A.1 THEORETICAL ANALYSIS

  • A smaller normalized calibration discrepancy indicates a smaller worst-case relative discrepancy between calibration and decode-time reconstruction objectives over the projected directions.The normalization uses the decode-time Hessian’s eigenvalues after projection onto its top-k eigenvectors.Source: §A.1 · p. 13
  • Projected reconstruction loss is defined layer-wise over the top-k eigenvectors of the decode-time Hessian for decode-time, AR, and TF activation sources.The setup distinguishes decode-step activations from autoregressive calibration prompts and fixed calibration sequences.Source: §A.1 · pp. 12–13
  • When the theorem’s condition holds, AR calibration has a smaller worst-case relative reconstruction-loss discrepancy with respect to the decode-time objective than TF calibration.The analysis considers a layer and retained dimension k.Source: §A.1 · p. 14

A.2 EMPIRICAL FINDINGS

The empirical analysis compares autoregressive (AR) calibration with fixed C4 calibration on Qwen3-14B and Qwen3-32B. Across both models and all retained fractions, AR calibration shows smaller decode-time reconstruction-objective discrepancy in most prunable modules.

A.2 EMPIRICAL FINDINGS

  • LongWriter prompts generate the autoregressive calibration activations, while fixed calibration uses C4 sequences.The analysis evaluates retained fractions of 0.01, 0.02, 0.05, 0.10, 0.20, and 0.50.Source: §A.2 · p. 14
  • The empirical discrepancy estimate substitutes activation-based empirical Hessians for the corresponding population Hessians.A smaller estimated value indicates a smaller empirical worst-case reconstruction-loss discrepancy.Source: §A.2 · p. 14
  • In at least 87% of prunable modules across both models and all retained fractions, AR calibration has lower discrepancy, with median C4-to-AR ratios from 1.81 to 3.26.The comparison uses Qwen3-14B and Qwen3-32B; a ratio above 1 indicates smaller discrepancy for AR.Source: Table 6 · p. 15, Table 7 · p. 15, §A.2 · p. 15

B ADDITIONAL EXPERIMENTAL RESULTS

Additional experiments find that decoding-aware calibration improves generation quality beyond the primary sparsity setting and across comparisons with other fixed calibration corpora.

B ADDITIONAL EXPERIMENTAL RESULTS

  • Under 8:16 sparsity, SparseDecoding consistently improves generation quality compared with C4 calibration across evaluated models, showing decoding-aware calibration benefits extend beyond 2:4 sparsity to this pattern.Source: B ADDITIONAL EXPERIMENTAL RESULTS · p. 15
  • SparseDecoding achieves higher overall WritingBench scores than Pile or RedPajama calibration across both Qwen3 models and the evaluated sparsity settings.The comparison covers 50% unstructured and 2:4 semi-structured sparsity.Source: B ADDITIONAL EXPERIMENTAL RESULTS · p. 15

C KERNEL IMPLEMENTATION DETAILS

The N:M sparse matrix-vector kernel is implemented and tuned for batch-one autoregressive decoding, with pattern-specific compile-time specialization and runtime configuration tuning.

C KERNEL IMPLEMENTATION DETAILS

  • The Triton N:M SpMV kernel targets batch-one autoregressive decoding and avoids explicitly staging the input vector in shared memory.Input loads use the .ca cache policy; sparse weights and bitmask metadata use .cg.Source: C KERNEL IMPLEMENTATION DETAILS · p. 15
  • Fixing N and M at compile time enables unrolling of the bitmask-decoding loops, with nonzero positions recovered by bit-scan operations.The kernel compiles separately for each N:M pattern.Source: C KERNEL IMPLEMENTATION DETAILS · pp. 15–16
  • The implementation autotunes output tile size, warp count, pipeline stages, and reduction strategy for each matrix shape and sparsity pattern.For configurations benefiting from more parallelism, it splits reductions across programs and combines partial sums with atomicAdd.Source: C KERNEL IMPLEMENTATION DETAILS · p. 16
Loading 2610.12327v1…