PaperlayerView exact revision on arXiv

Source-linked AI summary

Hidden in Plain Sight: The Overlooked Significance of Canonical Elements for Extreme LLM Sparsity

Hyeondo Jang, Kwanhee Lee, Dongyeop Lee, Namhoon Lee

arXiv:2609.06557v1cs.LG

TL;DR

LLM pruning often degrades beyond moderate sparsity, leaving the attainable extreme-sparsity regime underexplored. BEST revisits second-order saliency and progressive sparsification with continued training coordinated to sparsity progression, improving perplexity and downstream accuracy up to 99% sparsity across LLaMA-2 and Qwen-3.

  • Problem

    LLM pruning has often shown sharp degradation beyond moderate sparsity, while the potential of carefully optimized extreme sparsity remains underexplored.

  • Method

    BEST combines global second-order saliency, progressive sparsification, and continued training with learning-rate scheduling coordinated to sparsity progression.

  • Results

    Across LLaMA-2 and Qwen-3, BEST improves perplexity and downstream accuracy over prior approaches from 70% to 99% sparsity, with 3.23× decoding speedup and 6.21× lower memory footprint on LLaMA-2-7B.

  • Takeaways & Limitations

    The results show that pretrained LLMs can retain strong performance at extreme sparsity, providing a foundation for further sparse-model improvements.

  • Takeaways & Limitations

    BEST requires continued training, and downstream performance declines in free-form generation and reasoning, particularly at extreme sparsity.

Abstract

from arXiv · show

Large language models (LLMs) are often considered fragile under aggressive sparsification, and maintaining reliable performance typically requires sticking to moderate sparsity levels. However, recent studies suggest that LLMs are more resilient to high sparsity than previously thought, reframing the problem as a design challenge rather than a fundamental limitation. In this work, we challenge the perceived limits of unstructured post-training LLM pruning by revisiting elementary pruning strategies that have remained relatively underexplored at this scale. Through a progressive sparsification framework with second-order saliency and continued training coordinated with sparsity progression, we show that pretrained LLMs can retain strong performance far beyond commonly studied sparsity regimes. Across LLaMA-2 and Qwen-3 model families, our approach improves perplexity and downstream accuracy up to 99\% sparsity, surpassing both the current state-of-the-art and representative baselines. Precisely, on LLaMA-2-7B, our approach achieves WikiText-2 perplexities of 13.48 and 19.67 at 95\% and 99\% sparsity, respectively, while delivering 3.23$\times$ decoding speedup and 6.21$\times$ memory savings at 95\% sparsity. Taken together, our results show that LLMs can be pushed into extreme sparsity while retaining strong performance, providing a foundation for further improving sparse models in this regime.

1 Introduction

LLM sparsification is motivated by deployment costs but has often shown sharp degradation beyond 50–70% sparsity. This work argues that coordinated pruning and continued training can push attainable sparsity substantially further.

  • LLM deployment incurs substantial compute and memory costs, motivating sparsification to reduce model size and accelerate inference.
  • Performance has often degraded sharply beyond roughly 50–70% sparsity, fostering the perception that extreme sparsity may be intrinsically limited.
  • Recent evidence indicates that continued training can preserve functional performance beyond moderate sparsity regimes, including at 90% sparsity.
  • The paper asks how far extreme sparsity can be pushed with carefully tuned optimization, noting that resource-efficient pruning protocols may reveal only part of LLM sparsity potential.
  • BEST revisits second-order saliency and progressive sparsification, coordinating them with continued training to push attainable sparsity beyond previously demonstrated levels.
  • Across LLaMA-2 and Qwen-3, BEST improves perplexity and downstream accuracy over prior approaches from 70% to 99% sparsity, while achieving 3.23× decoding speedup and 6.21× lower memory footprint on LLaMA-2-7B.

2 Background

Pruning reduces computation and memory by removing weights, but LLM pruning has largely favored inexpensive one-shot, layerwise procedures. The paper identifies this restricted paradigm as a likely contributor to the perceived sparsity ceiling.

  • Pruning seeks mostly-zero weight matrices so corresponding computation and memory allocation can be skipped while preserving task performance.
  • Early pruning methods commonly used a single one-shot pruning step either at initialization or after pretraining.
  • Stronger sparse subnetworks often result when pruning interleaves with optimization, through gradual sparsity increases, regrowth, or iterative pruning and retraining.
  • Parameter-selection methods range from magnitude and gradient–weight scores to second-order curvature and learned sparsity patterns.
  • LLM pruning converged toward inexpensive post-training, one-shot, layerwise surrogate objectives that established moderate sparsity with minimal overhead.
  • Refinements to this paradigm provide modest gains but do not reliably prevent collapse at extreme sparsity.
  • The paper therefore treats the prevailing sparsity limit as potentially arising from a restricted pruning paradigm rather than an intrinsic LLM constraint.

3 Method

BEST formulates pruning as alternating masking, a sparsity schedule, and weight training. It combines global second-order saliency, gradual sparsification, and coordinated learning-rate updates to support extreme sparsity.

  • Formulation: The pruning problem optimizes model parameters under an L0 constraint limiting the number of non-zero weights.
  • Framework: BEST builds on progressive sparsification, alternating masking with training of the remaining parameters from pretrained dense weights.
  • Framework: The framework exposes masking strategy, sparsity schedule, and training strategy as the three components governing sparsification performance.
  • Masking strategy: BEST uses second-order saliency with an optimizer-aware proxy and global pruning because exact constrained optimization and Hessian computation are impractical at LLM scale.
  • Masking strategy: Adam’s running squared-gradient statistics provide a zero-cost curvature proxy, enabling curvature-aware pruning without additional gradient-computation cost.
  • Masking strategy: Global comparison ranks parameters jointly, allocating sparsity without predetermined per-layer ratios and using a distributed binary search for tractable implementation.
  • Sparsity schedule: A cubic gradual sparsity schedule avoids sudden loss of pretrained knowledge while eventually freezing the mask at the target sparsity.
  • Training strategy: The learning rate starts small, grows with the sparsity ramp, and decays toward convergence through warmup followed by linear decay.

4 Experiments

Experiments evaluate BEST across model families, sparsity levels, downstream tasks, deployment metrics, quantization, ablations, and training cost. BEST consistently improves extreme-sparsity quality and practical efficiency, while progressive sparsification, second-order saliency, and coordinated training each contribute to the results.

  • Experimental setup: BEST is evaluated on LLaMA-2, Qwen2.5, and Qwen-3 using language-modeling and zero-shot generalization benchmarks.The experiments compare BEST with one-shot pruning, retrained one-shot methods, and recent pruning approaches.
  • Language modeling: 19.67 WikiText-2 perplexity is achieved by BEST at 99% sparsity on LLaMA-2-7B, compared with 35.53 for SparseGPT+.At 99% sparsity, BEST also reaches 22.07 on LLaMA-2-13B and 25.96 on Qwen3-4B-Base, versus 31.36 and 60.13 for the respective strongest baselines.
  • Zero-shot downstream performance: 42.21 and 29.49 zero-shot accuracy are retained by BEST at 99% sparsity on two reported Qwen3-8B tasks, outperforming SparseGPT+ on most tasks and on average.At 95% sparsity, BEST retains 50.76 and 34.91 on those tasks while SparseGPT+ drops by 32.7 and 21.2 percentage points from its 70% results.
  • Comparison with recent methods: 13.48 and 19.67 WikiText-2 perplexities are achieved by BEST at 95% and 99% sparsity, compared with 38.91 and 55.94 for ELSA.At 90% sparsity, BEST also reaches 11.60 WikiText-2 perplexity, 13.71 C4 perplexity, and 42.34 average zero-shot accuracy, versus 26.97, 23.14, and 38.52 for ELSA.
  • Inference efficiency: 3.23× decoding throughput and 6.21× lower memory footprint are obtained at 95% sparsity versus the dense LLaMA-2-7B baseline.The throughput comparison is 176.18 versus 54.47 tokens/s and includes sparse index metadata in the memory measurement.
  • Quantization: INT8 quantization is almost lossless, while INT4 increases perplexity by +0.41 at 70% sparsity and +0.18 at 90% sparsity.The quantization experiment applies round-to-nearest quantization to the remaining nonzero weights of BEST-pruned Qwen-3-8B-Base.
  • Ablations: Global second-order pruning achieves the best configuration across sparsity levels, while magnitude-based global allocation nearly collapses attention projections.Magnitude is biased by module-scale differences; second-order saliency produces more balanced allocations and benefits from global comparison.
  • Ablations: 2.38 loss and 11.60 perplexity at 90% sparsity outperform 2nd-order+ results of 2.49 and 14.11, supporting progressive sparsification.One-shot pruning drives optimization into a high-loss regime early, whereas progressive sparsification introduces sparsity incrementally.

5 Conclusion

The work finds that faithfully applying essential pruning components can establish a new state of the art in extreme LLM sparsity, while producing measurable inference speedups on commodity hardware.

  • BEST establishes a new state of the art in extreme LLM sparsity by faithfully applying essential pruning components.
  • The resulting sparse models yield measurable inference speedup on commodity hardware.

Limitations

The paper identifies limitations involving training cost, downstream capability loss, architectural scope, deployment dependence on dedicated kernels, and narrow margins over small dense models.

  • BEST requires continued training, incurring additional computational cost that the authors identify for future reduction.
  • Downstream performance declines despite minimal perplexity increases, especially for free-form generation and reasoning tasks such as GSM8K and NQ-Open.
  • The experiments focus on dense Transformers, leaving extension to architectures such as Mixture-of-Experts models unresolved.
  • Practical inference gains depend on a dedicated sparse kernel because native hardware support for irregular sparsity is absent.
  • Although BEST outperforms small dense models at comparable effective parameter counts, the margin remains narrow.

B.2 Detailed experimental setup

The experiments use controlled calibration, continued-training, evaluation, and scheduling protocols to assess BEST across sparsity levels. Ablations indicate that cubic sparsity progression and short warm-up improve performance, especially at higher sparsity.

  • Data and evaluation: 128 C4 calibration sequences of length 2048 are used solely for one-shot pruning with Wanda+ and SparseGPT+.
  • Data and evaluation: Except for BEST‡, continued-training experiments use SlimPajama, while BEST‡ matches ELSA with C4 and approximately 67M training tokens.
  • Data and evaluation: The main training budget is fixed at 64 × 4000 × 2048 tokens, with a separate ELSA-matched setting using batch size 8 and 4096 steps.
  • Progressive sparsification: BEST updates masks 50 times during the first half of training, using a cubic schedule to progressively increase sparsity.
  • Data and evaluation: Zero-shot accuracy is measured across seven lm-eval-harness benchmarks and reported as the mean across tasks.
  • Scheduling ablations: A short warm-up of approximately 10% of 4000 steps consistently achieves stable OPT-125M performance across sparsity levels.
  • Scheduling ablations: Cubic sparsity scheduling achieves lower final loss and better perplexity than linear scheduling, with the advantage widening at higher sparsity.

C.3 Additional results on the sparsity schedule

Progressive sparsification consistently outperforms one-shot alternatives across evaluated sparsity levels, with its advantage increasing in the extreme regime.

  • Figure 8 compares BEST with 2nd-order+ across all evaluated sparsity levels, isolating the contribution of the gradual schedule.
  • The perplexity gap grows from 0.01 to 8.68 against 2nd-order+ and from 0.22 to 15.86 against SparseGPT+ between 70% and 99% sparsity.
  • The gradual schedule becomes increasingly important as sparsity rises because one-shot pruning imposes sparsity in a single step.

C.4 Diagonal Fisher approximation via Adam second moment

Reusing Adam’s accumulated second-moment state provides an efficient diagonal-Fisher proxy, while global thresholding outperforms the tested layer-wise allocation strategy.

  • Adam second-moment reuse: Perplexity differences remain marginal even with 2048 calibration sequences across the evaluated sparsity regimes.
  • Adam second-moment reuse: Recomputing the empirical Fisher introduces non-trivial computational overhead from additional forward-backward passes.
  • Sparsity allocation: The controlled ATP comparison changes only the allocation rule while retaining BEST’s saliency criterion, gradual schedule, optimization, and training budget.
  • Sparsity allocation: Global thresholding achieves lower perplexity than ATP at both tested sparsity levels on both Wikitext-2 and C4.

D.1 Extended comparison with ELSA

Under ELSA’s matched training setting, BEST outperforms ELSA throughout the evaluated sparsity range and retains strong perplexity at 95% and 99% sparsity.

  • BEST‡ outperforms ELSA at every evaluated sparsity level from 70% to 99%.BEST‡ uses ELSA’s exact training setting, including C4 and a 67M-token budget.
  • At 95% sparsity, BEST‡ reaches lower perplexity than ELSA at 90% sparsity, 24.34 versus 26.97.
  • With a 524M-token SlimPajama training setup, BEST achieves WikiText-2 perplexities of 13.48 at 95% sparsity and 19.67 at 99% sparsity.

D.2 Why magnitude pruning fails: sparsity allocation and module-level weight distributions

Magnitude pruning fails under global pruning because heterogeneous module weight distributions distort sparsity allocation, while BEST supports compact models but does not preserve all free-form reasoning capabilities.

  • Allocation failure: Magnitude pruning over-allocates sparsity to vproj and outproj while leaving qproj and kproj relatively dense.
  • Allocation failure: This uneven allocation causes magnitude pruning to fail under global pruning.
  • Allocation failure: Module-level weight distributions are heterogeneous across Transformer blocks, with the strongest differences in early blocks and flatter patterns later.
  • Compact model construction: BEST provides a compute-efficient route to competitive compact models when a strong pretrained model is already available.
  • Compact model construction: Large sparse models achieve lower Wikitext-2 perplexity than dense counterparts with comparable effective parameter counts across OPT and Qwen2.5.
  • Open-ended capabilities: GSM8K accuracy falls from 87.64 to 28.05 at 70% sparsity and approaches zero beyond that level, while NQ-Open degrades more gradually.
  • Open-ended capabilities: Preserving perplexity and multiple-choice accuracy under extreme sparsity does not ensure robust free-form generation or reasoning.The evaluations used pruned Qwen3-8B-Base models without task-specific data curation or adaptation.

D.5 Numerical results of zero-shot downstream task accuracy

This section provides full per-task zero-shot accuracy results across multiple sparsity regimes for four model configurations. The results are organized in Tables 16–19, with several comparison methods identified for LLaMA-2-7B.

  • Full per-task zero-shot results are provided for LLaMA-2-7B/13B and Qwen3-4B/8B-Base.These results are discussed in Section 4.
  • Tables 16–19 report zero-shot accuracy (%) across multiple tasks and sparsity regimes.The tables cover LLaMA-2-7B, LLaMA-2-13B, Qwen3-4B-Base, and Qwen3-8B-Base.
  • Table 16 compares L-ADMM, ALPS, SAFE, SparseLLM, and ELSA alongside LLaMA-2-7B results.The comparison numbers for these methods are taken from Lee et al. (2026).
Loading 2609.06557v1…