PaperlayerView exact revision on arXiv

Source-linked AI summary

Building a Nested LADDER for LLM Unstructured Post-Training Pruning

Nicolás Mauricio Cuadrado Ávila, Ivo Gollini Navarrete, Martin Takáč, Samuel Horváth

arXiv:2610.28773v1cs.LG

TL;DR

Existing non-uniform pruning can deteriorate sharply at aggressive sparsity, while fixed, independently estimated layer costs do not capture interacting pruning effects. LADDER allocates sparsity through repeated layer updates chosen by end-to-end degradation. Across five models, it is more robust under aggressive pruning, and that advantage largely persists after distillation while improving aggregate zero-shot accuracy retention.

  • Problem

    Existing non-uniform pruning methods deteriorate sharply under aggressive sparsity, and interacting binary pruning decisions make fixed, independently estimated layer costs inadequate for determining the profile.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    LADDER incrementally selects the next layer pruning update by marginal end-to-end degradation, producing nested mask supports across sparsity budgets.

    Source: §1 · p. 2

  • Results

    Across five OPT, Llama, and Qwen models, LADDER is substantially more robust under aggressive pruning, with the advantage largely surviving posterior distillation and improving aggregate accuracy retention across seven zero-shot tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluated results indicate more consistent downstream accuracy retention across models and tasks, rather than dominance in every configuration.

    Source: §4.3 · p. 7

    Larger model evaluations are still necessary, and a naive implementation becomes costlier with wider layers.

    Source: §4.5 · p. 7

Abstract

from arXiv · show

Non-uniform allocation can improve post-training pruning of large language models, yet quality can still degrade sharply at high sparsity. To control degradation as sparsity increases, we introduce LADDER (Layer-Wise Allocation via Damage-Driven Evaluation and Refinement), which allocates sparsity incrementally. At each step, it evaluates pruning an additional fraction of weights from each eligible layer and commits the layer update that causes the least end-to-end degradation per removed parameter. This process produces non-uniform sparsity profiles and exactly nested masks across multiple budgets. We evaluate LADDER on five OPT, Llama, and Qwen models across 10-90% sparsity, using SparseGPT as its candidate-generation backend and comparing against uniform and adaptive allocation baselines. Results reveal a clear operating-regime trade-off: LADDER is comparable or slightly weaker at benign sparsity but becomes substantially more robust from 70% onward. This high-sparsity advantage largely persists after posterior distillation across four of five models and translates into improved retention of aggregate zero-shot accuracy.

1 INTRODUCTION

Existing non-uniform pruning methods can deteriorate sharply at aggressive sparsity. LADDER makes layer allocations incrementally and produces nested masks; across five models, it is more robust under aggressive pruning, with benefits largely persisting after distillation.

1 INTRODUCTION

  • Rising training and deployment costs, alongside edge-device memory, bandwidth, and latency constraints, make post-training compression more attractive as open pretrained models become more available.The passage frames training from scratch as increasingly restricted for some organizations.Source: §1 · p. 1
  • Evidence that pruning an overparameterized model can outperform training a smaller model from scratch motivates using pretrained LLMs to create efficient subnetworks.The cited result is presented for high-dimensional settings, even when relevant features are known.Source: §1 · p. 1
  • Sparse unstructured pruning has practical efficiency potential: MACKO-SpMV reports memory reduction and faster inference for a 50% sparse LLM.The passage says runtime and storage benefits improve as sparsity increases.Source: §1 · p. 1
  • Although non-uniform allocation can exploit layer redundancy, existing methods deteriorate sharply under aggressive sparsity.The introduction notes that some deep layers can be removed with minimal negative impact, a finding termed the Curse of Depth.Source: §1 · p. 2
  • LADDER repeatedly commits the layer update with the least end-to-end degradation per removed parameter, producing exactly nested mask supports across sparsity budgets.In the main experiments, SparseGPT generates the local pruning candidates.Source: §1 · p. 2, Figure 1 · p. 2
  • Across five OPT, Llama, and Qwen models, LADDER is substantially more robust under aggressive pruning, with its advantage largely surviving posterior distillation and improving aggregate accuracy retention across seven zero-shot tasks.The authors describe a trade-off: LADDER is comparable or slightly weaker at benign sparsity, but more robust from 70% onward.Source: §1 · p. 2

2 BACKGROUND

The background establishes a common layer-degradation notation, distinguishes global allocation from local pruning, and surveys pruning methods, layer-wise allocation, nested model families, and sparse-model refinement.

2 BACKGROUND

  • The layer-local degradation functional provides common notation for comparing pruning methods through mask selection, surviving-weight reconstruction, and sparsity allocation.The background distinguishes global pruning, which allocates a model-level sparsity budget, from local pruning, which minimizes damage within a fixed layer-wise budget.Source: §2 · p. 2
  • Global pruning allocates a target sparsity across layers to minimize expected model degradation, while local pruning seeks a minimally damaging mask for a fixed layer budget.Jointly optimizing masks and surviving weights is described as intractable, so existing methods typically address only one variable.Source: §2 · pp. 2–3

2.1 Pioneering Pruning Methods

  • Score-based methods such as magnitude pruning and Wanda select weights for removal but leave surviving values unchanged, whereas SparseGPT reconstructs surviving weights using approximate second-order information.Reconstruction-based methods aim to maintain pre-pruning activation dynamics.Source: §2.1 · p. 3

2.2 Layer-Wise Pruning Allocation Methods

  • Layer-wise allocation methods assign non-uniform sparsity using signals such as activation outliers, layer statistics, spectral metrics, reconstruction error, or evolutionary search.The surveyed methods include OWL, DSA, AlphaPruning, LSA, BESA, and EvoPress, among others.Source: §2.2 · p. 3

2.3 Elastic Model Families

  • Elastic model families construct nested subnetworks with shared parameters to support multiple deployment budgets, including width-adjustable networks and nested Transformers.The examples include Slimmable Networks, MatFormer, and Flextron.Source: §2.3 · p. 3

2.4 Sparse-Model Refinement and Knowledge Transfer

  • Knowledge-transfer methods such as CAP and AST apply distillation to improve sparse-model quality, but the surveyed approaches target a single sparse endpoint.CAP uses dense models and intermediate pruning states for supervision; AST combines post-pruning retraining with dense-teacher supervision.Source: §2.4 · pp. 3–4

3 METHODOLOGY

LADDER frames layer-wise sparsity allocation as minimizing end-to-end calibration degradation at a target global sparsity, then greedily commits the least-damaging candidate update. Increasing sparsity budgets yield nested mask supports, although the greedy trajectory does not guarantee optimality.

3.1 Problem Setup

  • The allocation minimizes end-to-end calibration degradation over layer-wise sparsities subject to the requested global sparsity over prunable parameters.At an interior optimum, marginal end-to-end damage per removed parameter must be equal across eligible layers; uniform sparsity meets this only when those marginal damages agree.Source: §3.1 · p. 4

3.2 Layer-Wise Allocation

  • Each round evaluates candidate layer updates from the same committed state and commits the one with least degradation until reaching the global sparsity target.Each prunable layer proposes removing its next step of least-important surviving parameters.Source: §3.2 · p. 4

3.3 Damage-Driven Evaluation

  • Candidates are ranked by relative degradation per added zero, using local pruning backends; SparseGPT also reconstructs retained weights.Wanda and SparseGPT refresh their activation-dependent scores or curvature information from the current committed model.Source: §3.3 · pp. 4–5
  • Because candidate assessments use the current committed state, each decision reflects interactions evaluated so far, but greedy selection can miss options that become favorable later.Exact nesting provides structural assurance, not optimality.Source: §3.3 · p. 5

3.4 Refinement

  • Processing sparsity budgets in increasing order and restricting each run to the preceding checkpoint’s support produces nested masks; reconstruction may still change surviving values.The authors empirically verify zero support violations.Source: §3.4 · p. 5
  • Posterior distillation initializes one shared student from the pretrained dense model and updates it using averaged temperature-scaled KL losses across fixed masks and dense-teacher outputs.The same procedure is applied to every baseline.Source: §3.4 · p. 5

4 EXPERIMENTS & RESULTS

The experiments compare LADDER with non-uniform allocation baselines across language modeling, downstream tasks, and ablations. LADDER’s advantage is most consistent at high sparsity, while distillation narrows differences and results remain architecture- and task-dependent.

4.1 Experimental Setup

  • The evaluation covers five OPT, Qwen, and Llama models, three language-modeling benchmarks, seven downstream tasks, and requested global sparsities from 10% to 90%.Methods share SparseGPT as the local pruning backend; LADDER also uses calibration-sample perplexity for candidate selection.Source: §4.1 · p. 6

4.2 Language-Modeling Across Regimes

  • LADDER is often matched by non-uniform baselines at lower sparsity, has mixed results at 50–60%, and separates most consistently from 70% onward before distillation.Distillation narrows the differences, especially at moderate sparsity, but high-sparsity ordering remains architecture-dependent; OWL and LSA are competitive in some Llama-3.2-1B settings.Source: §4.2 · p. 6

4.3 Downstream Transfer

  • Across models and tasks, LADDER generally retains zero-shot accuracy better as sparsity increases, with lower macro-averaged degradation from dense performance in the high-sparsity regime.The strength and ordering of methods vary by task, and results are interpreted as more consistent retention rather than dominance in every configuration.Source: §4.3 · p. 7, Figure 4 · p. 8

4.4 Ablations

  • LADDER lowers test loss in 41 of 45 backend–dataset–sparsity comparisons and beats each standalone backend at 70%, 80%, and 90% sparsity.SparseGPT significantly outperforms the other tested backends and is used for the main experiments.Source: §4.4 · p. 7, Figure 5 · p. 8
  • Continuing LADDER from baselines’ 70% masks yields lower-loss checkpoints at 80% and 90% than directly generated baseline masks at those sparsities.OWL is the strongest tested initialization and closely aligns with standard LADDER at 80%, while dense-to-sparse LADDER performs better at 90%.Source: §4.4 · p. 7, Figure 6 · p. 8
  • A nested trajectory reduces highest-sparsity perplexity by 10.6% on WikiText-2, 8.6% on PTB, and 15.6% on C4 versus independently initialized target searches.Producing all checkpoints from 10% to 90% takes 333 search rounds, approximately the 327 rounds for one independently optimized 90% mask.Source: §4.4 · p. 7

4.5 Scalability and Deployment

  • LADDER is a one-time post-training procedure using a small calibration set without retraining, but naive implementations become costlier with wider layers and larger-model evaluations remain necessary.The study prioritizes pruning quality over direct inference-throughput measurement and relies on existing sparse-kernel results for acceleration claims.Source: §4.5 · pp. 7–8

5 CONCLUSION

LADDER controls sparsity growth by repeatedly choosing the least damaging pruning update, limiting quality degradation at 70–90% sparsity across five models and producing nested mask supports. Distillation narrows the differences, while downstream evaluation shows more consistent accuracy retention across models and tasks.

5 CONCLUSION

  • Across five models and three language-modeling datasets, LADDER limits quality degradation at 70–90% sparsity compared with OWL, DSA, and LSA.The conclusion positions LADDER as a strategy for constructing robust high-sparsity model families.Source: §5 · p. 8
  • LADDER repeatedly selects the least damaging pruning update as sparsity increases, complementing different local pruning rules and producing nested mask supports across budgets.SparseGPT is identified as the strongest backend.Source: §5 · p. 8
  • Distillation narrows the differences between LADDER and the compared allocation methods, while downstream evaluation shows more consistent accuracy retention across models and tasks.Source: §5 · p. 8
  • LADDER can refine masks initialized by existing allocation methods, although its full dense-to-sparse trajectory remains strongest at 90%.The conclusion notes the importance of earlier allocation decisions.Source: §5 · p. 8

AI Use Statement

  • Generative AI assisted with software implementation and debugging and manuscript editing and review, but did not generate experimental measurements or replace the authors’ scientific judgment.The authors say they reviewed and verified AI-assisted code and text and take responsibility for the work.Source: AI Use Statement · p. 9

REPRODUCIBILITY STATEMENT

An anonymized implementation of LADDER and scripts for the reported results are available in a repository. It includes the allocation procedure, baselines, distillation pipeline, and evaluation code.

REPRODUCIBILITY STATEMENT

Shared protocol

The shared protocol evaluates unstructured pruning on five pretrained models, with language-modeling and zero-shot evaluation and posterior distillation over fixed masks. The supplied settings also list allocation configurations and ablation comparisons.

Shared protocol

  • The protocol uses pretrained OPT-125M, OPT-350M, OPT-1.3B, Qwen3-0.6B, and Llama-3.2-1B models without additional training before pruning.It prunes unstructured transformer-block Linear weights, excluding biases and output heads, at requested sparsities from 10% to 90% in 10-point increments.Source: Shared protocol · p. 13
  • Evaluation uses WikiText-2 and PTB test splits, C4 validation, and seven zero-shot lm-eval tasks with batch size 1.The language-modeling evaluations use contiguous 2,048-token windows.Source: Shared protocol · p. 13
  • Posterior distillation uses a dense teacher, logit KL with T = 2, AdamW at learning rate 10−4, gradient clipping 1, and five epochs over nine fixed masks.Source: Shared protocol · p. 13
  • The listed configuration uses γ = 0.20 for OPT/Qwen, γ = 0.10 for Llama, and zmin = 1.Source: Shared protocol · p. 13
  • Other listed protocol fragments include projection-level reconstruction error, calibration perplexity for tensor-level pruning choices, layer-wise outlier distribution, and LSAC settings of λ = 0.08, 50% probe sparsity, and profile block size 128.The settings also list M = 10 and λ = 0.08.Source: Shared protocol · p. 13
  • The OPT-125M, γ = 0.20 ablation compares Magnitude, Wanda, and SparseGPT alone and within LADDER, and continues OWL, DSA, and LSA masks from 70% to 80% and 90%.Standard dense-to-sparse LADDER is the reference for the initialization comparison.Source: Shared protocol · p. 13

A SUPPLEMENTARY MATERIAL

The supplementary material supplies experimental settings and results omitted from the main paper. It covers pruning, evaluation, distillation, language-modeling and downstream results, ablations, and mask analyses.

A SUPPLEMENTARY MATERIAL

A.1 Reproducibility and Exact Configurations

The appendix fixes shared pruning and evaluation settings to support reproducible comparisons. It also specifies LADDER’s calibration, posterior-distillation, and implementation procedures.

A.1 Reproducibility and Exact Configurations

  • OWL, DSA, LSA, and LADDER share SparseGPT as the local pruning backend, with 128-column blocks and mean-diagonal Hessian damping of 0.01.The common backend and fixed local mask selection and weight reconstruction isolate global sparsity allocation.Source: §A.1 · p. 13
  • All allocation methods use the same 128 C4 training sequences of 2,048 tokens; LADDER also uses their corpus perplexity to select candidates.The calibration samples are not used to compute reported language-modeling loss.Source: §A.1 · p. 13
  • Posterior distillation fixes nine masks from 10% to 90% sparsity and jointly optimizes one shared student initialized from pretrained dense weights.Training uses five epochs of AdamW, learning rate 10^-4, and temperature T = 2.Source: §A.1 · p. 14
  • Runs use seed 0 and deterministic algorithms, so each reported model–method pair is one run rather than an average across independent runs.Distillation accumulates equally weighted gradients over nine masks before each update, for 640 total updates.Source: §A.1 · p. 14

A.2 Downstream Summary

The downstream evaluation aggregates accuracy across seven tasks and compares methods from 50% to 90% sparsity. At 70–90%, LADDER has the highest mean accuracy in seven of nine model–sparsity comparisons, with task-level variation and occasional reversals.

A.2 Downstream Summary

  • Figure 7 compares post-distillation mean accuracy across seven tasks from 50% to 90% sparsity, against dense and random-accuracy references.Shading denotes propagated lm-eval uncertainty; the seven-task mean random accuracy is 35.7%.Source: Figure 7 · p. 15
  • LADDER has the highest mean accuracy in seven of nine model–sparsity comparisons at 70–90% sparsity.OWL leads for Llama-3.2-1B at 80%, and LSA leads for that model at 90%.Source: §A.2 · p. 15, Figure 7 · p. 15

A.3 Ablations

Across the ablations and appendix results, LADDER improves high-sparsity pruning outcomes, while posterior distillation usually reduces test loss. Its downstream advantage is aggregate rather than universal, and its masks remain nested across sparsity levels.

A.3 Ablations

  • At 70–90% sparsity, LADDER lowers test loss in all 27 matched local-rule–dataset comparisons, and SparseGPT yields the lowest LADDER loss in all nine dataset–sparsity comparisons.Table 3 compares each local pruning rule used alone and within LADDER.Source: §A.3 · p. 15, Table 3 · p. 16
  • Continuing LADDER from OWL, DSA, or LSA 70% masks lowers loss versus extending the original method in all 18 comparisons; dense-to-sparse LADDER has the lowest loss on every dataset at 90%.OWL provides the strongest external starting point.Source: §A.3 · p. 15, Table 4 · p. 16

A.4 Complete Language-Modeling Results

  • Posterior distillation reduces test loss in 178 of 180 model, dataset, method, and sparsity combinations at 70–90%.Both exceptions are 90%-sparse OPT-350M LADDER models.Source: §A.4 · p. 15, Table 6 · p. 17

A.5 Task-Level Downstream Results

  • Downstream results support stronger aggregate accuracy retention at high sparsity, not superiority on every task or configuration.The leading method varies across tasks and sparsity levels, with ties and occasional reversals.Source: §A.5 · pp. 15–16

A.6 Allocation Patterns and Mask Nestedness

  • Across four measured transitions from 50% to 90%, LADDER reintroduces no weights for any evaluated model, while OWL, DSA, and LSA reintroduce some.Figure 14 partitions higher-sparsity kept weights into those retained from the next lower sparsity and those reintroduced; the comparison verifies nestedness, not its effect on language-modeling performance.Source: §A.6 · p. 16, Figure 14 · p. 24

Related papers

Loading 2610.28773v1…