PaperlayerView exact revision on arXiv

Source-linked AI summary

Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too

Mehdi Makni, Natalia Ponomareva, Rahul Mazumder

arXiv:2610.18516v1cs.LGstat.ML

TL;DR

Conventional post-training pruning does not explicitly preserve gradient alignment, and curvature-aware compression has been costly to optimize. CACTUS uses ADMM to optimize a curvature-aware objective for pruning and sparse-plus-low-rank compression. Across settings it improves downstream utility while retaining gradient fidelity, and the authors identify several compression settings outside the paper's scope.

  • Problem

    Layerwise reconstruction is gradient-blind, while curvature-aware compression incorporates downstream gradients but can require multiple GPU hours per model.

    Source: §1 · p. 1

  • Method

    CACTUS uses ADMM to optimize a curvature-aware objective for pruning and sparse-plus-low-rank compression.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Across compression settings, architectures, and gradient projection strategies, CACTUS improves downstream utility while retaining gradient fidelity.

    Source: §5 · p. 9

  • Takeaways & Limitations

    At 50% pruning on Llama-3.2-1B, CACTUS raises WikiText-2 per-example gradient cosine similarity from 0.445 with LLM-Surgeon to 0.512.

    Source: §1 · p. 2

    The study focuses on pruning and sparse-plus-low-rank compression, uniform budgets, and no post-compression full-model or LoRA-style fine-tuning.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Post-training pruning enables efficient deployment, but conventional methods do not explicitly preserve internal-gradient alignment between sparse and dense models. We study novel connections between downstream performance of compressed models and the ability to retain gradient information via influence-function preservation. Central to this connection is a curvature-aware objective that accounts for both layer activations and backpropagated gradients. Despite its promise, this objective remains underexplored in pruning due to a lack of efficient solvers. We introduce $\texttt{CACTUS}$, a novel ADMM-based optimizer supporting both pruning and sparse-plus-low-rank decompositions of pre-trained weights. Compared to LLM-Surgeon which targets the same objective, $\texttt{CACTUS}$ can achieve over an order-of-magnitude speedup. At $50\%$ pruning of Qwen3-4B, $\texttt{CACTUS}$ reduces the C4 perplexity gap to the dense model by $45\%$, improves compressed-to-dense per-sample gradient cosine similarity on WikiText-2 from $0.670$ to $0.755$, and runs over $28\times$ faster than LLM-Surgeon on substantially weaker hardware. By preserving influence functions, $\texttt{CACTUS}$ supports effective data selection. On Llama-3.2-3B, fine-tuning a fresh dense model on the $5\%$ of data selected by $\texttt{CACTUS}$ achieves $49.44\%$ accuracy on BIG-Bench Hard, compared with $47.22\%$ using ALPS and $49.54\%$ using dense-model selection. Finally, integrating $\texttt{CACTUS}$ with LoGRA, a recent gradient-projection strategy, on Qwen3-4B with MLP layers pruned to 2:4 sparsity and a rank-64 correction retains a $0.711$ cosine alignment with dense LoGRA-projected C4 gradients.

1 INTRODUCTION

The paper connects post-training compression with influence-function preservation through a curvature-aware objective that uses both layer activations and downstream gradients. CACTUS makes this objective practical for pruning and sparse-plus-low-rank compression, with reported gains in compression quality and gradient fidelity.

Motivation

  • Conventional layerwise reconstruction matches layer outputs but is gradient-blind, while curvature-aware compression also incorporates calibration-set downstream gradients.LLM-Surgeon and YAQA incorporate downstream gradients, but optimizing the objective can require multiple GPU hours per model.Source: §1 · pp. 1, 3
  • Influence-function estimation uses per-example gradients and an inverse-Hessian term, but per-sample gradient computations remain costly at LLM scale.IProX uses curvature-aware compression to preserve per-sample gradients in a low-rank proxy.Source: §1 · p. 2

Contributions

  • CACTUS runs over an order of magnitude faster than LLM-Surgeon on substantially weaker hardware while supporting pruning and sparse-plus-low-rank compression.The algorithm uses ADMM and targets the curvature-aware objective.Source: §1 · p. 2

Compression quality

  • At 50% compression on Llama-3.2-1B, CACTUS reduces the C4 perplexity gap by 25% for pruning and 28% for sparse-plus-low-rank decompositions versus corresponding baselines.For Qwen3.8-27B under 2:4 sparsity, it reduces the C4 perplexity gap to dense by over 25% versus SparseGPT.Source: §1 · p. 2

Influence preservation

  • At 50% pruning on Llama-3.2-1B, CACTUS improves WikiText-2 per-example gradient cosine similarity between compressed and dense models from 0.445 with LLM-Surgeon to 0.512.Using 50%-sparse pruned models to select 5% of an instruction-tuning mixture for a fresh Llama-3.2-3B, CACTUS yields 49.44% BBH accuracy versus 47.22% with ALPS.Source: §1 · p. 2

2 CURVATURE-AWARE MODEL COMPRESSION

CACTUS minimizes a KFAC curvature-aware loss using activations and downstream-gradient statistics while enforcing pruning or sparse-plus-low-rank constraints. Its ADMM updates exploit eigenspaces to make the coupled solve tractable, and the stated assumptions yield convergence to a feasible decomposition.

2.1 PROBLEM FORMULATION

  • The KFAC loss uses input-activation and downstream-gradient covariances to choose a structurally constrained compressed weight.The method collects gradient covariance factors once, then constructs activation covariance factors sequentially as layers are compressed.Source: §2.1 · p. 3

2.2 CACTUS ALGORITHM DERIVATION

  • CACTUS applies ADMM to the sparse-plus-low-rank problem, splitting the sparse constraint from the smooth objective through a consensus equality.Pure pruning is the special case with rank r = 0.Source: §2.2 · pp. 3–4, §2.1 · p. 3
  • The S-update solves a two-sided generalized Sylvester equation exactly in the eigenspaces of H and G, avoiding a naive O((din dout)^3) vectorized solve.Reusing eigenspaces reduces the smooth solve to matrix multiplications and elementwise division.Source: §2.2 · p. 4
  • The low-rank update uses a weighted rank-r SVD, while the sparse projection retains top-magnitude entries or top-n entries in each n:m block.A candidate low-rank update is accepted only when its movement is bounded relative to the smooth-block movement.Source: §2.2 · p. 4

2.3 CONVERGENCE

  • Under positive-definite factors, exact smooth solves and sparse projections, and the theorem's penalty and movement-bound conditions, the iterates converge to a feasible decomposition.Changes in S and L and the consensus residual are O(1/ρ(t−1)) in Frobenius norm.Source: §2.3 · p. 5

3 EXPERIMENTAL RESULTS

The experiments evaluate compressed-model utility and influence-function preservation, then test data selection and LoGRA-projected gradients. Results include gains in TracIn retention, data-selection outcomes, and LoGRA-projected influence metrics.

3.1 EXPERIMENTAL SETUP

  • Evaluation covers C4 and WikiText-2 perplexity plus TracIn and KFAC-weighted influence metrics, including per-example cosine fidelity and query-ranking Spearman and recall.The setup uses separate C4 training examples and WikiText-2 queries for influence evaluation.Source: §3.1 · pp. 5–6

3.2 ONE-SHOT COMPRESSION RESULTS

  • At 50% compression on Llama-3.2-1B, CACTUS reduces the C4 perplexity gap to dense by 28% over the closest S+L method and 25% over LLM-Surgeon(5sh).CACTUS-prune runs over 15× faster than LLM-Surgeon(5sh).Source: §3.2 · p. 6
  • 38% higher per-query TracIn retention over the closest competitor with CACTUS-SLR on Llama-3-8B at 2:4+r256.The configuration is reported with 1.32× inference speedup and 29% memory reduction using specialized kernels.Source: §3.2 · p. 6, Figure 3 · p. 7
  • At 50% Qwen3-4B compression, 3BASiL has better WikiText-2 perplexity than CACTUS-prune (15.34 vs. 16.27) but lower dense-gradient cosine alignment (0.743 vs. 0.762).The comparison shows that perplexity and gradient preservation do not always rank methods alike.Source: §3.2 · p. 6

3.3 DATA SELECTION RESULTS

  • 49.44 BBH EM on Llama-3.2-3B versus 47.22 for ALPS; CACTUS also led score-rank correlation (0.638 vs. 0.594) and dense-selection overlap (47.20% vs. 44.07%).The comparison evaluates subsets selected with the LESS procedure after fine-tuning a fresh dense model.Source: Table 3 · p. 7, §3.3 · p. 7

3.4 LOGRA INTEGRATION

  • LoGRA projects layer activations and backpropagated gradients without materializing full per-sample gradients; shared rank-64 probes support direct dense-versus-compressed comparisons.The comparisons use the same projection probes for dense and compressed models.Source: §3.4 · p. 8

LoGRA+CACTUS-SLR

  • For CACTUS-SLR, the low-rank correction is incorporated into a widened LoGRA adapter, while only the original probe’s gradient block is retained.The augmented branch contributes the deployed low-rank correction to the forward pass.Source: LoGRA+CACTUS-SLR · p. 8
  • On Qwen3-4B, CACTUS-SLR improves all LoGRA-projected influence-preservation metrics over pruning, reaching 0.804 per-query Spearman and 0.597 recall at 5%.CACTUS-prune uses one L40 GPU versus two A100 GPUs for LLM-Surgeon and runs roughly 5× and 25× faster than its 1-shot and 5-shot variants.Source: LoGRA+CACTUS-SLR · p. 8

4 RELATED WORK

Prior compression methods differ in their constraints and solvers, while curvature-aware methods incorporate downstream gradients. CACTUS applies the curvature-aware objective to pruning and sparse-plus-low-rank compression and extends compressed proxies for influence estimation.

One-shot pruning and low-rank decompositions for LLMs

  • One-shot post-training compression minimizes layerwise reconstruction error on calibration data, with methods differing by constraint family and per-layer solver.SparseGPT uses sequential column sparsification, while ALPS formulates a constrained quadratic problem solved by ADMM.Source: §4 · pp. 8–9

Curvature-Aware Compression

  • CACTUS optimizes the curvature-aware loss for pruning and sparse-plus-low-rank compression using multi-block ADMM and efficient subroutines.Prior methods include LLM-Surgeon for pruning, IProX’s weighted-SVD low-rank proxy, and YAQA for weight-only quantization.Source: §4 · p. 9
  • CACTUS-prune is over 28× faster than five-shot LLM-Surgeon on Qwen3-4B at 50% compression, using one L40 instead of two A100 GPUs.CACTUS pruning and sparse-plus-low-rank representations also achieve substantially lower perplexity than IProX’s aligned low-rank proxy.Source: §4 · p. 9

Efficient Influence Function Estimation

  • At LLM scale, exact per-sample gradients and inverse-Hessian-vector products are prohibitively expensive to compute and store.Compressed proxies and gradient projections are two complementary approaches to tractable influence estimation.Source: §4 · p. 9

5 CONCLUSION AND LIMITATIONS

CACTUS connects downstream utility with influence preservation through a curvature-aware objective and makes its optimization practical for pruning and sparse-plus-low-rank decomposition. The paper identifies compression settings and refinements it leaves for future work, and includes an AI-use disclosure.

5 CONCLUSION AND LIMITATIONS

  • Across compression settings, architectures, and gradient projection strategies, CACTUS consistently improves downstream utility while retaining gradient fidelity.The conclusion connects these outcomes to the curvature-aware objective and its lower cost relative to prior state-of-the-art curvature-aware pruning methods.Source: §5 · p. 9
  • The study focuses on pruning and sparse-plus-low-rank compression; quantization and mixed quantization-sparsity extensions remain future work.The authors note particular difficulty when reparameterizations such as rotations change gradient geometry.Source: §5 · pp. 9–10
  • The study uses uniform compression budgets and does not evaluate post-compression full-model or LoRA-style fine-tuning.The authors leave non-uniform allocation and possible deployment-quality versus gradient-alignment trade-offs from refinement for future work.Source: §5 · p. 10

AI USE STATEMENT

  • The authors report using generative AI for code development, manuscript revision and polishing, and figure plotting and formatting.They state that the authors are responsible for the final content and verification of reported results.Source: AI USE STATEMENT · p. 10

A THE LLM-SURGEON AND IPROX DERIVATIONS

Figure 5 presents one curvature-aware objective with two derivations, connecting model compression and influence preservation. Its setup defines the layer, weight perturbation, and dense-model input and loss gradient used in the derivation.

A THE LLM-SURGEON AND IPROX DERIVATIONS

  • For a layer y = W^T x, the setup defines the weight perturbation as ΔW = cW − W and g as the dense-model loss gradient with respect to y.The input x and gradient g are evaluated at the dense model.Source: A THE LLM-SURGEON AND IPROX DERIVATIONS · p. 13
  • Figure 5 connects LLM-Surgeon’s model-compression motivation and IProX’s influence-preservation motivation to Equation (1).The caption states that the influence-distortion bound uses IProX’s assumptions, Proposition 4.1, and Appendix D.Source: Figure 5 · p. 13, A THE LLM-SURGEON AND IPROX DERIVATIONS · p. 13
  • The figure identifies H = X^T X and G = A^T A, with Cκ the finite constant in the influence-distortion bound.That bound relies on the assumptions of IProX.Source: Figure 5 · p. 13

B CONVERGENCE OF CACTUS

The convergence proof follows 3BASiL’s argument while modifying the low-rank and smooth-step bounds for CACTUS. The remaining recurrence, movement bound, and feasibility argument are unchanged.

B CONVERGENCE OF CACTUS

  • The rescaled dual update matches 3BASiL, and the proof identifies two bounds that change.The variables cW, D(t), and V(t) correspond to W, Δt, and Yt = ρtVt, respectively.Source: B CONVERGENCE OF CACTUS · p. 14
  • The low-rank bound replaces 3BASiL’s selection of constant CF in Equation (9).The passage also notes that if the candidate is rejected, Lt = Lt−1, as in Lemma A.1.Source: B CONVERGENCE OF CACTUS · p. 14
  • The smooth-step bound uses the Sylvester equation from Equation (5), whose entries in two orthonormal eigenbases are divided by 2λH,iλG,j + ρt ≥ ρt.The proof then applies the projection inequality to obtain the bounds of Lemma A.2.Source: B CONVERGENCE OF CACTUS · p. 14
  • The proof retains the same recurrence and movement bound as 3BASiL, while closed constraint sets ensure feasibility of the limits.The recurrence follows from the proof of Lemma A.3, and Equation (8) gives the movement bound.Source: B CONVERGENCE OF CACTUS · p. 14

C ADDITIONAL EXPERIMENTAL PROTOCOL DETAILS

The appendix specifies compression and evaluation settings for influence preservation, data selection, and LoGRA integration. It also documents damping and ADMM schedules and the baselines used for comparison.

ADMM Settings

Compression and Damping Details

  • The main compression comparisons use matched layerwise frameworks and calibration samples, with activation-only baselines distinguished from methods that also use gradients.LLM-Surgeon uses row-wise correlated updates and five-shot settings in specified comparisons; Qwen3-4B MLP 2:4 results include both one-shot and five-shot variants. Ridge damping is generally λH = 0.01 and λG = 0.1, though 0.005 can cause decomposition issues for large layers or architectures.Source: C ADDITIONAL EXPERIMENTAL PROTOCOL… · pp. 15–16

Compression Protocol

  • 128 C4 sequences of length 2048 provide calibration statistics, while perplexity uses C4 and WikiText-2 and influence metrics use projected per-example gradients.Training and test gradients come from 1,000 C4 sequences and 100 WikiText-2 sequences, respectively, at length 512.Source: C ADDITIONAL EXPERIMENTAL PROTOCOL… · p. 16

Data Selection Protocol

  • The LESS selection protocol compares dense and compressed scorers on 270,679 candidates, with shared warm-up data and LoRA training settings.Candidate updates and query gradients use a shared 8192-dimensional TRAK projection.Source: C ADDITIONAL EXPERIMENTAL PROTOCOL… · p. 16

LoGRA Integration Protocol

  • The LoGRA integration evaluates Qwen3-4B with MLP linears at 2:4 sparsity; CACTUS-SLR adds rank-64 corrections, while attention projections remain dense.Adapters use shared random subspaces across methods, and CACTUS-SLR’s rank-2r adapter yields the standard rank-r gradient statistic by slicing its top-left block.Source: C ADDITIONAL EXPERIMENTAL PROTOCOL… · pp. 16–17

D ADDITIONAL EXPERIMENTAL RESULTS

The additional-results appendix documents influence-preservation and perplexity comparisons across several models, compression levels, and representations. It also includes a broad Qwen3.8-27B comparison across decoder matrices and zero-shot tasks.

Influence-Preservation Results

Llama Perplexity and Influence Results

Qwen3.8-27B Results

  • Table 14 covers Qwen3.8-27B at 2:4 sparsity across 496 decoder linear matrices, reporting C4 and WikiText-2 perplexities and seven zero-shot accuracies.The table also reports total runtime using two NVIDIA L40S GPUs.Source: Table 14 · p. 23

D.1 LESS SELECTION SCORES DISTRIBUTION

Figures 6 and 7 present dense and CACTUS-prune LESS scores for Llama-3.2-3B, emphasizing score distributions relative to dense-model ranking and the selection cutoff.

Figure 6

  • Figure 6 orders candidates by the dense model’s aggregate score and marks the top-5% cutoff with a dashed line.The top panel shows the full instruction mixture and the bottom panel the first 1,000 candidates.Source: Figure 6 · p. 24

Figure 7

  • Figure 7 displays the dense LESS and CACTUS-prune scores from Figure 6 separately.The figure description does not state the direction or magnitude of any score difference.Source: Figure 7 · p. 24

D.2 PER-LAYER LOSS DECREASE

The per-layer KFAC objective may rise early in CACTUS’s ADMM iterations, then stabilizes as the penalty increases and feasibility tightens.

Per-layer Loss Decrease

  • The KFAC objective can initially increase while ADMM explores feasible sparse candidates under a loose penalty, then decrease and converge as the penalty grows.Figure 8 compares representative trajectories for pure pruning and sparse-plus-low-rank compression.Source: Figure 8 · p. 25, §D.2 · p. 25

D.3 EFFECT OF CALIBRATION-SET SIZE

For CACTUS-prune at 50% unstructured sparsity, perplexity improves as the C4 calibration set grows from 32 to 256 sequences, with modest gains from 128 to 256 supporting 128 as a practical budget.

D.3 EFFECT OF CALIBRATION-SET SIZE

  • Doubling C4 calibration sequences from 128 to 256 yields modest perplexity improvements, supporting 128 sequences as a practical calibration budget.Perplexity improves with more calibration data on all three datasets.Source: §D.3 · p. 25, Table 15 · p. 25
  • The calibration-size study tests CACTUS-prune at 50% unstructured sparsity on Llama-3.2-1B and Qwen3-4B, using 32–256 C4 sequences.Source: §D.3 · p. 25, Table 15 · p. 25
  • All other settings are held fixed for each model; runs use sequence length 2048 and ridge coefficients λH = 0.01 and λG = 0.1.Source: §D.3 · p. 25, Table 15 · p. 25

Related papers

Loading 2610.18516v1…