OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction

• One-shot pruning seeks to reduce deployment and inference costs, but fast zeroing heuristics often lose accuracy while second-order optimization is infeasible at modern LLM scale.Approximate one-shot methods such as SparseGPT offer a practical middle ground but sacrifice reconstruction optimality.Source: §1 · p. 1
• OPTIMA reformulates post-mask layer reconstruction as column-wise QPs sharing a layer Hessian, yielding per-column global optimality for the estimated-Hessian objective.The reconstruction objective decomposes across columns, while their linear constraints differ according to the removed mask entries.Source: §1 · pp. 1–2
• A first-order primal–dual solver, incremental layer-Hessian accumulation, and batched column solves make the shared-QP pipeline practical on a single accelerator.The implementation uses tight bounds for fixed entries and processes thousands of small QPs in parallel.Source: §1 · p. 2
• Across eight zero-shot downstream benchmarks, OPTIMA reports up to 2.53 percentage-point gains without post-pruning fine-tuning.It is modular and plugs into existing mask selectors, including Wanda, SparseGPT, and Thanos.Source: §1 · p. 2
• Up to 3.97% average accuracy gains across tasks and model sizes, with robustness up to 60% sparsity and pruning of billion-parameter models on one H100 in under 40 hours.The paper also reports consistent zero-shot improvements without post-pruning fine-tuning.Source: §1 · p. 2

• Post-training pruning decomposes the reconstruction task into layer-wise subproblems that choose a binary mask and updated weights to minimize output reconstruction error.Each layer uses the previous layer's pruned output as its input.Source: §2 · p. 2
• Finding the optimal binary sparsity mask is NP-hard, motivating heuristic mask selection.Source: §2 · p. 2
• After mask selection, the weight-update problem is convex least squares, but solving it directly is computationally expensive for large LLM weights.Source: §2 · p. 3
• Existing approaches trade update quality for feasibility: Wanda sets selected weights to zero, while SparseGPT and Thanos update only a small subset.Source: §2 · p. 3

3 OPTIMA: OPTIMAL WEIGHT UPDATES VIA QUADRATIC PROGRAMMING

• OPTIMA updates all remaining weights after mask selection by reformulating reconstruction as independent QPs solvable in parallel on accelerators.The method also derives constrained and equivalent unconstrained formulations, implementing the constrained form for accelerator suitability.Source: §3 · p. 3

3.1 REFORMULATION AS A QUADRATIC PROGRAM WITH LINEAR CONSTRAINTS

• After mask selection, reconstruction separates by column; each QP fixes pruned entries to the negative original weights while optimizing the remaining changes.All column-wise problems share the positive-semidefinite Hessian H = X_l^T X_l, while their constraints vary with the mask.Source: §3.1 · pp. 3–4

3.2 REFORMULATION AS AN UNCONSTRAINED QUADRATIC PROGRAM

• The unconstrained formulation substitutes fixed pruned-weight changes into the objective, but variable problem dimensions make its QPs harder to batch.The dimensions vary with the number of unpruned weights in each column, motivating the uniform constrained formulation for parallel execution.Source: §3.2 · p. 4

3.3 SOLVING THE QUADRATIC PROGRAMS

• OPTIMA uses the rAPDHG first-order solver, whose Hessian matrix-vector operations run efficiently on GPUs and TPUs and support parallel solution of thousands of column QPs.The passage also reports provably optimal linear convergence and an open-source JAX implementation in MPAX.Source: §3.3 · p. 5

3.4 EFFICIENT IMPLEMENTATION

• Fixed-variable constraints use bounds instead of large equality matrices; column-wise QP batching reduces memory overhead in memory-limited settings; per-sequence covariance accumulation matches direct computation without storing the entire output tensor.Source: §3.4 · pp. 5–6

Model, datasets, and evaluation

• Evaluation covers LLaMA and Gemma families, zero-shot downstream tasks, and WikiText2 perplexity, comparing OPTIMA updates with Wanda, SparseGPT, Thanos, and ProxSparse where applicable.The evaluation uses the Language Model Evaluation Harness; OPTIMA is applied to masks generated by existing pruning methods.Source: §4 · p. 6

Model quality

• At 50% unstructured sparsity, OPTIMA improves accuracy across tasks, with up to 1.80% average accuracy improvement across six downstream tasks (Gemma-3-1B).Table 1 reports WikiText2 perplexity and zero-shot downstream-task accuracy.Source: §4 · pp. 6–7, Table 1 · p. 7

2:4 semi-structured sparsity

• Under 2:4 sparsity, OPTIMA consistently improves accuracy when pruning MLP weights and leaving attention dense, yielding 38% to 41% end-to-end sparsity.The reported setup selectively prunes MLP components to maintain accuracy above a practical threshold.Source: §4 · p. 7, Table 2 · p. 8

Comparison with alternative optimizers

• Compared with ADAM, OPTIMA produces stable, high-quality QP updates and is reported to guarantee convergence, while ADAM can be suboptimal or diverge.ADAM is occasionally competitive on larger models but can diverge on smaller ones.Source: §4 · p. 8

Higher sparsity ratios

• At 60% unstructured sparsity, OPTIMA gains up to 2.53% average accuracy across downstream tasks (LLaMA-3.2-1B); on LLaMA-3.2-3B, Wanda rises from 38.77% to 42.74%.The reported comparison covers WikiText2 perplexity and zero-shot accuracy.Source: §4 · pp. 7–8, Table 3 · p. 9

Layer-wise error improvement

• OPTIMA's relative output error improves across pruned LLaMA-3.2 1B layers, and the paper reports superior downstream-task accuracy.Figure 2 compares layer-wise errors with and without OPTIMA, using mean squared error across the calibration dataset.Source: §4 · p. 9

Pruning time analysis

• On one NVIDIA H100 with 80GB memory, pruning took 2.5 hours for LLaMA 3.2 1B and Gemma 3 1B, and 40 hours for LLaMA 3.1 8B.Other reported times were 5.5 hours for Gemma 2 2B and 7.0 hours for LLaMA 3.2 3B; pruning time scales with model size.Source: §4 · p. 9

Conclusion

• OPTIMA reformulates post-training weight reconstruction as batched, column-wise QPs sharing a layer Hessian, yielding per-column optimal updates for the least-squares objective given the estimated Hessian.The shared-Hessian structure enables parallel solves on GPUs and TPUs.Source: §5 · p. 10
• An accelerator-friendly primal-dual solver batches many small QPs, and OPTIMA serves as a drop-in weight-update step for Wanda, SparseGPT, and Thanos mask selectors.The solver accumulates one Hessian per layer.Source: §5 · p. 10
• Across LLM families, OPTIMA improves zero-shot accuracy by up to 3.97 percentage points, including at sparsity levels of at least 60%, without post-pruning fine-tuning.The experiments were run on a single NVIDIA H100.Source: §5 · p. 10

A EXTENDED EVALUATION ON THE QWEN-2.5 MODEL FAMILY

• The extended evaluation covers Qwen-2.5 models from 0.5B to 14B parameters at 50% and 60% unstructured sparsity and 2:4 sparsity.The authors describe these models as outside the main paper’s analysis and use them to test whether benefits extend across architectures.Source: A EXTENDED EVALUATION ON THE QWEN-2.5… · p. 13

A.1 UNSTRUCTURED SPARSITY (50% AND 60%)

• At 50% unstructured sparsity, Qwen-2.5 3B Wanda average accuracy rises from 54.02% to 55.33%, and SparseGPT rises from 54.70% to 55.69%.OPTIMA consistently improves zero-shot performance across all model sizes and mask selectors.Source: §A.1 · p. 13
• 4.19% absolute gain raises Wanda’s Qwen-2.5 3B average accuracy from 43.67% to 47.86% at 60% unstructured sparsity; Thanos rises from 48.45% to 49.98%, a 1.53% gain.The passage describes gains as more pronounced at this more aggressive sparsity level.Source: §A.1 · p. 13

A.2 SEMI-STRUCTURED SPARSITY (2:4)

• At 2:4 sparsity, Qwen-2.5 3B Wanda average accuracy increases from 49.48% to 50.63%, and 1.5B accuracy increases from 46.01% to 47.26%.Only MLP layers are pruned in this setting.Source: §A.2 · p. 13
• On the larger 7B and 14B models, results are more varied, with performance differing based on the underlying mask selector.The passage suggests a complex interaction between mask-selection heuristics and OPTIMA reconstruction for structured sparsity at this scale, a potential avenue for future investigation.Source: §A.2 · p. 13

B COMPARISON WITH ALTERNATIVE OPTIMIZERS

• ADAM uses MSE minimization with four learning rates from 10^-2 to 10^-5, linear scheduling, and per-layer exhaustive selection of the best configuration.The paper notes that ADAM lacks convergence guarantees and is sensitive to hyperparameters.Source: B COMPARISON WITH ALTERNATIVE OPTIMIZERS · p. 13
• On Gemma 3 1B, SparseGPT+ADAM reaches lower perplexity than OPTIMA, 27.12 versus 27.35, while Wanda+OPTIMA has higher accuracy than Wanda+ADAM, 44.01% versus 43.72%.The reported Gemma comparison therefore differs by metric and pruning setup.Source: B COMPARISON WITH ALTERNATIVE OPTIMIZERS · p. 13
• On OPT 125M, Wanda+ADAM diverges and reaches perplexity 205.82, compared with 35.44 for Wanda+OPTIMA.The passage characterizes ADAM as producing suboptimal solutions on Gemma 3 1B or divergence on OPT 125M.Source: B COMPARISON WITH ALTERNATIVE OPTIMIZERS · p. 13

C RELATED WORK

• The overview frames pruning as reducing deployment computation and memory by removing redundant weights from trained networks.It distinguishes layer-wise approaches such as OBS from end-to-end approaches such as OBD.Source: C RELATED WORK · p. 14

Layer-wise model pruning

• Layer-wise pruning targets within-layer redundancy: OBS adjusts remaining weights after removal, while SparseGPT approximates sparse regression for LLM scale and Thanos uses multi-column pruning.Wanda instead scores weights using magnitudes and calibration activations, but does not update weights after pruning.Source: C RELATED WORK · p. 14

End-to-end model pruning

• End-to-end methods use global weight importance; OBD relies on second-order loss derivatives, while MaskLLM reframes pruning as classification and ProxSparse uses regularizers.The passage describes end-to-end optimization as resource-intensive and says ProxSparse trades accuracy for speed relative to MaskLLM.Source: C RELATED WORK · p. 15

Other model compression methods

• Quantization, low-rank adapters, and knowledge distillation complement pruning by reducing precision, factorizing weights, or transferring knowledge to compact students.The paper presents these approaches as addressing different aspects of redundancy and as potential components of hybrid compression frameworks.Source: C RELATED WORK · pp. 15–16

D IMPLEMENTATION DETAILS AND HYPERPARAMETERS

• Calibration uses 128 samples of 2048 tokens each from C4.The paper says this follows previous work.Source: D IMPLEMENTATION DETAILS AND… · p. 16
• The rAPDHG solver uses 0.01 relative and absolute tolerances and a maximum of 100,000 iterations.OPTIMA skips a layer update if most problems fail to converge within the limit or if final layer error exceeds initial error.Source: D IMPLEMENTATION DETAILS AND… · p. 16
• Other baselines use publicly available checkpoints or are reproduced from their repositories with default hyperparameters.The implementation appendix points to Table 8 for OPTIMA’s key hyperparameters.Source: D IMPLEMENTATION DETAILS AND… · p. 16, Table 8 · p. 17

E CALIBRATION DATASET SIZE SENSITIVITY

• OPTIMA’s calibration-sample sensitivity is measured by WikiText2 perplexity for LLaMA-3.2-1B using varying numbers of C4 calibration samples.The authors state that OPTIMA, like SparseGPT, Wanda, and Thanos, uses C4 calibration data.Source: E CALIBRATION DATASET SIZE SENSITIVITY · p. 17
• As calibration samples increase, OPTIMA shows consistent but nonsignificant model-quality improvement, while Wanda and SparseGPT behave stochastically.The authors interpret the small improvements as indicating robustness to dataset size.Source: E CALIBRATION DATASET SIZE SENSITIVITY · p. 17

F LANGUAGE MODEL USAGE IN PAPER

• Language models were used to improve writing clarity, correct grammar and typographical issues, and check ICLR guideline adherence.Outside benchmark evaluations and experimental analyses, they were not applied to other components of the work.Source: F LANGUAGE MODEL USAGE IN PAPER · p. 17

G REPRODUCIBILITY STATEMENT

• The anonymous repository linked in the abstract footnote provides source code and scripts for reproducing all experiments.The authors present these materials as supporting independent reproduction of their findings.Source: G REPRODUCIBILITY STATEMENT · p. 17
• Sections 3 and 4 describe the method and experimental setup, while Appendix D gives implementation details, hyperparameters, and model configurations.Source: G REPRODUCIBILITY STATEMENT · p. 17
• The authors state that these resources ensure independent researchers can reproduce the findings with minimal effort.Source: G REPRODUCIBILITY STATEMENT · p. 17