PaperlayerView exact revision on arXiv

Source-linked AI summary

ROCKET: Rapid Optimization via Calibration-guided Knapsack Enhanced Truncation for Efficient Model Compression

Ammar Ali, Baher Mohammad, Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip, Stamatios Lefkimmiatis

arXiv:2602.11008v1cs.LGcs.AIcs.CL

TL;DR

Modern LLMs impose substantial computational and memory demands, motivating compression methods that reduce parameters without costly retraining. ROCKET combines single-step sparse factorization with knapsack-based layer allocation and consistently outperforms baselines across architectures and compression ratios, retaining over 90% of original accuracy at 30% compression without fine-tuning.

  • Problem

    Modern LLMs require substantial computational and memory resources, while low-rank factorization reduces parameters but imposes representational rigidity.

  • Method

    ROCKET uses calibration-guided structured sparsification with closed-form dictionary updates and formulates layer-wise compression allocation as a multi-choice knapsack problem.

  • Results

    ROCKET consistently outperforms baseline methods across multiple architectures and 20%–50% compression ratios, retaining over 90% of original accuracy at 30% compression without fine-tuning.

  • Takeaways & Limitations

    ROCKET enables high-fidelity, training-free compression of billion-parameter LLMs with minimal overhead under a global parameter budget.

  • Takeaways & Limitations

    The dynamic programming allocation is difficult to scale to architectures with very large numbers of compressible components, such as Mixture-of-Experts models with 128 or more experts per block.

Abstract

from arXiv · show

We present ROCKET, a training-free model compression method that achieves state-of-the-art performance in comparison with factorization, structured-sparsification and dynamic compression baselines. Operating under a global compression budget, ROCKET comprises two key innovations: First, it formulates layer-wise compression allocation as a multi-choice knapsack problem, selecting the optimal compression level for each layer to minimize total reconstruction error while adhering to a target model size. Second, it introduces a single-step sparse matrix factorization inspired by dictionary learning: using only a small calibration set, it sparsifies weight coefficients based on activation-weights sensitivity and then updates the dictionary in closed form via least squares bypassing iterative optimization, sparse coding, or backpropagation entirely. ROCKET consistently outperforms existing compression approaches across different model architectures at 20-50\% compression rates. Notably, it retains over 90\% of the original model's performance at 30\% compression without any fine-tuning. Moreover, when applying a light fine-tuning phase, recovery is substantially enhanced: for instance, compressing Qwen3-14B to an 8B-parameter model and healing it with just 30 million tokens yields performance nearly on par with the original Qwen3-8B. The code for ROCKET is at github.com/mts-ai/ROCKET/tree/main.

1. Introduction

ROCKET addresses the resource demands of billion-parameter models with a training-free compression method that combines sparse factorization and globally optimized layer-wise allocation. It targets the rigidity of conventional low-rank compression while preserving practical deployment efficiency.

  • Modern LLMs create substantial computational and memory demands that hinder deployment on constrained hardware and in latency-sensitive applications.
  • Post-training low-rank factorization reduces parameters without costly retraining, but truncated SVD imposes a rigid shared subspace representation.
  • Union-of-subspaces representations offer greater flexibility by modeling weight matrices with sparse combinations of dictionary atoms or basis matrices.
  • ROCKET uses single-step structured sparsification and multi-choice knapsack allocation to minimize reconstruction error under a target model-size constraint.
  • ROCKET is training-free, uses calibration-guided coefficient sparsification, and dynamically distributes compression budgets across layers.

2. Related Work

Related work spans dynamic budget allocation, structured factorization, and sparsification. ROCKET combines these directions in a post-training framework that replaces iterative dictionary learning with a faster single-step procedure and optimizes layer choices globally.

  • Compression research combines dynamic per-layer allocation, structured matrix factorization, and sparsification to reduce model size under resource constraints.
  • Structured Matrix Factorization for Weight Approximation: Truncated SVD provides low-rank approximations, while data-aware methods use calibration-derived activation structure for more effective compression.
  • Structured Matrix Factorization for Weight Approximation: Sparse dictionary learning represents columns with different subsets of dictionary atoms, providing a flexible union-of-subspaces model beyond fixed layer-invariant bases.
  • Structured Matrix Factorization for Weight Approximation: ROCKET replaces iterative K-SVD and OMP optimization with a single-step greedy algorithm intended to improve reconstruction accuracy and accelerate compression.
  • Budget Allocation and Layer Importance: Unlike training-based allocation methods, ROCKET uses dynamic programming for multi-choice knapsack optimization with global error minimization and per-layer error bounds.

3. Method

ROCKET compresses weights through calibration-aware structured sparsification in whitened activation space, then allocates layer configurations under a constrained global budget. Its pipeline combines adaptive factorization, closed-form updating, and dynamic programming.

  • ROCKET targets the efficiency-flexibility trade-off by combining spectral compression speed with the expressive union-of-subspaces structure of sparse dictionary learning.
  • Structured Sparsification: Given calibration inputs and weights, ROCKET minimizes activation-aware reconstruction error using a data-adaptive representation.
  • Whitened Activation Space: Whitening decorrelates calibration inputs, making YᵀY = I_d1 and simplifying the reconstruction objective.
  • Factorization: The whitened weight is approximated as BC, where B is column-orthogonal and C is obtained by orthogonal projection.
  • Structured Sparsification: Structured sparsification operates on C so each output can activate a distinct subset of basis vectors, yielding a union-of-subspaces representation.
  • Importance Scoring: ROCKET combines whitened- and original-space importance measures to account for directional reconstruction sensitivity.
  • Structured Sparsification: Column-wise thresholding followed by global refinement produces a coefficient matrix at the exact target compression ratio.
  • Closed-Form Update: After sparsification, a ridge-regularized least-squares update computes the final dictionary without requiring orthogonality of the left factor.

4. Experiments

ROCKET is evaluated across architectures, compression ratios, allocation strategies, pruning methods, healing settings, and modalities. It consistently preserves or recovers model performance under strict parameter constraints, including after substantial compression and lightweight fine-tuning.

  • Experimental setup: Evaluations use LLaMA and Qwen models in zero-shot settings across eight accuracy benchmarks and perplexity measures, with compression ratios from 0.2 to 0.5.Calibration-based methods use 256 randomly sampled RefinedWeb sequences, and dense linear layers across attention and feed-forward blocks are compressed.
  • Comparison with factorization baselines: ROCKET consistently outperforms SVD-LLM and CoSpaDi across Qwen3-8B, Llama3-8B, and Llama3.2-1B at 20%–50% compression in zero-shot accuracy and perplexity.At 50% compression on Qwen3-8B, ROCKET achieves 51.3 average accuracy versus 38.1 for SVD-LLM and 42.0 for CoSpaDi.
  • Comparison with pruning methods: At 60% compression, ROCKET reaches 0.56 average accuracy against depth- and sparsity-based pruning baselines, and with quantization leads Dobi-SVD 0.60 to 0.52.At 40% compression with quantization, ROCKET scores 0.65 versus Dobi-SVD’s 0.63.
  • Post-compression healing: Healing a compressed Qwen3-14B model reduced to 8B parameters with 30 million tokens raises average accuracy from 63.56 to 67.96, approaching native Qwen3-8B at 70.46.The healing phase keeps the sparsity pattern fixed while fine-tuning unmasked factorized weights and the dictionary.
  • Budget allocation: ROCKET retains the most normalized performance under equivalent parameter constraints across Qwen3-8B and LLaMA2-7B configurations when compared with Uniform, ARS, Dobi-SVD, and ARA allocation strategies.The comparison covers Qwen3-8B at 20% and 40% pruning and LLaMA2-7B at 40% pruning across eight benchmarks.
  • Generalization to other modalities: ROCKET generalizes beyond language models: Qwen3-4B-VL retains 65.75 average accuracy, while VibeVoice shows nearly stable WER and only a small UTMOS decrease.For VibeVoice, WER is 0.149 versus 0.148 and UTMOS is 3.43 versus 3.52, close to the 3.73 groundtruth reference.

5. Ablations

ROCKET’s ablations show that reconstruction-error choice, activation-aware sparsification, and global budget allocation each materially affect compression quality. Relative Frobenius error performs best for profiling, while full ROCKET achieves the strongest combined accuracy and perplexity.

  • Reconstruction error metric: Relative Frobenius error yields the best downstream performance among the four layer-profiling metrics tested at 20% compression.The alternatives were ℓ1 distance, mean cosine distance, and spectral distance.
  • Reconstruction error metric: ℓ1-based error estimates cause significant downstream degradation compared with the other profiling metrics.
  • Core components: Sparsification improves both average accuracy and perplexity in the core-component ablation.
  • Core components: 45.4 average accuracy and 27 perplexity result when ROCKET’s closed-form activation-aware sparsification replaces CoSpaDi’s iterative sparsification.The corresponding CoSpaDi values are 42.7 average accuracy and 64 perplexity.
  • Core components: 52.4 average accuracy and 18 perplexity are reached after adding optimal knapsack-based budget allocation to ROCKET.This exceeds the uniform-compression variant in the reported ablation.

6. Conclusion and Limitations

ROCKET combines calibration-guided structured factorization with knapsack-based layer-wise budget allocation and retains over 90% of original accuracy at 30% compression without fine-tuning. Its dynamic-programming allocation is difficult to scale to very large Mixture-of-Experts models, and fixed sparsity during healing is sub-optimal.

  • ROCKET combines calibration-guided structured weight factorization with optimal layer-wise budget allocation through a knapsack formulation.
  • Over 90% of original accuracy is retained at 30% compression without fine-tuning.
  • Dynamic programming is hard to scale to Mixture-of-Experts architectures with 128 or more experts per block.The limitation arises from combinatorial growth in compression options; scalable alternatives are left for future work.
  • Fixed sparsity patterns during healing are sub-optimal, while jointly learning adaptive patterns may improve recovery.

7. Ethical Statement and Broader Impact

ROCKET is presented as a training-free compression pipeline intended to improve LLM efficiency and accessibility under resource constraints. Its formal algorithm uses calibration data, whitening, candidate factorization and sparsification options, and dynamic-programming allocation, while the paper notes that it preserves the base model’s existing biases and safety limitations.

  • ROCKET is designed to improve LLM efficiency and accessibility without additional data or extensive fine-tuning computation.
  • ROCKET does not introduce new data collection, annotation, or model behaviors, so it preserves potential biases and safety limitations of the original model.
  • The pipeline begins with calibration data and computes a whitening transform before processing each layer.
  • For each layer, candidate ranks and sparsity ratios define factorization options, with coefficient importance based on coefficient magnitude and activation sensitivity.
  • Candidate coefficients are oversparsified and then globally reactivated to reach the exact target sparsity ratio.
  • Dynamic programming filters candidates by an error threshold, records cost and error, prunes dominated states, and backtracks the selected per-layer choices.

B.1. Relative Error of Rank-k Truncated SVD

ROCKET constructs its approximation in a whitened space where the eigenvalue-derived basis is equivalent to the truncated-SVD basis, then sparsifies coefficients and maps the result back to the original space. Its graph formulation encodes layer choices and global feasibility for exact optimization up to discretization.

  • Whitened-space approximation: ROCKET whitens the weight matrix using calibration activations before applying eigenvalue decomposition to a symmetric positive semidefinite matrix.This avoids applying eigenvalue decomposition directly to the generally rectangular, non-symmetric weight matrix.
  • Whitened-space approximation: The eigenvalue-derived basis coincides with the left singular vectors of the whitened weight matrix.
  • Whitened-space approximation: The resulting reconstruction is identical to rank-k truncated SVD before coefficient sparsification.
  • Whitened-space approximation: Coefficient sparsification defines a whitened-space approximation while preserving the norm-based properties used in the reconstruction analysis.
  • Whitened-space approximation: The compressed weight is recovered in the original space by applying L^-1 to the whitened reconstruction.
  • Graph-based budget allocation: The graph’s terminal sink is reachable only when cumulative retained parameters meet the minimum target, enforcing the global compression constraint by construction.Layer choices become graph transitions, and shortest-path search solves the discretized constrained allocation problem.

D. Inference Optimization

ROCKET’s heterogeneous, sensitivity-guided allocation concentrates compression in robust MLP layers while preserving capacity for sensitive attention layers, supporting efficient sparse inference. Its reported comparisons include favorable runtime, throughput, compression, and environmental outcomes.

  • Layer assignment: MLP gate and up projections receive higher sparsity and lower rank than attention projections because they are larger and their reconstruction error grows more slowly with sparsification.The optimizer therefore preferentially compresses MLP layers.
  • Sparse inference: MACKO consistently speeds up large, moderately sparse gate and up coefficient matrices, while matching PyTorch on smaller or denser down projections.Attention layers retain PyTorch’s native implementation because it is faster in practice there.
  • Budget allocation: ROCKET dynamically assigns heterogeneous (k_l, s_l) pairs by reconstruction sensitivity, lowering FLOPs in large robust layers and retaining higher FLOPs in smaller sensitive layers.This allocation produces a more balanced per-layer runtime profile despite comparable theoretical operation counts with CoSpaDi.
  • Comparison: ROCKET and CoSpaDi have comparable theoretical operation counts under a fixed global parameter budget, but ROCKET reports a consistent throughput advantage through dynamic allocation and sparse-kernel utilization.Both represent W as BC with a dense dictionary and column-wise sparse coefficients.
  • Environmental impact: ROCKET consumes over 100 times less energy, completes compression 96 times faster, and produces 23 times lower CO2 emissions than COSPADI.These measurements use the Llama3-1B model.
  • Compression performance: Across different benchmarks and compression ratios, ROCKET outperforms CosPaDi and SVD-LLM by a large margin.The cited comparisons include Llama3.2-1B and Llama3-8B evaluation settings.

E.3. Evaluation on Advanced Benchmarks

On newer, more rigorous benchmarks, ROCKET is compared with CoSpaDi and SVD-LLM, while limited healing enables a compressed Qwen model to approach a from-scratch baseline.

  • Advanced benchmarks: ROCKET is evaluated against CoSpaDi and SVD-LLM on a new set of benchmarks targeting rigorous, diverse, and realistic LLM capabilities.The benchmark set includes IFEVal, BBH, MATH, GPQA, MuSR, and MMLU-Pro.
  • Post-compression healing: Compressing Qwen-14B to 8B with approximately 30 million fine-tuning tokens achieves performance comparable to the original Qwen3-8B trained from scratch.The sparsity pattern remained fixed during this limited healing phase.
  • Future directions: The authors expect higher-quality, carefully curated fine-tuning data to further improve results, while trainable sparsity patterns remain future work.

F.1. Ablation on calibration data

ROCKET is relatively robust to the calibration dataset choice, with only modest average-accuracy differences across four tested sources.

  • Experimental setting: The ablation evaluates calibration data on Llama3-1B at 20% compression, with the first row representing the uncompressed baseline.
  • Calibration robustness: Average accuracy differs only modestly across RefinedWeb, PTB, WikiText, and Alpaca calibration datasets, supporting robustness to calibration-data choice.Alpaca yields 53.8 average accuracy versus 52.4 for the comparison reported in the passage.
  • Calibration robustness: Instruction-tuned Alpaca data produces slightly higher average accuracy, 53.8 versus 52.4, than the comparison calibration setting.

F.2. Ablation on the Sparsification Strategy

ROCKET’s sparsification strategy combines column-wise structure with activation-and-weight-balanced importance scoring, outperforming alternative coefficient-pruning schemes under the same budget.

  • Sparsification comparison: ROCKET’s column-aware, activation-and-weight-balanced sparsification outperforms per-row, global-importance, and whitened-space-only alternatives.The comparison uses the same calibration data and global parameter budget.
  • Ablation variants: Per-row sparsification breaks column-wise structure, global importance ignores structural constraints, and whitened-space-only scoring removes the original-space fidelity term.
Loading 2602.11008v1…