PaperlayerView exact revision on arXiv

Source-linked AI summary

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

Sankar Behera, Dhruv Singh, Anshika Agnihotri, Raj Kumar Choudhary, Satyadev Ahlawat, Yamuna Prasad

arXiv:2609.11163v1cs.LGcs.CL

TL;DR

LLM deployment is constrained by substantial computational and memory requirements, while calibration data may be unavailable. LILA addresses this with weight-only spectral neuron scoring and reports strong pruning performance without calibration data, policy training, or architectural modifications.

  • Problem

    LLM deployment is constrained by substantial computational and memory requirements, and the calibration-free setting assumes the target data distribution is unavailable.

  • Method

    LILA scores FFN neuron importance using the KS distance between the singular-value spectra of full and neuron-ablated weight matrices, using intrinsic spectral geometry without model forward passes.

  • Results

    LILA-Spectrum outperforms WikiText-2-calibrated SliceGPT by up to 6.0 pp and PruneNet by 1.57 pp without calibration data, policy training, or architectural modifications; post-recovery, it matches SOTA within 0.48 pp.

  • Takeaways & Limitations

    Weight spectral geometry provides a practical data-free importance signal, while dynamic KS-based sparsity allocation preserves generation at moderate compression and exposes architectural bottlenecks at high compression.

  • Takeaways & Limitations

    The calibration-free formulation assumes the target data distribution is unavailable and therefore approximates the training objective using weight-only importance scores; the Act variant requires calibration data.

Abstract

from arXiv · show

Structured pruning of large language models (LLMs) offers hardware-efficient compression, yet existing methods require calibration data, gradient computation, or large auxiliary policy networks at pruning time. LILA (\emph{Latent-Informed Layer Analysis}) scores neuron importance via the Kolmogorov--Smirnov (KS) distance between empirical singular value distributions of the full and neuron-ablated feed-forward network (FFN) weight matrix, providing a closed-form spectral rule requiring no training, calibration data, or auxiliary network. Without any fine-tuning, LILA surpasses PruneNet (45M-parameter RL policy) by 1.57~pp in zero-shot accuracy on LLaMA-2-7B at 25\% sparsity, and outperforms WikiText-2-calibrated SliceGPT by up to 6.0~pp across all sparsity levels, while preserving the original architecture. After one epoch of LoRA recovery fine-tuning, LILA achieves highly competitive performance, matching the heavily calibrated SliceGPT baseline to within a 0.48~pp margin across LLaMA-2-7B and Phi-2, despite using zero calibration data. A Neural Tangent Kernel analysis confirms a 22$\times$ reduction in functional distortion versus random pruning, providing theoretical grounding for the spectral importance criterion. Finally, extending LILA to dynamically allocate sparsity budgets via KS-scores yields state-of-the-art generative preservation at moderate compression, while uncovering fundamental single-layer architectural bottlenecks at higher compression regimes.

1 Introduction

LILA introduces calibration-free structured pruning that scores FFN neurons from intrinsic spectral geometry, avoiding calibration data, training, and architecture changes. Across zero-shot and recovery settings, it outperforms or closely matches calibrated and policy-based baselines while retaining hardware-efficient structure.

  • Motivation: Structured pruning removes entire neurons or other model components, producing dense tensors that can deliver hardware efficiency without specialized sparse software.This motivates LILA relative to irregular unstructured sparsity.
  • Research gap: Existing structured pruning methods depend on calibration sequences, activation statistics, gradients, or a large reinforcement-learning policy, creating deployment obstacles under inaccessible or mismatched data.Calibration distributions can be unavailable in privacy-sensitive settings or mismatched to the target task.
  • Proposed framework: LILA derives per-neuron FFN importance from NMF-revealed spectral geometry without model forward passes or calibration data.It treats a neuron as important when ablation substantially perturbs the singular-value spectrum.
  • Framework design: The framework unifies three matrix variants and three scoring criteria, spanning an approximately 18× pruning-speed range while data-free variants match or exceed calibrated Variant Act.Reported LLaMA-2-7B pruning times range from 2.4 to 44 minutes.
  • Method and headline result: 1.57 pp higher zero-shot accuracy than the 45M-parameter PruneNet policy at 25% sparsity on LLaMA-2-7B, without policy training.The KS-distance spectrum score provides the data-free importance criterion underlying this comparison.
  • Empirical validation: Up to 6.0 pp higher zero-shot accuracy than WikiText-2-calibrated SliceGPT, while preserving the original architecture and using no calibration data.LILA-Spectrum also remains within 0.71 pp of Alpaca-calibrated SliceGPT without recovery fine-tuning.

2 Related Work

Prior pruning methods trade off hardware efficiency, calibration requirements, architectural changes, and scoring complexity. LILA instead uses data-free spectral geometry to rank structured neuron masks without modifying the model graph.

  • Unstructured pruning: Unstructured methods such as SparseGPT and Wanda can achieve high sparsity but require calibration data and produce irregular tensors lacking ordinary hardware speedups.Their speed advantages depend on specialized sparse-tensor support.
  • Structured pruning: Structured methods produce dense tensors compatible with standard BLAS, but representative approaches use gradient dependency analysis, whole-layer removal, PCA rotations, or learned policy networks.SliceGPT permanently restructures the model graph, while PruneNet learns a 45M-parameter policy.
  • NMF-based scoring: LILA uses NMF as an importance-scoring instrument for discrete neuron masking rather than as direct low-rank weight replacement.This extends NMF-based analysis from prior convolutional-network applications to large-scale transformer FFN pruning.
  • Theoretical context: The paper extends NTK-based pruning analysis to transformer FFNs through an NTK trace ratio, while post-training quantization remains orthogonal and composable with spectral pruning.The NTK connection supplies a function-space perspective distinct from the pruning rule itself.
  • LILA positioning: LILA-Spectrum derives importance from higher-order singular-value geometry, unlike Wanda’s element-wise magnitude criterion, while preserving the model graph unlike SliceGPT.It also avoids forward passes and calibration data required by SparseGPT.

3 Method

LILA formulates structured FFN pruning as a calibration-free neuron-selection problem using NMF-based spectral geometry. It scores neurons by how their ablation changes singular-value distributions, retains the most important neurons, and preserves the dense transformer architecture.

  • Structured Pruning: The pruned model reduces each FFN hidden dimension to ⌊(1 −s)dff⌋ while preserving the original computation graph and dense architecture.LoRA fine-tuning can optionally be applied for recovery after pruning.
  • Calibration-Free Scoring: The calibration-free objective replaces unavailable data-distribution loss optimization with importance scores computed directly from weight matrices.The scoring pipeline uses randomized-SVD initialization and rank-1 downdates to reduce per-neuron spectral computation to O(r^2).
  • NMF Matrix Construction: LILA constructs a non-negative FFN weight matrix, factorizes it with rank-r NMF, and derives neuron importance from the resulting latent representation.Abs, Split, and Act variants encode different weight-geometry assumptions; Abs and Split are calibration-free, whereas Act uses calibration activations.
  • Neuron Importance Scoring: The KS spectrum score measures the distance between singular-value ECDFs of the full and neuron-ablated matrices, identifying neurons whose removal changes spectral energy.Large spectral shifts indicate spectrally indispensable neurons, while invariant spectra indicate redundancy.
  • Pruning Mask Construction: LILA ranks neurons by importance and retains the top-(1 −s) fraction in each FFN layer, removing corresponding weight rows and columns.The uniform mask preserves a controlled global sparsity ratio, while KS scores can also support adaptive layer-wise budgets.

4 Experiments

Across LLaMA-2-7B, Phi-2, and OPT-1.3B, LILA is evaluated at 20–40% sparsity against calibration-dependent and architecture-modifying baselines. LILA-Spectrum leads zero-shot comparisons, post-recovery performance approaches SliceGPT, and adaptive sparsity improves moderate-compression perplexity but exposes high-compression bottlenecks.

  • Zero-shot accuracy: Up to 6.00 pp: LILA-Spectrum surpasses WikiText-2-calibrated SliceGPT without fine-tuning, while preserving the original architecture.It also exceeds PruneNet at every reported LLaMA-2-7B sparsity level.
  • Recovery fine-tuning: Within 0.48 pp: post-RFT LILA matches SliceGPT across the reported LLaMA-2-7B and Phi-2 settings.The best configurations include 62.56% versus 63.04% on LLaMA-2-7B at 25% sparsity and 64.67% versus 65.24% on Phi-2.
  • Calibration robustness: Adding WikiText-2 calibration to the Spectrum criterion provides no consistent benefit over data-free variants, with all reported gaps at or below 0.9 pp.The direction of any advantage reverses across models and between noRFT and RFT settings.
  • Computational efficiency: 2.4–44 minutes: LILA configurations span a broad pruning-time range, with faster Sensitivity offering over 98% of Spectrum’s relative performance after recovery.The cited LLaMA-2-7B comparison is 61.42% versus 62.56% at 25% sparsity.
  • Adaptive sparsity: At 25% compression, adaptive KS-budgeting reduces LLaMA-2 perplexity from 11.14 to 9.50, whereas at 30% LLaMA-2 encounters catastrophic single-layer bottlenecks.The reported result supports uniform sparsity as the safer default at higher compression.

5 Conclusion

LILA uses singular-value KS divergence to perform calibration-free structured pruning while retaining the model architecture. It outperforms key no-fine-tuning baselines, approaches SliceGPT after recovery, reduces functional distortion, and reveals compression-dependent architectural bottlenecks.

  • Core contribution: LILA scores neuron importance with singular-value KS divergence, using spectral geometry rather than calibration data or auxiliary policy training.The evaluation includes Spectrum, Sensitivity, and Residual criteria across Abs, Split, and Act variants.
  • Theoretical analysis: 22×: NTK analysis reports lower functional distortion for LILA than random pruning.This analysis provides theoretical grounding for the spectral importance criterion.
  • Scope and implications: Adaptive sparsity preserves generation better at moderate compression but exposes architectural bottlenecks at higher compression.The paper also reports evaluation using WikiText-2 perplexity and five-benchmark zero-shot accuracy, with one-epoch Alpaca LoRA recovery.
  • Practical trade-offs: LILA-Sensitivity offers a faster practical alternative because recovery fine-tuning substantially narrows its performance gap with Spectrum.On Phi-2 at 25% sparsity, Sensitivity with RFT slightly exceeds Spectrum, 64.67% versus 63.52%.

C Full Per-Benchmark Results

Across LLaMA-2-7B, Phi-2, and OPT-1.3B, LILA-Spectrum preserves strong zero-shot performance without calibration, while matching calibrated SliceGPT closely after recovery fine-tuning. Results also show that instruction-tuning data improves recovery more than unstructured text.

  • Full Per-Benchmark Results: The study reports complete LILA evaluations for LLaMA-2-7B and Phi-2, with Spectrum and Wanda fully swept across OPT-1.3B sparsity levels.LLaMA-2-7B and Phi-2 include full per-benchmark tables, while OPT-1.3B coverage is narrower.
  • Full Per-Benchmark Results: LILA-Spectrum beats WikiText-2-calibrated SliceGPT by 5.2–6.0 pp across zero-fine-tuning sparsity settings.The comparison is evaluated on WikiText-2 while LILA uses no calibration data and preserves the architecture.
  • Full Per-Benchmark Results: LILA-Spectrum closes the gap to calibrated SliceGPT to ≤0.48 pp after recovery fine-tuning on LLaMA-2-7B and Phi-2.The comparison uses zero calibration data and preserves the original architecture.
  • Full Per-Benchmark Results: Alpaca recovery improves LILA accuracy by +0.81 to +2.47 pp over identical LoRA recovery on WikiText-2.Both recovery settings use the same one-epoch LoRA protocol.

E Extended Comparison with PruneNet

Against PruneNet, LILA achieves stronger or comparable zero-shot performance while replacing a trained reinforcement-learning policy with a closed-form spectral criterion. The advantage is clearest on LLaMA-2-7B and narrows on Phi-2.

  • Extended Comparison with PruneNet: LILA-Spectrum surpasses PruneNet on LLaMA-2-7B by +1.75 pp at 25% and +2.05 pp at 30% sparsity without fine-tuning.PruneNet uses a 45M-parameter policy and a 15-minute RL training phase, whereas LILA uses neither policy training nor calibration data.
  • Extended Comparison with PruneNet: With matched WikiText-2 recovery, LILA-Spectrum Split beats PruneNet on LLaMA-2-7B by +0.35, +0.71, and +2.11 pp at 20%, 25%, and 30% sparsity.On Phi-2, the methods are near-identical at 20% and 30%, while PruneNet leads by 1.43 pp at 25%.
  • Extended Comparison with PruneNet: LILA matches or exceeds an RL-policy pruning method using zero policy training, zero calibration data, and no architectural modification.This comparison summarizes the reported calibration-free trade-off against PruneNet.

F Ablation: NMF Variant × Scoring Method

The ablation identifies KS-distance Spectrum scoring as the dominant design choice: it remains effective across NMF variants, whereas Sensitivity and Residual scoring can collapse without calibration. Data-free variants are generally competitive with calibration-guided scoring, but activation scaling can fail at aggressive sparsity.

  • Ablation: NMF Variant × Scoring Method: Spectrum scoring achieves 60.01–60.38% accuracy and 11.04–11.06 perplexity across all NMF variants at 25% sparsity.Sensitivity and Residual scoring collapse to 37% and 36% for data-free Abs/Split configurations.
  • Ablation: NMF Variant × Scoring Method: The gap among Spectrum variants Abs, Split, and Act is ≤0.20 pp, indicating scoring matters more than the NMF construction choice.Split is marginally best at 25% sparsity, with 11.10 perplexity.
  • Ablation: NMF Variant × Scoring Method: Data-free Abs and Split generally match or exceed calibration-guided Act, with calibration gaps usually below 1 pp.On LLaMA-2-7B, data-free variants meet or exceed Act across sparsity levels; Phi-2 shows only a small no-fine-tuning edge for Act.
  • Ablation: NMF Variant × Scoring Method: At LLaMA-2-7B 30% sparsity without fine-tuning, Act falls to 45.97%, nearly 12 pp below Abs at 57.50%.The reported failure is associated with activation-weighted scaling destabilizing the NMF basis under aggressive sparsity.

G NTK Preservation Analysis

The NTK analysis links pruning criteria to preservation of the model’s linearized learning dynamics. Sensitivity has the lowest measured distortion, while Spectrum and Wanda could not be evaluated in this configuration because of memory limits.

  • NTK Preservation Analysis: NMF-Sensitivity records an NTK trace ratio of 2.10, compared with 47.38 for random pruning.Lower trace ratio indicates less functional distortion relative to the dense model.
  • NTK Preservation Analysis: All evaluated LILA methods substantially reduce NTK distortion relative to random pruning, while preserving kernel direction with cosine similarity 1.000.The analysis is reported for Phi-2 at 25% sparsity.
  • NTK Preservation Analysis: Spectrum and Wanda are excluded from the NTK comparison because their gradient evaluation exceeded GPU memory limits.The Spectrum calculation requires storing empirical spectral CDFs per neuron during intensive Jacobian accumulation.

I Layer-wise Spectral Analysis

Layer-wise KS analysis identifies early and final layers as especially sensitive to neuron removal, while top-scored-neuron ablations sharply reduce accuracy. The NTK analysis links retaining spectrally dominant neurons to lower functional distortion.

  • Layer-wise spectral analysis: Early (0–5) and final (28–31) layers have higher KS-distance scores, and removing their highest-scored neurons sharply degrades accuracy.These results indicate non-uniform layer sensitivity and support the KS score as a marker of functionally critical neurons.
  • NTK distortion: Retaining neurons that span each layer’s dominant spectral subspace minimizes NTK distortion under the stated spectral-pruning proposition.The proposition characterizes low spectral mass in pruned neurons as the condition for minimizing distortion.
  • NTK distortion: The KS score κj measures how much neuron j shifts the empirical singular-value distribution and prioritizes neurons with greater spectral mass.The analysis connects this score to lower Jacobian perturbation and reduced NTK Frobenius distortion.

K Scaling Validation: LLaMA-2-13B and OPT-2.7B

Scaling evaluations show that LILA’s performance pattern extends from LLaMA-2-7B to LLaMA-2-13B and remains competitive on OPT-2.7B. Recovery fine-tuning generally strengthens results, although OPT lacks a direct SliceGPT fine-tuning comparison.

  • LLaMA-2-13B: LLaMA-2-13B LILA configurations exceed WikiText-2-calibrated SliceGPT by +1.31 to +1.97 pp without fine-tuning.The best configurations are Abs Spectrum at 20%, Act Spectrum at 25%, and Act Sensitivity at 30%.
  • LLaMA-2-13B: After one LoRA recovery epoch, LILA-Residual Act reaches 67.63% / 66.07% at 20%/25% sparsity, surpassing SliceGPT+RFT by up to +5.50 pp.Against the best SliceGPT condition, LILA’s gap is at most −2.11 pp while using no calibration data and preserving the original architecture.
  • OPT-2.7B: On OPT-2.7B, post-recovery LILA-Spectrum Act achieves 53.18% / 52.43% / 52.19% at 20%/25%/30% sparsity, meeting or exceeding SliceGPT’s noRFT accuracy.The comparison values for SliceGPT are 53.68% / 52.86% / 51.13%, respectively.
  • OPT-2.7B: SliceGPT reports no recovery-fine-tuning results for OPT models, preventing a direct +RFT comparison.LILA nevertheless exceeds SliceGPT’s best reported noRFT figure at 25% and 30% sparsity.

L Benchmarking Throughput Experiment

LILA’s structured neuron removal produces dense smaller transformers that accelerate generation on standard hardware. Adaptive KS budgeting improves generative preservation at 25% sparsity but encounters a severe single-layer bottleneck at 30% compression.

  • Throughput: Structured pruning accelerates inference by removing entire FFN neurons and permanently reducing dff, leaving a dense transformer compatible with standard kernels.This avoids the specialized sparse hardware or custom kernels required for practical unstructured-pruning speedups.
  • Throughput: At 25% sparsity, LLaMA-2-7B generation rises from 25.60 to 33.96 tok/s, a 1.33× speedup, while LLaMA-2-13B reaches 1.64×.Throughput is measured on a single NVIDIA A100 using prompt lengths from 32 to 512 tokens at batch size 1.
  • Adaptive sparsity: At 25% global sparsity, adaptive KS budgeting lowers perplexity from 11.14 to 9.50 on LLaMA-2-7B and from 54.07 to 40.03 on Phi-2.The method assigns more surviving neurons to high-KS layers while preserving the overall sparsity target.
  • Adaptive sparsity: At 30% LLaMA-2 sparsity, adaptive budgeting causes perplexity to jump to 281.47 when robust layers reach the mmin = 0.65 survival bound.The result indicates a critical single-layer bottleneck near ∼33% sparsity for the LLaMA-2 backbone.
  • Adaptive sparsity: Uniform sparsity remains the safer default for brittle architectures like LLaMA-2 at higher compression, whereas adaptive KS budgeting is effective at moderate compression.The conclusion distinguishes generative preservation at 25% from the architectural limitation observed at 30%.
Loading 2609.11163v1…