PaperlayerView exact revision on arXiv

Source-linked AI summary

COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models

Peiqi Yu, Nam Ling, Wei Wang, Wei Jiang

arXiv:2608.21142v1cs.LG

TL;DR

Structured pruning lowers LLM cost by removing columns, but existing training-free corrections cannot adapt the retained weight’s input singular frame. COEC compensates pruning with two-sided rotations, singular-value rescaling, tempered calibration statistics, and inter-layer alignment, improving perplexity on every evaluated model and zero-shot accuracy in most settings. The method requires only small-set second-order statistics, not backpropagation or retraining.

  • Problem

    Existing training-free compensation methods use additive bias or one-sided output rotations, leaving the retained weight’s input singular frame unchanged after column removal.

  • Method

    COEC alternates left and reduced-Stiefel right orthogonal rotations, applies GCV-selected singular-value rescaling, tempers calibration statistics, and penalizes inter-layer misalignment.

  • Results

    COEC improves perplexity on every Llama-3, Llama-3.1, and Qwen2.5 model evaluated and zero-shot accuracy in most settings, with larger gains at higher pruning ratios.

  • Takeaways & Limitations

    COEC is a criterion-agnostic, training-free plug-in that recovers part of the performance lost to structured column pruning without changing model architecture.

  • Takeaways & Limitations

    Least-squares refitting can overfit the small calibration set by reshaping the retained weight’s singular spectrum and generalizing unstably across models.

Abstract

from arXiv · show

Structured pruning reduces the size and inference cost of large language models (LLMs) by removing weight columns, but the resulting output error can degrade accuracy. Existing training-free compensation methods use an additive bias or a single orthogonal rotation on the output side of the retained weight. These corrections leave its input singular frame unchanged and therefore limit how the retained weight can adapt after column removal. We propose COEC (Calibrated Orthogonal-Equivalence Compensation), a training-free compensation framework that applies alternating left and right orthogonal rotations to the retained weight. The right rotation is optimized on a reduced Stiefel manifold, while singular values are rescaled using generalized cross-validation to select the regularization strength for each layer. COEC further tempers the calibration Gram matrix to reduce the dominance of high-energy activation directions and introduces an alignment penalty that preserves the geometric relation between adjacent attention projections.All components use second-order statistics from a small calibration set and require neither backpropagation through the LLM nor retraining of the model parameters. COEC is independent of the column pruning criterion and can be applied to multiple structured pruning methods. Experiments on the Llama-3, Llama-3.1, and Qwen2.5 model families across multiple structured sparsity levels show that COEC improves perplexity on every model and zero-shot accuracy in most settings over existing compensation methods, with larger gains at higher sparsity. These results show that post-pruning compensation can recover part of the performance lost to column removal.

1 Introduction

COEC addresses the limitations of one-sided post-pruning compensation by jointly adapting retained weights’ input and output singular frames. It provides a criterion-agnostic, training-free plug-in that improves perplexity across evaluated models and zero-shot accuracy in most settings.

  • Motivation: Structured pruning removes weight-matrix columns to reduce parameter count, memory usage, and computation, but compensation is needed to reduce pruning-induced output error.Training-free pruning methods are attractive because retraining at LLM scale is expensive.
  • Limitations of Existing Methods: Existing compensation methods leave important adaptation constrained: RCPU rotates only the output frame, while pruning substantially changes the input singular subspace.A left rotation changes the output frame but leaves the input frame unchanged, and reconstruction-based compensation generally does not restore the pruned input subspace.
  • COEC: COEC applies alternating left and right orthogonal rotations with per-mode singular-value rescaling, allowing retained weights to adapt both singular frames while preserving a structured parameterization.The right rotation is optimized on a reduced Stiefel manifold, and scaling is recomputed during the alternating procedure.
  • COEC: COEC uses calibration statistics only, requires neither LLM backpropagation nor retraining, and can follow any column-selection rule without changing model architecture.The method is demonstrated with Wanda-sp, FLAP, and RCPU selections.
  • Results: COEC improves perplexity on every evaluated model and zero-shot accuracy in most settings over existing compensation methods, with larger gains at higher pruning ratios.Evaluation covers Llama-3, Llama-3.1, and Qwen2.5 across multiple structured sparsity levels.

2 Related Work

COEC differs from prior rotation and calibration approaches by operating after column pruning and combining two-sided rotations with criterion-independent, GCV-selected rescaling.

  • Structured Pruning: Structured pruning removes entire columns, rows, or blocks and directly shrinks dense models, unlike unstructured methods that require sparse-kernel support for speedups.Related methods include per-row least-squares reconstruction and activation-aware scoring.
  • Rotation-based Transformations: Prior lossless-rotation methods rotate before compression under functional equivalence, whereas COEC rotates after pruning to compensate removal error.COEC extends one-sided rotation to a converged two-sided update with per-mode rescaling independent of the selection criterion.
  • Calibration-aware Rescaling: COEC uses diagonal singular-frame rescaling with per-layer generalized cross-validation rather than the scale or bias corrections used in earlier calibration-aware methods.The paper characterizes the right rotation as a weighted orthogonal Procrustes problem solved on a reduced Stiefel manifold.

3 Problem Formulation

The formulation represents pruning compensation as a structured reconstruction problem over retained channels. COEC uses calibration second moments while constraining updates to rotations and regularized singular-mode rescaling instead of unrestricted least-squares refitting.

  • 3.1 Notation and Setup: Pruning removes structured channels in MLP and grouped-query attention sub-layers while preserving the grouped-query attention structure.MLP pruning deletes corresponding projection rows and output columns; attention pruning removes complete KV groups and associated query heads.
  • 3.1 Notation and Setup: Given pretrained weight W0, calibration activations X, outputs Y = W0X, and retained indices K, compensation constructs a reduced weight acting on XK and evaluates relative reconstruction error.The retained activations XK contain the rows of X indexed by K.
  • 3.1 Notation and Setup: The objective depends on second-moment statistics G, Gxx, and Gyx accumulated in one calibration pass, so token-level activations need not be retained afterward.The passage reports that 128 calibration sequences are sufficient in practice.
  • 3.2 Limitation of Least-Squares Compensation: Regularized least-squares refitting can overfit the small calibration set by reshaping the retained weight’s singular spectrum, producing unstable generalization across models.The practical ridge solution is W_ridge = Gyx(Gxx + λlsI)^−1.
  • 3.2 Limitation of Least-Squares Compensation: COEC instead parameterizes the compensated weight with left and right orthogonal transforms plus regularized singular-mode rescaling, limiting unrestricted deformation.The rotations adjust output and input singular frames, while mode-wise rescaling controls changes to the singular spectrum.
  • 3.2 Limitation of Least-Squares Compensation: The objective includes an inter-layer alignment penalty for attention output projections, with its coefficient set to zero elsewhere.This penalty is described in Sec. 4.5 and is applied only to the attention output projection.

4 Method

COEC alternates orthogonal updates to the output and input singular frames, while rescaling singular modes and regularizing calibration statistics to improve post-pruning compensation. It additionally preserves inter-layer attention geometry through an alignment penalty.

  • Two-Sided Rotation: Identity initialization is important at mild sparsity, where the optimal right rotation typically remains close to the original input frame.Initializing from the one-sided Procrustes solution increased reconstruction error on attention output projections from 0.019 to 0.159.
  • In-Loop Scaling: COEC recomputes the global scale after right-rotation updates so the rotation and scale converge jointly.The alternating procedure updates the left rotation, scale, right rotation, and scale again, with early stopping when the objective change falls below 10^-4.
  • Two-Sided Rotation: COEC alternates a closed-form left rotation with an iterative right rotation optimized on a reduced Stiefel manifold.The reduced formulation removes rotation directions that do not affect output slices and reduces memory and per-step computation by approximately b/k.
  • GCV-Based Singular-Value Rescaling: Per-mode singular-value rescaling adapts individual singular modes instead of applying one global scale to all singular values.The mode response reflects output-input alignment under the target cross-covariance, while mode energy reflects calibration energy along the input singular direction.
  • GCV-Based Singular-Value Rescaling: Generalized cross-validation selects the regularization strength independently for each layer to limit overfitting to calibration statistics.The effective degrees of freedom penalize solutions that fit calibration statistics too closely, whereas unadjusted calibration residual is minimized at λ = 0.
  • GCV-Based Singular-Value Rescaling: Layer-specific regularization matches the performance of the best globally tuned value on both validation models.The regularization is selected per layer rather than using one shared global value.
  • Spectral Tempering: Spectral tempering compresses the calibration Gram-matrix spectrum, reducing high-energy directions’ dominance and relatively increasing lower-energy directions’ weight.When α = 1, the original activation statistics are preserved; the evaluated best values lie between 0.3 and 0.9, with α = 0.9 used by default.
  • Anchored Inter-Layer Alignment: An alignment penalty regularizes the attention output projection toward the original geometric relation between adjacent value and output projections.With λa = 50, the alignment penalty after compensation is a median 7.2× lower than for the uncompensated pruned weight.

5 Experiments

COEC is evaluated across model families, pruning criteria, sparsity levels, calibration sizes, ablations, geometric measures, and deployment costs. It consistently improves perplexity, usually improves zero-shot accuracy, remains stable with limited calibration data, and adds no inference-shape cost.

  • 5.2 Main Results: COEC reduces perplexity on every evaluated model under Wanda-sp, FLAP, and RCPU selections, while improving zero-shot accuracy in most comparisons.At 30% sparsity, it achieves the largest Wanda-sp perplexity gain on Qwen2.5-32B, from 13.90 to 9.64; under RCPU, Qwen2.5-14B is the sole accuracy exception.
  • 5.2 Main Results: COEC gives larger gains at higher sparsity, where pruning leaves more error for compensation to recover.The main tables report 30% sparsity, with 10% and 20% results following the same protocol in the supplement.
  • 5.3 Effect of Calibration Size: COEC achieves the lowest perplexity at every tested calibration size and sparsity ratio, improving monotonically as calibration size increases.At 30% sparsity, reducing calibration size from 256 to 32 costs about 0.6 PPL, while N=128 lies on the flat part of every COEC curve.
  • 5.4 Ablation Study: At 30% sparsity, cumulative ablations reduce perplexity from 10.06 with RCPU compensation to 9.65 for full COEC and increase accuracy from 50.9 to 52.9.GCV rescaling and two-sided rotation drive the perplexity reduction; the reduced-Stiefel solve contributes most of the zero-shot gain, while Gram tempering recovers much of its perplexity cost.
  • 5.5 Efficiency Analysis: At 30% column sparsity, structured pruning reduces parameters, peak inference memory, and dense FLOPs per token by 23–29%, independently of compensation.COEC changes retained weights in place without changing model shape; compensating a 7–8B model takes 45–86 GPU-minutes, and alignment adds 0.5% overhead.
  • 5.6 Geometric Analysis: On DeepSeek-R1-0528-Qwen3-8B, pruning rotates input directions by about 22° on average, while the anchored penalty reduces mismatch by a median factor of 7.2.COEC preserves input-output direction pairing within each layer with 1.1–1.2% discrepancy without an explicit pairing term.

6 Conclusion

COEC provides training-free compensation for structured pruning by rotating and rescaling retained weights while preserving model structure. It uses calibration statistics without retraining or backpropagation and can recover performance lost through column removal.

  • 6 Conclusion: COEC rotates retained weights on both sides and rescales singular values to reroute signal lost through removed columns.Its rescaling strength is selected from calibration statistics, and an alignment penalty maintains geometric consistency with a neighboring layer.
  • 6 Conclusion: COEC requires no LLM backpropagation or parameter retraining and compensates a 7–8B model in under 1.5 GPU-hours without changing model architecture.

A Results at 10% and 20% Column Sparsity

At 10% and 20% sparsity, COEC generally improves perplexity across model families, with gains over RCPU increasing at higher sparsity; FLAP remains competitive at lower ratios.

  • COEC lowers perplexity on all six models under Wanda-sp selection at both 10% and 20% sparsity.
  • Under RCPU selection, COEC improves perplexity on five of six models at 10% and all six at 20%.The mean gap over RCPU is 0.06 at 10% and 0.20 at 20%.
  • At 10% and 20% sparsity, FLAP leads COEC on perplexity for five of six models, while zero-shot accuracy is evenly split.

B Implementation Details

COEC’s implementation uses low-precision manifold retractions, a logarithmic GCV search, fixed hyperparameters, and numerical safeguards for degenerate layers.

  • The GCV ridge λ is minimized over a 120-point logarithmic grid spanning [10^-4, 10^5].
  • COEC uses Newton–Schulz polar retraction for rectangular Stiefel frames and Cayley retraction for square frames.
  • Numerical safeguards skip invalid scale updates, cap retraction steps, and provide an SVD fallback chain for degenerate slices.
  • Reported experiments use one configuration with α = 0.9, rp = 16, δ = 7·10^-4, at most 50 alternation rounds, and κ = 10.

C Geometric Measurements

Measurements show substantial input-frame rotation after pruning, while COEC keeps the paired frame changes closely matched; its alignment penalty reduces projector mismatch without removing within-subspace rotation.

  • Pruning rotates the retained input subspace by average angles of 22.2° for attention output projections and 22.6° for MLP down projections.Per-module spans are 8–48° and 2–32°, respectively.
  • At λa = 50, the attention-projector alignment penalty falls by a median factor of 7.2, up to 975.
  • The O–C angles remain 22–24°, indicating that the penalty does not remove rotation within the retained subspace.
  • The rank cut is near-degenerate, so rp should be placed at a spectral gap where one exists.

D Additional Model Families: Llama-2 and Qwen3

On Llama-2 and Qwen3, COEC improves perplexity broadly across sparsity levels and compensation baselines, while its accuracy gains are strongest for selected settings and Qwen3 models.

  • Under FLAP, COEC lowers perplexity on every Llama-2 and Qwen3 model at 10%, 20%, and 30% sparsity.
  • Under Wanda-sp, COEC lowers perplexity on all six models at 10% and 20% sparsity and five of six at 30%.
  • Under RCPU, COEC leads on five of six models at 10%, four at 20%, and three at 30%, with gains concentrated on Qwen3.
  • The largest reported recovery is on Qwen3-32B at 30% sparsity, improving perplexity from 11.86 to 8.94 and zero-shot accuracy by 6.6 points.
Loading 2608.21142v1…