Source-linked AI summary
Forward-Free LLM Depth Pruning via Weight Redundancy
Vincent-Daniel Yun, Woosang Lim
TL;DR
Depth pruning needs to reduce LLM inference cost, but activation-based selection requires calibration execution and existing forward-free criteria do not measure inter-layer similarity. WRP estimates redundancy from checkpoint weights, and across six settings, three model families, and nine tasks it outperforms forward-free magnitude pruning while remaining close to activation-based methods.
Problem
Activation-based depth pruning requires calibration forward passes, while existing forward-free methods score blocks independently without measuring inter-layer redundancy.
Method
WRP compares attention-output and MLP down-projection weights across layers and combines pairwise similarities with projection-scale information for global block selection without calibration data or model forward passes.
Results
Across six pruning settings on three model families and nine tasks, WRP substantially outperforms forward-free magnitude pruning while remaining close to activation-based methods without post-pruning recovery.
Takeaways & Limitations
Checkpoint weights alone provide useful inter-layer redundancy signals for effective forward-free depth pruning.
Abstract
from arXiv · showhide
Depth pruning reduces large language model (LLM) inference cost by removing complete Transformer blocks. Activation-based methods collect hidden states through forward passes on calibration data, while existing forward-free methods score each Transformer block separately without measuring similarity between blocks. We propose Weight-Redundancy Pruning (WRP), a forward-free depth-pruning method that estimates inter-layer redundancy from checkpoint weights to select blocks without calibration data or model forward passes. WRP compares attention output and MLP down-projection weights across layers and combines their pairwise similarities with relative projection-scale information. The resulting all-pairs similarity matrix guides layer grouping and block selection. Across multiple pruning settings, model families, and downstream tasks, WRP consistently outperforms existing forward-free magnitude pruning and approaches the performance of activation-based methods.
1. INTRODUCTION
Depth pruning offers flexible cost control, but activation-based selection can require unavailable memory and existing forward-free methods miss inter-layer redundancy. WRP instead uses checkpoint-weight similarities for global block selection and performs close to activation-based pruning across broad evaluations.
- Depth pruning removes complete Transformer blocks to adjust model size to available memory and computation while retaining dense operators.
- Activation-based pruning requires calibration forward passes and stored activations, which can exceed the GPU memory available for running the pruned model.
- Existing forward-free criteria score blocks independently by weight magnitude rather than measuring redundancy between layers.
- WRP combines attention-output and MLP down-projection weight similarities with projection-scale information to build an all-pairs matrix for global selection without calibration data or model forward passes.
- Across six pruning settings, three model families, and nine tasks, WRP substantially outperforms forward-free magnitude pruning while remaining close to activation-based methods without post-pruning recovery.
2. RELATED WORK
Prior depth-pruning methods either remove blocks using checkpoint-derived magnitudes or estimate inter-layer redundancy from activations. WRP occupies the forward-free alternative by estimating pairwise redundancy directly from checkpoint weights.
- Magnitude-based structured pruning can select complete Transformer blocks directly from checkpoint weights without calibration forward passes.
- Activation-based methods compare block states, boundary activations, or global pairwise activation similarities, but require model execution and calibration data.
- WRP estimates pairwise inter-layer redundancy directly from checkpoint weights instead of using activation collection.
3. FORWARD-FREE WEIGHT REDUNDANCY
WRP estimates inter-layer redundancy from checkpoint weights by combining projection similarity and relative scale information, then clusters layers and allocates removals without model execution. Its weight-space similarity matrix supports global block selection and recovery-free pruning.
- Weight-Space Block Descriptors: WRP compares attention-output and MLP down-projection weights across layers because they share a hidden-state output space.Other projections map into layer-specific internal spaces, so WRP uses their scale information instead.
- Weight-Space Block Descriptors: The method uses linear CKA on output-space Gram matrices to compare cross-layer projection similarity.The Gram construction is invariant to permutations of intermediate dimensions, while CKA normalizes scale differences.
- Weight-Space Block Descriptors: WRP represents each block with Frobenius norms of seven projections and centers them to capture relative attention and MLP scales.The resulting similarity matrix combines projection similarity with relative scale information.
- Weight-Only Clustering and Allocation: WRP selects between K=2 and K=4 clusters from spectral evidence in the all-pairs similarity matrix without calibration data or forward passes.Depth-distance-adjusted reference gaps determine whether coarse or finer structure is used.
- Weight-Only Clustering and Allocation: The allocation procedure protects the first and last blocks, removes redundant eligible blocks across clusters, and stops at the requested pruning budget.Selected blocks are removed without modifying the remaining parameters.
4. EXPERIMENTAL RESULTS
Across six pruning settings, three model families, and nine tasks, WRP substantially outperforms forward-free Mag+ while remaining close to activation-based methods. At 25% pruning, it also reduces latency and peak GPU memory without custom kernels or runtime changes.
- Main Results: WRP outperforms Mag+ across all six settings by 10.68–17.15 points, averaging a 14.38-point gain.WRP averages 57.41 compared with 57.93 for LoRP and achieves the best pruned average on Qwen3-14B at 8/40.
- Inference Cost: At 25% pruning, WRP reduces prefill latency by 21–23%, decode latency by 23–25%, and peak GPU memory by 21–22%.These measurements use one NVIDIA RTX 6000 Ada Generation GPU with FP16 and batch size 1.
5. CONCLUSION
WRP enables forward-free depth pruning by estimating inter-layer redundancy directly from checkpoint weights. Across six pruning settings, it consistently outperforms magnitude-based pruning while remaining close to activation-based methods.
- WRP estimates inter-layer redundancy directly from checkpoint weights to enable forward-free depth pruning.
- Across six pruning settings, WRP consistently outperforms magnitude-based pruning while remaining close to activation-based methods.
- Checkpoint weights alone provide useful signals for effective depth pruning.