LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning

1 Introduction

• Unstructured pruning can be hard to accelerate on general hardware and is incompatible with directly merging dense LoRA weights; pruning and fine-tuning separately can also yield sub-optimal results.Computing the full BA product and mask for unstructured pruning can require intermediate storage equivalent to three NVIDIA A100 (80G) GPUs when pruning LLaMA-65b.Source: §1 · pp. 1–2
• LoRAPrune jointly applies structured pruning and LoRA, pruning heads or channels and corresponding LoRA weights without first computing BA, so the weights can be merged for inference.Its criterion uses LoRA gradients to approximate pre-trained-weight gradients, which are not computed when those weights are frozen.Source: §1 · p. 2
• LoRAPrune updates importance with a moving average and periodically removes unimportant structures while fine-tuning, using only a small calibration set.This iterative process continues until the desired sparsity is reached.Source: §1 · p. 2
• The LoRA-guided criterion uses LoRA weights and gradients to estimate importance, avoiding the need to compute gradients of pre-trained weights.The pruning process aggregates importance scores within groups and removes low-scoring groups.Source: §1 · p. 3, Figure 2 · p. 3
• At up to 65 billion weights, LoRAPrune prunes on one GPU and, versus LLM-Pruner, uses 52.6% of the memory while lowering perplexity by 4.81 on WikiText2 and 3.46 on PTB.It also matches semi-structural pruning performance across various LLMs.Source: §1 · p. 3

Parameter-efficient fine-tuning

• LoRA inserts trainable low-rank matrices that can be merged into pre-trained weights to avoid extra inference computation, while other LoRA-based methods address long contexts, personalization, or quantized fine-tuning.The passage describes LoRAPrune as compatible with extension to other LoRA-based PEFT methods.Source: §2 · p. 3

Neural network pruning

• Unstructured pruning can compress models but leaves structures that are hard to deploy, whereas structured pruning removes grouped parameters such as heads or channels for deploy-friendly models.The passage notes that structured pruning may require extensive fine-tuning to recover performance; one cited approach used 50B sampled tokens for continued pretraining.Source: §2 · p. 4

Pruning criterion

• Pruning criteria include parameter magnitude and Taylor-based error approximations, but first-order approaches such as LLM-Pruner require gradients of pre-trained weights.PST avoids computing those gradients with an extra low-rank matrix, but its unstructured pruning requires a large mask in each forward pass.Source: §2 · p. 4
• Unlike LLM-Pruner and PST, LoRAPrune relies only on LoRA gradients and avoids expensive mask computation.This is presented as making the criterion memory-efficient.Source: §2 · p. 4

3.1 Preliminary

• LoRA keeps pretrained weights frozen and adds a low-rank update, while Taylor-based pruning estimates importance from the loss impact of removing a weight.For structured pruning, connected weights are grouped to account for dependencies among neurons.Source: §3.1 · p. 4

3.2 Pruning with Low-rank Adaption

• LoRA gradients and weights approximate pretrained-weight importance without computing gradients of the pretrained weights.The criterion uses gradients of the low-rank matrices A and B; computing gradients for BA would retain the pretrained-weight-sized complexity.Source: §3.2 · pp. 5–6
• LoRAPrune’s pruning criterion is integrated into an iterative structured-pruning process that progressively removes groups.The method computes group importance from the LoRA-guided criterion and applies a binary mask to pruned layers.Source: §3.2 · pp. 6–7
• A moving average combines historical and current group-importance statistics before groups below a threshold are pruned.The process incrementally removes less critical groups using a binary mask for each pruned layer.Source: §3.2 · p. 6

4.1 Experimental Setup

• Evaluation covers LLaMA-7B, 13B, 30B, and 65B on perplexity, zero-shot reasoning, and 5-shot MMLU tasks.Iterative pruning uses LaMini for LLaMA-7B and 20k sampled C4 data for the larger models; pre-trained weights may be quantized to 8 bits.Source: §4.1 · p. 7

4.2 Main Results

• 11.60 perplexity on WikiText2 at 50% compression, versus 16.41 for LLM-Pruner.The reported comparison is under structured sparsity.Source: §4.2 · p. 8
• WANDA under structured pruning falls short of gradient-based LLM-Pruner and LoRAPrune in the reported experiments.LoRAPrune retains pruning results when weights are quantized to 8 bits, reducing pruning memory requirements.Source: §4.2 · p. 8
• LoRAPrune achieves higher 5-shot MMLU scores than other pruning methods across sparsity ratios, matching unpruned LLaMA-7B at 20% sparsity.Source: §4.2 · p. 8
• Structured pruning reduces inference time by 24.7% at 20% compression and 49.5% at 50% compression.With 20% pruning, inference without merging LoRA is slower than inference with LoRA merged and no pruning.Source: §4.2 · p. 8
• LoRAPrune outperforms magnitude-based pruning across larger LLaMA models and matches 2:4 sparsity results at 50% sparsity.The LLaMA-30B and LLaMA-65B pre-trained weights were quantized to 8 bits; 2:4 pruning requires specialized hardware and has LoRA-merging overhead.Source: §4.2 · p. 8

4.3 Ablation Study

• LoRA-guided and 8-bit LoRA-guided criteria save 52.6% and 64.2% GPU memory, respectively, while the LoRA-guided criterion increases throughput by 64.6% versus vanilla.The throughput comparison reports comparable performance.Source: §4.3 · p. 8, Table 4 · p. 8
• LoRA-guided and vanilla masks are highly consistent at 10% compression, with similarity potentially decreasing at higher compression rates.Iterative pruning needs to identify only the least important weights—about the top 5%—per iteration, supporting results on par with the vanilla criterion at lower cost.Source: §4.3 · p. 9

• LoRAPrune combines iterative structured pruning and fine-tuning with a LoRA-guided criterion that avoids computing pre-trained-weight gradients.Source: §5 · p. 9
• Experiments across LLMs and benchmarks report superiority over other pruning methods and efficiency gains over the vanilla criterion.Source: §5 · p. 9
• LoRAPrune requires fine-tuning to restore model performance, limiting its application where fine-tuning is unavailable.Source: §5 · p. 9

• Pruning a Query attention head also requires pruning the matching Key, Value, and Out weights; pruning an Up-layer channel requires matching Gate and Down channels.The method prunes attention heads and FFN channels while coordinating these dependent weights.Source: A Weight Dependency for LLaMA · p. 12

• On 20k sampled C4 data, LoRAPrune beats LLM-Pruner and WANDA on most zero-shot reasoning datasets and has the highest average score.It exceeds LLM-Pruner by 0.82% and 1.02%, respectively, as reported in the passage.Source: B More Ablation Studies · p. 12
• Increasing the moving-average parameter λ reduces perplexity, with the effect especially pronounced at λ = 0.The experiments use LLaMA-7B and 20k sampled C4 data.Source: B More Ablation Studies · p. 12
• Excessive pruning iterations can decrease zero-shot performance, potentially through overfitting on calibration data.Source: B More Ablation Studies · p. 12
• At 50% compression, the model requires more iterations to regain performance.Source: B More Ablation Studies · p. 12
• LoRAPrune is 8.19× faster than LLM-Pruner with CPU offloading and 2.75× faster without it.The passage notes that memory-access costs and computational overhead remain substantial for offloading.Source: B More Ablation Studies · p. 12
• Joint pruning and fine-tuning performs better than one-shot pruning followed by LoRA recovery fine-tuning, especially at high compression.Source: B More Ablation Studies · p. 12
• Pruning frequency 10 gives the best reported result; frequency 1 may leave too little recovery time, while frequency 20 can overfit calibration data.Source: B More Ablation Studies · p. 12

C Generative Results

• LoRAPrune’s generalization is assessed on instructions spanning common-sense, translation, and coding tasks.The results are presented in Table 9.Source: C Generative Results · p. 12
• Figure 5 depicts weight dependency in the attention and FFN layers.Source: Figure 5 · p. 13
• Table 5 reports zero-shot performance for compressed LLaMA models fine-tuned on 20k sampled C4 examples, averaging accuracy across seven classification datasets.Boldface marks the best performance at each compression rate; ⋆ marks reproduced results.Source: Table 5 · p. 13
• Figure 6 reports ablations of the moving-average λ value and fine-tuning iterations.Source: Figure 6 · p. 13
• Table 6 compares LoRAPrune and LLM-Pruner efficiency with CPU off-loading.Source: Table 6 · p. 13
• Table 7 examines joint pruning and fine-tuning, while Table 8 reports results under different pruning frequencies; both use average performance across seven classification datasets.Source: Table 7 · p. 13, Table 8 · p. 13
• Table 9 presents generated examples from the pruned models.Source: Table 9 · p. 14