PaperlayerView exact revision on arXiv

Source-linked AI summary

Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation

Peng Liu, Huibing Zeng, Yiqun Zhang, Yang Yi, Jigang Wu

arXiv:2608.24973v1cs.LGcs.AIcs.ET

TL;DR

Large Transformer models make full-gradient importance estimation and full-parameter finetuning costly. REP-LIE uses LoRA gradients, stability-aware iterative pruning, and LoRA-only recovery, achieving competitive results across encoder and generative models, including reported gains at higher sparsity.

  • Problem

    Existing pruning workflows rely on full-gradient importance estimation and full-model finetuning, creating substantial memory and computational overhead for large language models.

  • Method

    REP-LIE estimates importance from LoRA low-rank gradients, uses stability scores for iterative pruning, and recovers pruned models through LoRA-only updates.

  • Results

    REP-LIE achieves competitive performance across medium-scale encoders and large-scale generative models; at 50% sparsity, it improves average performance by approximately 5.2% over SlimGPT and about 17% over LLM-Pruner.

  • Takeaways & Limitations

    REP-LIE unifies importance estimation and performance recovery within the low-rank parameter space, shifting pruning toward simultaneous pruning-and-finetuning.

Abstract

from arXiv · show

With the rapid development of large-scale pre-trained language models based on Transformer architectures, their high computational and memory costs have become a major obstacle to deployment, especially in resource-constrained environments. Traditional pruning methods typically depend on full gradient-based importance estimation, and they necessitate prior finetuning of the model to achieve satisfactory performance. This process often results in intolerable resource consumption. This paper proposes REP-LIE, a new approach to enable resource-efficient pruning during the process of finetuning. REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation. To address the inherent randomness in importance estimation, a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters. The pruned model is further finetuned through lightweight updates, eliminating the need for full-parameter optimization in the process of finetuning. Extensive experiments on both medium-scale encoder models and large-scale generative models (LLaMA-7B and Mistral-7B) demonstrate that REP-LIE still achieves competitive performance compared to existing approaches.

I. INTRODUCTION

Transformer models offer strong task performance but create substantial deployment and training costs as their parameter counts grow. REP-LIE addresses pruning overhead by estimating importance with LoRA gradients and recovering pruned models through lightweight updates.

  • Motivation: Large Transformer language models create substantial computational and memory challenges for practical development and deployment.GPT-3 is cited as an example with 175 billion parameters.
  • Motivation: Structured pruning removes model components at hardware-friendly granularity, whereas unstructured sparsity faces irregularity and hardware constraints.Both approaches estimate importance and remove the least important weights, but structured pruning is more suitable for acceleration.
  • Existing pruning workflows: Pre-finetune and post-finetune pruning both rely on gradient-based importance estimation and full-model finetuning, imposing substantial memory and computational overhead.Post-finetune pruning first finetunes the full model and then the pruned model, while pre-finetune pruning reverses that order.
  • REP-LIE: REP-LIE estimates weight importance using only LoRA low-rank gradients and restores pruned-model performance by updating low-rank matrices instead of the full parameter set.This framework is designed to reduce pruning overhead and avoid full-parameter optimization during recovery.
  • Evaluation: REP-LIE is evaluated on medium-scale encoder models and large-scale generative models, where it achieves competitive performance across scales.The supplied introduction describes the evaluation as demonstrating scalability and robustness.

B. Pruning Criterion

The paper motivates low-rank importance estimation by contrasting expensive or incomplete pruning criteria with LoRA’s parameter-efficient adaptation. LoRA freezes pretrained weights and learns a low-rank update with substantially lower trainable complexity.

  • Importance estimation: Second-order Taylor importance estimation is expensive for large networks, while weight magnitude can overlook parameters that significantly affect performance.The supplied passage introduces these as alternative importance-estimation strategies.
  • Existing methods: Existing structured-pruning methods improve flexibility or recovery but may still require full-model finetuning, distillation, or gradient-based importance estimation.RECAP reduces memory overhead yet retains gradient-based importance estimation, which introduces additional overhead.
  • LoRA: LoRA freezes the pretrained weight matrix and adds two trainable low-rank matrices, A and B, with rank r much smaller than the layer dimensions.This reparameterizes adaptation without directly updating the original matrix.
  • LoRA: LoRA reduces trainable-parameter complexity from O(dk) to O((d + k)r), enabling parameter-efficient adaptation.The original weight is represented as W = W0 + BA, with the update initialized to preserve the initial output.
  • LoRA: Low-rank adaptation can approach full-finetuning performance; one cited example uses r = 8 with less than 0.1% additional parameters.LoRA has been applied to attention projections and extended to feed-forward layers.

III. PROPOSED METHOD

REP-LIE performs structured pruning during finetuning by estimating importance from LoRA gradients, stabilizing those estimates, and recovering performance through LoRA-only updates. Transformer attention heads and FFN hidden units serve as pruning units.

  • III. PROPOSED METHOD: REP-LIE combines low-rank-gradient importance estimation, stability-aware progressive pruning, and lightweight LoRA finetuning in one framework.The framework is designed to support compression and performance recovery without substantial computational or memory overhead.
  • Importance estimation: Importance is estimated from gradients of LoRA low-rank matrices rather than full model gradients.The criterion integrates Taylor expansion with low-rank gradients for computationally efficient parameter assessment.
  • Stabilization and pruning: A stability score mitigates randomness in importance estimation, and unimportant parameters are iteratively pruned according to the resulting scores.The method applies this stabilization mechanism during progressive structured pruning.
  • Lightweight finetuning: After each pruning round, REP-LIE restores model performance by updating only the LoRA parameters.This avoids updating the entire model parameter set during recovery.
  • Structured pruning units: For Transformer encoders, each attention head is a pruning unit in MHA, while each FFN hidden unit is treated as a pruning unit.An attention-head unit includes its query, key, value, and output projection matrices.

1) Importance Estimation via Loss Function:

REP-LIE estimates pruning importance through a first-order loss approximation while replacing full-weight gradients with gradients from LoRA low-rank matrices. This reduces importance-estimation overhead and supports compatibility across Transformer architectures equipped with LoRA.

  • First-order Taylor expansion approximates each weight’s importance using its weight-gradient product.This captures both the parameter’s magnitude and its estimated effect on loss.
  • Full pretrained-weight gradients remain costly, motivating a low-rank compensation approximation using LoRA matrices A and B.The approximation assumes the compensation term −(BA)i,j equals Wi,j.
  • Directly retaining gradients of BA does not reduce complexity because BA has the same shape as W.REP-LIE therefore expands the gradient through the chain rule.
  • The resulting pruning-unit importance uses only LoRA parameters A and B, minimizing memory and computational overhead.The formulation is illustrated in Fig. 3(a).
  • REP-LIE can be applied to Transformer models equipped with LoRA, including encoder-only, decoder-only, and encoder-decoder architectures.LoRA replaces W with W0 + BA without modifying the original architecture.

C. Stabilization Mechanism

REP-LIE introduces a stability score to address randomness in iterative importance estimation. Units with higher instability are treated as less important and become stronger pruning candidates.

  • Parameter updates and changing input-batch sequences can make importance estimates inconsistent across iterations.The randomness particularly affects low-significance pruning units.
  • The stability score measures ranking variation over repeated importance computations, with higher values indicating greater instability and lower importance.It uses each unit’s iteration-specific rank and mean rank.
  • Unimportant parameters are iteratively pruned according to their stability scores, with LoRA-only updates restoring performance after each pruning round.The recovery step avoids updating the full parameter set.

D. Adaptive Pruning under Resource Constraints

REP-LIE performs adaptive structured pruning under computational and sparsity budgets. Dynamic scheduling progressively changes sparsity, evaluates intermediate masks, and selects the highest-performing mask for the target sparsity.

  • Resource-aware pruning incorporates computational constraints such as FLOPs together with sparsity constraints.These constraints guide the model-pruning process.
  • The pruning objective minimizes task loss under a resource budget while controlling retained attention heads and feed-forward units with a binary mask.The resource cost of retained components is constrained by the total budget.
  • Dynamic sparsity scheduling controls sparsity through distinct training phases using initial and final rates over total steps.The schedule is represented as Scheduling(ti, tf, T, vi, vf).
  • After all iterations, the algorithm chooses the recorded step with the highest accuracy and returns its binary mask.The selected mask identifies the final heads and filters to prune.
  • Each iteration computes a target rate, selects the least-important heads and filters, updates their masks, and records performance and pruning plans.The procedure progressively adjusts pruning toward the target rate.

IV. EXPERIMENTAL EVALUATION

REP-LIE is evaluated across BERT-base, LLaMA-7B, and Mistral-7B using task, language-modeling, efficiency, and criterion comparisons. The reported results show superior or competitive performance, robustness under high compression, and low memory usage across these settings.

  • Experimental scope: REP-LIE is evaluated across three representative model families and six research questions covering effectiveness, efficiency, scalability, and design choices.Experiments include medium-scale encoders and large-scale generative models.
  • Medium-scale performance: On BERT-base GLUE tasks, REP-LIE is significantly superior to other compared compression methods despite performance drops from reducing model size.Comparisons include CoFi, PGB, DynaBERT, EBERT, TinyBERT, and RECAP.
  • Large-scale scalability: Large-scale evaluation tests LLaMA-7B and Mistral-7B using language modeling and zero-shot commonsense reasoning across multiple compression levels.The original dense models serve as baselines, with perplexity evaluated on WikiText-2.
  • High-compression robustness: Under high sparsity and varying FLOPs reductions, REP-LIE maintains stable GLUE performance and significantly outperforms CoFi and EBERT.This evaluates robustness under aggressive compression ratios.
  • Memory efficiency: At 50% sparsity, REP-LIE achieves the lowest reported GPU memory footprint, 476MB, while maintaining competitive performance.The comparison includes Head-FT, Pre-FT, Post-FT, and RECAP.
  • Criterion effectiveness: The LoRA-based importance criterion outperforms weight-magnitude and conventional gradient-based criteria in average performance and memory efficiency.The study also compares the resulting pruned model structures.

B. Medium-Scale Model Performance Comparison

On BERT-base, REP-LIE achieves the best results across GLUE tasks while preserving the model’s parameter count and limiting performance loss.

  • REP-LIE achieves the best results across all GLUE tasks at the same 67M parameter count.For QQP, RTE, and MRPC, accuracy decreases by only 0.17%, 0.35%, and 0.58%, respectively.
  • REP-LIE shows no significant performance degradation compared with the full BERT-base model.The Wilcoxon signed-rank test reports p > 0.05 for this comparison.
  • The Wilcoxon signed-rank test confirms that REP-LIE significantly outperforms the other compression methods in most comparisons.The reported consistent p-value is 0.0156, excluding comparisons involving EBERT.
  • EBERT performs worst among the compared methods, reaching 72.08% accuracy on MRPC and 52.75% on RTE.

C. Large-Scale Model Performance Comparison

On LLaMA-7B, REP-LIE maintains strong language-modeling and reasoning performance across sparsity levels, with larger advantages over baselines at higher sparsity.

  • REP-LIE achieves superior language-modeling and reasoning performance across most LLaMA-7B sparsity levels and subtasks.Models are evaluated at 20%, 25%, 33%, and 50% sparsity, producing 5.4B, 5.0B, 4.5B, and 3.4B parameters.
  • At 20% sparsity, REP-LIE attains the best PPL and highest average score, closely matching the original LLaMA-7B.
  • At 50% sparsity, REP-LIE improves average performance by approximately 5.2% over SlimGPT and about 17% over LLM-Pruner.
  • At 50% sparsity, REP-LIE surpasses SlimGPT by over 5% on both BoolQ and ARC-easy.
  • Under high sparsity and varying FLOPs reduction ratios, REP-LIE maintains stable GLUE performance and significantly outperforms mainstream compression methods.At 88% sparsity, average performance remains 78.64%, while MRPC declines by only 3.96% versus the baseline.

E. Memory Usage Comparison

REP-LIE reduces memory and pruning overhead by using low-rank gradients and lightweight LoRA updates, while its pruned models deliver deployment speedups.

  • REP-LIE achieves the lowest BERT-base memory footprint, using 476MB while maintaining competitive performance.At 50% sparsity, this reduces memory consumption by 69.4% compared with Post-FT.
  • REP-LIE avoids full-gradient computation by estimating importance from low-rank LoRA gradients.It selectively updates LoRA matrices instead of performing full-model finetuning during performance recovery.
  • REP-LIE achieves the fastest pruning time and lowest memory footprint across all tested sparsity levels against advanced baselines.The efficiency advantage is attributed to compact LoRA-based importance estimation and lightweight finetuning.
  • Pruning 50% of LLaMA-7B parameters consumes 5.4GB of GPU memory and completes in 16 minutes.This is 57.6% of the memory required to load the full model.
  • For BERT-base, the same pruning procedure completes in approximately 2 minutes.
  • At 50% sparsity, pruned LLaMA-7B reduces peak GPU memory by 47% and latency by 40%, while increasing throughput by 68%.Inference FLOPs are nearly halved relative to the dense model.

G. Pruning Criterion Analysis

LoRA-based importance estimation matches gradient-based performance at much lower cost and becomes increasingly advantageous under aggressive compression.

  • Gradient-based and LoRA-based criteria achieve comparable task performance, while weight magnitude lags across all evaluated models.
  • LoRA-based importance estimation reduces GPU memory by 49%–50% and importance-estimation FLOPs by over 96% versus gradient-based estimation.
  • At 80% FLOPs reduction on BERT-base, accuracy gaps reach about 15.7% on CoLA and 8.2% on MRPC.
  • On LLaMA-7B, magnitude pruning shows moderate PPL degradation but a steeper decline in zero-shot reasoning performance.
  • At r = 64, low-rank gradients achieve ρ > 0.75 overall and ρ > 0.85 for attention layers relative to full-rank estimation.At r = 2, overall correlation already exceeds 0.5; gains beyond r = 64 are marginal.
  • Attention layers exhibit higher low-rank approximability than FFN layers, while pruning methods generally retain more attention heads than FFN channels.

V. CONCLUSION

REP-LIE integrates low-rank importance estimation, structured pruning, and lightweight finetuning to improve the resource efficiency of Transformer compression. Its effectiveness is demonstrated across encoder and decoder-only language models, while broader generalizability remains underexplored.

  • REP-LIE evaluates weight importance with LoRA gradients, reducing memory and computational overhead by avoiding full-gradient computation.The framework uses low-rank matrices for importance estimation during pruning.
  • A stability score supports reliable importance estimation, while lightweight finetuning recovers pruned-model performance with minimal resource consumption.
  • Experiments on encoder models and large-scale decoder-only LLMs show that REP-LIE achieves state-of-the-art performance across diverse architectures.
  • REP-LIE unifies importance estimation and performance recovery within the same low-rank parameter space, integrating pruning and finetuning rather than treating them as sequential stages.
  • The method’s generalizability to diverse learning scenarios and compositional compression remains underexplored.Future work includes evaluating REP-LIE on knowledge-distilled and quantized models.
Loading 2608.24973v1…