PaperlayerView exact revision on arXiv

Source-linked AI summary

Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation

Peng Liu, Huibing Zeng, Yiqun Zhang, Yang Yi, Jigang Wu

arXiv:2608.24973v1cs.LGcs.AIcs.ET

TL;DR

Large Transformer models make full-gradient importance estimation and full-parameter finetuning costly. REP-LIE uses LoRA gradients, stability-aware iterative pruning, and LoRA-only recovery, achieving competitive results across encoder and generative models, including reported gains at higher sparsity.

  • Problem

    Existing pruning workflows rely on full-gradient importance estimation and full-model finetuning, creating substantial memory and computational overhead for large language models.

    Source: §I · p. 1

  • Method

    REP-LIE estimates importance from LoRA low-rank gradients, uses stability scores for iterative pruning, and recovers pruned models through LoRA-only updates.

    Source: §I · p. 2, §III · p. 4, C. Stabilization Mechanism · p. 5

  • Results

    REP-LIE achieves competitive performance across medium-scale encoders and large-scale generative models; at 50% sparsity, it improves average performance by approximately 5.2% over SlimGPT and about 17% over LLM-Pruner.

    Source: §I · p. 2, C. Large-Scale Model Performance… · p. 8

  • Takeaways & Limitations

    REP-LIE unifies importance estimation and performance recovery within the low-rank parameter space, shifting pruning toward simultaneous pruning-and-finetuning.

    Source: §V · p. 13

Abstract

from arXiv · show

With the rapid development of large-scale pre-trained language models based on Transformer architectures, their high computational and memory costs have become a major obstacle to deployment, especially in resource-constrained environments. Traditional pruning methods typically depend on full gradient-based importance estimation, and they necessitate prior finetuning of the model to achieve satisfactory performance. This process often results in intolerable resource consumption. This paper proposes REP-LIE, a new approach to enable resource-efficient pruning during the process of finetuning. REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation. To address the inherent randomness in importance estimation, a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters. The pruned model is further finetuned through lightweight updates, eliminating the need for full-parameter optimization in the process of finetuning. Extensive experiments on both medium-scale encoder models and large-scale generative models (LLaMA-7B and Mistral-7B) demonstrate that REP-LIE still achieves competitive performance compared to existing approaches.

I. INTRODUCTION

Transformer models offer strong task performance but create substantial deployment and training costs as their parameter counts grow. REP-LIE addresses pruning overhead by estimating importance with LoRA gradients and recovering pruned models through lightweight updates.

Motivation

  • Large Transformer language models create substantial computational and memory challenges for practical development and deployment.GPT-3 is cited as an example with 175 billion parameters.Source: §I · p. 1
  • Structured pruning removes model components at hardware-friendly granularity, whereas unstructured sparsity faces irregularity and hardware constraints.Both approaches estimate importance and remove the least important weights, but structured pruning is more suitable for acceleration.Source: §I · p. 1

Existing pruning workflows

  • Pre-finetune and post-finetune pruning both rely on gradient-based importance estimation and full-model finetuning, imposing substantial memory and computational overhead.Post-finetune pruning first finetunes the full model and then the pruned model, while pre-finetune pruning reverses that order.Source: §I · p. 1

REP-LIE

  • REP-LIE estimates weight importance using only LoRA low-rank gradients and restores pruned-model performance by updating low-rank matrices instead of the full parameter set.This framework is designed to reduce pruning overhead and avoid full-parameter optimization during recovery.Source: §I · p. 2

Evaluation

  • REP-LIE is evaluated on medium-scale encoder models and large-scale generative models, where it achieves competitive performance across scales.The supplied introduction describes the evaluation as demonstrating scalability and robustness.Source: §I · p. 2

B. Pruning Criterion

The paper motivates low-rank importance estimation by contrasting expensive or incomplete pruning criteria with LoRA’s parameter-efficient adaptation. LoRA freezes pretrained weights and learns a low-rank update with substantially lower trainable complexity.

Importance estimation

  • Second-order Taylor importance estimation is expensive for large networks, while weight magnitude can overlook parameters that significantly affect performance.The supplied passage introduces these as alternative importance-estimation strategies.Source: B. Pruning Criterion · p. 3

Existing methods

  • Existing structured-pruning methods improve flexibility or recovery but may still require full-model finetuning, distillation, or gradient-based importance estimation.RECAP reduces memory overhead yet retains gradient-based importance estimation, which introduces additional overhead.Source: C. Structured Pruning · p. 3

LoRA

  • LoRA freezes the pretrained weight matrix and adds two trainable low-rank matrices, A and B, with rank r much smaller than the layer dimensions.This reparameterizes adaptation without directly updating the original matrix.Source: D. Low-Rank Adaptation · p. 3
  • LoRA reduces trainable-parameter complexity from O(dk) to O((d + k)r), enabling parameter-efficient adaptation.The original weight is represented as W = W0 + BA, with the update initialized to preserve the initial output.Source: D. Low-Rank Adaptation · p. 3
  • Low-rank adaptation can approach full-finetuning performance; one cited example uses r = 8 with less than 0.1% additional parameters.LoRA has been applied to attention projections and extended to feed-forward layers.Source: D. Low-Rank Adaptation · p. 3

III. PROPOSED METHOD

REP-LIE performs structured pruning during finetuning by estimating importance from LoRA gradients, stabilizing those estimates, and recovering performance through LoRA-only updates. Transformer attention heads and FFN hidden units serve as pruning units.

III. PROPOSED METHOD

  • REP-LIE combines low-rank-gradient importance estimation, stability-aware progressive pruning, and lightweight LoRA finetuning in one framework.The framework is designed to support compression and performance recovery without substantial computational or memory overhead.Source: §III · pp. 3–4, Figure 3 · p. 4

Importance estimation

  • Importance is estimated from gradients of LoRA low-rank matrices rather than full model gradients.The criterion integrates Taylor expansion with low-rank gradients for computationally efficient parameter assessment.Source: §III · p. 3, B. Low-Rank Pruning Criterion · p. 4

Stabilization and pruning

  • A stability score mitigates randomness in importance estimation, and unimportant parameters are iteratively pruned according to the resulting scores.The method applies this stabilization mechanism during progressive structured pruning.Source: §III · p. 4, Figure 3 · p. 4

Lightweight finetuning

  • After each pruning round, REP-LIE restores model performance by updating only the LoRA parameters.This avoids updating the entire model parameter set during recovery.Source: §III · p. 4

Structured pruning units

  • For Transformer encoders, each attention head is a pruning unit in MHA, while each FFN hidden unit is treated as a pruning unit.An attention-head unit includes its query, key, value, and output projection matrices.Source: A. Grouping Model Weights · p. 4

1) Importance Estimation via Loss Function:

REP-LIE estimates pruning importance through a first-order loss approximation while replacing full-weight gradients with gradients from LoRA low-rank matrices. This reduces importance-estimation overhead and supports compatibility across Transformer architectures equipped with LoRA.

C. Stabilization Mechanism

REP-LIE introduces a stability score to address randomness in iterative importance estimation. Units with higher instability are treated as less important and become stronger pruning candidates.

  • Parameter updates and changing input-batch sequences can make importance estimates inconsistent across iterations.The randomness particularly affects low-significance pruning units.Source: C. Stabilization Mechanism · p. 5
  • The stability score measures ranking variation over repeated importance computations, with higher values indicating greater instability and lower importance.It uses each unit’s iteration-specific rank and mean rank.Source: C. Stabilization Mechanism · p. 5
  • Unimportant parameters are iteratively pruned according to their stability scores, with LoRA-only updates restoring performance after each pruning round.The recovery step avoids updating the full parameter set.Source: C. Stabilization Mechanism · p. 5

D. Adaptive Pruning under Resource Constraints

REP-LIE performs adaptive structured pruning under computational and sparsity budgets. Dynamic scheduling progressively changes sparsity, evaluates intermediate masks, and selects the highest-performing mask for the target sparsity.

  • Resource-aware pruning incorporates computational constraints such as FLOPs together with sparsity constraints.These constraints guide the model-pruning process.Source: D. Adaptive Pruning under Resource… · p. 5
  • The pruning objective minimizes task loss under a resource budget while controlling retained attention heads and feed-forward units with a binary mask.The resource cost of retained components is constrained by the total budget.Source: D. Adaptive Pruning under Resource… · p. 5
  • Dynamic sparsity scheduling controls sparsity through distinct training phases using initial and final rates over total steps.The schedule is represented as Scheduling(ti, tf, T, vi, vf).Source: D. Adaptive Pruning under Resource… · p. 5
  • After all iterations, the algorithm chooses the recorded step with the highest accuracy and returns its binary mask.The selected mask identifies the final heads and filters to prune.Source: D. Adaptive Pruning under Resource… · p. 6
  • Each iteration computes a target rate, selects the least-important heads and filters, updates their masks, and records performance and pruning plans.The procedure progressively adjusts pruning toward the target rate.Source: D. Adaptive Pruning under Resource… · p. 6

IV. EXPERIMENTAL EVALUATION

REP-LIE is evaluated across BERT-base, LLaMA-7B, and Mistral-7B using task, language-modeling, efficiency, and criterion comparisons. The reported results show superior or competitive performance, robustness under high compression, and low memory usage across these settings.

Experimental scope

  • REP-LIE is evaluated across three representative model families and six research questions covering effectiveness, efficiency, scalability, and design choices.Experiments include medium-scale encoders and large-scale generative models.Source: §IV · p. 6, A. Experimental Settings · p. 6

Medium-scale performance

  • On BERT-base GLUE tasks, REP-LIE is significantly superior to other compared compression methods despite performance drops from reducing model size.Comparisons include CoFi, PGB, DynaBERT, EBERT, TinyBERT, and RECAP.Source: A. Experimental Settings · pp. 6–7

Large-scale scalability

  • Large-scale evaluation tests LLaMA-7B and Mistral-7B using language modeling and zero-shot commonsense reasoning across multiple compression levels.The original dense models serve as baselines, with perplexity evaluated on WikiText-2.Source: A. Experimental Settings · pp. 6–7, Table II · p. 7

High-compression robustness

  • Under high sparsity and varying FLOPs reductions, REP-LIE maintains stable GLUE performance and significantly outperforms CoFi and EBERT.This evaluates robustness under aggressive compression ratios.Source: A. Experimental Settings · p. 6

Memory efficiency

  • At 50% sparsity, REP-LIE achieves the lowest reported GPU memory footprint, 476MB, while maintaining competitive performance.The comparison includes Head-FT, Pre-FT, Post-FT, and RECAP.Source: A. Experimental Settings · p. 6

Criterion effectiveness

  • The LoRA-based importance criterion outperforms weight-magnitude and conventional gradient-based criteria in average performance and memory efficiency.The study also compares the resulting pruned model structures.Source: A. Experimental Settings · p. 6

B. Medium-Scale Model Performance Comparison

On BERT-base, REP-LIE achieves the best results across GLUE tasks while preserving the model’s parameter count and limiting performance loss.

C. Large-Scale Model Performance Comparison

On LLaMA-7B, REP-LIE maintains strong language-modeling and reasoning performance across sparsity levels, with larger advantages over baselines at higher sparsity.

E. Memory Usage Comparison

REP-LIE reduces memory and pruning overhead by using low-rank gradients and lightweight LoRA updates, while its pruned models deliver deployment speedups.

  • REP-LIE achieves the lowest BERT-base memory footprint, using 476MB while maintaining competitive performance.At 50% sparsity, this reduces memory consumption by 69.4% compared with Post-FT.Source: E. Memory Usage Comparison · p. 9
  • REP-LIE avoids full-gradient computation by estimating importance from low-rank LoRA gradients.It selectively updates LoRA matrices instead of performing full-model finetuning during performance recovery.Source: E. Memory Usage Comparison · p. 9
  • REP-LIE achieves the fastest pruning time and lowest memory footprint across all tested sparsity levels against advanced baselines.The efficiency advantage is attributed to compact LoRA-based importance estimation and lightweight finetuning.Source: F. Efficiency Evaluation on Medium and… · p. 10
  • Pruning 50% of LLaMA-7B parameters consumes 5.4GB of GPU memory and completes in 16 minutes.This is 57.6% of the memory required to load the full model.Source: F. Efficiency Evaluation on Medium and… · p. 10
  • For BERT-base, the same pruning procedure completes in approximately 2 minutes.Source: F. Efficiency Evaluation on Medium and… · p. 11
  • At 50% sparsity, pruned LLaMA-7B reduces peak GPU memory by 47% and latency by 40%, while increasing throughput by 68%.Inference FLOPs are nearly halved relative to the dense model.Source: F. Efficiency Evaluation on Medium and… · p. 11

G. Pruning Criterion Analysis

LoRA-based importance estimation matches gradient-based performance at much lower cost and becomes increasingly advantageous under aggressive compression.

V. CONCLUSION

REP-LIE integrates low-rank importance estimation, structured pruning, and lightweight finetuning to improve the resource efficiency of Transformer compression. Its effectiveness is demonstrated across encoder and decoder-only language models, while broader generalizability remains underexplored.

  • REP-LIE evaluates weight importance with LoRA gradients, reducing memory and computational overhead by avoiding full-gradient computation.The framework uses low-rank matrices for importance estimation during pruning.Source: §V · p. 13
  • A stability score supports reliable importance estimation, while lightweight finetuning recovers pruned-model performance with minimal resource consumption.Source: §V · p. 13
  • Experiments on encoder models and large-scale decoder-only LLMs show that REP-LIE achieves state-of-the-art performance across diverse architectures.Source: §V · p. 13
  • REP-LIE unifies importance estimation and performance recovery within the same low-rank parameter space, integrating pruning and finetuning rather than treating them as sequential stages.Source: §V · p. 13
  • The method’s generalizability to diverse learning scenarios and compositional compression remains underexplored.Future work includes evaluating REP-LIE on knowledge-distilled and quantized models.Source: §V · pp. 13–14

Related papers

Loading 2608.24973v1…