PaperlayerView exact revision on arXiv

Source-linked AI summary

COPAL: Continual Pruning in Large Language Generative Models

Srikanth Malla, Joon Hee Choi, Chiho Choi

arXiv:2405.02347v2cs.LGcs.AIcs.CL

TL;DR

Adapting pre-trained LLMs across domains is challenged by computational demands and limited continual adaptability. COPAL addresses both through training-free continual pruning guided by sensitivity analysis, and evaluations across LLaMA models report improved BWT and PPL performance over existing methods. The method is intended to preserve prior knowledge while adapting to new datasets without retraining.

  • Problem

    Adapting pre-trained LLMs to new domains requires substantial computation and can cause catastrophic forgetting of previously learned capabilities.

    Source: §1 · p. 1

  • Method

    COPAL uses training-free continual pruning guided by sensitivity analysis and small calibration sets, without storing or reusing past datasets.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    COPAL significantly outperformed existing methods in BWT reduction and PPL performance across pruning structures, datasets, and LLaMA-7B, 13B, 30B, and 65B models.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    COPAL enables continual model adaptation while reducing model size and preserving performance across previously encountered datasets.

    Source: §1 · p. 2, §6 · p. 9

    Updating pruned weights with a new calibration dataset can deteriorate performance on previously encountered datasets, motivating COPAL's sensitivity-based treatment of forgetting.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Adapting pre-trained large language models to different domains in natural language processing requires two key considerations: high computational demands and model's inability to continual adaptation. To simultaneously address both issues, this paper presents COPAL (COntinual Pruning in Adaptive Language settings), an algorithm developed for pruning large language generative models under a continual model adaptation setting. While avoiding resource-heavy finetuning or retraining, our pruning process is guided by the proposed sensitivity analysis. The sensitivity effectively measures model's ability to withstand perturbations introduced by the new dataset and finds model's weights that are relevant for all encountered datasets. As a result, COPAL allows seamless model adaptation to new domains while enhancing the resource efficiency. Our empirical evaluation on a various size of LLMs show that COPAL outperforms baseline models, demonstrating its efficacy in efficiency and adaptability.

1. Introduction

Pre-trained LLM adaptation is constrained by computational cost and catastrophic forgetting, while existing pruning and continual-learning approaches address these challenges separately. COPAL combines continual pruning with sensitivity analysis to adapt models without retraining, and evaluations report improved efficiency and adaptability.

  • Traditional pruning improves inference efficiency but lacks continual flexibility, whereas continual learning adapts to evolving data through finetuning that remains computationally expensive.Prior pruning approaches may require domain-specific training or calibration, while continual-learning methods require sequential dataset finetuning.Source: §1 · pp. 1–2
  • COPAL is designed to address computational inefficiency and limited model adaptability simultaneously through continual pruning in adaptive language settings.The framework avoids further training or finetuning and identifies weights relevant across encountered datasets.Source: §1 · p. 2
  • COPAL avoids the computational burden of retraining and does not require storing or reusing past datasets during continual adaptation.The method uses a small calibration set when transitioning to a new dataset.Source: §1 · pp. 1–2
  • Figure 1 reports that COPAL maintains performance in average and worst-case BWT scenarios as sparsity increases.The figure is presented as evidence that continual pruning can balance model complexity and performance.Source: Figure 1 · p. 1, §1 · p. 2
  • Sensitivity analysis guides pruning under continual model adaptation by identifying weights that remain relevant across encountered datasets.A small calibration set from the new dataset guides pruning, while previously pruned parameters retain information about past datasets.Source: §1 · p. 2
  • Empirical evaluations on LLaMA-7B, 13B, 30B, and 65B report that COPAL outperforms baseline methods in efficiency and adaptability.Source: §1 · p. 2

2. Prior Works

Prior work studies pruning for efficiency and continual learning for adaptation, but both approaches face limitations when applied to evolving LLM workloads. Existing continual-pruning efforts still involve retraining or risk reduced generalizability, motivating a training-free alternative.

Pruning

  • Post-training pruning reduces model size after training and avoids retraining, while unstructured pruning removes individual weights and structured pruning removes larger network elements.The prior-work discussion distinguishes structured and unstructured pruning and identifies post-training methods as especially relevant to LLM efficiency.Source: §2 · p. 2

Continual Learning

  • Continual learning adapts models to evolving tasks through rehearsal, regularization, or architectural methods intended to mitigate catastrophic forgetting.Source: §2 · p. 2

Continual Pruning

  • Continual pruning is defined as training-free pruning across evolving datasets that dynamically adjusts relevant weights without saving or reusing all past data.The supplied overview also depicts calibration data D and an incremental order D1→D2→D3.Source: Figure 2 · p. 3

Continual Learning

  • Continual learning still requires finetuning on new datasets, increasing training costs and risking diminished base-model generalizability as data and applications evolve.Source: §2 · pp. 2–3

Pruning in Continual Setting

  • Existing pruning-in-continual-learning studies emphasize efficiency and adaptability but may reduce base-model generalizability and retain increasing LLM training costs.The paper positions its method as bypassing retraining rather than continuing to retrain models for new tasks.Source: §2 · p. 3

3. Problem Formulation

Continual pruning targets training-free model sparsification across evolving datasets while preserving prior capabilities. The section identifies weight stasis and forgetting as the central obstacles and motivates COPAL’s sensitivity-based solution.

3.1. Continual Pruning

  • Continual pruning removes weights across evolving datasets without retaining all past data or retraining, while aiming to preserve previously acquired performance.Source: §3.1 · p. 3, §3 · p. 3

3.2. Weight Stasis in Continual Pruning

  • Calibration-guided pruning creates weight stasis because weights zeroed on one dataset remain below later thresholds and cannot respond to new data.Source: §3.2 · pp. 3–4
  • The pruning formulation ranks weights by importance W*_i = |W_i · R_i| and applies a mask that retains weights above a sparsity-dependent threshold.Source: §3.2 · pp. 3–4

3.3. Forgetting in Continual Pruning

  • Table 1 evaluates pruning methods on LLaMA-7B at 50% sparsity, defining A-BWT as average backward transfer and S-INIT as sequential initialization.Source: Table 1 · p. 4

3.2. Weight Stasis in Continual Pruning

  • Sequential initialization starts each new dataset from an already pruned matrix, so the resulting mask preserves previously zeroed weights during transitions.Source: §3.2 · p. 4

3.3. Forgetting in Continual Pruning

  • Non-sequential initialization can reduce weight stasis, but updating pruned weights with new calibration data then deteriorates performance on earlier datasets.Source: §3.3 · p. 4
  • Under sequential initialization, all baselines exhibit weight stasis, while their higher A-BWT values indicate forgetting; COPAL reports A-BWT of 0.016 in the corresponding evaluation.Source: §3.3 · p. 4, Table 1 · p. 4

4. Methodology

COPAL uses sensitivity analysis to identify weights whose perturbation affects outputs and losses across datasets. The resulting algorithm accumulates weight importance, thresholds it for a target sparsity, and prunes progressively.

4. Methodology

  • COPAL addresses continual-pruning weight stasis and forgetting through a sensitivity-analysis framework for identifying weights important across datasets.Source: §4 · p. 4

4.1. Theoretical Framework for Continual Sensitivity Analysis

  • The theoretical framework measures output sensitivity to infinitesimal perturbations in both weights W and input features x.Source: §4.1 · p. 4
  • Finite-difference approximations replace analytically difficult derivatives, converting perturbation effects into practical sensitivity terms.Source: §4.1 · pp. 4–5

4.2. Identification of Crucial Weights

  • Weight importance is based on the magnitude of directional derivatives of dataset loss, so larger values identify weights to which performance is more sensitive.Source: §4.2 · p. 5
  • The loss-gradient formulation determines perturbations in W that minimize output change and accumulates absolute gradients through the dataset sequence.Source: §4.2 · p. 5

4.3. Algorithm

  • For each dataset, COPAL computes sensitivity, updates weight importance, sets a sparsity-based threshold, and masks weights below it.Source: §4.3 · p. 5

5. Experiments

COPAL is evaluated on LLaMA models from 7B to 65B across three datasets, pruning settings, dataset permutations, and continual-learning metrics. Across these experiments, it generally reduces backward transfer and perplexity relative to baselines, while addressing forgetting at increased sparsity.

5.1. Experimental Setting

  • The evaluation uses Wikitext-2, PTB, and C4 with LLaMA models at 7B, 13B, 30B, and 65B parameters, measuring perplexity and backward transfer across dataset permutations.Experiments include unstructured 50% sparsity and semi-structured 2:4 and 4:8 sparsity, with sequential layer sparsification performed without fine-tuning.Source: §5.1 · p. 6

5.3. Discussions

  • The dense models provide reference average PPL values of 7.714 for LLaMA-7B, 6.131 for LLaMA-30B, and 6.139 for LLaMA-65B.The results discussion also reports that increasing sparsity causes forgetting or poorer previous-task performance for other pruning methods, whereas COPAL is more robust.Source: §5.2 · p. 7, §5.3 · p. 9

5.2. Results

  • COPAL reduces average BWT to 0.016 versus Wanda’s 0.569 on LLaMA-7B unstructured pruning, a 97.2% improvement at 50% sparsity.It also reports PPL values of 7.144 on Wikitext2 and 12.567 on PTB, compared with baseline values of 7.208 and 13.141, respectively.Source: §5.2 · p. 7, Table 2 · p. 8
  • COPAL maintains strong results across LLaMA-13B, 30B, and 65B, including average BWT values of 0.029 and 0.001 for the 13B and 65B models.For LLaMA-30B, it improves average BWT over SparseGPT by 98.2%; for LLaMA-65B, it achieves a 99.7% improvement over SparseGPT.Source: §5.2 · p. 7
  • In semi-structured pruning, COPAL improves both BWT and PPL across 2:4 and 4:8 configurations, including a 98.2% BWT improvement for LLaMA-65B at 2:4 sparsity.For LLaMA-65B, average BWT falls to 0.038 at 2:4 and 0.019 at 4:8, while average PPL falls to 9.161 and 8.291, respectively.Source: §5.2 · pp. 7–8
  • For LLaMA-7B semi-structured pruning, COPAL lowers average BWT to 0.009 and average PPL to 15.335 at 2:4 sparsity.These correspond to a 99.4% BWT improvement over Wanda and a 5.1% PPL improvement; performance also improves under the 4:8 pattern.Source: §5.2 · p. 8

6. Conclusion

The paper introduces continual pruning to adapt LLMs without retraining. COPAL addresses weight stasis and forgetting, and evaluations across models, datasets, and pruning structures report consistent gains in BWT and PPL.

  • COPAL introduces continual pruning for model adaptation without requiring model retraining.Source: §6 · p. 9
  • The framework addresses weight stasis and forgetting through mathematical formulations and sensitivity-guided pruning.Source: §6 · p. 9
  • Across LLaMA-7B, 13B, 30B, and 65B models, COPAL outperforms existing methods in BWT reduction and PPL performance across pruning structures and datasets.Source: §6 · p. 9

Impact Statement

COPAL’s efficiency gains may broaden access to powerful language models, but continual adaptation raises fairness, transparency, explainability, and accountability concerns.

  • COPAL’s reduced computational demands may make powerful LLMs more accessible to users and organizations.Source: Impact Statement · p. 10
  • Dynamic pruning can challenge transparency and explainability, creating implications for user trust and accountability.Source: Impact Statement · p. 10

A. Theoretical Sensitivity to Practical Metrics

The paper defines sensitivity as the response of a model output to parameter perturbations and replaces difficult derivatives with a computable finite-difference metric.

B. Loss Function to its Gradient

This subsection expands the local loss expression and differentiates its components with respect to dW, using ∆W as a small parameter change.

B. Loss Function to its Gradient

2. The term

  • For a small change, ∆W is treated as the increment dW, whose derivative with respect to W is 1 or the identity matrix for matrix-valued W.Source: §2 · p. 13

C. Approximating the Gradient of f with Respect to W

The practical gradient approximation perturbs W, computes the resulting change in f, handles non-full-rank perturbations with a pseudoinverse, and automates sparsity thresholding.

C. Approximating the Gradient of f with Respect to W

C.1. Full-rank of ∆W

  • ∆W must be full rank, with rank equal to min(M, D), to be invertible in the approximation.Source: §C.1 · p. 13

C.2. Moore-Penrose Pseudoinverse

  • When ∆W is non-square or not full rank, the method uses the Moore-Penrose pseudoinverse computed through SVD.Source: §C.2 · p. 13

D. Automating threshold for r% sparsity

E. Overall standard deviations

Table 4 reports overall means and standard deviations for continual pruning across three datasets, four LLaMA model sizes, and all incremental permutations at 50% unstructured sparsity.

  • Table 4 covers wikitext2, ptb, and c4 under continual pruning.Source: Table 4 · p. 14
  • The evaluation includes LLaMA-7B, 13B, 30B, and 65B models.Source: Table 4 · p. 14
  • Results aggregate all incremental permutations and report both overall means and standard deviations at 50% unstructured sparsity.Source: Table 4 · p. 14

Related papers

Loading 2405.02347v2…