PaperlayerView exact revision on arXiv

Source-linked AI summary

Magnitude Profile Pruning: Calibration-Free Structured Attention Head Removal for Transformer Compression

Kasun Dewage, Marianna Pensky, Heranga K. Rathnasekara, Suranadi De Silva

arXiv:2609.26177v1cs.CLcs.AIcs.LG

TL;DR

Transformer compression faces deployment costs, while some head-importance methods require calibration data and Hessian estimation. Magnitude Profile scoring uses weight-norm outliers to identify heads to prune, with an extension for grouped-query attention. Across five models, MP-G performs best in several settings and ranks in the top two in most primary model-sparsity settings without calibration overhead.

  • Problem

    Transformer computational and memory footprints challenge deployment, while a Hessian-based head-scoring method requires calibration data and Hessian estimation.

    Source: §I · p. 1, E. Baseline Scoring Methods · p. 4

  • Method

    Magnitude Profile scores heads by excess weight-row norms beyond a statistical threshold, while MP-G distributes shared key-value group scores across associated query heads.

    Source: §I · p. 1

  • Results

    Across encoder and decoder architectures, MP-G achieves the best perplexity in several settings and top-two performance in most primary model-sparsity settings at 12.5%–50% head sparsity, with zero calibration overhead.

    Source: §3 · p. 2, §VII · p. 6

  • Takeaways & Limitations

    MP-G offers a fast, zero-cost initial pruning strategy, particularly when calibration data is unavailable, domain-mismatched, or rapid pruning iteration is needed.

    Source: §VII · p. 6

    Theoretical FLOP savings still require future work on model surgery or sparse-kernel support to translate into measured speedup.

    Source: §VII · p. 6

Abstract

from arXiv · show

Structured pruning of attention heads provides a hardware-friendly way to compress Transformer language models. However, existing methods for measuring head-level importance require calibration data, gradient computation, or Hessian estimation. These requirements add extra overhead and make the methods depend on the data. Our work presents Magnitude Profile (MP) scoring, a training-free criterion for head importance that identifies dispensable heads through statistical outlier detection on weight row norms. Heads whose projection weights fall within the population bulk are pruned, while heads exhibiting outlier norms, which carry disproportionate representational capacity, are preserved. Our work further gives MP-G, a variant that handles Grouped Query Attention (GQA) by distributing shared key-value group scores across associated query heads. Across five models evaluated on WikiText-2 perplexity at 12.5%-50% head sparsity, MP-G achieves the best perplexity on OPT-6.7B at all sparsity levels (18.46 at 12.5%, 27.87 at 25%, 152.0 at 50%). MP-G also gives the best results on RoBERTa-large at 12.5% and 25% sparsity, with perplexity values of 7.27 and 10.28, outperforming calibration-dependent baselines including Wanda-Head, SparseGPT-Head, and Gradient-Head. It requires zero forward passes, calibration samples, or gradient computation. At 50% sparsity, head pruning yields up to 16% parameter reduction with 50% attention FLOP savings. Our results show that weight-only statistical scoring can match or outperform data-dependent methods for structured head pruning, providing a practical, zero-cost criterion for Transformer compression.

I. INTRODUCTION

Structured head pruning can reduce Transformer computation, but many existing importance criteria require calibration data or gradient-related computation. The paper proposes weight-only Magnitude Profile scoring and a GQA-aware variant, reporting competitive results across five models with no calibration overhead.

I. INTRODUCTION

  • Structured head pruning can directly reduce projection operations and attention computations, unlike unstructured sparsity that may need specialized hardware or software for speedup.Source: §I · p. 1
  • MP scores head importance by weight-norm excess beyond a statistical threshold, treating typical norms as zero-score and pruning them first.Source: §I · p. 1
  • The criterion uses model weights only, with O(p) computation and no calibration data, forward passes, gradients, or Hessian estimation.Source: §I · p. 1, Figure 1 · p. 2
  • MP-G extends the scoring approach to Grouped Query Attention by accounting for shared key-value heads across associated query heads.Source: §I · p. 1

3. We demonstrate through experiments on five models

  • MP-G achieves best perplexity in several settings and top-two performance in most primary model-sparsity settings across encoder and decoder architectures.The evaluation covers 12.5%–50% head sparsity and reports zero calibration overhead.Source: §3 · p. 2
  • 50% head pruning yields approximately 7–16% total parameter reduction with 50% attention FLOP savings.Source: §3 · p. 2

II. RELATED WORK

Prior head-pruning methods use training or task data, while unstructured pruning criteria target individual weights and broader structural methods often require substantial computation. This section frames head-level pruning as globally scoring and zeroing selected attention-head weight slices.

II. RELATED WORK

  • Gradient-based head importance and differentiable gates use task-specific data or training, while SparseGPT and Wanda were designed for element-level sparsity.Source: §II · p. 2
  • Other structured-pruning approaches target broader units such as layers or hidden dimensions and typically require substantial computation.Source: §II · p. 2

A. Problem Formulation

  • The pruning objective is to globally score heads and zero the Q, K, V, and O slices of the lowest-scoring ⌊s·L·H⌋ heads.The experiments zero slices in place; physical removal of head dimensions is left to deployment model surgery.Source: A. Problem Formulation · p. 2

B. Magnitude Profile Scoring

Magnitude Profile scoring identifies heads whose weight norms are statistical outliers relative to the projection population. It aggregates norm excesses into head-level scores, with the threshold parameter controlling which norms count as outliers.

B. Magnitude Profile Scoring

C. MP-G: GQA-Aware Extension

MP-G addresses the shared key-value structure in GQA by computing a per-group score and distributing it across associated query heads. It accounts for group sharing while retaining standard MP behavior for MHA.

C. MP-G: GQA-Aware Extension

  • MP-G computes a per-KV-group score and distributes it equally across associated query heads to account for GQA’s shared key-value structure.Standard MP applies the same K/V excess to every query head in a group without accounting for that structural coupling.Source: C. MP-G: GQA-Aware Extension · p. 3
  • αq and αkv control the relative weighting of query-side and KV-side contributions, and both default to 1.0.The KV-group excess is the total excess for that group.Source: C. MP-G: GQA-Aware Extension · p. 3
  • The division by group size prevents the shared KV signal from being over-counted across query heads.When every query head in a KV group is pruned, the corresponding K/V weights are also zeroed.Source: C. MP-G: GQA-Aware Extension · p. 3
  • For MHA models, where Hkv = H, MP-G reduces to standard MP.Source: C. MP-G: GQA-Aware Extension · p. 3

D. Pruning Procedure

The pruning procedure scores all heads, ranks them globally, and zeros the lowest-scoring heads until the target sparsity is reached, without fine-tuning or weight reconstruction.

D. Pruning Procedure

  • MP-G scores heads across all layers, globally ranks them in ascending score order, then zeros the lowest-scoring heads until target sparsity is reached.Source: D. Pruning Procedure · p. 3
  • The procedure takes a model and target sparsity as inputs and applies the ranking across all layers.Source: D. Pruning Procedure · p. 3
  • Pruning requires neither fine-tuning nor weight reconstruction.Source: D. Pruning Procedure · p. 3

E. Baseline Scoring Methods

The paper compares four head-level scoring baselines: activation-aware Wanda, Hessian-based SparseGPT, gradient-based scoring, and a hybrid of magnitude, MP, and Wanda signals.

E. Baseline Scoring Methods

  • Wanda-Head aggregates weight-magnitude and input-activation-norm products across each head’s Q/K/V rows and O columns, requiring calibration-data forward passes.Source: E. Baseline Scoring Methods · pp. 3–4
  • SparseGPT-Head aggregates a Hessian-diagonal importance score at head level and requires calibration data and Hessian estimation.Hjj is the diagonal of the empirical Hessian of the layer input.Source: E. Baseline Scoring Methods · p. 4
  • Gradient-Head uses the per-weight first-order criterion |wij · gij|,, aggregated per head, and requires a backward pass over calibration data.Source: E. Baseline Scoring Methods · p. 4
  • Hybrid min-max normalizes Magnitude, MP, and Wanda scores to [0, 1] and combines them with equal weights of 1.0.The combination tests whether fusing weight-only and calibration signals improves over either alone.Source: E. Baseline Scoring Methods · p. 4

IV. EXPERIMENTAL SETUP

Experiments span five encoder and decoder models, multiple attention-head sparsity levels, and perplexity and GLUE evaluations. Calibration-dependent baselines use WikiText-2 training samples, while MP and MP-G do not require calibration data.

IV. EXPERIMENTAL SETUP

  • The evaluated models are RoBERTa-base, RoBERTa-large, OPT-6.7B, LLaMA-3-8B, and Mistral-7B-v0.1, covering encoder, decoder, MHA, and GQA architectures.Source: §IV · p. 4
  • Head sparsity is set to 12.5%, 25%, or 50% by global ranking across layers; decoder perplexity uses WikiText-2 validation, while RoBERTa uses masked-LM pseudo-perplexity.Source: §IV · p. 4
  • GLUE runs fine-tune RoBERTa-large on SST-2, MRPC, and RTE before pruning and evaluation, using a learning rate of 10−5 and batch size 16.The fine-tuning durations are 3 epochs for SST-2, 5 for MRPC, and 10 for RTE.Source: §IV · p. 4
  • Calibration-based methods use WikiText-2 training samples at sequence length 512: 64 samples for Wanda-Head, SparseGPT-Head, and Hybrid, and 32 for Gradient-Head; MP and MP-G use none.Source: §IV · p. 4
  • MP and MP-G use default threshold z=2.0, Hybrid uses equal combination weights λ=1.0, and all experiments use seed 0.Source: §IV · p. 4

V. RESULTS

Across the evaluated models, MP-G performs best on OPT-6.7B at every tested sparsity and on RoBERTa-large at lower sparsities, while Gradient-Head leads on RoBERTa-base and the GQA models.

A. Perplexity Across Models

  • 7.27 and 10.28: MP-G achieves the lowest perplexity on RoBERTa-large at 12.5% and 25% sparsity, respectively.At 50%, MP-G's 45.76 is close to SparseGPT-Head's 42.23.Source: A. Perplexity Across Models · p. 4
  • Gradient-Head performs best on RoBERTa-base at every sparsity level; at 50%, MP-G ranks second with 71.25 versus 64.74.At 50%, MP-G also beats Wanda-Head (103.8) and SparseGPT-Head (4730).Source: A. Perplexity Across Models · p. 4
  • 18.46, 27.87, and 152.0: MP-G achieves the lowest perplexity on OPT-6.7B at 12.5%, 25%, and 50% sparsity, respectively.At 12.5% sparsity, it improves on Hybrid (25.64) by 28%; Gradient-Head results are unavailable at 25% and 50%.Source: A. Perplexity Across Models · p. 4
  • 11.12, 17.69, and 47.94: Gradient-Head leads on LLaMA-3-8B at 12.5%, 25%, and 50% sparsity, respectively.MP-G ranks second at 12.5% (20.39) and 25% (41.41).Source: A. Perplexity Across Models · p. 4
  • 22.48: MP-G is competitive on Mistral-7B at 12.5% sparsity, while Gradient-Head leads at 25% (13.30) and 50% (54.11%).At 50%, Wanda-Head (552.8) outperforms MP-G (1072) and SparseGPT-Head (871.8) among non-gradient baselines.Source: A. Perplexity Across Models · p. 4

B. Effect of the Threshold z

The threshold z controls which weight-norm outliers MP-G preserves, and its best setting varies by model. Downstream results also differ from perplexity comparisons, particularly at aggressive sparsity.

B. Effect of the Threshold z

  • z=2.5 changes LLaMA-3-8B perplexity at 50% sparsity from 7227 to 282.8, showing the threshold's importance at extreme pruning.MHA RoBERTa-base often prefers z=1.5, while GQA LLaMA-3 and RoBERTa-large benefit from z=2.0–2.5.Source: B. Effect of the Threshold z · p. 4, Table II · p. 5
  • The optimal z is model-dependent: RoBERTa-base often prefers 1.5, while LLaMA-3 and RoBERTa-large benefit from 2.0–2.5.The authors suggest model-adaptive z selection as future work.Source: B. Effect of the Threshold z · p. 4, Table II · p. 5

C. Downstream Task Performance

  • At 25% sparsity, reported methods retain 92.8–95.2% SST-2 accuracy; Hybrid achieves the highest SST-2 accuracy (95.2%).At 50%, SparseGPT-Head retains 89.6% SST-2 accuracy and 81.4 MRPC F1 while MP-G, Wanda-Head, and Hybrid collapse on MRPC.Source: C. Downstream Task Performance · p. 5, Table III · p. 5
  • At 50% sparsity on RTE, all reported methods converge to near-chance accuracy (47.3%).MP-G has competitive perplexity (45.76 versus SparseGPT-Head's 42.23), but SparseGPT-Head shows better downstream robustness after fine-tune-then-prune.Source: C. Downstream Task Performance · p. 5

D. Efficiency Analysis

At 50% head sparsity, pruning cuts attention FLOPs by half, but measured speedups remain modest without physically removing pruned dimensions or using sparse inference support. MP-G's weight-norm signal also avoids calibration computation.

D. Efficiency Analysis

  • 50% head sparsity yields 50% attention FLOP reduction and 8%–16% parameter reduction, but measured wall-clock change ranges from small slowdowns to about 3.5% speedup.Standard PyTorch attention kernels do not dynamically skip zeroed heads; model surgery or sparse inference engines are needed to realize theoretical FLOP savings.Source: D. Efficiency Analysis · p. 5

A. Why Does MP-G Work?

  • MP-G exploits the observation that a minority of trained heads develop disproportionately large weight norms, while typical-norm heads are mostly redundant.The authors relate these weight-norm signatures to specialized heads identified in prior work, without observing attention patterns on data.Source: A. Why Does MP-G Work? · p. 5

B. Calibration-Free Advantage

  • MP-G requires no calibration dataset or forward/backward passes and computes weight norms in seconds, supporting rapid pruning iteration.Compared methods use 32–64 calibration sequences and forward passes; Gradient-Head also uses backward passes.Source: B. Calibration-Free Advantage · pp. 5–6

C. Method Robustness Across Architectures

MP-G is top-two on most primary model-sparsity settings and produces finite perplexity across tested configurations, despite Gradient-Head having the most first-place finishes. The conclusion positions it as a fast initial pruning option, with fine-tuning available when task performance is critical.

C. Method Robustness Across Architectures

  • MP-G achieves top-two perplexity in most primary model-sparsity settings and finite perplexity on every tested configuration; Gradient-Head has eight first-place finishes.The comparison covers 15 model-sparsity settings with at least two finite results at z=2.0; OPT-6.7B Gradient-Head results at 25% and 50% are unavailable.Source: C. Method Robustness Across… · p. 6

VII. CONCLUSION

  • MP-G needs only model weights and achieves competitive or superior perplexity in several settings without calibration data, forward passes, gradients, or Hessian estimation.The authors describe it as a fast, zero-cost initial pruning strategy and suggest optional fine-tuning when downstream task performance is critical.Source: §VII · p. 6

Related papers

Loading 2609.26177v1…