PaperlayerView exact revision on arXiv

Source-linked AI summary

SVD-Attention: Efficiently Analyzing Attention Blocks in Transformers

Debopam Sanyal, Hongjie Chen, Riya Elizabeth Kuriyan, Arth Tak, Divya Kothandaraman, Akshay Mehra, Joshua Kimball, Alexey Tumanov

arXiv:2610.34971v1cs.CLcs.AIcs.LG

TL;DR

Pretrained attention heads have an algebraic rank budget, but it is unclear which directions real inputs use. SVD-Attention scores per-head interaction modes using both singular values and token activation variance, then diagnoses task-dependent redundancy. Across seven models, the layers flagged vary by task, and acting on the diagnostic yields modest FLOP reductions alone or larger reductions with weight pruning.

  • Problem

    Although per-head interaction matrices have a bounded rank budget, the paper asks which directions real input tokens actually use across layers and tasks.

    Source: §1 · p. 1

  • Method

    SVD-Attention scores singular modes of each per-head interaction matrix by singular value and the geometric mean of query- and key-side activation variances, then retains modes meeting a target importance fraction without finetuning.

    Source: §1 · p. 1, §3.3 · p. 3

  • Results

    Redundancy lies early to middle for sentiment and image classification and generation, but late for dense prediction and extractive QA; FLOPs fall 0.8–1.5% alone or 8–11% with pruning, with task quality within 0–4% of baseline.

    Source: §1 · p. 1, §6 · p. 9

  • Takeaways & Limitations

    The authors position SVD-Attention primarily as a measurement tool that diagnoses where compression fails, rather than as a high-ratio compression method.

    Source: §6 · p. 9

    Reported FLOPs are analytic; wall-clock speedup requires a reduced-dimension attention kernel, since padding discarded dimensions back erases the savings.

    Source: §7 · p. 9

Abstract

from arXiv · show

Pretrained transformers spend a large share of their inference compute inside self-attention, yet the per-head query-key interaction is structurally low-rank by construction. We ask a diagnostic question: across layers and architectures, which heads actually use the full rank budget on real data, and which do not? We propose SVD-Attention, a post-hoc method that decomposes the per-head interaction matrix $M_h$ and scores every singular mode by both its spectral magnitude and its activation variance across tokens on a small calibration set. The score keeps the modes the head can amplify and that real data actually activates, and discards the rest; the resulting factorization is exact for the retained modes and needs no finetuning. Across seven pretrained models and five task types, the diagnostic finds a task-dependent pattern: redundancy sits in the early to middle layers for sentiment and image classification and for generation, and in the late layers for dense prediction and extractive question answering. It is also predictive: truncating the layers it flags costs at most $1.75$ F1, while truncating the same number of layers elsewhere in the same model costs up to $50$ F1. SVD-Attention is an analysis tool rather than a compression method: acting on it reduces inference FLOPs by $0.8$-$1.5\%$ on its own and by $8$-$11\%$ when composed with weight pruning, with $0$-$4\%$ change in task quality.

1 Introduction

SVD-Attention examines which per-head interaction directions real tokens use, then uses that diagnostic to locate task-dependent redundancy and assess its predictive value.

1 Introduction

  • Unlike methods that decompose query and key weights independently, SVD-Attention decomposes their product, the per-head interaction matrix consumed by softmax.That matrix has rank at most the head dimension, although real inputs may use fewer directions.Source: §1 · p. 1
  • Each mode is scored by its singular value and the geometric mean of query- and key-side activation variances, retaining the smallest set that captures a chosen share of importance without finetuning.Source: §1 · p. 1
  • Redundancy shifts by task—early to middle layers for sentiment, image classification, and generation, but late layers for dense prediction and extractive QA—and FLOPs fall 0.8–1.5% alone or 8–11% with pruning.Task quality remains within 0–4% of baseline when composed with weight pruning.Source: §1 · p. 1
  • For RoBERTa-base on SQuAD 2.0, more than half the average head’s rank budget can be discarded for 0.16 F1, and the diagnostic predicts where truncation fails.Source: §1 · p. 2

2 Related Work

SVD-Attention is positioned as a post-training analysis of per-head interaction redundancy, distinct from methods that alter attention computation, reduce tokens, or compress weights independently.

2 Related Work

  • Efficient-attention methods change the computation, whereas SVD-Attention keeps softmax intact and studies redundancy in pretrained projection weights (Wang et al., 2020a; Choromanski et al., 2020; Liu et al., 2021).Source: §2 · p. 2
  • Token Merging reduces sequence length, while SVD-Attention reduces the per-head channel dimension, making the two compression axes complementary (Bolya et al., 2022).Source: §2 · p. 2
  • ASVD (Yuan et al., 2023) and SVD-LLM (Wang et al., 2025) decompose weights separately; SVD-Attention scores the softmax-consumed interaction matrix using activation variance.Its variance term distinguishes modes that vary across tokens from modes that contribute only a near-constant offset.Source: §2 · p. 2
  • WANDA prunes individual weight entries using weight magnitude and input activation norm, while SVD-Attention removes whole modes of the per-head interaction matrix (Sun et al., 2024).The paper composes with WANDA as a stronger baseline.Source: §2 · p. 2
  • Head pruning removes whole heads, but a necessary head may still contain unused rank directions; reconstruction-based compressors can prioritize spectrally large but data-inactive modes (Voita et al., 2019; Hsu et al., 2022).The paper frames its data-aware mode analysis as complementary to these coarser or reconstruction-oriented approaches.Source: §2 · p. 2

3 SVD-Attention Method

The method starts from the per-head matrix governing attention logits, decomposes it, and uses token variation in query and key singular directions to identify modes that matter on data.

3.1 Setup

  • The head’s attention logits depend on inputs through M_h, and its algebraic rank is at most d_h; the method asks whether real inputs use that rank budget fully.Source: §3.1 · p. 2

3.2 Decomposition

  • SVD decomposes M_h into singular values and distinct query and key bases, with at most d_h nonzero singular values.Source: §3.2 · p. 3

3.3 Data-Aware Importance

  • A mode’s importance combines its singular value with the geometric mean of query- and key-side activation variances, because a zero variance on either side prevents token-level differentiation.A mode that adds only a near-constant offset is washed out by softmax.Source: §3.3 · p. 3

3.4 Adaptive Rank Selection and Replacement

Each head receives an adaptive retained rank based on the importance captured by its top modes; the retained factorization is exact, while inference savings depend on computing at the reduced dimensions.

3.4 Adaptive Rank Selection and Replacement

  • Each head retains the smallest rank r_h whose top-ranked modes capture at least a (1−ε) fraction of total importance, then replaces its projections with factors for those modes.The factorization is exact for retained modes; approximation error comes only from discarded modes.Source: §3.4 · p. 3

3.5 Inference Cost

  • Reducing each head from d_h to r_h lowers projection and attention-logit FLOPs by a factor of r_h/d_h; realizing wall-clock savings requires a reduced-dimension attention kernel.The decomposition is performed offline, and the reported reductions assume discarded dimensions are not computed.Source: §3.5 · p. 3

3.6 Truncation Error Bound

The section characterizes truncation error under data-aware mode selection and explains why it accepts a gap from spectral-norm optimality: data-inactive modes can contribute only a constant offset to attention logits.

3.6 Truncation Error Bound

  • When data-aware and spectral rankings agree, truncation is spectrally optimal; when they disagree, the bound’s gap reflects the difference between the retained modes and the top-rank spectral set.The gap is zero when no high-spectral mode is discarded in favor of a lower-spectral one.Source: §3.6 · p. 4
  • A mode whose token projections are nearly constant adds a near-constant logit offset that softmax absorbs, so retaining it can waste rank without changing attention.The variance term in the importance score measures each mode’s contribution beyond that offset.Source: §3.6 · p. 4
  • The data-aware Frobenius error exceeds the Eckart–Young lower bound by the spectral mass sacrificed to retain data-active modes.The section also applies the same argument to the Frobenius norm.Source: §3.6 · p. 4

3.7 SVD-Attention with RoPE

For RoPE models, SVD-Attention uses the position-independent interaction matrix as a practical diagnostic proxy, not an exact factorization of the position-dependent interaction. The section gives reasons for this approximation, tests it on Llama-2-7B, and identifies scope limits.

3.7 SVD-Attention with RoPE

  • RoPE truncation is a practical approximation: the exact factorization applies to the position-independent matrix, not the full position-dependent interaction.The truncated projections are followed by the standard RoPE operation, preserving the original model’s RoPE form.Source: §3.7 · p. 4
  • RoPE makes the effective query-key interaction depend on relative position, with the position-independent interaction as its zero-offset member.The rotations are orthogonal and reweight directions in the head space without adding rank; the diagnostic’s activation variances use layer inputs that RoPE does not alter.Source: §3.7 · pp. 4–5
  • The truncation-error bound is uniform in position because the shared RoPE rotation preserves vector length, preventing error from growing with distance.This relies on the same rotation acting on exact and truncated queries and keys in the original head coordinates.Source: §3.7 · p. 5
  • A discarded mode with nearly constant token projections can leave a distance-varying positional bias under RoPE, rather than a constant offset absorbed exactly by softmax.The bias carries no information about which tokens occupy those positions.Source: §3.7 · p. 5
  • For Llama-2-7B, flagged layers {8, . . . , 11} keep WikiText-2 perplexity within 0.11 of baseline (5.58 against 5.47) and MMLU within 0.65 points (44.65 against 45.30).Truncating non-flagged layers {4, . . . , 7} costs 2.64 perplexity (8.11) and 4.45 MMLU (40.85).Source: §3.7 · p. 5
  • A RoPE-aware extension to grouped-query attention remains future work because it must account for shared key projections and rotary coordinate structure.The evaluation described here uses standard multi-head attention in Llama-2-7B.Source: §3.7 · p. 5

3.8 Algorithm

Algorithm 1 applies SVD-Attention independently to each head and selected layer, using a calibration pass to obtain layer inputs before decomposition.

3.8 Algorithm

  • The procedure is applied independently across heads and across the layers selected by the diagnostic.Algorithm 1 summarizes the procedure for one attention layer and one head.Source: §3.8 · p. 5
  • The resulting layer inputs are the calibration data used by the per-head procedure.The procedure is then applied to the selected layers and their heads.Source: §3.8 · p. 5
  • Calibration inputs are collected in a single forward pass per example, with hooks at each attention-layer input.This pass runs once before any decomposition.Source: §3.8 · p. 5

4 Experimental Setup

The evaluation covers seven pretrained transformers across five task types, using public checkpoints without finetuning or architectural changes. It also specifies calibration data, diagnostic settings, and comparison baselines.

4 Experimental Setup

  • The study evaluates seven pretrained transformers across five task types, including language understanding, extractive QA, generation, vision classification, and dense prediction.The listed models include DistilBERT, RoBERTa-base, MiniLM, Llama-2-7B, Swin-B, ViT-B/16, and DPT-Large; all are public checkpoints evaluated without finetuning or architectural changes.Source: §4 · p. 5
  • Calibration uses 20,000 ImageNet-1K, 2,000 SST-5, 200 TruthfulQA, and 50 ADE20K examples; extractive QA uses 1,303 SQuAD 2.0 examples unless stated otherwise.For extractive QA, the layer-level answer is unchanged from 100 to 19,548 calibration examples.Source: §4 · pp. 5–6
  • The diagnostic sweeps energy threshold ϵ over {0.1, . . . , 0.6} and compares against the unmodified model, WANDA at sparsity 0.1, and the method composed with WANDA.Reported metrics include task quality and inference-FLOP change relative to the unmodified model.Source: §4 · p. 6

5 Results

Across seven models, the diagnostic finds different layer locations for redundancy by task and predicts which layer groups can be truncated while preserving quality. Its robustness checks and rank-energy sweep show that calibration choices are stable and quality degrades gradually within flagged layers.

5.1 Answering the Research Questions

  • Data-aware ranking beats singular-value-only ranking at every tested ϵ, with the Top-1 gap widening from +0.10 to +0.74 as ϵ increases from 0.1 to 0.3.On NewsQA, singular-value-only ranking keeps roughly ten more modes per head yet scores 20.73 against 59.77 for the data-aware score.Source: §5.1 · p. 6
  • Redundancy is concentrated in early-to-middle layers for classification and generation, but in late layers for dense prediction and extractive QA.The reported layer groups include DistilBERT {0, 1, 2}, Llama-2-7B {8, . . . , 11}, DPT-Large {16, . . . , 19}, and late groups in RoBERTa-base and MiniLM.Source: §5.1 · pp. 6–7
  • At ϵ = 0.3, flagged-layer truncation costs at most 1.75 F1, versus up to 50 F1 for the same number of non-flagged layers.The comparison is reported across QA settings and contrasts layer groups selected by the diagnostic with matched-size groups elsewhere in the same model.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 Layer-Wise Sweep

  • Across layer-wise sweeps, flagged windows offer the best accuracy-FLOPs tradeoff; later ViT-B truncation drops accuracy, while late DPT-Large truncation preserves baseline mIoU.The sweeps also report quality preservation for the flagged DistilBERT group and in Swin-B’s first three blocks.Source: §5.2 · pp. 7–8, Table 4 · p. 8, Table 5 · p. 8

5.3 Robustness of the Diagnostic

  • Flagged layer groups remain stable across calibration distributions, example draws, and sizes, including RoBERTa-base calibration from 100 to 19,548 examples.The reported layer-group ordering is identical across calibration sizes and across five same-size draws for DistilBERT.Source: §5.3 · pp. 8–9, Table 8 · p. 8

5.4 Rank-Energy Frontier in the Flagged Layers

  • At ϵ = 0.3, retaining mean rank 29 of 64 gives F1 82.85 against baseline 83.01, a 0.16 loss; quality degrades gradually within flagged layers.The passage contrasts this smooth frontier with up to 50 F1 lost when the same ϵ is applied to non-flagged layers.Source: §5.4 · p. 9, Table 11 · p. 9

6 Conclusion

The conclusion presents SVD-Attention as a diagnostic for attention redundancy and reports task-dependent layer patterns across the tested models. It also describes the diagnostic as a measurement tool first, since it predicts where compression fails.

6 Conclusion

  • SVD-Attention scores singular modes of each per-head interaction matrix using singular values and query- and key-side activation variances, then discards modes without finetuning.Source: §6 · p. 9
  • Redundancy concentrates in early to middle layers for sentiment, image classification, and generation, but in late layers for dense prediction and extractive question answering.The conclusion reports measurable FLOP reductions across the seven tested models and says the diagnostic composes with weight-level pruning such as WANDA.Source: §6 · p. 9
  • The diagnostic is positioned primarily as a measurement tool because it predicts where compression fails.Source: §6 · p. 9

7 Limitations

The limitations section notes that the method compresses only multi-head self-attention, so its absolute gains are modest compared with token-level methods. It also distinguishes analytic FLOP reductions from wall-clock speedups and identifies boundaries involving RoPE, grouped-query attention, and unevaluated combinations with other methods.

7 Limitations

  • Absolute compression gains are modest compared with token-level methods because the method compresses only multi-head self-attention.Source: §7 · p. 9
  • Analytic FLOP reductions do not establish wall-clock speedup; a reduced-dimension attention kernel is needed, since padding discarded dimensions back erases the saving.Source: §7 · p. 9
  • Under RoPE, the query-key interaction is a relative-position family, and the method does not claim an exact factorization of that full family.Source: §7 · pp. 9–10
  • The method does not treat grouped-query attention or evaluate interactions with quantization or attention-pattern modifications such as linear attention.The authors expect gains to compose with those methods, but report that this has not been verified empirically.Source: §7 · p. 10

Related papers

Loading 2610.34971v1…