PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Attention Patterns Exist: A Unifying Temporal Perspective Analysis

Qingyue Yang, Jie Wang, Xing Li, Yinqi Bai, Xialiang Tong, Huiling Zhen, Jianye Hao, Mingxuan Yuan, Bin Li

arXiv:2601.21709v1cs.CL

TL;DR

Existing studies identify diverse attention patterns but lack a unifying explanation of their emergence. TAPPA analyzes patterns temporally through query self-similarity, keys, and RoPE, and its q-similarity metric improves KV cache compression and LLM pruning performance.

  • Problem

    Prior studies identify diverse attention patterns, but the factors determining which pattern emerges under the same attention formulation remain unclear.

  • Method

    TAPPA explains attention patterns through temporal query self-similarity and the joint effects of queries, keys, and RoPE.

  • Results

    A TAPPA-derived metric consistently improves performance in both KV cache compression and LLM pruning.

  • Takeaways & Limitations

    Temporal query self-similarity provides a unifying explanation for predictable and unpredictable attention patterns and informs compression and pruning.

Abstract

from arXiv · show

Attention patterns play a crucial role in both training and inference of large language models (LLMs). Prior works have identified individual patterns such as retrieval heads, sink heads, and diagonal traces, yet these observations remain fragmented and lack a unifying explanation. To bridge this gap, we introduce \textbf{Temporal Attention Pattern Predictability Analysis (TAPPA), a unifying framework that explains diverse attention patterns by analyzing their underlying mathematical formulations} from a temporally continuous perspective. TAPPA both deepens the understanding of attention behavior and guides inference acceleration approaches. Specifically, TAPPA characterizes attention patterns as predictable patterns with clear regularities and unpredictable patterns that appear effectively random. Our analysis further reveals that this distinction can be explained by the degree of query self-similarity along the temporal dimension. Focusing on the predictable patterns, we further provide a detailed mathematical analysis of three representative cases through the joint effect of queries, keys, and Rotary Positional Embeddings (RoPE). We validate TAPPA by applying its insights to KV cache compression and LLM pruning tasks. Across these tasks, a simple metric motivated by TAPPA consistently improves performance over baseline methods. The code is available at https://github.com/MIRALab-USTC/LLM-TAPPA.

1 INTRODUCTION

TAPPA provides a unified temporal framework for explaining diverse attention patterns through query self-similarity, key continuity, and RoPE, and applies these insights to KV cache compression and LLM pruning. It distinguishes predictable from unpredictable patterns and analyzes re-access, sequential, seasonal, and periodic sequential behavior.

  • Motivation and framework: TAPPA addresses fragmented prior analyses by explaining attention patterns from a unified temporal perspective.It treats query vectors and attention distributions as time series and focuses on their temporal continuity.
  • Motivation and framework: Query self-similarity is the key factor distinguishing predictable attention patterns from effectively unpredictable ones.The framework relates predictable behavior to temporal continuity in queries and to the response of RoPE channels.
  • Downstream applications: TAPPA applies its temporal insights to KV cache compression and LLM pruning, using stable attention behavior to exploit redundant or predictable allocation.These downstream settings target stored states and model parameters through selective retention or pruning.
  • Pattern taxonomy: TAPPA identifies unpredictable patterns alongside re-access, sequential, and seasonal predictable patterns, while explaining periodic sequential diagonals through RoPE’s rotation period.Stable patterns emerge from the continuity of queries and keys combined with RoPE.

2 RELATED WORK

Prior work studies recurring sparse-attention patterns for inference optimization, while separate efforts examine RoPE and input-token dynamics as possible mechanisms. TAPPA unifies these perspectives by mathematically connecting temporal query self-similarity with RoPE’s rotational mechanics.

  • Pattern-based inference optimization: Prior research identifies recurring sparse-attention patterns in LLMs and uses them to guide inference optimization, including attention sinks on initial tokens.Attention sinks receive consistently high attention scores from later positions.
  • RoPE-based explanations: RoPE-focused studies link its frequency components to particular attention shapes, including diagonal or previous-token patterns associated with high frequencies.The passage also notes that other studies examine low-frequency components, but the supplied text truncates their specific findings.
  • Input-driven explanations: AttentionPredictor identifies temporal query continuity as a driver of pattern formation, while related work connects attention sinks to query-key angular relationships.The passage characterizes AttentionPredictor as lacking deep mathematical analysis and not considering RoPE’s interplay.
  • TAPPA framework: TAPPA unifies input dynamics and positional encoding by showing that temporal variations in query self-similarity combined with RoPE can mathematically explain diverse attention patterns.This framework bridges the input-feature and RoPE perspectives developed in prior work.

3 BACKGROUND

The background defines RoPE-based attention through relative-position logits and their softmax distribution. It further decomposes scores across frequency channels and frames the study around sparse attention patterns in Llama-3.1-8B and Qwen-2.5-7B.

  • Attention Mechanism: At decoding step t, attention logits compare query q_t with RoPE-rotated keys R_(t−j)k_j before softmax normalization.The rotation encodes a phase proportional to the relative position t−j.
  • Attention Mechanism: Because softmax is monotonic in the logits, their relative ordering across positions is preserved in the final attention distribution.The analysis therefore focuses on logits while extending conclusions directly to attention weights.
  • RoPE: RoPE applies channel-wise 2D rotations to embedding-dimension pairs, with channel frequency θ_m = c^(-2m/d).Large-scale models such as Llama and Qwen2 use half-split pairing for computational efficiency.
  • Decomposition View of Attention: The RoPE formulation decomposes each attention logit additively across frequency channels, where temporal shifts are modulated by channel-dependent phases.Each two-dimensional query-key pair corresponds to a frequency channel, and its contribution depends on the associated angular frequency.
  • Attention Patterns: The study examines sparse attention patterns, particularly in Llama-3.1-8B and Qwen-2.5-7B using GSM8K and AIGC datasets.RoPE’s relative-position identity makes attention depend on relative distance rather than absolute positions.

4 WHY PREDICTABLE AND UNPREDICTABLE ATTENTION PATTERNS EXIST

TAPPA offers a unifying temporal perspective that divides attention patterns into predictable and unpredictable regimes. Query self-similarity explains this distinction: high similarity supports stable patterns, whereas low similarity produces retrieval-like, unpredictable behavior.

  • Temporal attention regimes: TAPPA reframes diverse attention patterns through time-series analysis of their underlying attention mechanisms, dividing them into predictable and unpredictable categories.This contrasts with prior functionality-based and geometric-morphology analyses.
  • Temporal attention regimes: Retrieval heads exemplify unpredictable attention by jumping across the context, while predictable heads repeatedly attend to locally structured or frequently accessed tokens.Retrieval behavior supports semantic retrieval but reduces predictability; predictable behavior reflects stable patterns exploitable for inference acceleration.
  • Query self-similarity: Query self-similarity is TAPPA’s key differentiator: close successive queries yield smoothly changing attention indices, whereas drifting queries can cause unpredictable jumps despite RoPE.When query representations remain close, attention maps are more predictable; strong query drift violates inequalities defining structured patterns.
  • Query self-similarity: High q-similarity correlates with stable, predictable heads, while low q-similarity produces retrieval-like, unpredictable behavior across examined layers, heads, models, and datasets.High-continuity heads are common but not universal, and Figure 3 shows stable patterns under high similarity and random patterns under low similarity.
  • Query self-similarity: Proposition 4.1 establishes that large, non-orthogonal query changes create large logit-vector differences, making high q-similarity necessary for predictable attention.TAPPA therefore treats q-similarity as a quantitative indicator of whether an attention head is predictable or unpredictable.

5 PREDICTABLE ATTENTION PATTERNS

Predictable attention patterns arise from temporal query continuity, with re-access and sequential patterns explained by query/key self-similarity jointly with RoPE. Their regularities support KV-cache compression, while RoPE frequency determines periodicity and patterns weaken as relative distance grows.

  • Inference acceleration: Predictable temporal regularities enable inference acceleration methods such as Minference, H2O, and SnapKV to compress KV caches with little performance loss.These empirical results support temporal stability as an important signal for effective KV compression.
  • Re-access patterns: Re-access patterns repeatedly attend to a small set of keys because consecutive queries remain self-similar and low-frequency RoPE preserves query–key alignment over time.This produces vertically aligned attention weights and corresponds to the attention-sink phenomenon often seen at the first token.
  • Sequential patterns: Sequential patterns emerge when queries and keys both have high self-similarity, while RoPE’s relative-position structure propagates attention along the (+1, +1) diagonal.Empirical results show that heads with high query and key similarity exhibit clear diagonal structures.
  • Periodic sequential patterns: Periodic sequential patterns have spacing T = 2π/θm⋆, determined by the dominant RoPE channel’s angular frequency.Lower channel indices produce higher frequencies and shorter diagonal spacing, whereas higher indices produce lower frequencies and greater spacing.
  • Periodic sequential patterns: Diagonal patterns fade over long distances because query–key self-similarity diminishes with relative distance, disrupting the continuity needed to sustain them.This limitation persists even for high-frequency RoPE channels.

6 DOWNSTREAM TASKS

TAPPA-derived q-similarity guides both dynamic KV-cache budget allocation and structured whole-layer pruning. It improves information preservation or model accuracy by distinguishing retrieval likelihood from stable, predictable, and redundant patterns.

  • KV Cache Compression: KV cache compression uses q-similarity to dynamically allocate per-layer cache budgets under limited memory while maintaining model accuracy.Lower query similarity indicates a higher likelihood of scattered retrieval patterns, which require larger cache budgets.
  • KV Cache Compression: The method consistently outperforms CAKE and four other baselines across three budget settings.Allocating more cache budget to layers with lower query similarity better preserves critical information.
  • LLM Pruning: Structured pruning combines Block Influence with q-similarity to guide whole-layer removal for higher accuracy at the same compression ratio.Higher q-similarity indicates more stable, predictable patterns and suggests that a layer extracts less novel information.
  • LLM Pruning: The pruning method consistently outperforms ShortGPT across different pruning ratios and models.High-q-similarity layers exhibit greater redundancy and can therefore be pruned with minimal impact.

7 CONCLUSION

The paper introduces TAPPA as a unifying framework for analyzing diverse LLM attention patterns. It explains predictable versus unpredictable patterns through temporal query self-similarity and attributes stable predictable patterns to query-key continuity combined with RoPE.

  • TAPPA systematically analyzes diverse attention patterns within large language models.
  • Temporal self-similarity of queries explains the distinction between predictable and unpredictable attention patterns.
  • Stable, predictable patterns arise from the combined effects of query-key continuity and Rotary Positional Embeddings (RoPE).

ETHICS STATEMENT

The research uses no personally identifiable information and relies on publicly available datasets with research-permitting licenses. It follows ICLR ethical principles and discloses limited LLM use for text polishing while retaining human responsibility for contributions and validation.

  • ETHICS STATEMENT: The research does not involve any personally identifiable information.
  • ETHICS STATEMENT: All datasets are publicly available, widely adopted, and licensed for research use.
  • ETHICS STATEMENT: The work adheres to ICLR Code of Ethics principles of fairness, transparency, and responsible AI research.
  • ETHICS STATEMENT: LLMs were used for text polishing, while conceptual contributions and validation remain the researchers’ responsibility.

REPRODUCIBILITY STATEMENT · A PROOF OF UNPREDICTABLE PATTERN · B PROOF OF RE-ACCESS PATTERN

The paper supports reproducibility through publicly available resources and proves conditions for both unpredictable and vertically stable attention patterns. The proofs connect query changes, RoPE, key geometry, channel dominance, and low frequencies to attention-logit behavior.

  • REPRODUCIBILITY STATEMENT: The authors will release source code, configuration files, preprocessing scripts, dataset versions, and reproduction instructions for the main experiments.All datasets are publicly available, with exact preprocessing steps specified.
  • A PROOF OF UNPREDICTABLE PATTERN: A large query change that is nonorthogonal to rotated keys necessarily produces a large difference between consecutive logit vectors.The proposition formalizes unpredictable patterns through query increments and their alignment with RoPE-rotated keys.
  • A PROOF OF UNPREDICTABLE PATTERN: Writing the query increment as Δq separates each logit change into a query-difference term and a RoPE-difference term.The proof denotes these contributions T1,j and T2,j.
  • A PROOF OF UNPREDICTABLE PATTERN: Orthogonality of RoPE rotations bounds the RoPE-difference contribution when key norms are bounded.The operator-norm argument gives a factor-of-two bound for the difference between consecutive rotations.
  • A PROOF OF UNPREDICTABLE PATTERN: If the query increment has significant projection onto at least one rotated key with a minimum norm, the query-difference contribution is bounded below.The proof combines this lower bound with the RoPE term using the reverse triangle inequality.
  • A PROOF OF UNPREDICTABLE PATTERN: The proposition concludes with constants c1 = αBk,min and c2 = 2∥qt∥BK controlling the resulting logit-difference bound.The infinity norm then converts the componentwise bound into a vector-level statement.
  • B PROOF OF RE-ACCESS PATTERN: Under continuous query evolution, fixed keys, a dominant low-frequency channel, and channel-wise decomposition, per-key attention-logit differences are uniformly small.As ε → 0 and the dominant RoPE frequency becomes small, the theorem’s uniform bound can be made arbitrarily small.

C PROOF OF SEQUENTIAL PATTERN · D PROOF OF PERIODIC SEQUENTIAL PATTERN

The sequential-pattern proof shows that high query/key self-similarity under RoPE makes attention logits approximately shift-invariant along the (+1, +1) diagonal. The periodic extension shows that a dominant RoPE channel determines the spacing between repeated attention diagonals through its rotation frequency.

  • C PROOF OF SEQUENTIAL PATTERN: Under RoPE, high self-similarity of queries and keys is the central hypothesis for deriving sequential attention patterns.The theorem assumes sufficiently small self-similarity error ε.
  • C PROOF OF SEQUENTIAL PATTERN: Consequently, attention logits are approximately shift-invariant along the (+1, +1) diagonal, producing a sequential diagonal structure.The stated bound limits the shift error to Cε.
  • C PROOF OF SEQUENTIAL PATTERN: The RoPE relative-position identity preserves the rotation under a simultaneous (+1, +1) shift of query and key positions.This follows from R(t+1)−(i+1) = R(t−i).
  • D PROOF OF PERIODIC SEQUENTIAL PATTERN: A periodic sequential pattern is analyzed when the corresponding key contains a massive channel at index m⋆.The dominant channel controls the periodic spacing.
  • D PROOF OF PERIODIC SEQUENTIAL PATTERN: The interval between adjacent attention diagonals is exactly determined by the rotation frequency of the dominant RoPE channel.Using θ_m = c^−2m/d and m = m⋆ links the spacing to channel m⋆.
  • D PROOF OF PERIODIC SEQUENTIAL PATTERN: When the dominant channel governs the logits, slowly varying query and key components allow the attention pattern to repeat across diagonals.The proof identifies repetition through the dominant channel’s phase behavior.

E PROOF OF SEASONAL PATTERN · F EMPIRICAL SUPPORT · F.1 EMPIRICAL VALIDATION OF THE DOMINANT-CHANNEL ASSUMPTION OF RE-ACCESS PATTERN

Theorem 5.4 shows that approximately periodic queries and keys, combined with near resonance to a dominant RoPE frequency, produce approximately seasonal attention logits with period L. Empirical spectrum analysis supports the related single-low-frequency-channel assumption for re-access heads, with one channel contributing about 51% of attention mass.

  • E PROOF OF SEASONAL PATTERN: If queries and keys are approximately periodic with interval L and L is near resonance with the dominant RoPE frequency, attention logits approximately repeat with period L.The theorem assumes sufficiently small periodicity errors εq and εk and resonance error δ.
  • E PROOF OF SEASONAL PATTERN: The proof uses channel-wise RoPE decomposition and orthogonality to express each channel’s contribution as a rotated query-key dot product.This isolates the dominant channel’s variation under temporal shifts.
  • E PROOF OF SEASONAL PATTERN: Near resonance makes the dominant-channel RoPE rotation close to identity, while non-dominant channels contribute uniformly bounded residual variation absorbed into the theorem’s constants.The argument applies symmetrically along query and key dimensions, yielding the stated seasonal repetition.
  • E PROOF OF SEASONAL PATTERN: The resulting bounds show that the dominant component of attention logits approximately repeats every L steps along both query and key dimensions, producing a seasonal pattern.This is the concluding implication of the proof.
  • F EMPIRICAL SUPPORT: Spectrum analysis directly tests the single-channel assumption for a head whose attention map displays a clear re-access pattern.The analysis decomposes query and key vectors into M = D/2 RoPE channels and normalizes their aggregated contributions over decoding steps.
  • F.1 EMPIRICAL VALIDATION OF THE DOMINANT-CHANNEL ASSUMPTION OF RE-ACCESS PATTERN: About 51% of total attention mass is carried by one low-frequency channel m∗, while the remaining channels form a much smaller long tail.The channel weights are measured at the attention sink corresponding to the re-access stripe.
  • F.1 EMPIRICAL VALIDATION OF THE DOMINANT-CHANNEL ASSUMPTION OF RE-ACCESS PATTERN: The concentrated spectrum provides direct empirical evidence that the analyzed re-access heads’ attention logits are governed by a single low-frequency channel.Figure 6(b) visualizes the normalized contribution of each RoPE channel.

F.2 DISENTANGLING QUERY DYNAMICS AND ROPE IN SEQUENTIAL PATTERN. … J COMPARISON WITH EXPECTED ATTENTION

The paper validates TAPPA by disentangling query dynamics from RoPE, characterizing q-similarity distributions, and applying the resulting signal to compression, pruning, ablations, and baseline comparisons. Across experiments, q-similarity supports stable, low-overhead allocation decisions and improves performance under constrained budgets.

  • F.2 DISENTANGLING QUERY DYNAMICS AND ROPE IN SEQUENTIAL PATTERN.: Only smooth q-dynamics combined with RoPE produces a clean sequential attention pattern; removing RoPE roughens it, while perturbing queries adds unpredictable spikes.The ablation head has consecutive-query cosine similarity approximately 0.99; perturbation reduces it to 0.97.
  • F.3 Q-SIMILARITY DISTRIBUTION: Average q-similarity is around 0.80 for Llama-3.1 and 0.86 for Qwen2.5-7B, with model-specific yet layer-structured distributions that motivate layer-wise calibration.For each model, heads within the same layer cluster closely, while average q-similarity differs substantially across layers.
  • G EXPERIMENT DETAILS: TAPPA adjusts CAKE’s layer preference using q-similarity, allocating larger budgets to layers with lower similarity and evaluating Llama-3.1-8B and Qwen2.5-7B on 16 LongBench tasks.Baselines include StreamingLLM, H2O, SnapKV, PyramidKV, and CAKE; pruning experiments use PG19 calibration and common-sense reasoning datasets.
  • G.2 DETAILS FOR LLM PRUNING: The pruning proxy incorporates 1−q into Block Influence, ranks layers by adjusted scores, and removes the lowest-scoring layers, with pruning evaluated across three LLMs.The calibration set is PG19, and downstream tasks are PIQA, HellaSwag, WinoGrande, and ARC-easy.
  • G.2.1 COMPARISON WITH ADDITIONAL STRUCTURED PRUNING BASELINES: The method achieves the best average performance at a higher pruning ratio than additional structured-pruning baselines while maintaining strong WSC and RACE-H accuracy.Compared baselines are LLMPruner, SliceGPT, LaCo, and ShortGPT; removed layers are listed for benchmark models.
  • H.3 COMPUTATIONAL OVERHEAD OF Q-SIMILARITY COMPUTATION: Q-similarity adds below 0.2 ms per-layer latency and about 8.69 MB memory, with effectively context-length-independent computation from a fixed window.Measurements use Llama-3.1-8B, window size 32, and context lengths up to 32K tokens; CAKE stores per-token attention statistics.
Loading 2601.21709v1…