Not All Attention Is Equal: A Quantitative Survey of the EEI Trade-off

1 Introduction

• The survey follows attention from Transformer sequence modeling through vision adaptations and into hybrid architectures combining adaptive routing with attention–SSM convergence.Its coverage includes Bahdanau–Luong alignment, Transformers, ViT, DETR, Swin, sparse and linear attention, FlashAttention, state-space alternatives, and interpretability.Source: §1 · pp. 1, 3, Figure 1 · p. 2
• Quadratic pairwise attention costs O(L^2) time and memory, making long-context inference a central motivation for efficiency innovations.The cost arises because scaled dot-product attention computes similarities between every pair of tokens; 128K-token contexts can make this cost dominate inference budgets.Source: §1 · p. 1

Survey Methodology

• The literature search covered four databases through 31 May 2026, but reconstructed counts and screening records are not independently reproducible.Approximately 1,265 records were retrieved, 1,123 unique records screened, and 94 retained after title/abstract screening; the search process was not preregistered and underlying exports and logs were not retained.Source: Survey Methodology · p. 3, Figure 2 · p. 4
• The review retained 63 papers for narrative synthesis and selected 21 methods for an EEI panel spanning six taxonomy families and multiple architectural levels.The panel is purposive rather than statistically representative, and its comparisons are most meaningful within architectural layers rather than as a head-to-head performance ranking.Source: Survey Methodology · p. 3, Table 2 · p. 3

EEI axes

• EEI defines Efficiency through complexity, hardware overhead, and accelerator compatibility; Expressiveness through interaction capacity, retrieval, induction, positional sensitivity, and exactness; and Interpretability through observability, tooling, theory, and causal validation.The interpretability axis is explicitly an aspirational, tooling-dependent construct, and observable attention weights are not treated as faithful causal explanations.Source: §2 · pp. 4–5

Trade-off structure

• The score map exposes a frontier: full attention maximizes expressiveness but scales poorly, Mamba and Performer improve efficiency with lower expressiveness, and Focus and DashAttention occupy an intermediate near-full-expressiveness zone.No surveyed method combines the strongest observed levels on all three axes; progress is framed as expanding the frontier rather than finding a universal optimum.Source: §2 · p. 5, Figure 3 · p. 6

Exploratory quantitative analysis

• The exploratory Expressiveness–LRA association is positive but unconfirmed: ρ = 0.54 (p = 0.21, n = 7), or ρ = 0.45 (p = 0.27, n = 8) after adding RWKV’s non-official-protocol result.Efficiency correlations are omitted because speedups use incompatible scopes, baselines, hardware, sequence lengths, and precisions; interpretability is not benchmarked quantitatively.Source: §2 · pp. 8–9, Table 5 · p. 10

Formal EEI score

• Under equal weights, FlashAttention-2/3, PagedAttention, DashAttention, Swin Transformer, and Mamba-2 tie at EEI = 8.0; retrieval-focused weights favor FlashAttention and PagedAttention at EEI = 8.8, while deployment-focused weights favor Mamba-2 at EEI = 8.8.The ranking shifts reflect the methods’ differing strengths, and the composite remains exploratory because the axes are ordinal rather than commensurate cardinal quantities.Source: Table 4 · p. 7, §2 · p. 9

Score consistency and limitations

• Across 200,000 perturbation samples, 67% of equal-weight cases changed a method’s rank by more than one position, with mid-tier methods showing roughly 65–80% instability.The protocol independently samples each axis from {score−1, score, score+1}, clips to [1,10], and recomputes rankings under all three weight configurations.Source: §2 · p. 11
• A rank-matched null reproduces the inverted-U stability profile, so extreme-rank stability largely reflects rank censoring; consequently, the scores support coarse top-, mid-, and bottom-tier groupings rather than fine-grained comparisons.Claims that one method outperforms another within the 7.0–7.7 composite range are unsupported under the single-rater perturbation model, while dense ties increase instability above the null expectation.Source: §2 · p. 11

Proposed EEI Reporting Protocol

• The proposed EEI protocol standardizes efficiency reporting across GPU generations and sequence lengths, requiring conventional and optimized exact-attention baselines with fully specified implementations.It requires reporting OOM when full attention exceeds memory and normalizing throughput against a specified exact full-attention implementation under identical conditions.Source: §2 · pp. 12–13

EEI Axes

• Expressiveness combines retrieval, language-modeling, and LRA metrics, while interpretability remains four separate fields: component evidence, tooling, theory, and causal validation.These expressiveness measures are complementary rather than interchangeable, and interpretability fields must not be collapsed without validation.Source: §2 · pp. 12–13

EEI Frontier Pattern

• The EEI Frontier Pattern is a rubric-level consistency check: exact methods can improve efficiency beyond E ≥6 without expressiveness loss, whereas approximate methods are scored with degradation unless input-adaptive.The pattern is not an independently verified law because approximation methods are penalized on Ex by construction.Source: §2 · p. 13
• Adaptive sparsity reaches E = 7–8 with Ex = 9, while fixed sparsity reaches Ex = 7–8 and score or kernel approximation at E ≥7 receives Ex = 6 under the rubric.Focus and DashAttention use learned input-dependent routing; exact methods such as FlashAttention, PagedAttention, and Ring Attention retain Ex = 9–10.Source: §2 · pp. 13–14

Limitations of EEI

• The framework remains subjective and unevenly grounded: integer scores depend on one rater, interpretability lacks standardized benchmarks, and future axes such as energy, robustness, and training stability may be needed.EEI is therefore a conceptual vocabulary for organizing trade-offs and guiding method selection, not a definitive replacement for task-specific empirical evaluation.Source: §2 · p. 14
• EEI rankings are sensitive to weights and ordinal conventions: Mamba-2 ranks first under deployment weights at EEI = 8.8 but ties at 8.0 under retrieval weights, while Mamba falls near rank 12 with EEI = 7.2.Axis variance also makes effective contributions unequal, with deployment weights assigning approximately 87% of composite variance to E and retrieval weights approximately 86% to Ex.Source: §2 · pp. 12, 14

3.1 The Bahdanau–Luong Heritage

• Bahdanau et al. (2015) introduced nonlinear additive alignment, while Luong et al. (2015) reduced cost with multiplicative dot-product attention and localized windowed alignment.Additive attention is more expressive but heavier, whereas multiplicative attention is simpler, faster, and suited to hardware-optimized matrix multiplication.Source: §3.1 · pp. 14–15, Figure 4 · p. 16

3.2 Scaled Dot-Product and Multi-Head Attention

• Scaled dot-product multi-head attention became the canonical formulation, using 1/√d_k scaling for stable gradients and multiple heads for distinct representation subspaces.KV caching enables efficient autoregressive decoding but requires O(h·L·d_k) memory; MQA and GQA reduce this footprint with an expressiveness trade-off.Source: §3.2 · p. 15

3.3 Positional Encoding

• RoPE (Su et al., 2021) became the dominant modern positional encoding, while ALiBi (Press et al., 2022) provides length extrapolation through fixed distance-proportional score biases.RoPE interpolation methods such as PI (Chen et al., 2023) and YaRN (Peng et al., 2023b) extend context windows but do not reduce quadratic attention cost.Source: §3.3 · pp. 15, 17

3.4 Causal vs. Bidirectional Architectures

• Attention patterns define architectural roles: bidirectional encoders support contextual understanding, causal decoders enable left-to-right generation, and encoder-decoders connect representations through cross-attention.Bidirectional and causal self-attention remain O(L^2), while causal decoding benefits from KV caching and cross-attention costs O(L_decL_encd).Source: §3.4 · p. 17

4 Attention in Computer Vision

• The section is illustrative rather than exhaustive, covering ViT, DETR, Swin, and selected vision-language patterns while leaving several methods without dedicated subsections or EEI scores.Excluded examples include SAM (Kirillov et al., 2023), Perceiver IO (Jaegle et al., 2022), DeiT (Touvron et al., 2021), DINOv2 (Oquab et al., 2024), VideoMAE (Tong et al., 2022), and TimeSformer (Bertasius et al., 2021).Source: §4 · p. 18, §4.4 · p. 20

4.1 Vision Transformer

• ViT tokenizes image patches for standard bidirectional self-attention, matching or exceeding CNNs with sufficient data but requiring later efficiency research for high-resolution images.A 224×224 image yields 196 tokens, whereas 1024×1024 yields 4096 tokens and approximately 440× more attention computation.Source: §4.1 · p. 18

4.2 DETR: Object Queries as Learned Attention

• DETR replaces hand-crafted detection components with learned object queries, cross-attention, and Hungarian matching for end-to-end object prediction.Queries compete through decoder self-attention and specialize through cross-attention to encoder image features, establishing learned slots extendable to video, segmentation, and multimodal tasks.Source: §4.2 · pp. 18–19

4.3 Swin Transformer and Hierarchical Attention for Vision

• Swin Transformer uses shifted local windows and hierarchical downsampling to provide efficient attention, multi-scale features, and strong vision performance without ViT-scale pretraining data.Its complexity is O(HpWpM^2d), and fixed-size windows make scaling effectively linear in pixel count for high-resolution inputs.Source: §4.3 · p. 19

4.4 Vision-Language Cross-Attention

• Vision-language architectures align modalities either through CLIP-style dual-encoder embeddings or Flamingo-style cross-attention that injects visual features into language decoders.Because high-resolution images produce thousands of visual tokens, Perceiver-style latent queries compress visual information before cross-attention.Source: §4.4 · pp. 19–20

5 Efficiency Innovations

• O(L^2) complexity is the central bottleneck for scaling Transformers to long sequences, motivating seven efficiency/architecture families with distinct efficiency, quality, and hardware-compatibility trade-offs.Table 8 presents the seven-family taxonomy, while Figure 7 summarizes it visually.Source: §5 · p. 20

Fixed Sparse Patterns

• Sparse Transformer introduced strided and fixed attention patterns that connect nearby tokens with every k-th token, enabling lower-cost long-range information propagation.The approach was an early replacement for dense attention with predetermined sparse patterns.Source: §5.1 · p. 20
• Fixed patterns reduce cost and guarantee predictable memory usage without learning overhead, but their static connectivity can miss important content-dependent interactions.The limitation applies when relevant tokens fall outside predetermined connections.Source: §5.1 · p. 20
• Longformer combines fixed local windows with task-specific global tokens, reducing complexity to O(Lw) while preserving a global view for critical tokens.It recovered most quality loss from pure local attention on document-level NLP tasks.Source: §5.1 · p. 20
• BigBird combines random, window, and global attention to approximate a full Turing machine while maintaining O(L) complexity.Its construction establishes theoretical expressiveness results under the stated assumptions.Source: §5.1 · p. 20
• Fixed sparse patterns struggle with heterogeneous attention distributions, and deployments often favor Longformer-style learned global tokens because theoretical guarantees offer limited practical guidance.This gap shows that sparsity-pattern selection matters beyond asymptotic complexity alone.Source: §5.1 · p. 20

5.2 Content-Dependent Sparsity

• Reformer (Kitaev et al., 2020) uses locality-sensitive hashing to attend within shared query-key buckets, reducing complexity from O(L2) to O(LlogL) under uniform bucket distribution.The shared hash form makes query-key comparisons well-defined without additional parameters.Source: §5.2 · pp. 20, 22
• Reformer’s non-differentiable hashing prevents direct gradient propagation and requires multiple hashing rounds with averaging, motivating differentiable sparsity approaches.The routing objective can therefore mismatch the task objective.Source: §5.2 · p. 22
• Routing Transformer (Roy et al., 2021) uses online k-means with momentum to learn query-key clusters, enabling stable, highly non-local attention with head specialization.Different heads attend to very different parts of the input.Source: §5.2 · p. 22
• Both methods preserve model quality at reduced complexity, but hard routing creates optimization challenges and routing overhead can diminish wall-clock speedups.Hashing and clustering overhead can consume substantial theoretical savings, making these methods competitive with full attention mainly at longer sequence lengths.Source: §5.2 · p. 22

5.3 Linear Attention

• Kernelized linear attention reduces computation to O(Ld^2) by reusing key-value and normalization aggregates across queries.The noncausal form suits bidirectional attention, while causal attention computes the aggregates incrementally.Source: §5.3 · p. 22
• Performer’s FAVOR+ uses positive orthogonal random features to approximate softmax attention, with uniform convergence requiring feature count dependent on d and precision but not L.The approximation assumes queries and keys lie within a radius-R ball; fixed feature budgets can still incur high constants.Source: §5.3 · p. 22
• Practical overhead and approximation error can erase linear attention’s advantage over optimized standard attention, especially below 8K tokens.Kernel feature computation contributes a substantial constant factor to the theoretical O(L) complexity.Source: §5.3 · p. 22
• Kernel approximations can dilute attention concentration and impair precise token selection, while fixed-state alternatives struggle on multi-query associative recall.These limitations affect copying, retrieval, entity extraction, and code completion, indicating an expressiveness ceiling beyond approximation error alone.Source: §5.3 · p. 23
• GLA and Griffin restore sharper token selection through data-dependent gating, with Griffin additionally combining gated recurrences and sliding-window local attention.Both have shown strong results in large-scale language modeling.Source: §5.3 · p. 23

FlashAttention

• FlashAttention uses SRAM-sized tiling and online softmax to compute exact attention while minimizing HBM reads and writes, with backward recomputation avoiding full attention-matrix storage.FlashAttention-2 further improves parallelism and work partitioning across row and column blocks, warps, and tensor cores.Source: §5.4 · p. 23
• 2–3× wall-clock speedups over standard PyTorch attention accompany exact O(L2) compute, linear memory, and reduced HBM traffic from tiled execution.FlashAttention’s traffic is Θ(L2d2M−1) with SRAM size M; quadratic compute remains, but the constant is smaller than materializing the attention matrix.Source: §5.4 · p. 23
• FlashAttention-3 adds asynchronous HBM–SRAM overlap and low-precision computation, achieving 1.5–2.0× speedups over FlashAttention-2 in FP16 and up to 740 TFLOPs/s.FP8 reaches close to 1.2 PFLOPs/s on H100, while block-wise quantization reduces kernel-level RMSE 2.6× versus baseline FP8 attention.Source: §5.4 · p. 23

Systems extensions

• PagedAttention, vLLM, and StreamingLLM extend systems-level efficiency to serving by reducing KV-cache waste, enabling production deployment, and stabilizing streaming generation with bounded memory.Ring Attention distributes computation across devices while overlapping communication for sequences exceeding single-device memory capacity, including 100K+ tokens.Source: §5.4 · p. 23

Limitations

• FlashAttention’s gains depend on hardware-specific SRAM and FP8 support, while persistent O(L2) compute eventually becomes the bottleneck and motivates sub-quadratic alternatives.Advantages are strongest on modern Hopper and Blackwell GPUs and more modest on older architectures.Source: §5.4 · p. 24

Hyena

• Hyena uses implicit long convolutions to achieve attention-like language-modeling quality at 2K with 20% less training compute and 100× FlashAttention speedup at 64K.Its sub-quadratic design replaces structured state matrices with feedforward-parameterized convolutions.Source: §5.5 · p. 24

Mamba

• Mamba matches or exceeds Transformer language-modeling quality while providing O(L) time and O(1) inference memory, though precise retrieval evidence remains limited.Evaluated configurations include scaling-law studies up to 1B parameters and a trained 3B model; Mamba-3B exceeds Pythia-7B on several tasks, while 6.9B is throughput-only.Source: §5.5 · p. 24

Mamba-2 and SSD

• Mamba-2’s SSD framework expresses selective SSMs as structured linear attention, placing attention and SSMs on a spectrum of data-dependent token-mixing structures.Attention uses an unstructured matrix, whereas SSMs use a highly structured semiseparable matrix; RetNet previously introduced the parallel/recurrent duality.Source: §5.5 · p. 24

xLSTM

• xLSTM combines exponential gating and matrix memory with O(L) recurrent inference, achieving competitive language-modeling perplexity against Transformers and Mamba.Together with SSMs, it broadens sub-quadratic sequence modeling beyond the attention–SSM spectrum.Source: §5.5 · p. 24

Mamba-3

• Mamba-3 adds exponential-trapezoidal discretization, complex-valued updates, and MIMO to improve retrieval and state tracking without increasing inference latency.It is discussed historically but excluded from EEI scoring because its evaluations lacked comparable efficiency and interpretability evidence at the survey cutoff.Source: §5.5 · pp. 24–25

Limitations and hybrids

• SSMs achieve competitive aggregate language modeling but have limited direct evidence for exact token retrieval, making attention–SSM hybrids a practical compromise whose optimal ratio remains task-dependent.Mamba lacks original Long Range Arena results, and broader public evaluations are not directly comparable to its task suite.Source: §5.5 · pp. 24–25

• Focus partitions tokens into learned semantic groups while retaining local-window attention, adding composable routing to frozen pretrained models with zero reported benchmark degradation and 2× H100 speedup.Its routing overhead is 148K parameters for the reported GPT-2 124M configuration, about 0.12%, and as low as 0.015% at 70B scale.Source: §5.6 · p. 25
• DashAttention replaces fixed hard top-k routing with α-entmax, whose variable support enables differentiable block selection followed by fine-grained softmax attention.It achieves 75% sparsity, up to 93.75% at higher ratios, with comparable accuracy to full attention and up to 3.3× kernel speedup over FlashAttention-3.Source: §5.6 · p. 25
• Learned routing directs each query to selected tokens through lightweight clustering or attention, suggesting better quality-efficiency Pareto frontiers than fixed sparsity or linear attention.Native Sparse Attention (NSA; Yuan et al., 2025) is identified as a prominent hardware-native reference design for this adaptive frontier.Source: §5.6 · p. 25
• Generalization beyond training-context lengths, performance at 1M+ tokens, and favorable amortization in batched serving remain unresolved empirical questions, including interaction with hardware optimizations.These limitations qualify the early evidence for learned routing’s efficiency advantages.Source: §5.6 · p. 25
• Adaptive methods incur routing overhead that must be amortized against reduced attention computation, including Focus’s Na(dmodel + K)dg parameters and DashAttention’s separate chunk-level routing pass.DashAttention’s speedup comes primarily from its high sparsity ratio rather than eliminating computation altogether.Source: §5.6 · p. 25

Deployment guidance

• For 2025–2026 production inference, FlashAttention-3 with GQA is a strong default, while SSMs or adaptive sparsity suit extreme-length regimes where quadratic computation is prohibitive.Selection depends primarily on sequence length, hardware generation, and quality requirements; retrieval-heavy tasks require exact attention or carefully designed sparsity.Source: §5.7 · pp. 25, 27

Quantization

• Quantization requires distinct safeguards: W8A8 preserves near-lossless standard attention quality, whereas low-bit KV caches need per-channel keys, per-token values, and a full-precision recent-token window.FlashAttention-3 FP8 kernels reach roughly 1.6× FP16 throughput and reduce FP8 numerical error by up to 2.6× versus baseline FP8 attention.Source: §5.7 · p. 27

Scope and limitations

• Inference efficiency is the guidance’s focus, so training-stage decisions require separate measurements of gradient memory, recomputation, mixed precision, optimizer state, and sharding.Inference-efficient methods such as Mamba’s O(L) recurrence can have different training profiles, and practitioners should consult original training benchmarks.Source: §5.7 · pp. 27–28

Cross-study caveats

• Published speedups and benchmark values are approximate reference points rather than directly comparable rankings because hardware, sequence length, baselines, models, datasets, and protocols differ.Publication bias likely makes reported gains upward-biased, while missing LRA results limit common comparison across newer efficiency methods.Source: §5.7 · pp. 27, 28, 32

Quantitative synthesis

• Efficiency and expressiveness show no significant monotonic relationship across 21 methods, with Spearman ρ ≈−0.12, p ≈0.61, n = 21.Exact IO-aware methods achieve Ex ≥8 at E ≥7, while adaptive methods are assigned Ex ≥9 at E ≥7 in the EEI rubric.Source: §5.7 · pp. 27–28

Chronological trends

• Attention research evolved from approximation, through IO-aware quality-preserving co-design, toward adaptive selection that learns where to apply exact attention.Whether adaptive methods dominate fixed sparsity at matched efficiency remains unresolved without a common benchmark.Source: §5.7 · p. 32

Generalization 1

• Exact full-attention transformations preserve quality relative to approximation because they compute full softmax attention, whereas GQA/MQA can incur small costs from shared K/V parameterization.This applies to global, tiled, distributed, and paged exact computation; quality differences otherwise reflect training, scale, or hyperparameters.Source: Cross-Family Empirical Generalizations · p. 32

Generalization 2

• LRA averages of 51.41, 50.67, and 51.36 for Performer, Linformer, and Reformer trail the Transformer’s 54.39 despite competitive language-modeling perplexities.These methods provide O(L) or expected O(LlogL) complexity, but fixed approximation or hashing budgets can compound error with sequence length.Source: Cross-Family Empirical Generalizations · pp. 32, 35

Generalization 3

• RWKV reports an LRA average of 72.07 versus the official Transformer’s 54.39, while Mamba reports Pile validation PPL 10.56 at 130M scale without comparable retrieval evidence.The RWKV comparison is not on Tay et al.’s (2021) official protocol; SSD’s O(L) scan kernel is reported 2–8× faster than Mamba’s fused associative scan (Dao & Gu, 2024).Source: Cross-Family Empirical Generalizations · p. 32

Generalization 4

• FlashAttention-2 is approximately 2× faster than FlashAttention-1, and FlashAttention-3 adds 1.5–2× on H100, while preserving identical numerical output in supported paths.IO-aware methods improve efficiency by reducing HBM traffic without approximation-induced quality loss; FP8 introduces finite-precision differences, and GQA/MQA trade memory savings for small quality costs.Source: Cross-Family Empirical Generalizations · pp. 32–33

Generalization 5

• Adaptive sparsity has not been shown superior to fixed patterns on a common benchmark, although Focus and DashAttention report comparable quality to full attention in their own suites.This remains a hypothesis based on two unverified preprints, whereas Longformer and BigBird are established and reproduced.Source: Cross-Family Empirical Generalizations · pp. 33, 35

Scope and reproducibility

• The five regularities are provisional because each rests on n = 2–6 methods and heterogeneous benchmarks, while custom-kernel and adaptive-sparsity methods receive limited independent reproduction.The survey identifies the publication–verification gap as a central reproducibility challenge, with adaptive methods lacking independent implementations outside the original repositories.Source: Cross-Family Empirical Generalizations · pp. 32–34

6.1 Mechanistic Interpretability

• Induction heads implement in-context copying, while syntactically specialized heads and recurring circuits provide converging interpretability evidence strongest in small models.Induction heads copy the token following a previous occurrence of the same token, and their emergence coincides with a sharp increase in in-context learning ability; extrapolation to frontier-scale models is not established.Source: §6.1 · p. 35, §6.4 · p. 36

6.2 Expressiveness Limits

• Transformers face expressiveness ceilings on parity, periodic finite-state languages, and deep compositional reasoning because content-dependent attention and bounded depth and width limit required computations.Under Lipschitz-continuity and bounded-activation assumptions, self-attention cannot compute parity or recognize periodic finite-state languages at arbitrary lengths; finite-feature softmax approximations also generally cannot exactly reproduce the full kernel at finite width.Source: §6.2 · p. 35, §6.4 · p. 36

6.3 Superposition and Sparse Autoencoders

• Superposition compresses more features than dimensions into overlapping directions, complicating attention analysis, while sparse autoencoders recover interpretable features that drive individual heads.Superposition emerges as feature sparsity increases and organizes features into geometric structures; sparse autoencoders disentangle otherwise polysemantic hidden representations.Source: §6.3 · p. 36

6.4 Attention-SSM Duality

• SSD expresses attention and state-space sequence mappings as structured matrix operations, framing a continuum from flexible O(L^2) attention to efficient O(L) SSMs and motivating task-specialized hybrids.Attention layers are suited to precise retrieval, copying, and long-range interactions, whereas SSM layers support compression and streaming; the framework provides algebraic equivalence and expressiveness guarantees, not a task-level guarantee about layer allocation.Source: §6.4 · p. 36

7.1 Unified Efficiency Benchmark

• A unified efficiency benchmark should standardize hardware, sequence lengths, tasks, metrics, training forward passes, and inference decoding to enable fair regime-specific comparisons.Existing speedups vary with GPU generation, sequence length, batch size, and precision; quality benchmarks such as LongBench, RULER, and HELMET do not standardize efficiency.Source: §7.1 · pp. 36–37

7.2 Learned Routing Policies for Hybrid Architectures

• Learned routing should adapt attention–SSM allocation by layer and input, while transferring load-balancing and capacity insights from MoE gating to sparse-attention routing.Early layers may favor attention, middle layers hybrid mechanisms, and late layers SSMs; the broader attention–MoE design space remains largely unexplored.Source: §7.2 · p. 37, §7.3 · p. 38

7.4 Length Generalization in Sub-Quadratic Models

• Sub-quadratic models often fail beyond training lengths: linear attention accumulates approximation error, sparse patterns miss unseen distances, and SSM states may not extrapolate to longer dependencies.A model trained with 8K-token windows may degrade catastrophically at 128K tokens, motivating extrapolable positional encodings, length-adaptive routing, and curriculum learning.Source: §7.4 · p. 38

7.5 Scaling Mechanistic Interpretability

• Scaling mechanistic interpretability to 70B+ models is limited by combinatorial circuit composition, more severe superposition, and diffuse attribution across many heads and layers.Activation patching and attribution patching show promise, but their reliability and completeness at frontier scale remain open questions.Source: §7.5 · p. 38

7.6 Explicit Research Gap Analysis

• The five central research gaps are retrieval-efficient sub-quadratic attention, reliable sparse attention at 1M+ tokens, theory predicting when sparsity succeeds, task-adaptive attention–SSM routing, and interpretability-guided efficiency.No surveyed method simultaneously combines arbitrary-position exact retrieval, sub-quadratic complexity, and hardware efficiency; interpretability-guided head-level routing remains almost entirely unexplored.Source: §7.6 · pp. 38–39

7.7 Reproducibility and Deployment Considerations

• Reproducible evaluation requires at least two GPU generations, end-to-end throughput, permissively licensed implementations, standardized quality benchmarks, and quality measurements across sequence lengths.Reported performance depends on GPU models, software settings, and custom kernels, while deployment constraints differ across FlashAttention, PagedAttention, Mamba, and KV-cache compression.Source: §7.7 · p. 39

7.8 Threats to Validity

• Survey conclusions are limited by publication and preprint bias, inconsistent cross-paper protocols, subjective EEI scores, rapid frontier change, and NVIDIA-dominated hardware evidence.The EEI panel captures a mid-2026 snapshot, and rankings should not be generalized to non-NVIDIA hardware without independent validation.Source: §7.8 · pp. 39–40

7.9 Outlook Toward 2030

• Future attention systems may combine exact retrieval with sub-quadratic computation, learned routing, hybrid attention–SSM designs, diverse hardware, standardized interpretability benchmarks, and hardware-aware kernels.These are forecasts or hypotheses: learned-routing advantages remain unverified on common benchmarks, and the 2015–2026 timeline shows accelerating innovation toward hybrid and systems-oriented methods.Source: §7.9 · pp. 40–41

Design Principles for Future Attention Architectures

• HBM traffic can dominate wall-clock speed, so attention designs should evaluate memory movement alongside asymptotic complexity, especially for GPU contexts below 128K.FlashAttention can match O(L) alternatives at many lengths up to ~128K, while Mamba-2 reported a 6× speedup over FlashAttention-2 at 16K under its benchmark configuration.Source: Design Principles for Future Attention… · pp. 41–42
• Adaptive sparse methods may outperform fixed-pattern methods at similar efficiency, motivating scalable learned routing that allocates attention according to token content.The proposed causal distinction is that fixed windows, strides, and random connections ignore token importance, whereas clustering, entmax routing, and learned grouping adapt selection; the comparison awaits independent verification.Source: Design Principles for Future Attention… · p. 42
• Interpretability should be a design constraint because it is the least-developed EEI axis: median I = 6 versus median E = 7 and median Ex = 8.None of the 21 scored methods prioritized interpretability; Swin’s spatial locality and BigBird’s explicit attention-type decomposition instead create analyzable structure as side effects.Source: Design Principles for Future Attention… · p. 42
• No surveyed method maximizes all three EEI axes, while equal-weight leaders score 8.0 and deployment weighting makes Mamba-2 lead at 8.8.Retrieval weighting favors FlashAttention-2/3 and PagedAttention at 8.8, supporting hybrid architectures that combine exact attention, state-space context bandwidth, and hardware-aware kernels.Source: Design Principles for Future Attention… · p. 42
• Future architectures should dynamically route tokens among mechanisms using input content, sequence position, and hardware state, converging exact computation, learned routing, hardware awareness, and interpretability.The section frames hybridization as the logical consequence of the EEI frontier and identifies Jamba, Samba, and Mamba-3 as examples of the trend.Source: Design Principles for Future Attention… · pp. 42–43

Scope and evolution

• Attention research expanded from machine-translation alignment to a dominant deep-learning primitive, organized across four connected threads through the EEI framework.The foundational mechanism remains widely used, while positional encodings, architectural variants, and vision adaptations define different operational envelopes.Source: Conclusion · p. 43

EEI frontier

• The surveyed trajectory moves from standard attention’s expressiveness, through efficient approximations and IO-aware exact computation, toward adaptive routing that may partially avoid the trade-off in retrieval-focused regimes.IO-aware exact attention improves hardware efficiency without expressiveness loss, while state-space models add extreme efficiency with structural interpretability gaps; adaptive results remain conditional on reproduction.Source: Conclusion · p. 43

Interpretability

• Interpretability research connects attention heads, induction mechanisms, superposition, sparse autoencoders, and state-space models through structured token-mixing perspectives.The SSD framework mathematically links certain structured state-space models to attention-like matrix operations, suggesting a continuum based on token-mixing structure.Source: Conclusion · p. 43

Open challenges

• The field still lacks a resolved retrieval-efficiency frontier, common large-context evaluation, principled sparsity guidance, reliable attention-SSM routing, cross-architecture benchmarks, and reproducible evidence.Interpretability-informed efficiency is also largely unexplored, and structural reproducibility challenges remain.Source: Conclusion · p. 43

Future directions

• Future systems are expected to combine learned routing, hardware-aware kernels, and structured state-space mechanisms into adaptive hybrid architectures.Interpretability may guide selective computation by identifying important tokens and features, while theory unifies apparently disparate models under common mathematical frameworks.Source: Conclusion · p. 43

A Detailed EEI Score Justification

• EEI assignments are grounded in rubric criteria, complexity, benchmark evidence, hardware measurements, and interpretability research, with Table 16 mapping scores to evidence types.Expressiveness uses complexity class and, where available, LRA or perplexity; efficiency uses asymptotic and hardware/kernel evidence, while throughput remains exploratory validation context.Source: A Detailed EEI Score Justification · p. 44, Table 16 · p. 45

Full Attention Methods

• Transformer (Vaswani et al., 2017) scores (4, 10, 6): exact pairwise attention maximizes expressiveness, but O(L^2d) time and O(L^2) memory limit efficiency while head-level mechanisms remain only partly understood.Attention weights are directly analyzable, yet superposition and incomplete attribution constrain interpretability.Source: Full Attention Methods · p. 44

IO-Aware Methods

• FlashAttention-2/3 (Dao, 2024; Shah et al., 2024) scores (8, 10, 6): IO-aware tiling improves memory efficiency and speed without changing exact attention or its interpretability.FlashAttention-2 achieves approximately 2.0× speedup over FA1 and up to 73% of peak FLOPs on A100, but computation remains O(L^2).Source: IO-Aware Methods · pp. 44–45
• GQA and MQA reduce KV-cache memory and bandwidth costs while retaining O(L^2) computation, but shared projections impose expressiveness and head-attribution trade-offs.GQA scores (7, 9, 6), whereas MQA scores (7, 8, 6) because its single shared KV projection causes larger quality degradation at matched model size.Source: IO-Aware Methods · pp. 44–46
• PagedAttention scores (8, 10, 6) and Ring Attention (7, 10, 6): both preserve exact attention, while serving or distributed execution improves practical scalability without removing quadratic total computation.PagedAttention achieves 2–4× serving-throughput improvement; Ring Attention overlaps communication and computation to support sequences exceeding a single device’s memory.Source: IO-Aware Methods · pp. 45–46

Sparse and Window Methods

• Longformer scores (8, 7, 7), BigBird (7, 8, 7), and Sparse Transformer (7, 7, 6), trading linear or structured sparsity for restricted or pattern-dependent interactions.BigBird’s LRA average is 55.01 versus Transformer’s 54.39, while Longformer’s is 53.46; fixed patterns improve analyzability but limit adaptive retrieval.Source: Sparse and Window Methods · p. 46
• Routing Transformer scores (6, 8, 6): online k-means enables content-adaptive clustering with O(L^1.5d) complexity, but recomputation overhead and hard cluster boundaries limit practical efficiency and retrieval.Cluster assignments aid interpretation, while stochastic, nondifferentiable k-means complicates gradient-based analysis.Source: Sparse and Window Methods · pp. 46–47

Hashing, Kernel, Recurrent, Structured SSM, Long Convolution, and Adaptive Sparsity Methods

• Reformer, Performer, Linformer, and xLSTM improve efficiency through hashing, kernel features, low-rank projections, or recurrence, but incur approximation, retrieval, or interpretability limitations.Reformer’s LSH attention is O(LlogL) yet can be slower than vanilla attention below 4K tokens; Performer and Linformer are O(L) or O(Lk), while xLSTM provides O(L) recurrent inference.Source: Hashing, Kernel, Recurrent, Structured… · pp. 47–48

State-Space and Recurrent Models

• Mamba achieves O(L) inference with constant state memory, reporting 5× generation throughput over a comparable Transformer and up to 3× over convolution-based SSMs.Its selective recurrent state supports aggregate long-range dependencies but does not establish arbitrary pointwise retrieval, and its compressed state lacks pairwise attention interpretability.Source: State-Space and Recurrent Models · p. 48
• Mamba-2 combines O(L) structured linear attention with tensor-core-friendly computation, yielding 2–8× speedup over Mamba-1 and 6× over FlashAttention-2 at 16K.SSD provides an attention-related theoretical interpretability bridge, although its structured parameterization is less general than Mamba’s and practical analysis tools remain limited.Source: State-Space and Recurrent Models · p. 48
• Hyena and RWKV provide efficient alternatives through long convolution and recurrent weighted key-value computation, but their token-level retrieval evidence and interpretability remain limited.Hyena is roughly O(LlogL), while RWKV is O(L); RWKV reports an LRA average of 72.07 but no long-context retrieval numbers or wall-clock speedups.Source: State-Space and Recurrent Models · p. 48

Adaptive Sparsity Methods

• Focus achieves a 2.0× speedup over full attention while retaining exact softmax attention within learned token groups and reporting comparable quality to its full-attention baseline.Its learned centroid routing is more interpretable at the group level, though coarse clustering can miss boundary-case targets and the cited benchmark evidence overlaps with Table 11 validation kernels.Source: Adaptive Sparsity Methods · p. 49
• DashAttention reaches up to 3.3× speedup over FlashAttention-3 on GH200 while maintaining accuracy comparable to full attention at 75% sparsity.Its differentiable coarse-to-fine α-entmax routing supports group-level analysis, but the reported efficiency comes from controlled-mask kernel benchmarks rather than application-level end-to-end measurement.Source: Adaptive Sparsity Methods · p. 49

Vision Methods

• Swin Transformer uses shifted local windows to obtain effectively linear patch complexity, ImageNet top-1 accuracy of 83.5% for Swin-B, and the survey’s highest interpretability score.Its hierarchical windows expand receptive fields across layers, while spatially grounded attention and shifts make routing comparatively direct to inspect; the score is not directly comparable to NLP methods.Source: Vision Methods · p. 49

Code and Data Availability

• All data, scripts, scoring records, reproducibility artifacts, and manuscript source are publicly available at github.com/Nixon-H/not-all-attention-isequal.The release is intended to reproduce the computational analyses reported in the study.Source: Code and Data Availability · p. 49