PaperlayerView exact revision on arXiv

Source-linked AI summary

The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends

Zhentao Tan, Jingyi Shen, Yanbo Li, Yao Liu, Yue Wu, Jieping Ye

arXiv:2609.39661v1cs.CL

TL;DR

Dense self-attention offers query-dependent access to context but has quadratic prefill comparisons and a KV cache that grows with context. This survey compares memory mechanisms and documented model architectures through a five-dimensional lens. It finds that LLM attention designs are diversifying rather than converging, motivating a multidimensional memory-routing hypothesis.

  • Problem

    Dense self-attention incurs quadratic prefill comparisons and a KV cache that grows with context, while longer nominal context alone does not ensure reliable use of distant evidence.

    Source: §1 · p. 4

  • Method

    The survey uses five functional memory dimensions to compare sequence mechanisms and descriptive analyses of documented model architectures and high-performing open-weight endpoints.

    Source: §8 · p. 48, §10 · p. 56

  • Results

    Documented LLM attention design is diversifying rather than converging: layer-wise hybrids have become more common, while cross-layer reuse remains limited.

    Source: §8.4 · p. 50

  • Takeaways & Limitations

    The survey proposes a hypothesis that persistent contextual memory could be organized across temporal scope, network depth, substrate type, and representation granularity, with Sparse Write and Sparse Read coordinating its lifecycle and use.

    Source: §10 · p. 56

    The architecture analyses are descriptive and purposively curated, so they characterize adoption and coexistence rather than attributing model quality causally to attention mechanisms.

    Source: §8 · p. 48

Abstract

from arXiv · show

Self-attention gives LLMs fine-grained, query-dependent access to context, but dense token interactions incur quadratic prefill cost and a key--value cache growing with context length. Research thus spans explicit-memory compression, sparse access, recurrent state construction, structured state dynamics, and heterogeneous mechanism composition. This survey analyzes these developments as model-internal contextual memory. We introduce a five-dimensional lens---Memory Representation, Memory Update, Access, Readout, and Integration---describing what is represented, how it changes, what is query-eligible, how it is read, and how readouts form outputs. This lens compares overlapping research lines without imposing one computational model. We reconstruct mechanism-level developments and architectural adoption using 59 release-level records from 14 major model lineages and 11 high-performing open-weight endpoints. First, explicit-memory and recurrent-state methods retain distinct interfaces but increasingly control overlapping memory functions. Second, heterogeneous architectures increasingly coordinate across network depth: layer-wise composition distributes complementary memory processing across representational stages, while cross-layer reuse carries selected memory and routing artifacts forward. Depth thus becomes a dimension along which contextual memory is constructed and managed. Third, these developments motivate a stateful multidimensional memory-routing hypothesis: persistent memory is organized across temporal scope, network depth, substrate type, and representation granularity, while coordinated Sparse Write and Sparse Read determine what is maintained and what contributes to each query. Overall, efficient sequence architecture design increasingly concerns the organization, lifecycle, and selective use of contextual memory rather than an isolated Attention operator.

1 Introduction

Dense self-attention provides query-dependent access to context but incurs quadratic prefill comparisons and a KV cache that grows with context. This survey compares evolving memory mechanisms and documented LLM architectures, finding heterogeneous designs and increasingly overlapping memory functions.

1 Introduction

  • Dense attention incurs quadratic query-key comparisons during prefill and a KV cache that grows linearly with accumulated context during decoding.A longer nominal context window alone does not ensure reliable use of distant evidence, because retrieval interference can weaken long-range recall.Source: §1 · p. 4
  • Five overlapping lines cover explicit-memory Softmax Attention, sparse access, recurrent Linear Attention, structured state-space memory, and Hybrid Architecture composition [1] [10] [17] [25] [28].Their trade-offs include growing cost with retained explicit units, dependence on selecting relevant evidence, loss of direct token retrieval in compressed states, and added coordination decisions in hybrids.Source: §1 · p. 4
  • This review complements architecture-, efficiency-, context-, state-, cache-, and memory-centered surveys by focusing on attention and adjacent sequence mixers through contextual memory.Its literature coverage is structured but not exhaustive and excludes several topics from the core taxonomy unless they alter contextual-memory semantics.Source: §1 · pp. 4–5
  • The survey introduces five analytical dimensions—Memory Representation, Memory Update, Access, Readout, and Integration—to compare sequence mechanisms without reducing them to one computational model.It reconstructs four mechanism-centered research lines and one composition-centered line while preserving their overlaps.Source: §1 · pp. 5–6
  • A longitudinal inventory of 59 release-level records across 14 model lineages and a frozen comparison of 11 high-performing open-weight endpoints characterize architectural adoption and coexistence.The analyses are descriptive, not evidence that attention mechanisms cause model quality.Source: §1 · pp. 5–6
  • The survey synthesizes a shift from distinct explicit-memory and state-based interfaces toward overlapping memory functions, with depth emerging as a dimension of memory construction and management.Layer-wise composition distributes complementary processing across stages, while cross-layer reuse extends the lifetime of selected memory and routing artifacts.Source: §1 · p. 6

2 A Unified Memory-Centric View of Attention

The survey treats sequence mechanisms as systems that maintain and use model-internal contextual memory. It compares them through five functional dimensions, then maps where different research lines concentrate their design effort.

2 A Unified Memory-Centric View of Attention

  • Sequence mechanisms can be compared as systems that maintain and use input-dependent, model-internal contextual memory in forms ranging from token KVs to recurrent states.The term excludes pretrained parameters, external retrieval corpora, and biological memory.Source: §2 · p. 6

2.1 Five Analytical Dimensions

  • The five dimensions ask what remains represented, how memory changes, what is eligible for a query, how eligible information is read, and how readouts form the output.They are functional roles, not mandatory separate components or a fixed execution sequence; Figure 1 maps representative Linear and Sparse Attention operations onto them.Source: §2 · p. 6, Figure 1 · p. 7

2.2 Classification Principles and Family-Level Mapping

  • Design effort differs by research line: Softmax Attention emphasizes explicit-memory representation and update, Sparse Attention emphasizes access, and Linear Attention and State Space Models emphasize recurrent-state representation and update.Hybrid Architecture primarily diversifies and allocates representations; individual methods can modify multiple dimensions, so the lines are not exclusive partitions.Source: §2.2 · p. 10

3 Softmax Attention

Softmax Attention retrieves from explicit, enumerable memory, but its growing cache motivates changes to how memory is represented, updated, read, and integrated. The surveyed approaches compress or share memory, consolidate history, or modulate readout and output contributions.

3.1.1 Sharing across Heads

  • GQA is a continuum between MHA and MQA: sharing KV heads reduces stored representations per token without changing the token candidate set.The query-head count remains fixed as independently stored KV heads decrease.Source: §3.1.1 · p. 11, Figure 2 · p. 12

3.1.2 Compression along Channels

  • MLA caches a compact KV latent rather than expanded per-head keys and values, preserving token-level addressability through a low-rank bottleneck.It separates content and positional components to accommodate rotary positional encoding.Source: §3.1.2 · pp. 11–12

3.1.3 Sharing across Layers

  • Cross-Layer Attention computes KV activations in selected source layers for reuse by consumers; sharing can reduce cached KV layers while preserving each consumer’s attention computation.Head sharing and layer sharing are orthogonal, and can be combined.Source: §3.1.3 · p. 12

3.2.1 Fixed-Capacity Recurrent Memory

  • Fixed-capacity recurrence updates bounded latent-state or KV-slot memory, so consolidated historical tokens cease to be separately addressable.RMT and TransformerFAM update latent states at segment or block boundaries, while Trellis and Lattice use explicit KV slots and finer-grained updates [45] [49] [42].Source: §3.2.1 · p. 13

3.2.2 Dynamic-Resolution Memory

  • Dynamic-resolution memory keeps recent tokens fine-grained and consolidates remote history into coarser summaries, which may still grow with history.KSA combines a recent-token path with summary memory; HCA Core attends densely over compressed global entries [6] [50].Source: §3.2.2 · p. 13

3.3 Readout and Integration Modulation

  • Score and map methods mix, subtract, or forget attention weights, while MoH routes and Gated Attention rescales head outputs; runtime savings depend on implementation support.These interventions change readout or integration without directly shrinking stored memory or the candidate set [51, 52, 53, 54, 55, 44].Source: §3.3 · p. 14, §3.3.1 · p. 14, §3.3.2 · p. 14

3.4 Summary

  • The surveyed architectures use GQA, latent-KV compression, compressed memory, and output gating across several model lineages, including Llama, Qwen, DeepSeek, and Kimi.Where recurrent or sparse paths are also present, those components are treated separately in later sections.Source: §3.4 · p. 15

4 Sparse Attention

Sparse Attention targets the cost of dense causal access by restricting each query to a budgeted subset of memory units. Methods vary in how they construct and reuse that support, while often retaining candidate-local Softmax readout.

4 Sparse Attention

  • Dense causal attention considers all eligible positions, giving Θ(L^2d) training and prefill work; Sparse Attention limits each query to a budgeted subset to target computation and data movement.Dense decoding also reads a KV cache growing as Θ(Ld_KV) per layer.Source: §4 · p. 15
  • Sparse methods differ in how they produce support and how long decisions remain valid; temporal and cross-layer reuse can combine with structural, self-routing, or auxiliary-proxy evidence.These subsection labels are editorial groupings, not mutually exclusive classes.Source: §4 · p. 15
  • Dense access retains every causal pair, while the illustrated structure-based and routing-based patterns restrict which positions each query can access.The figure shows a four-token sliding window plus a sink for structure sparsity, and diagonal-plus-selected-token support for routing sparsity.Source: §4 · p. 15, Figure 3 · p. 16

4.1 Structure-Constrained Sparse Attention

Structure-constrained sparse attention limits eligible positions through a pattern established before unrestricted query-specific search. Such patterns may be prescribed before training or inferred from a trained dense model for inference.

4.1 Structure-Constrained Sparse Attention

  • Structure-constrained attention instantiates a sparse support from a designed or observed pattern, optionally using input-dependent information to choose positions within that pattern family.Architecture-prescribed patterns are fixed before training; post-hoc patterns are inferred from a dense checkpoint and applied at inference.Source: §4.1 · p. 16

4.1.1 Architecture-Prescribed Patterns

  • Architecture-prescribed patterns combine local windows with global or random edges, or use strided and dilated schedules; their fixed topology provides training–inference consistency but limits input adaptivity.Expanding windows or adding global edges improves coverage while consuming compute and bandwidth savings.Source: §4.1.1 · p. 16

4.1.2 Post-hoc Discovered Patterns

  • Post-hoc patterns can accelerate dense checkpoints without sparse pretraining, but deleting connections and changing Softmax normalization creates training–inference mismatch.Fixed rules limit selection overhead but adapt little; pattern families recover some adaptivity with runtime statistics and candidate construction.Source: §4.1.2 · p. 17

4.2 Self-Routing Sparse Attention

Self-routing sparse attention derives candidate evidence from the model’s own representations and narrows exact attention to selected tokens, blocks, or summarized chunks. Its designs trade routing granularity and adaptability against indexing, layout, and memory costs.

4.2.1 Dynamic Token Grouping

  • Balanced group sizes and separate local heads address work balance and missed nearby evidence, but assignment, sorting, and irregular gathers add overhead.Dynamic token grouping assigns similar queries and keys to content-dependent groups, then applies exact token-level Softmax within each causal group.Source: §4.2.1 · p. 18

4.2.2 Block Aggregation

  • Block aggregation scores contiguous blocks before expanding selected blocks to original KVs; dense-checkpoint scoring is retrofit-friendly, while sparse training can adapt backbone Q/K representations.MoBA uses mean backbone keys to score blocks and exact token-level Softmax over selected original KVs; FlashMoBA improves the selected-block computation with tiled Top-k and gather-and-densify kernels.Source: §4.2.2 · p. 19
  • NSA combines compressed summaries, selected original-token KVs, and sliding-window context with input-dependent gates; summaries also provide routing evidence [12].When compression and selection block boundaries differ, NSA aggregates overlapping compression-block probabilities to score selection blocks.Source: §4.2.2 · pp. 19–20

4.2.3 Summary-Token Encoding

  • Landmark and gist tokens summarize chunks for retrieval, while HiLS uses landmark-derived surrogate chunk masses to select distant chunks and allocate probability across them [14] [85] [86].HiLS applies exact token-level Softmax inside selected chunks, but uses surrogate masses to distribute probability across chunks.Source: §4.2.3 · p. 20
  • Summary-token methods require specialized sequence layouts and training, consume token capacity, and risk an information bottleneck when one vector represents a long or heterogeneous chunk.In HiLS, the chunk-summary bias participates in forward attention weights, allowing the language-modeling loss to train landmark summaries end to end.Source: §4.2.3 · p. 20

4.3 Auxiliary-Proxy Sparse Routing

Auxiliary-proxy routing separates the address representation used to select content from the main attention projections. Designs vary in selection granularity and whether they compress only addresses or both addresses and content.

4.3 Auxiliary-Proxy Sparse Routing

  • A separately parameterized indexer scores backbone states in a compact address space, and its selected indices determine which original K/V entries are accessed.The indexer’s dimensionality, heads, update schedule, supervision, and routing granularity can be tuned independently, but its relevance to content memory must be learned and maintained.Source: §4.3 · p. 21

4.3.1 Token-Granularity Routing

  • DSA’s Lightning Indexer selects one support shared across attention heads, whereas LSA adds streaming-aware and hierarchical indexing to address full-length scans and discontinuous KV access [64] [16].DSA distills the indexer toward main-attention scores, then trains under sparse support.Source: §4.3.1 · pp. 21–22
  • Token-granularity routers can recover isolated evidence and minimize block overfetch, but scanning proxy entries and gathering scattered KVs can make index bandwidth the bottleneck.LongCat Sparse Attention combines a fixed sink and sliding window with dynamic token selection, then uses hierarchical indexing to narrow token selection to recalled blocks.Source: §4.3.1 · p. 22

4.3.2 Block-Granularity and Compressed-Entry Routing

  • MSA trains the backbone under block-sparse support and selects blocks per GQA group, but still scans every visible token for index evidence [94].Its block scores take the maximum token-level index score within each contiguous KV block; regular KV reads do not remove the context-length cost of the evidence scan.Source: §4.3.2 · p. 22
  • QSA shortens the index scan by pooling index keys before scoring while retaining original-token content; CSA compresses both index and content entries, trading item-level fidelity for smaller memory and attention work [13] [50].CSA combines compressed entries with an uncompressed sliding window for the query’s incomplete block.Source: §4.3.2 · pp. 23–24

4.4 Temporal and Cross-Layer Reuse

Temporal and cross-layer reuse change when support decisions and content states are refreshed. They reduce repeated candidate discovery or duplicated memory, while introducing the risk that reused support becomes stale or insufficiently specific.

4.4.1 Temporal Support Reuse

  • Temporal reuse uses prior query–support relations to propose candidates, then recomputes current-query scores, support, and sparse Softmax over original KVs.ReTopK merges supports from similar cached queries with a recent window before reranking candidates using the current query.Source: §4.4.1 · p. 24
  • Temporal reuse can drift as queries change, so ReTopK falls back to exact full-history Top-K below a similarity threshold and periodically refreshes support.Its efficiency depends on cache capacity, recalled supports, recent-window width, fallback rate, and refresh interval.Source: §4.4.1 · pp. 24–25

4.4.2 Cross-Layer Reuse

  • Cross-layer designs reuse selected positions or attention evidence from anchor layers; more extensive variants also share KVs and routing artifacts across layer groups [98] [99] [102] [103].HySparse2 reuses full-layer KVs and selected indices, and uses one unified sparse-attention operation for local and retrieved tokens.Source: §4.4.2 · pp. 25–26
  • IndexCache layers inherit indices from the nearest preceding Full layer, while CSA2 distinguishes Full, Reindex, and Reuse modes to refresh KVs, address entries, and support selectively [15] [101].CSA2 Reindex layers reuse main KVs and indexer keys but refresh support within a shared candidate pool; Reuse layers inherit the latest support.Source: §4.4.2 · p. 25
  • Reuse trades repeated selection and duplicated memory for specificity: query drift or layer-specific retrieval needs can make shared decisions less suitable.Selection or reindex frequency, reuse span, representation compatibility, and correction policy should be reported alongside accuracy and throughput.Source: §4.4.2 · p. 26

4.5 Summary

Sparse attention has progressed from fixed support patterns to learned routing and reuse of address and content memory. The survey reports conditional quality parity with dense attention for trained or distilled sparse methods, alongside deployment in recent production LLMs.

4.5 Summary

  • The trajectory moves from structural patterns through self-routing and learned proxy indices toward lifecycle management that shares address and content memory across steps and layers [70] [10] [11] [71] [76] [77] [78] [79] [12] [80] [91] [64] [50] [101] [100].Training-free sparsity remains an approximation with training–inference mismatch, while trained sparse methods’ margins over dense attention remain task-, budget-, and implementation-dependent.Source: §4.5 · p. 26
  • Trained sparse attention and distilled-then-continued indexers report quality on par with dense attention, and occasionally above it at long context, while cutting prefill and per-step decoding cost [12] [80] [64].The reported parity depends on end-to-end router training, a budget matched to task evidence density, and speed accounting that includes evidence construction, index traffic, and gathering.Source: §4.5 · p. 26
  • Recent production LLMs deploy auxiliary-proxy routing at token, block, or compressed-entry granularity, with some releases adding cross-layer reuse [64] [50] [94] [16].The surveyed recipe uses compact learned indices, optionally compresses them across positions, aligns them through training or distillation, and increasingly reuses them across depth.Source: §4.5 · p. 26

5 Linear Attention

Linear Attention replaces token-wise KV history with recurrent associative states, shifting design toward how compressed memory is represented, updated, accessed, and read. Its development refines state updates and expands memory organization while adding routing and coordination choices.

5 Linear Attention

  • Linear Attention compresses historical key–value associations into recurrent associative states, trading item-wise token addressability for a state whose size can be independent of context length.Later designs enlarge or partition the associative space and reorganize it across temporal ranges or blocks.Source: §5 · pp. 26–27

5.1 Memory Update: From Accumulation to Controlled State Editing

  • Single-state updates progress from additive accumulation toward controlled retention, correction, erasure, and writing, changing which historical distinctions persist in the state.The survey compares update roles analytically; it does not assume every method implements three independent modules or uses the same order.Source: §5 · p. 27, §5.1 · p. 27

5.1.1 Retain / Decay

5.1.2 Erase / Correct

  • Erase mechanisms progress from no address-specific removal to coupled delta correction, then to decoupled directions or multiple erase components [17] [19] [105] [20] [113] [106] [114] [115].The distinctions concern whether old content is read and removed directionally, whether read and update directions align, and whether erase components are combined.Source: §5.1.2 · pp. 28–29

5.1.3 Write / Commit

5.1.4 Representative Recurrent Update Rules

  • The representative recurrent-rule comparison uses unified single-head notation and omits method-specific normalization, feature generation, and output gating.For DeltaProduct, R denotes the number of per-token delta transformations; the passage identifies HGRN2's channel-wise forget gate and candidate content.Source: §5.1.4 · p. 30, Table 7 · p. 31

5.1.5 Loss-Based Optimization View

  • Early objectives maximize state-prediction correlation, while delta-style methods minimize reconstruction error and make writes residual-dependent; EDA fits an erase target before the current value.The progression also changes the optimization state, number of steps, and whether erase and write share an address and target.Source: §5.1.5 · pp. 31–32

5.2 Capacity Expansion

Capacity Expansion relaxes the single-dense-state assumption by adding memory rows and using sparse operations. SSE imposes grouped routing, while SDM uses a flat row space, trading regularity against addressing freedom and coordination demands.

5.2 Capacity Expansion

  • A single dense state has finite capacity, while enlarging it directly increases every token's read and write cost; Capacity Expansion adds memory units with sparse selection.The central design challenge becomes expanding available memory without requiring every token to interact with every row.Source: §5.2 · p. 32

5.2.1 Sparse State Expansion (SSE)

  • SSE organizes added state rows into groups and routes each token to a small number of groups, with separate coefficients for row-level writing and reading.Fixed group boundaries regularize execution but can exclude potentially relevant rows in inactive groups and limit reorganization.Source: §5.2.1 · p. 32

5.2.2 Sparse Delta Memory (SDM)

  • SDM uses a flat recurrent row memory with Product-Key routing, selecting rows for delta-style writes and using a separate router to select read rows.Independent write and read supports allow storage location and later retrieval location to differ, but can create mismatches in row use and router conventions.Source: §5.2.2 · p. 32

5.2.3 Comparison and Open Design Space

  • Both methods expand a dense matrix into row memory and use sparse reads and writes; SSE uses fixed-group routing, whereas SDM retrieves independently in a flat row space.Fixed groups regularize and narrow search but restrict reorganization; flat routing offers addressing freedom but must maintain low-cost, high-recall retrieval, and read–write coordination remains unresolved.Source: §5.2.3 · p. 33, Table 9 · p. 33

5.3 Temporal Expansion

Temporal Expansion preserves multiple associative states for different historical intervals, with methods differing in segmentation, state management, and query-conditioned integration. Related coordination mechanisms distribute writes across feature heads or route memory-derived signals across layers.

5.3 Temporal Expansion

  • Temporal Expansion keeps multiple states for historical intervals so recent detail and distant context need not share one temporally uniform representation.The query reads from retained states separately, then Integration combines their readouts.Source: §5.3 · pp. 33–34

5.3.1 Log-Linear Attention

  • Log-Linear Attention maintains O(log t) states in a deterministic multiscale hierarchy, preserving recent history in short segments and compressing distant history into longer ones.Because segmentation depends on position, a fixed merge can cross a semantic boundary.Source: §5.3.1 · p. 34, Figure 8 · p. 34

5.3.2 Dynamic Linear Attention (DLA)

  • DLA segments according to content change and, at its state bound K, merges adjacent states with the lowest information density, keeping Mt ≤ K.Dynamic boundaries and merges allocate more of the bounded state budget to semantic change, but merged history generally cannot regain its original temporal resolution.Source: §5.3.2 · p. 34

5.3.3 Multi-Head Linear Attention

  • MHLA retains separate key–value summaries for fixed token blocks and lets each query block form a learned mixture of those summaries.Unlike the merging approaches, it does not intrinsically bound the number of retained summaries, so efficiency depends on block count and inter-block mixing cost.Source: §5.3.3 · p. 34

5.3.4 Comparison and Development Trends

  • Log-Linear Attention uses deterministic multiscale intervals, DLA content-dependent segments, and MHLA fixed-granularity blocks with query-conditioned mixing.These choices trade off temporal resolution, bounded growth, adaptive segmentation, and query-dependent retrieval; the comparison is also presented in Table 10.Source: §5.3.4 · pp. 34–35, Table 10 · p. 35

5.4 Auxiliary Coordination and Routing Enhancements

  • SLA coordinates writing and output contribution across feature heads, while CLVR routes an internal write-associated value through the residual stream to later layers.CLVR does not primarily add persistent memory units or replace its host Linear Attention recurrence.Source: §5.4 · p. 35

5.5 Summary

  • The section’s synthesis connects state-update refinement and capacity or temporal expansion with coordination across feature heads and network depth.The resulting memory system’s practical value depends on balancing compression, capacity, temporal resolution, access cost, readout quality, and implementation efficiency.Source: §5.5 · p. 36

6 State Space Models

SSMs retain sequence history in an evolving state, with structured or input-conditioned dynamics governing what persists and how each position is read. Later designs enrich the state’s read–write interfaces, temporal organization, and update geometry.

6 State Space Models

  • SSMs evolve a hidden state as inputs arrive and read it for contextual representations; the state records the current sequence, distinct from weights learned across examples.Transitions govern persistence and interaction, while input and readout maps connect the state to the surrounding network.Source: §6 · p. 36, §6.1 · p. 36

6.1.1 Structured Time-Invariant Dynamics

  • Time-invariant SSMs share operators across positions, supporting convolutional whole-sequence processing and recurrent state updates during incremental decoding.HiPPO supplies history-preserving dynamics, and S4 uses structured transitions to make long-sequence computation practical.Source: §6.1.1 · p. 37

6.1.2 Input-Conditioned State Dynamics

  • Input-conditioned dynamics let current inputs modulate state transitions, writing, and reading, addressing content-dependent retention that shared temporal rules do not provide.Mamba generates its step size and input and readout maps from the current input, and uses a selective scan for the resulting kernel.Source: §6.1.2 · p. 38
  • Mamba-2’s head-wise scalar transition factors each earlier input’s contribution into its match with the current read direction and the intervening transitions.The resulting lower-triangular coefficient matrix connects the SSM recurrence to associative computation used in Linear Attention.Source: §6.1.2 · p. 39
  • For fixed head count and state dimensions, Mamba-2 decoding state and per-step work are context-length-independent, while total sequence work grows linearly with sequence length.The N × P head state sets retained storage; increasing state dimensions raises storage and per-step arithmetic costs.Source: §6.1.2 · p. 39

6.2.1 Read–Write Interfaces

  • Read–write interfaces determine how state information reaches the layer output; Mamba-3 MIMO adds multiple read and write directions to the principal state.This raises arithmetic, which can improve hardware utilization when state movement dominates decoding time; realized latency depends on implementation, device, and batch size.Source: §6.2.1 · p. 40

6.2.2 State Organization and Update Geometry

  • SSM layers can organize complementary temporal characteristics across heads or condition the geometry of incoming updates while retaining the learned transition as a separate component.HADES combines shared low-pass and expert high-pass filters; MuonSSM uses a momentum pathway and lightweight Newton–Schulz iterations on low-rank input injections.Source: §6.2.2 · p. 40

6.3 Summary

  • SSM designs combine retained state and temporal dynamics with selective controls, richer state interfaces, head organization, and conditioned input updates; these components also enter hybrid architectures.Attamba uses SSMs for block-level keys and values, DART retains local SSM block-state contributions, and Samba combines Mamba with sliding-window attention.Source: §6.3 · p. 41

7 Hybrid Architecture: Composing Heterogeneous Memory Mechanisms

Hybrid architectures compose heterogeneous sequence mixers at different structural granularities, from layers to individual tokens. Their designs range from arranging largely self-contained mechanisms to coordinating memory representations, routing, readouts, and outputs.

7 Hybrid Architecture: Composing Heterogeneous Memory Mechanisms

  • Hybrid Architecture is a compositional design space: different mechanisms can be combined across layers, heads, branches, or tokens, with the five memory dimensions describing effects of that composition.These granularities can coexist, extending contextual memory beyond a single substrate or processing path.Source: §7 · p. 42

7.1 Layer-wise Hybrid

  • Layer-wise hybrids assign different mixers across network depth, using either predefined layer schedules or selection informed by pretrained-model behavior.Selection approaches include direct diagnosis, alignment-based evaluation, and iterative distillation-guided selection.Source: §7.1 · p. 42, §7.1.4 · p. 44

7.1.1 Predefined Allocation

  • Predefined schedules place mixers at fixed depths, while diagnosis, alignment, and iterative distillation use different signals to decide which attention layers to retain or replace.Diagnosis examines original layers; alignment-based methods assess replacement after alignment; iterative selection uses changes in teacher–student KL divergence after restoring attention layers.Source: §7.1.1 · p. 43, §7.1.2 · p. 44, §7.1.3 · p. 44

7.2 Head-wise Hybrid

  • Head-wise hybrids divide mixer capacity within a layer through fixed budgets, function-aware selection, input-adaptive routing, or depth-adaptive allocation.DuoAttention preserves retrieval heads with full attention and a complete KV cache while streaming heads retain attention sinks and recent tokens; Elastic Attention routes heads by current input.Source: §7.2 · p. 44, §7.2.1 · p. 44, §7.2.2 · p. 45

7.3 Branch-wise Hybrid

  • Branch-wise hybrids preserve identifiable memory paths, distinguishing fusion of contextual readouts before a shared projection from fusion of independently formed path outputs.Block-Recurrent Transformer combines token and state readouts; Titans-MAG combines its sliding-window attention and neural long-term memory branch outputs.Source: §7.3 · p. 45, §7.3.1 · p. 46, §7.3.2 · p. 46

7.3.1 Readout-Level Branch Fusion

  • Readout-level branch fusion combines path readouts before the enclosing module completes its output transformation, even when paths share state or read interfaces.DART adds a scalar-gated State-Memory Attention correction to the SSM readout.Source: §7.3.1 · p. 46

7.4 Token-wise Hybrid

  • Token-wise hybrids route tokens or chunks by temporal boundary, content score, or learned operation assignment, changing their representation or processing path.LoLCATs moves tokens leaving a local window into a Linear Attention state; LoLA uses self-recall error to choose between sparse historical KV and state compression.Source: §7.4 · p. 47, §7.4.1 · p. 47, §7.4.2 · p. 47

7.5 Summary

  • Hybrid granularities are composable: architectures can coordinate complementary retrieval and state-propagation capabilities across structural axes, not merely place separate modules side by side.Finer-grained designs can jointly organize memory formation, access, lifecycle, readout, and combination.Source: §7.5 · p. 48

8 Attention Designs in Publicly Documented LLM Architectures: Evolution, Coordination, and Frontier Adoption

The architecture inventory and open-weight comparison show attention design diversifying rather than converging, with hybrid mechanisms and cross-layer reuse adding ways to coordinate memory across depth. The analyses characterize adoption and coexistence, not causal links between attention mechanisms and model quality.

8.1 Diversification Rather Than Convergence in Attention Design

  • Dense, compressed, sparse, and recurrent-state approaches persist as distinct design routes, and model lineages may change direction rather than follow a single replacement sequence.GQA reduces KV-head redundancy while retaining full token retrieval; other routes compress KVs, restrict access, or replace some repeated retrieval with fixed-size state.Source: §8.1 · pp. 48–49

8.2 Increasing Hybridization and Emerging Cross-Layer Reuse

  • Cross-layer reuse appears in five release records, carrying selected retrieval indices, KV representations, or token-selection decisions forward across layers.This differs from hybrid composition, which concerns where complementary mechanisms are placed.Source: §8.2 · pp. 49–50
  • Hybrid records rose from none in 2022–2023 to 9 of 25 in 2024–2025 and 17 of 27 in 2026, mainly through layer-wise schedules.Of 58 classifiable records, 26 are Hybrid; the schedules assign complementary capabilities to different layers, coordinated through the hidden-state stream.Source: §8.2 · p. 49, Table 17 · p. 50

8.3 Attention Designs among High-Performing Open-Weight Models

  • The eleven selected high-performing open-weight endpoints retain a heterogeneous mix of attention designs, and each preserves an explicit token-retrieval path.The comparison includes hybrid and non-hybrid designs; it demonstrates coexistence, not an attention-based quality ranking.Source: Table 18 · p. 50, §8.3 · p. 50

8.4 Summary

  • Across the inventory and frontier snapshot, attention design is diversifying; hybrid composition is common, while cross-layer reuse remains limited.High-performing open-weight models combine efficient state propagation or sparse access with explicit token retrieval.Source: §8.4 · pp. 50–51

9 Synthesis Across Mechanisms, Architectures, and Future Directions

This synthesis treats contextual memory as a shared lens for comparing sequence mechanisms and their architectural coordination. It traces expanding memory control across mechanisms and network depth, then proposes a multidimensional routing hypothesis and testable directions for developing it.

9 Synthesis Across Mechanisms, Architectures, and Future Directions

  • The survey frames sequence-architecture evolution as redesigning contextual memory’s representation, lifecycle, and use, rather than simply replacing one operator.Its five dimensions are Memory Representation, Memory Update, Access, Readout, and Integration.Source: §9 · p. 51

9.1 Mechanism Level: Distinct Starting Points and Expanding Control Scopes

  • Explicit-memory methods retain enumerable units, while state-based methods compress history into recurrently maintained states; these are predominant forms, not exclusive model classes.The routes begin with different memory forms and design pressures.Source: §9.1 · p. 51
  • Explicit-memory methods emphasize representation and access, while state-based methods emphasize representation and update; both extend control across broader memory functions.Explicit methods also address updates, readout, and integration; state-based methods develop access, readout, and integration through differentiated state units and coordination.Source: §9.1 · p. 51
  • The routes increasingly overlap in the memory functions they explicitly control, without converging on one substrate or computational mechanism.The five dimensions are Memory Representation, Memory Update, Access, Readout, and Integration.Source: §9.1 · p. 52

9.2 Architecture-Level Synthesis: From Complementary Placement to Memory-Artifact Lifecycles

  • Layer-wise composition distributes complementary memory operations across representational stages, coordinated through the hidden-state stream.Explicit retrieval, sparse access, and recurrent state propagation can perform different operations as representations move through the network.Source: §9.2 · p. 52
  • Cross-layer reuse extends the lifetime of selected KV, index, or candidate artifacts, a distinct form of coordination from placing mechanisms at different depths.The inventory documents five release-level records corresponding to three architectural patterns.Source: §9.2 · pp. 52–53
  • Layer-wise composition determines what memory processing occurs at each depth, while cross-layer reuse determines which artifacts persist; together they make depth a dimension of memory organization.This concerns where memory is produced, transformed, maintained, reused, and refreshed across the network.Source: §9.2 · p. 53

9.3 Forward-Looking Hypothesis: Stateful Multidimensional Memory Routing

  • The proposed hypothesis organizes persistent memory across temporal scope, network depth, substrate type, and representation granularity, with separate write and read routing.Write routing determines what information is preserved; read routing determines what represented units contribute to a query.Source: §9.3.3 · p. 54

10 Conclusion

The survey compares overlapping attention and sequence-mixing research lines as forms of contextual memory, then examines how these mechanisms are adopted and coordinated in model architectures. Its synthesis finds distinct memory interfaces with increasingly overlapping functions, and motivates a multidimensional memory-routing hypothesis.

10 Conclusion

  • The five-dimensional lens—Memory Representation, Memory Update, Access, Readout, and Integration—compares mechanisms as analytical roles, not a universal computational factorization.It distinguishes what remains represented, how memory changes, what is query-eligible, how it is read, and how readouts are coordinated into outputs.Source: §10 · p. 56
  • Explicit-memory and recurrent-state methods retain distinct predominant interfaces, but increasingly control overlapping memory functions.The survey describes expanding control over representation, update, access, readout, and integration without convergence on one substrate or operator.Source: §10 · p. 56
  • Across 59 release-level records from 14 major lineages, documented architectures show coexistence rather than a universal replacement path.Hybrid designs increasingly combine complementary memory capabilities; layer-wise composition is the principal pattern, while cross-layer reuse remains concentrated in a small number of records.Source: §10 · p. 56
  • Attention-centered architectures require comparison beyond asymptotic complexity, including memory, quality, retrieval, routing, and realized hardware utilization.Trade-offs include explicit-memory storage and movement, sparse-retrieval index quality and irregular execution, recurrent-state capacity and recoverability, and heterogeneous-design coordination costs.Source: §10 · p. 56
  • The synthesis motivates a hypothesis that persistent contextual memory is organized across temporal scope, network depth, substrate type, and representation granularity.Coordinated Sparse Write and Sparse Read would shape which addresses receive information and which represented units contribute to queries; the hypothesis depends on practical quality–efficiency gains without prohibitive costs.Source: §10 · p. 56

A Supplementary Materials: Publicly Documented Architecture Inventory

The appendix documents the architecture inventory used in Section 8 and specifies its counting and classification conventions. The inventory is purposively curated, not exhaustive or market-share weighted.

A Supplementary Materials: Publicly Documented Architecture Inventory

  • The inventory contains 59 publicly documented release-level architecture records analyzed in Section 8.Table S1 lists the records and the lineage and Hybrid labels used to derive Table 17.Source: A Supplementary Materials: Publicly… · p. 57, Table S1 · p. 57
  • A release record groups models released together when they share a language-model sequence-mixing architecture, while separately released versions remain separate records.Model lineage identifies the broader family used for the 14-lineage count.Source: A Supplementary Materials: Publicly… · p. 57
  • Hybrid labels require two or more distinguishable sequence-mixing or access regimes; multiple components forming one sparse-attention candidate set do not alone qualify.Undisclosed records are excluded from Non-Hybrid/Hybrid percentages, and cross-layer reuse is reported separately.Source: A Supplementary Materials: Publicly… · p. 57
  • The inventory is purposively curated rather than exhaustive or market-share weighted.Source: A Supplementary Materials: Publicly… · p. 57

Related papers

Loading 2609.39661v1…