Native Hybrid Attention for Efficient Sequence Modeling

Motivation

• NHA targets the gap between local sparse attention, which misses distant tokens, and linear models, whose fixed-size memory can lose precise information.Sparse attention reduces cost through selected or local tokens, whereas linear sequence models achieve O(n) efficiency by compressing sequence history into a fixed-size state.Source: §1 · p. 1

NHA design

• NHA integrates precise local tokens with compressed long-term memory in one unified softmax attention operation.A linear RNN updates fixed-size key–value slots, which are concatenated with tokens inside the sliding window.Source: §1 · p. 2

Intra-layer hybrid

• NHA dynamically allocates attention between short-term and long-term memory for each query without additional fusion parameters.Unlike prior hybrids that separately compute attentions and combine their outputs through weighted summation, NHA uses context-dependent softmax weighting.Source: §1 · p. 2

Inter-layer hybrid

• Sliding-window size controls NHA’s inter-layer behavior without changing its uniform architecture, from pure linear RNN processing to full attention.A zero window yields a pure linear RNN layer, while a full-sequence window recovers full attention.Source: §1 · p. 2

Evaluation

• The paper evaluates NHA across hybrid and non-hybrid architectures, reporting strongest overall performance for NHA on recall-intensive tasks.Hybrid architectures consistently surpass standard Transformers on those tasks.Source: §1 · p. 2

Pretrained-model hybridization

• Pretrained Transformer LLMs can be adapted into NHA with brief finetuning, retaining competitive performance while improving inference speed.The paper demonstrates this hybridization for pretrained models rather than only newly pretrained architectures.Source: §1 · p. 2

Intra-Layer Hybrid in NHA

• NHA compresses historical tokens into fixed-size memory slots, concatenates them with recent local context, and applies unified softmax attention.Figure 2 contrasts this single attention operation with prior hybrids that compute separate outputs and fuse them through weighted summation.Source: Figure 2 · p. 3

Sliding Window Attention

• Sliding Window Attention stores a fixed-size neighborhood precisely, reducing attention computation but limiting access to information outside the window.Its local focus captures immediate context accurately while restricting long-term sequence comprehension.Source: §2 · p. 3

Gated Slot Attention

• Gated Slot Attention compresses the entire sequence into memory slots updated through gating, trading precise token storage for broader contextual coverage.This provides broader context than localized attention while adaptively blending new and past information.Source: §2 · p. 3

3.1 Level 1: Architectural Hybrid of RNN Memory and Softmax Attention

• NHA compresses long-term information into fixed key–value slots, concatenates them with local tokens, and applies one softmax attention operation over the unified memory.The long-term slots are updated by a linear RNN, while the local tokens preserve precise recent context.Source: §3.1 · p. 3, §3.2 · pp. 3–4

3.2 Level 2: Intra-Layer Hybrid

• Unified softmax attention dynamically weights long-term and short-term memories jointly from query–key similarities, coupling their token-level attention rather than using separate output fusion.The weighting depends on similarities with keys from both memory types and incorporates information from preceding tokens.Source: §3.2 · p. 4

3.3 Level 3: Inter-Layer Hybrid via Window Size Adjustment

• Changing the sliding window size lets each structurally identical NHA layer range from pure linear attention at w = 0 to full attention at w = N, enabling inter-layer hybridization.A single model can switch among hybrid configurations during inference without retraining by adjusting window sizes.Source: §3.3 · pp. 4–5, Figure 3 · p. 4, §3.2 · p. 4

3.4 Chunkwise Parallel Form

• NHA supports chunkwise-parallel computation by combining gated linear-memory updates with sliding-window logits, concatenating their logits for a shared softmax and aggregating both value branches.The linear memory uses a gated recurrent update, while the sliding-window branch supplies shifted local attention logits.Source: §3.4 · p. 5

3.5 Connections to MesaNet and Atlas

• NHA preserves precise short-term memory while compressing long-term key and value information, distinguishing it from MesaNet and Atlas, whose KV updates preclude softmax attention.The paper relates NHA’s bounded accumulation range to the conjugate-gradient step size in MesaNet and the window size in Atlas.Source: §3.5 · p. 5

4.2 Recall-Intensive Tasks

• NHA consistently achieves the strongest reported results on recall-intensive benchmarks, where hybrid models outperform standard Transformers.Evaluation covers FDA, SWDE, SQuAD, NQ, TriviaQA, and Drop.Source: §4.2 · p. 6

4.3 Commonsense Reasoning Tasks

• NHA achieves the highest average score across commonsense reasoning tasks while preserving general reasoning abilities with improved efficiency.The evaluation includes WikiText, LAMBADA, ARC-Easy, ARC-Challenge, HellaSwag, PiQA, and WinoGrande.Source: §4.3 · p. 6

4.4 Long-Context Benchmarks

• On RULER long-context tasks trained at 2K and extrapolated to 8K, NHA achieves the best results across multiple tasks despite Transformers retaining an in-length advantage.The evaluated tasks include NIAH-MK, NIAH-MQ, RULER-CWE, and RULER-Hotpot.Source: §4.4 · p. 6, Table 2 · p. 7

4.5 Operator Efficiency

• NHA and GSA maintain near-linear operator scaling, with NHA matching GSA speed across tested lengths while FlashAttention grows sharply on long sequences.The comparison uses a Triton benchmark on one NVIDIA H100-80G GPU, with 32 long-term slots and a 32-token window for NHA.Source: §4.5 · p. 6, Figure 5 · p. 7

4.6.2 Results

• Structural hybridization replaces selected full-attention layers in pretrained Llama-3-8B and Qwen2.5-7B with NHA modules; NHA-Llama closely matches original Llama with a better accuracy–efficiency trade-off.Hybridizing Qwen3-30B-A3B also yields competitive results across diverse benchmarks.Source: §4.6 · p. 7, §4.6.2 · pp. 7–8, Table 3 · p. 8

4.6.3 Efficiency

• NHA-Llama3-8B shows slower growth in inference latency and GPU memory than Llama3-8B as input length increases.The comparison generates 1K tokens under varying input lengths.Source: §4.6.3 · p. 8, Figure 6 · p. 8

4.7.1 Inference-Time Architecture Search Exploration

• Inference-time window reconfiguration reveals layer-specific sensitivity to global information, allowing four full-attention layers to match a 12-layer baseline in one FDA configuration.The same exploration reports that a pure-Transformer configuration can surpass a standard Transformer pretrained from scratch.Source: §4.7.1 · p. 9, Table 5 · p. 9

4.7.2 Memory and Fusion Ablation

• Removing either memory harms performance, token shift further degrades results, and unified softmax outperforms fixed or learnable weighted fusion.These ablations support complementary long- and short-term memories and joint attention over both memories.Source: §4.7.2 · p. 9, §4.7.3 · p. 9, Table 6 · p. 9

• Future work could learn task-specific slot initial states and selectively compress chain-of-thought reasoning into long-term memory.The proposed directions aim to retain essential information while reducing computational overhead.Source: §5 · p. 9

• Slot size and window size may require careful tuning, while operator optimization and structured depth-wise window variation remain open limitations.The current evaluation primarily considers uniform window settings across layers.Source: §6 · p. 9

• NHA outperforms hybrid baselines on recall-intensive, commonsense reasoning, and long-context benchmarks.The paper presents this as evidence of stronger overall effectiveness.Source: A Answers to Research Questions · p. 12
• NHA matches standard Transformer accuracy competitively while reducing inference time and memory in pretrained-LLM applications.The efficiency evidence is reported for both general NHA operation and pretrained-model hybridization.Source: A Answers to Research Questions · p. 12
• Ablations identify both memories and token shift as essential, while unified softmax outperforms fixed or learned weighted fusion.Additional analyses examine long–short memory and slot- and window-size choices.Source: A Answers to Research Questions · p. 12
• Finetuned NHA-Llama and NHA-Qwen scale to billion-parameter LLMs with strong accuracy and efficiency gains, including Qwen3-30B-A3B.The paper also reports benefits from using fewer full-attention layers than other hybrid models.Source: A Answers to Research Questions · p. 12

B Related Work and Positioning

• NHA concatenates explicit local-window keys with compressed long-term slot keys and applies one softmax, enabling context-dependent allocation without manual fusion rules.This differs from prior hybrids that compute separate attentions and combine their outputs with fixed or learned coefficients.Source: B Related Work and Positioning · pp. 12–13
• NHA uses one unified layer type throughout the network, while setting each layer’s window from zero to full sequence length interpolates between linear and Transformer-like behavior.This avoids heterogeneous module alignment while retaining depth-wise control over attention behavior.Source: B Related Work and Positioning · p. 13

C.1 Pretrain Experiment Setup

• The study pretrains 340M-parameter models for 15B tokens and 1.3B-parameter models for 100B tokens on SlimPajama using matched optimization settings.Training uses AdamW, a 3e-4 learning rate, cosine scheduling, weight decay 0.01, gradient clipping 1.0, and seed 42.Source: §C.1 · pp. 13–14

C.2 Finetune Experiment Setup

• NHA-Llama3 and NHA-Qwen2.5 each use four full-attention layers through inherited Transformer weights and layer-specific full-length windows.The Llama3 configuration selects one full-attention layer per eight-layer block, while Qwen2.5 selects one per seven-layer block.Source: §C.2 · p. 14

C.3 Large-Scale Hybridization with Qwen3-30B-A3B

• Hybridized Qwen3-30B-A3B achieves competitive performance while reducing reliance on full-attention layers, supporting NHA scalability to tens-of-billions-parameter models.The large-model experiment uses a conservative 2:1 hybridization strategy and 5B SlimPajama tokens.Source: §C.3 · p. 14

C.4.1 Ablation on Slot Size and Window Size

• Larger slot counts and moderate windows generally improve recall-intensive tasks, whereas commonsense reasoning remains stable across configurations.Very small windows reduce short-term precision, while very large windows approach full attention and reduce efficiency gains.Source: §C.4.1 · p. 14, Table 7 · p. 14

C.4.2 Ablation on Positional Embedding

• Applying positional encoding solely to short-term memory yields the best performance in the evaluated NHA strategies.The result is attributed to enhancing softmax attention without interfering with positional information captured by long-term recursive updates.Source: §C.4.2 · pp. 14–15, Table 8 · p. 14

C.5 Relation Between Full Attention Layers and Recall Performance

• NHA-Llama3-8B attains the best recall performance with only 4 full attention layers.Figure 7 relates average recall accuracy to the number of full-attention layers.Source: Figure 7 · p. 15

C.6 Advantage over Mamba-H: Structural Compatibility

• At a 12.5% hybrid ratio, NHA-Llama-3 achieves a +9.06 average zero-shot score gap over Mamba-Llama-3 while using half the training tokens.NHA’s unified layer treats full attention as a full-length-window case, enabling parameter inheritance from Llama-3.Source: §C.6 · p. 15

D Gradient Coupling Analysis

• Unified softmax produces query-dependent long–short trade-offs that static or input-only fusion coefficients cannot generally reproduce.The allocation reflects the current query and aggregate similarity interactions with all stored keys.Source: D Gradient Coupling Analysis · p. 15
• Gradients adjusting long-term logits depend on short-term attention mass, while short-term gradients depend on long-term mass, creating cross-memory coupling.This coupling aligns both memories in the same similarity space.Source: D Gradient Coupling Analysis · p. 15
• Independent weighted fusion gives no allocation gradient to logits from the other memory type, producing weaker coadaptation between memories.The coefficient is applied after two separate softmax operations, so unrelated-branch logits receive no signal for adjusting the allocation.Source: D Gradient Coupling Analysis · p. 15

E Memory Analysis

• Different layers and heads specialize in memory usage, while later positions increasingly attend to long-term slots for distant-information retrieval.Early tokens before the sliding window fills rely more on long-term memory, and positional patterns appear across the network.Source: E Memory Analysis · p. 15, Figure 8 · p. 16
• NHA’s long-term-memory usage increases toward later positions, unlike the input-projection fusion variant’s more uniform distribution.This position-sensitive trend reflects adaptive weighting within the unified attention distribution rather than an externally parameterized trade-off.Source: E Memory Analysis · pp. 15–16, Figure 9 · p. 16

F Datasets

• Models are pretrained on cleaned, deduplicated SlimPajama subsets of 15B tokens for 340M parameters and 100B tokens for 1.3B parameters.SlimPajama contains Common Crawl, Wikipedia, books, and GitHub code.Source: F Datasets · p. 16
• The evaluation uses English benchmarks spanning standard language understanding and recall-intensive tasks, including WikiText, LAMBADA, ARC, HellaSwag, FDA, SWDE, SQuAD, Natural Questions, TriviaQA, and DROP.The datasets are publicly available and used under their original purposes and licenses.Source: F Datasets · pp. 16–17