WorldAttention: An Efficient Attention Architecture for Interactive Video World Models

1 Introduction

• Sliding-window caches discard earlier visual context, while retaining full-history KV for interactive recall quickly exceeds GPU memory capacity.The resulting challenge is to preserve history while efficiently managing both attention computation and caching.Source: §1 · pp. 1–2
• 0.9472 on VBench-Long and 0.9668 on InterVBench are WorldAttention’s subject consistency scores as it surpasses prior state-of-the-art methods.The paper also reports state-of-the-art performance for coherent, high-fidelity generation over minute-long horizons.Source: §1 · p. 2
• WorldAttention combines page-level Hierarchical KV Cache retrieval across GPU, CPU, and NVMe with head-adaptive Hybrid Sparse Attention and kernels that reorganize retrieved pages into contiguous buffers.HKV minimizes GPU memory residency; HSA reduces redundant computation within retrieved pages.Source: §1 · p. 2
• The HSA kernel integration achieves 14.02× speedup over FlashAttention-3, and HSA with HKV achieves 2.21× end-to-end speedup.Source: §1 · p. 2

2 Related Work

• Video-generation methods use sliding-window caches for chunk-wise conditioning or sparse retrieval for long-horizon interactive memory, but growing caches and coarse retrieval remain bottlenecks.Source: §2 · p. 2
• WorldAttention’s architecture pairs Hierarchical KV Cache for coarse-to-fine retrieval with Hybrid Sparse Attention for dual-branch attention computation.Source: Figure 1 · p. 3
• Sparse and linear attention methods reduce computation through pruned interactions or O(L) global-attention approximations, while hybrid methods combine coarse global modeling with local sparse refinement.The cited approaches are described as lacking explicit co-design with KV management and hardware-aware kernel execution for chunk-wise long-horizon generation.Source: §2 · pp. 2–3

3.1 Overview

• WorldAttention combines hierarchical KV retrieval, hybrid sparse attention over retrieved tokens, and customized kernel integration with memory-layout optimization.The global KV cache preserves entries from past chunks, from which relevant historical context is dynamically selected during decoding.Source: §3.1 · p. 3

3.2 Hierarchical KV Cache

• HKV partitions chunks into semantically coherent pages of 8 frames and retrieves pages in two stages: prompt-level chunk selection followed by page-level selection.It selects the top-1 chunk by prompt similarity, then the top 4 pages; at 1,560 tokens per frame, those pages contain 49,920 tokens.Source: §3.2 · p. 4
• HKV migrates inactive pages from GPU memory to CPU memory when capacity is exceeded, then to NVMe when CPU capacity is exceeded.NVMe retains a full backup of historical pages, while GPU occupancy remains bounded as the cache grows.Source: §3.2 · p. 4

3.3 Hybrid Sparse Attention

• HSA combines a linear global-attention branch with a block-sparse branch, reducing the linear branch's complexity from O(L^2) to O(Lr) for fixed rank r ≪ L.The sparse branch computes attention at original token resolution but restricts computation to selected blocks.Source: §3.3 · p. 5
• HSA selects the minimal set of blocks exceeding each head's attention-mass threshold, using coarser blocks for historical pages and finer blocks for the current chunk.The inter-chunk block size is 128 and the intra-chunk block size is 64.Source: §3.3 · p. 6
• Head-specific sigmoid gates fuse the linear and sparse branch outputs, allowing query-dependent modulation of their contributions.The linear branch supplies global information flow, while the sparse branch supports high-resolution modeling where needed.Source: §3.3 · p. 6

4.1 Benchmarks and Metrics

• Evaluation uses 160 interactive 60-second VBench-Long videos and InterVBench, while training extends a causal-attention model through interactive long tuning to 60 seconds.The implementation builds on Wan2.1-T2V-1.3B and uses switch prompts, with each training sample containing one prompt switch.Source: §4.1 · p. 6, §4.2 · p. 6

4.3 Main Results

• WorldAttention achieves the best performance across nearly all VBench-Long metrics and shows strong prompt adherence, smooth prompt transitions, and long-range consistency.The reported VBench-Long results include subject and background consistency, motion smoothness, dynamic degree, and image quality.Source: Table 1 · p. 7, Table 2 · p. 7, §4.3 · p. 7

4.4 Efficiency

• 86.55 quality score and 22.0 FPS make WorldAttention the highest-quality and fastest method in the single-prompt 30-second VBench-Long evaluation.The comparison reports both generation quality and end-to-end inference throughput.Source: Table 3 · p. 8, §4.3 · p. 8, §4.4 · p. 8
• HSAtk peaks at 14.02× over FlashAttention-3; WorldAttention reaches 2.21× end-to-end speedup, with gains persisting across B200 scales, resolutions, and horizons.The end-to-end speedup is attributed to system-oriented kernel customization; gains become more pronounced for longer sequences and larger models.Source: §4.4 · p. 8, Table 9 · p. 11, Hierarchical KV cache · p. 11

Hybrid sparse attention.

• Combining HSA’s linear and sparse branches improves most VBench-Long metrics; the (τmax = 1.00, τmin = 0.35) setting leads five listed metrics.The linear-only branch favors consistency, while sparse-only attention improves dynamic degree and aesthetic quality; sparse performance depends on its thresholds.Source: Hybrid sparse attention · pp. 8–9, Table 4 · p. 8, Figure 6 · p. 8

Hierarchical KV cache.

• Under the same four-page budget, HKV scores 85.53 overall and 25.01 CLIP in the 50–60-second segment, outperforming recent-, random-, and mismatched-page retrieval.The two-stage method first narrows the search by prompt similarity, then selects relevant pages using query–key similarity.Source: Table 6 · p. 9, Hierarchical KV cache · pp. 9–10
• HKV improves overall quality and segment-wise CLIP scores while maintaining high throughput, unlike sliding-window degradation or the throughput cost of re-caching more frames.The (Psize = 8, Kp = 4) configuration achieves the best overall performance in the reported ablation.Source: Hierarchical KV cache · p. 9, Table 5 · p. 9
• Top-1 chunk retrieval best balances precision and efficiency; with three similar historical chunks, quality falls only from 85.53 to 85.18 and late-segment CLIP remains 24.74.The evaluations use the same active budget of four pages; spreading that budget over more chunks can reduce fine-grained coverage or add irrelevant context.Source: Table 7 · p. 10, Hierarchical KV cache · p. 10, Table 8 · p. 10

• HSA combines linear global attention with head-adaptive block sparsity, while HKV uses multi-tier paging to balance historical context against restricted GPU resources.The paper reports superior temporal consistency, prompt alignment, and generation quality alongside improved decoding efficiency on VBench-Long and InterVBench.Source: §5 · p. 11

A.1.1 3D Full Attention

• Flattening a (T, H, W) video latent into L = THW tokens lets every token attend to every other token, with full attention having quadratic complexity in sequence length.The flattening maps spatial-temporal locations to sequence indices; full attention uses an all-zero mask.Source: §A.1.1 · p. 15

A.1.2 Sparse Attention

• Sparse attention masks selected interactions to reduce computation, but choosing critical tokens trades selection accuracy against prediction overhead.Full score computation is accurate but limits savings to the AV stage, while fixed patterns add no prediction overhead but may miss important long-range interactions.Source: §A.1.2 · p. 15

A.1.3 Block Sparse Attention

• Block-sparse attention partitions the mask into blocks that can be processed densely or skipped, aligning sparse computation with GPU execution.The video latent is grouped into contiguous 3D cubes mapped to GPU execution blocks; smaller blocks offer finer patterns, while larger blocks improve throughput at coarser resolution.Source: §A.1.3 · pp. 15–16

A.1.4 Global and Local Context in Sparse Attention

• Sparse connectivity limits receptive fields; lightweight global modules can capture coarse long-range information, while local inductive biases can enhance fine-grained feature learning.The described alternatives are to augment sparse attention with global context or incorporate locality priors.Source: §A.1.4 · p. 16

A.2 Semantic Sparse KV Cache

• BIFE retains the top-k tokens by aggregated attention importance from each chunk, then stores sparse KV entries in a global memory bank.The cache retrieves relevant context using semantic similarity between the current prompt embedding and past chunk embeddings.Source: §A.2 · pp. 16–17
• The top-l most similar historical chunks are selected, and their sparse KV states are combined with the most recent chunks to construct the context.Source: §A.2 · p. 17
• Despite reducing memory consumption and improving long-range coherence, storing sparse KV entries from all previous chunks still makes cache size grow with sequence length.This limitation motivates WorldAttention’s hierarchical KV management strategy.Source: §A.2 · p. 17

B Metrics

• VBench-Long evaluation adopts five complementary VBench metrics covering frame-level imaging quality, motion smoothness, aesthetics, background consistency, and subject consistency.Subject consistency evaluates whether appearance remains temporally coherent throughout the video.Source: B Metrics · p. 17
• InterVBench uses Video Drift Error (VDE) to measure temporal drift across video segments with five drift-aware metrics for clarity, motion, aesthetics, background, and subject.VDE scores indicate stability, with lower scores representing better stability; VDE Subject tracks identity drift and recognizability.Source: B Metrics · p. 17

C Limitation and Future Work

• WorldAttention’s current framework primarily uses textual prompts for control, leaving richer interaction signals such as actions, trajectories, and embodied cues for future work.The paper proposes that action-conditioned generation could support controllable environment simulation and decision-driven video prediction.Source: C Limitation and Future Work · p. 17

D Kernel Customization

• The kernel customization pairs global linear and head-adaptive local sparse attention kernels with semantically grouped KV pages mapped to contiguous memory after retrieval.The optimized KV layout is intended to minimize data movement and maximize GPU utilization.Source: D Kernel Customization · pp. 17–18

E Results on InterVBench

• WorldAttention achieves state-of-the-art performance across most InterVBench metrics, with improved long-horizon stability and overall generation quality versus prior methods.Table 12 reports five VDE metrics and five complementary VBench metrics.Source: E Results on InterVBench · p. 18, Table 12 · p. 18

F Sparse Branch

• HSA selects the smallest set of high-scoring blocks whose cumulative attention exceeds a head-specific threshold, replacing fixed Top-K selection.This produces different sparsity patterns across heads, from highly sparse local focus to broader-context attention.Source: F Sparse Branch · pp. 18–19
• The Gini coefficient measures each head’s attention-score concentration and maps to a bounded coverage threshold that controls how many blocks are selected.More peaked distributions yield higher Gini values; higher thresholds select more blocks, while lower thresholds limit selection.Source: F Sparse Branch · p. 19

G Linear Branch

• Retrieved KV tokens are divided into fixed-length segments, projected independently with low-rank matrices, and concatenated to form compressed context.The method is described as preserving spatiotemporal physical anchors and relative positional topologies for long-range information.Source: G Linear Branch · p. 19
• Linformer recovers long-range attention mass and key-context peaks closer to full attention than standard linear attention, and has higher output cosine similarity to the full-attention baseline.The reported comparison covers attention distributions at the initial frame and within local contexts, as well as the last chunk.Source: G Linear Branch · pp. 19–20

H Visualization

• Qualitative comparisons report stronger subject consistency, smoother motion transitions, and more stable scene structure across the entire sequence than existing approaches.Figures 8–13 present interactive long-video examples.Source: H Visualization · p. 21