PaperlayerView exact revision on arXiv

Source-linked AI summary

HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models

Zhuokun Chen, Feng Chen, Xi Lin, Xiyu Wu, Jiahao He, Jianfei Cai, Bohan Zhuang

arXiv:2610.05739v1cs.CVcs.AI

TL;DR

Long-horizon world models need to recover earlier scenes after they leave recent context, but recurrent linear attention can progressively lose distant scene information. HLA-WM uses compact chunk-wise transition summaries, camera-geometry retrieval, and selective chronological recomposition to construct query-specific recurrent states without retraining. On SANA-WM-Bench, HLA-WM improves revisit fidelity and camera controllability; on the 60-second benchmark, it improves the base autoregressive generator by 0.74 dB in PSNR and reduces rotation error by 28.5%.

  • Problem

    Long-horizon world models need to recover earlier scenes after they leave recent context, but recurrent linear attention can progressively lose distant scene information.

    Source: §1 · p. 1

  • Method

    HLA-WM uses compact chunk-wise transition summaries, camera-geometry retrieval, and selective chronological recomposition to construct query-specific recurrent states without retraining.

    Source: §1 · p. 2

  • Results

    On SANA-WM-Bench, HLA-WM improves revisit fidelity and camera controllability; on the 60-second benchmark, it improves the base autoregressive generator by 0.74 dB in PSNR and reduces rotation error by 28.5%.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    Selective recurrent memory improves long-range scene recall while retaining substantially lower memory cost than full KV caching and adding only modest inference overhead.

    Source: §1 · p. 3, §6 · p. 10

    HLA-WM is currently training-free, and its retrieval uses camera geometry and FOV overlap without explicitly modeling occlusion, visibility, or semantic relevance.

    Source: §6 · p. 10

Abstract

from arXiv · show

Long-horizon video world models require persistent memory to preserve scene consistency over extended rollouts. Softmax attention retains the full generation history through a growing KV cache, whereas recurrent linear attention compresses history into fixed-size states with substantially lower memory cost. However, we identify severe long-range forgetting in Gated DeltaNet (GDN), where information from distant but relevant scenes is progressively attenuated by subsequent state updates. To address this limitation, we propose HLA-WM, a training-free hybrid linear-attention framework that combines coarse-grained geometry-guided retrieval with fine-grained recurrent linear-state computation. HLA-WM exploits the affine structure of GDN to cache compact chunk-wise transition summaries, retrieve scene-relevant historical chunks using camera geometry, and recompose them into query-specific recurrent states. On the $60$-second SANA-WM-Bench, HLA-WM improves all six aggregate revisit-consistency and camera-control metrics of the base autoregressive generator without additional training, including a $0.74$ dB PSNR gain and a $28.5\%$ reduction in rotation error. The improvements persist after downstream refinement and generalize to MBench-A, where HLA-WM consistently improves all three revisit-consistency metrics across all four subsets and all evaluated inference modes over $547$ samples. At a $60$-second context, HLA-WM reduces historical-state memory by $12\times$ relative to full KV caching while incurring at most a $1.6\%$ reduction in inference throughput. These results demonstrate that selectively addressable recurrent memory can improve long-range scene recall while preserving the efficiency advantages of GDN. Project page: https://caesarhhh.github.io/hla-wm/

1 Introduction

Long-horizon world models need to recover earlier scenes after long excursions, but recurrent memory can progressively lose distant information. HLA-WM uses geometry-guided retrieval and recomposition of compact chunk summaries; experiments report improved revisit consistency and camera control without additional training.

1 Introduction

  • Long-horizon simulation requires recovering previously observed scenes after they leave recent context, while full-history KV storage grows with rollout length.Recurrent linear attention compresses history into fixed-size states, motivating the question of how to retain efficiency while recovering distant scenes.Source: §1 · p. 1
  • In GDN-based rollouts, revisit consistency declines as temporal gaps increase, and revisited scenes can change in structure after the camera returns.The paper attributes this forgetting to distant scene information surviving intervening state updates.Source: §1 · p. 2, Figure 1 · p. 2
  • Experiments report consistent long-range revisit-fidelity improvements on SANA-WM-Bench and MBench-A.At Stage 1, HLA-WM directly modifies recurrent memory; the introduction also reports that improvements largely persist after downstream refinement.Source: §1 · pp. 2–3
  • HLA-WM is a training-free hybrid framework that uses compact affine summaries, camera-guided retrieval, and chronological recomposition for selectively addressable historical memory.It changes historical memory storage and access rather than retraining the underlying generator.Source: §1 · pp. 2–3
  • On 60-second SANA-WM-Bench, HLA-WM improves PSNR by 0.74 dB and reduces rotation error by 28.5% without training.The paper also reports improvements across SANA-WM-Bench and MBench-A, with gains persisting after downstream refinement.Source: §1 · pp. 2–3

2 Related Work

Prior world-model approaches preserve history through selected frames, tokens or KV caches, features, or spatial representations. Recurrent linear attention offers fixed-size memory, but distant information remains susceptible to attenuation and interference during long rollouts.

2 Related Work

  • World models preserve history using selected frames, visual tokens or KV caches, compact feature memories, or queryable spatial representations.These approaches add storage and retrieval overhead; some frame-based methods also require re-encoding retrieved observations and recomputing attention caches.Source: §2 · p. 3
  • Linear attention compresses history into fixed-size recurrent states, while GDN combines data-dependent decay with delta-rule updates for adaptive forgetting and selective memory updates.The related work describes recurrent linear attention as compact cross-frame memory for autoregressive video generation.Source: §2 · p. 3
  • Despite their memory efficiency, recurrent linear-attention methods make distant information increasingly susceptible to attenuation and interference during long-horizon generation.The paper frames this as a consequence of continuously compressing historical observations into a limited recurrent state.Source: §2 · p. 3

3 Empirical Insights

GDN repeatedly transforms early scene information through subsequent chunk transitions, so information can weaken before a spatially relevant revisit. HLA-WM instead keeps chunk summaries independently addressable and selectively recomposes relevant history.

3 Empirical Insights

  • An early chunk's information is repeatedly transformed by later transition matrices, and smaller cumulative retention indicates stronger attenuation.The retention measure is the average norm of the intervening transition product.Source: §3 · p. 4
  • Chunk 3's retained influence falls to 0.0416 by Chunk 35, when the camera returns near the earlier viewpoint and the generated scene substantially deviates.The chunks in between mostly correspond to different viewpoints, yet their transitions are still applied to Chunk 3's memory.Source: §3 · p. 4
  • HLA-WM represents historical chunks with independent GDN summaries and uses geometry-guided retrieval to compose selected history into a query-specific recurrent state.The overview includes persistent sink and recent context, with selected summaries composed in chronological order.Source: Figure 3 · p. 5
  • GDN memory follows temporal progression, but revisit relevance can depend on spatial and geometric proximity, creating a mismatch in long-range rollouts.This motivates selectively preserving and composing scene-relevant historical states rather than propagating all past information uniformly.Source: §3 · p. 5

4 Method

HLA-WM organizes Gated DeltaNet history as independently addressable chunk memories, retrieves scene-relevant chunks using camera geometry, and selectively recomposes them for each query.

4.1 Overview

  • HLA-WM replaces one accumulated GDN memory with independently addressable chunk-wise memories, retrieved by camera geometry and selectively composed for each query.This combines coarse-grained geometry-guided retrieval with recurrent linear attention.Source: §4.1 · p. 5

4.2 Chunk-wise GDN Memory

  • Affine chunk summaries (A_i, B_i) encode memory transformation and chunk-written information, and associative composition permits recombining selected summaries without replaying visual tokens.Source: §4.2 · p. 6

4.3 Geometry-Guided Memory Retrieval

  • Geometry ranks eligible historical chunks by proxy frustum overlap; a first-chunk sink and recent context remain, with recent budget expanding to three chunks on abrupt camera turns.The overlap uses camera poses, intrinsics, and a scene-level median-depth estimate, and is a lightweight proxy rather than exact 3D intersection.Source: §4.3 · p. 6

4.4 Selective State Recomposition

  • Retrieved summaries return to chronological order and reconstruct historical state, shortening propagation through irrelevant chunks while preserving selected chunks’ original GDN dynamics.Cached chunk summaries allow recomposition without reforwarding selected historical chunks; storage scales with chunk count rather than visual-token count.Source: §4.4 · p. 7

5 Experiments

Experiments evaluate HLA-WM on SANA-WM-Bench and MBench-A, measuring revisit consistency, camera control, and memory cost. The reported improvements persist across benchmarks and refinement settings, with qualitative examples illustrating better scene preservation on revisits.

5.1 Experimental Setup

  • Experiments use official 1.6B SANA-WM, 961-frame SANA trajectories, and 547 MBench-A samples, evaluating revisit fidelity and camera control.The default retains a sink, recent context, and the top-1 geometrically ranked historical chunk; recent context expands to three chunks above a 25° turn threshold.Source: §5.1 · p. 8

5.2 Main Results

  • Across SANA-WM-Bench and MBench-A, HLA-WM improves revisit fidelity, with camera-control gains on SANA-WM-Bench and revisit-metric gains across MBench-A inference modes.The MBench-A comparison is summarized in Table 2.Source: §5.2 · p. 9, Table 2 · p. 9
  • On SANA-WM-Bench Hard trajectories at Stage 1, PSNR rises 0.74 dB, SSIM rises 0.0223, LPIPS falls 0.0196, and RotErr falls 26.0%.On Simple trajectories, PSNR gains 0.73 dB and RotErr falls 31.0%; full-sequence refinement improves all six metrics on both splits.Source: §5.2 · p. 9
  • On MBench-A, Stage 1 PSNR and SSIM rise 0.84 dB and 0.0303, while LPIPS falls 0.0428; all three revisit metrics improve across three inference modes.The gains remain after refinement, including PSNR gains of 0.37 dB under causal AR refinement and 0.58 dB under bidirectional refinement.Source: §5.2 · p. 9, Table 2 · p. 9
  • At a 60-second context, HLA-WM uses 2.15 GiB of historical-state memory, 12× less than full KV caching, with at most a 1.6% throughput reduction.Figure 4 presents the memory-scaling comparison.Source: §5.2 · p. 9, Figure 4 · p. 9
  • On long-range revisits, HLA-WM better preserves the indoor aisle and shelf layout under causal AR refinement and outdoor scene geometry under bidirectional refinement.Figure 5 shows first visits, intermediate views, and later revisits along the same camera trajectory.Source: §5.2 · p. 10, Figure 5 · p. 10

6 Conclusion

The paper identifies long-range forgetting in recurrent linear attention and introduces HLA-WM to retrieve and recompose geometrically relevant chunk memories. Experiments report improved revisit fidelity and camera controllability with substantially lower memory cost than full KV caching, while noting limits in training and retrieval signals.

6 Conclusion

  • Distant but relevant scene information in GDN is progressively attenuated by subsequent, often unrelated, state transitions.HLA-WM retrieves geometrically relevant chunk memories and selectively recomposes them for long-range recall.Source: §6 · p. 10
  • SANA-WM-Bench experiments show improved revisit fidelity and camera controllability with substantially lower memory cost than full KV caching and modest inference overhead.The method is currently training-free, which limits achievable gains; retrieval uses camera geometry and FOV overlap without explicit occlusion, visibility, or semantic relevance modeling.Source: §6 · p. 10
  • HLA-WM's retrieval relies solely on camera geometry and FOV overlap, without explicitly modeling occlusion, visibility, or semantic relevance.The paper suggests integrating selective state retrieval into training and using richer or learned retrieval signals as future work.Source: §6 · p. 10

A.1.1 Geometry-Aware Historical Retrieval

Geometry-aware retrieval estimates overlap between historical and query chunks using camera-trajectory metadata and proxy points. It ranks eligible chunks by that score while retaining sink and recent context, with the overlap serving as an approximation rather than exact scene intersection.

A.1.1 Geometry-Aware Historical Retrieval

  • Retrieval estimates overlap from camera intrinsics, extrinsics, and a scene-level median-depth scalar, without generated RGB frames or semantic features.Each camera frame uses a 7×7 image-plane grid; sampled rays are instantiated at three proxy depths, and proxy points are transformed into world coordinates.Source: §A.1.1 · p. 14
  • In the 60-second setting, 961 frames are divided into 40 chunks, giving approximately 24 camera poses per chunk.Each chunk is assigned starting and ending frame indices.Source: §A.1.1 · p. 14
  • The symmetric overlap score uses chunk-point visibility, but is only a proxy for geometric overlap and does not model occlusion or scene geometry.The directed visibility score is the fraction of a chunk's proxy points that project with positive depth into another chunk's image plane.Source: §A.1.1 · p. 14
  • For each query, retrieval keeps the sink and recent chunks, selects the top-K remaining historical chunks by overlap score, then restores chronological order.Source: §A.1.1 · p. 14

A.1.2 Turn-Aware Recent Context

The recent-history budget adapts to motion within the current query chunk to preserve local continuity during abrupt trajectory changes.

A.1.2 Turn-Aware Recent Context

  • The recent-history budget adapts to motion within the current query chunk to preserve local continuity under abrupt trajectory changes.The motion criterion uses the chunk's translation vectors and their angular difference.Source: §A.1.2 · p. 14
  • The budget is set using the angular difference computed for the current chunk's translation vectors.The supplied passages state that the angular difference is computed before setting the budget.Source: §A.1.2 · pp. 14–15
  • When the 25° turn threshold triggers expansion, the larger recent context applies only to the current chunk.Subsequent chunks return to the default budget unless triggered again.Source: §A.1.2 · p. 15

A.2 Implementation of State Recomposition

HLA-WM caches chunk-level affine statistics and uses them to reconstruct query-specific GDN states without replaying historical chunks through the Transformer.

A.2 Implementation of State Recomposition

  • After each chunk, HLA-WM caches frame-wise Phase-A sufficient statistics that characterize its affine effect on an incoming recurrent state.For a query, selected chunks' statistics are gathered in chronological order and passed to the original Phase-B forward scan.Source: §A.2 · p. 15
  • The original Phase-B forward scan recomposes the selected historical chunks into a query-specific GDN state without re-evaluating those chunks through the Transformer.Source: §A.2 · p. 15
  • Recomposition changes only the main GDN matrix and normalization states; camera-GDN and local convolutional states keep chronological updates.Compatible GDN blocks are batched to reduce kernel-launch and data-movement overhead without changing the recurrence.Source: §A.2 · p. 15

A.2.1 Overall Inference Procedure

HLA-WM combines geometric retrieval and recurrent-state recomposition with persistent and recent context; results span benchmark settings, ablations, retrieval choices, and refined outputs.

A.2.1 Overall Inference Procedure

  • The inference procedure selects historical chunks geometrically, recomposes their main-GDN state in chronological order, and generates the current chunk without replaying historical visual tokens.The component ablation finds that the full method performs best across all six metrics, with selective composition, persistent sink memory, and adaptive recent context contributing complementarily.Source: §A.2.1 · p. 15, §A.3 · pp. 15–16

A.4 GDN-State Recomposition and KV-Cache Alignment

  • GDN-only drives the main long-range recall gains, while KV alignment adds complementary stabilization; their combination gives the strongest results across the reported settings.On Hard trajectories under bidirectional refinement, GDN-only raises RotErr from 13.17° to 33.04°, while full HLA-WM achieves the best PSNR, LPIPS, and all three camera-control metrics.Source: §A.4 · pp. 16–17

A.5 Number of Retrieved Historical Chunks

  • With K = 1, HLA-WM achieves the best PSNR, SSIM, and LPIPS; larger retrieval budgets yield only modest camera-control gains in some metrics.Geometry-guided selection at K = 4 improves PSNR from 9.91 to 10.00, SSIM from 0.1910 to 0.1936, and reduces LPIPS from 0.5957 to 0.5885 versus Uniform, though Uniform has marginally lower TransErr.Source: §A.5 · p. 18, Table 6 · p. 18, §A.6 · p. 19

A.7 Effect of Revisit Interval

  • Across Stage 1, AR refinement, and bidirectional refinement, revisit gains generally grow for medium and long intervals; PSNR gains increase consistently with revisit interval.LPIPS shifts from minor degradation at short intervals to clear improvement at long intervals; SSIM's trend is not monotonic under AR refinement.Source: §A.7 · p. 19, Figure 6 · p. 19

A.8.1 SANA-WM-Bench Category Results

  • Stage 1 yields the most consistent category-level gains, while downstream refinement brings category-dependent trade-offs; aggregate revisit metrics improve under each evaluated mode.For MBench-A, all three revisit metrics improve in every mode-subset combination, although gain magnitude varies.Source: §A.8.1 · p. 20, §A.8.2 · p. 20

A.8.2 MBench-A Subset Results

  • On MBench-A's 547 samples, HLA-WM improves PSNR, SSIM, and LPIPS across all three inference modes and all four subsets.On SANA-WM-Bench, category-level gains are broadly distributed rather than concentrated in one scene type, with Stage 1 showing the most consistent improvements.Source: §A.8.2 · p. 20, §A.8.1 · p. 20

A.9.1 Stage 1

  • Qualitative comparisons show better preservation of scene structure and distinctive content at Stage 1, after causal AR refinement, and after bidirectional refinement.Examples include maintaining mountain contours, street geometry, room layouts, and revisited scene identity where the baseline drifts or changes structure.Source: §A.9.1 · p. 21, §A.9.2 · p. 22, §A.9.3 · p. 22
Loading 2610.05739v1…