PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry as Address: Routing Attention to Visual Memory for Long-Horizon Camera-Controlled Video Generation

Zesong Yang, Weikai Chen, Liyuan Cui, Lutao Jiang, Runze Zhang, Yingda Yin, Xiaoyang Huang, Kai Yan, Keyang Luo, Wangguandong Zheng, Xin Wang, Hujun Bao, Zhaopeng Cui

arXiv:2609.34722v1cs.CV

TL;DR

Long-horizon camera-controlled video generation must recover previously observed content from an expanding history. GEAR uses per-frame geometry to route attention to relevant historical visual features and filters occluded matches with visibility evidence. The paper reports state-of-the-art visual quality, precise camera control, and revisit consistency in minute-long generation along challenging trajectories.

  • Problem

    Long-horizon camera-controlled generation must recover previously observed scene content after temporal gaps while following prescribed camera trajectories.

    Source: §1 · p. 2

  • Method

    GEAR uses per-frame geometry to address historical visual tokens, Geometric Correspondence Attention to retrieve their features during denoising, and an Invisible Octree to filter occluded matches.

    Source: §1 · pp. 2–3, §3.3 · p. 5

  • Results

    GEAR achieves state-of-the-art visual quality, precise camera control, and revisit consistency, enabling minute-long generation along challenging trajectories.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    Using geometry as an address rather than persistent memory enables long-horizon generation without error accumulation from global 3D fusion.

    Source: §5 · p. 9

    GEAR relies on an external 3D model for depth estimation, adding computational overhead.

    Source: §5 · p. 10

Abstract

from arXiv · show

Long-horizon camera-controlled video generation requires recovering previously observed content from an ever-growing visual history. Existing approaches either search historical context implicitly or reconstruct it into persistent 3D memory, facing inefficient memory access or accumulated geometric errors. Our key insight is that geometry need not explain the scene--it only needs to determine where visual memory should be read from, while attention decides what should be recovered. Based on this insight, we introduce GEAR, a Geometry-Enabled Attention Routing framework that uses geometry as an explicit token-level address for visual memory. Rather than fusing historical observations into a persistent global 3D representation, GEAR retains them as frame latents and uses per-frame geometry only to establish token-level correspondences with target views, thereby avoiding persistent error accumulation from global fusion. Guided by these correspondences, Geometric Correspondence Attention (GCA) selectively injects geometrically matched historical features into noisy target patches during denoising. We further introduce an Invisible Octree to accumulate visibility evidence and reject geometrically plausible but occluded correspondences. Extensive experiments demonstrate that GEAR achieves state-of-the-art visual quality, precise camera control, and revisit consistency, enabling minute-long video generation along challenging trajectories.

1. Introduction

Long-horizon camera-controlled generation must retrieve previously observed scene content after temporal gaps, but existing memory strategies trade costly search against errors from persistent 3D fusion. GEAR uses geometry to address historical visual tokens, attention to select useful features, and visibility evidence to reject occluded matches.

1. Introduction

  • Long-horizon generation must recover previously observed scene content after temporal gaps while following prescribed camera trajectories.When a region is revisited outside the model’s temporal context, its appearance must come from past observations.Source: §1 · p. 2
  • Implicit history methods search increasingly large token sets, whereas global 3D memory risks persistent reconstruction and fusion errors.Frame retrieval or compression can reduce search burden but may be too coarse or discard information needed for precise revisitation.Source: §1 · p. 2, Figure 2 · p. 2
  • Prior geometry-based methods select, align, or construct conditioning signals, but leave access to underlying visual features indirect.This motivates using correspondence to determine directly which historical visual tokens each target token can access.Source: §1 · p. 2
  • GEAR uses per-frame geometry as a token-level address, letting visual attention choose useful features among sparse geometrically matched historical tokens.Geometric Correspondence Attention gathers matched features as keys and values, then injects their aggregated features through a residual branch during denoising.Source: §1 · pp. 2–3
  • The Invisible Octree accumulates visibility evidence to filter plausible but occluded correspondences; experiments report state-of-the-art visual quality, camera control, and revisit consistency.The paper reports minute-long generation along challenging user-specified trajectories with lightweight adaptation of a pretrained video model.Source: §1 · p. 3

2. Related Work

Related work spans implicit history retrieval or compression and explicit reconstruction into 3D memories. GEAR instead uses per-frame geometry to select visible historical tokens for sparse correspondence attention without global 3D fusion.

2. Related Work

  • Implicit methods retrieve or compress history but still require dense attention; explicit 3D methods provide spatial grounding while remaining vulnerable to accumulated reconstruction and fusion errors.The cited implicit approaches include Li et al. [20], Sun et al. [31], Xiao et al. [42], and Yu et al. [44]; explicit methods include [6] [18] [38] [41] [49] [50].Source: §2 · p. 3
  • UCM [43] warps positional encodings and Lyra 2.0 [30] injects warped correspondence coordinates, while GEAR selects visible historical tokens for sparse attention without global fusion.Source: §2 · p. 3

3. Methods

GEAR autoregressively generates video chunks from a bounded history of frame latents, using per-frame geometry to address historical patches and filter occluded matches. It combines this routing with geometric attention, degraded-history training, and streaming retrieval for long-horizon generation.

3.1. Problem Formulation and Preliminaries

  • Given an initial frame, prompts, and a camera trajectory, GEAR generates video chunks autoregressively while retaining historical latents and camera parameters as a streaming history bank.Frames are encoded independently to preserve alignment between visual observations and camera poses; training uses a flow-matching objective.Source: §3.1 · p. 3

3.2. Geometry-Addressed Patch Memory

  • GEAR derives patch-level memory addresses from each historical frame’s local geometry, matching source and target latent patches without constructing a globally fused scene representation.The source mesh is built at image resolution to preserve geometric discontinuities that latent-grid depth could blur; multiple valid historical correspondences are retained.Source: §3.2 · pp. 4–5

3.3. Visibility-Aware Correspondence

  • The Invisible Octree accumulates visibility evidence over time and rejects geometrically projectable correspondences that are occluded in the target view.It stores no appearance and serves only as a coarse binary correspondence filter.Source: §3.3 · p. 5

3.4. Geometric Correspondence Attention

  • Geometric Correspondence Attention selectively aggregates matched historical tokens and injects their features into noisy target tokens during denoising.Attention is normalized over matched candidates; with no matches, its output is zero, and the pathway is injected residually while backbone self-attention is retained.Source: §3.4 · pp. 5–6

3.5. Robust Training with Degraded History

  • To address errors accumulating in generated history at inference, GEAR trains with mildly degraded historical context while leaving the target chunk’s flow-matching objective unchanged.A reverse-flow step produces the degraded history used as conditioning, improving robustness to accumulated history errors in long-horizon inference.Source: §3.5 · p. 6

3.6. Long-Horizon Inference

  • For long-horizon inference, GEAR greedily retrieves history frames that maximize newly covered regions, retains the initial and latest frames, and appends generated observations for subsequent chunks.Depth estimates and newly observed geometry also update the Invisible Octree during streaming rollout.Source: §3.6 · p. 7

4. Experiments

GEAR is evaluated on camera-controlled generation, long-horizon revisitation, and component ablations. It achieves strong camera alignment and revisit results, while the ablations examine the roles of geometric attention, visibility filtering, and degraded-history training.

4.1. Experimental Setup

  • Training uses DL3DV-10K clips with I2V and H2V conditioning, a frozen Wan2.1-I2V-14B backbone, and jointly optimized LoRA adapters and GCA modules.H2V samples form 70% of training; degraded-history augmentation is applied to H2V samples from iteration 8K with probability 40%.Source: §4.1 · pp. 7–8

4.2. Quantitative Evaluation

  • Qualitative comparisons show different baseline weaknesses, while GEAR maintains coherent appearance and camera control over extended rollouts.Lyra2 shows degradation and drift under rapid or extended motion; Spatia and WorldStereo show scene distortions, and UCM gradually deviates during long rollouts.Source: §4.2 · pp. 8–9
  • GEAR reduces ATE by 80.3% versus the strongest baseline and achieves the best WorldScore content alignment, photometric consistency, style consistency, and revisit performance.It also maintains coherent appearance and accurate camera control during minute-long explorations with rapid camera motion.Source: Figure 9 · p. 8, §4.2 · pp. 8–9

4.3. Ablation Study

  • Removing GCA increases temporal instability and camera-motion deviation, while replacing correspondence-restricted attention with dense history attention degrades camera control and visual fidelity.Figure 10 illustrates the GCA ablation's effects on temporal stability and adherence to the prescribed trajectory.Source: §4.3 · p. 9, Figure 10 · p. 9
  • The Invisible Octree rejects candidates that conflict with accumulated visibility evidence, suppressing propagation of occlusion-related structural inconsistencies through autoregressive history.Source: §4.3 · p. 9
  • Degraded-history augmentation improves long-horizon stability, and GCA remains comparable under strong correspondence perturbations.The perturbation test retains only 2–3 valid matches out of 9 and shifts the other candidates by 64–128 pixels; Figure 8 illustrates the augmentation result.Source: Figure 8 · p. 8, §4.3 · p. 9, Figure 11 · p. 9

5. Conclusion

GEAR uses per-frame geometry to establish local history-to-target correspondences and retrieves relevant features through Geometric Correspondence Attention. The paper reports minute-long generation with precise camera control and consistent scene persistence, while noting the overhead of external depth estimation.

5. Conclusion

  • GEAR uses per-frame geometry for local history-to-target token correspondences and lightweight Geometric Correspondence Attention to integrate historical features during denoising.Source: §5 · p. 9
  • Treating geometry as an address rather than persistent memory avoids error accumulation from global 3D fusion while supporting minute-long generation with precise camera control and scene persistence.Source: §5 · p. 9
  • GEAR relies on an external 3D model for depth estimation, adding computational overhead; jointly learning geometry and memory addressing is proposed as future work.Source: §5 · p. 10

A. Additional Method Details

The streaming procedure maintains a history bank and visibility evidence, then retrieves geometrically relevant frames for each upcoming camera chunk. Filtered correspondences and latents condition denoising, after which generated observations update the history and octree.

A.1. End-to-End Streaming Generation

  • At each generation step, the history bank stores observed frames, frame-aligned VAE latents, camera parameters, and estimated depths, while the Invisible Octree accumulates visibility evidence.Source: §A.1 · p. 13
  • For each camera chunk, the procedure retrieves frames by geometric coverage, builds patch correspondences from local meshes, and removes candidates marked occluded by the projected Invisible Octree.Source: §A.1 · p. 13
  • Filtered correspondences and retrieved history latents condition the DiT through 25 denoising steps; decoded frames are depth-estimated and added to the history bank and octree.Source: §A.1 · p. 13

A.2. Frame-Aligned VAE Encoding

Temporal compression makes a latent frame combine observations from different camera poses, weakening its geometric correspondence to any one view. Encoding each video frame independently gives every latent frame a unique image, depth map, and camera pose for GCA.

A.2. Frame-Aligned VAE Encoding

  • Temporally compressed latent frames can aggregate observations from different poses, so frame-level depth and camera parameters cannot be unambiguously transferred to their tokens.The ambiguity is more pronounced under rapid camera motion.Source: §A.2 · p. 13
  • Independent single-frame VAE encoding bypasses temporal compression while retaining spatial encoding, producing one latent frame per video frame.Framewise decoding preserves the same alignment at inference.Source: §A.2 · p. 13
  • Each latent frame has a unique associated image, depth map, and camera pose, enabling per-frame geometry to address historical tokens during GCA.Spatial latent tokens also have well-defined image-grid locations.Source: §A.2 · p. 13

A.3. Geometry-Addressed Correspondence Construction

GEAR constructs source-to-target patch correspondences from per-frame geometry, filters occluded matches using accumulated visibility evidence, and retrieves historical frames to cover upcoming target views. It updates depth and visibility as generation proceeds.

A.3. Geometry-Addressed Correspondence Construction

  • Shared face identities in source- and target-view rasterizations provide patch correspondences, which are cached per source frame and filtered for occlusion before GCA accesses them.Local meshes are built from depth-derived geometry without fusing meshes across observations.Source: §A.3 · pp. 14–15, Figure 12 · p. 15

A.4. Invisible Octree Construction and Streaming Update

  • The Invisible Octree tracks unresolved invisible space alongside a global visible mesh, refining partially visible nodes and updating newly exposed regions as observations arrive.The proxies provide visibility information only; the octree mask removes correspondences that fall in target-view regions determined to be occluded.Source: §A.4 · pp. 15–16, Figure 13 · p. 16

A.5. Keyframe History Retrieval

  • History frames are selected by marginal contribution to target-view coverage, while retaining the first and latest frames and capping added value after three covers per location.Coverage is measured after filtering occluded projections with the target-view Invisible Octree mask.Source: §A.5 · p. 16

A.6. Depth Update for Streaming Generation

  • After each chunk, new-frame depths are estimated jointly with uniformly sampled historical anchor frames in pose-conditioned mode, and only new depths are appended to history.Stored historical depths remain unchanged and the new maps update visibility proxies for later chunks.Source: §A.6 · p. 16

B. Dataset and Implementation Details

GEAR is trained on processed DL3DV-10K clips using a frozen Wan2.1-I2V-14B backbone with jointly trained LoRA adapters and GCA modules. The reported inference setup adds little denoising overhead.

B. Dataset and Implementation Details

  • Approximately 30K processed clips are constructed from about 6.5K retained DL3DV-10K scenes, using 55-frame clips split into 32 history frames and 23 generation targets.Nine keyframes are retrieved from the history bank during H2V training.Source: B. Dataset and Implementation Details · pp. 16–17
  • Rank-32 LoRA adapters and GCA modules are jointly trained on a frozen Wan2.1-I2V-14B backbone, with GCA inserted after self-attention in every even-indexed DiT block.GCA's output projection is zero-initialized so its residual initially does not perturb backbone features.Source: B. Dataset and Implementation Details · p. 17, Table 3 · p. 17
  • GCA adds 1.9% to the backbone parameter count with minimal denoising overhead; inference uses 25 denoising steps and classifier-free guidance scale 5.Coverage retrieval and correspondence rasterization are reported to introduce negligible inference overhead.Source: B. Dataset and Implementation Details · p. 17

C.1. Baseline Configuration and Method Comparison

The comparisons report degradation in competing methods under challenging camera motion, while GEAR uses matched historical patches as attention keys and values rather than only geometric cues. Evaluation also applies depth and camera-scale alignment, with some baseline-specific qualifications.

C.1. Baseline Configuration and Method Comparison

  • Camera translations are normalized for implicit-memory baselines, and Infinite-World's trajectory adherence is qualified by its conversion of continuous transformations into discrete actions.Depth and camera scale are aligned for comparisons; AnchorWeave is excluded because inference weights were unavailable at evaluation time.Source: §C.1 · p. 17
  • UCM warps positional encodings for geometry-aware token interaction, whereas Lyra 2.0 adds encoded correspondence-map embeddings without explicitly restricting each target patch to matched historical patches.UCM assigns each historical frame a single relevant target viewpoint for warping.Source: §C.1 · p. 18
  • GEAR gathers matched historical features as keys and values for GCA, allowing each target patch to draw on multiple observations while the Invisible Octree filters occluded candidates.The attention output enters denoising through a residual branch, keeping depth errors local to individual source views.Source: §C.1 · pp. 18–19
  • UCM shows increasing drift and visual degradation under longer or faster motion; Lyra 2.0 also degrades with substantial viewpoint changes, while Spatia loses content on challenging long-horizon routes.Spatia can recover convincing observations on some simpler rotational trajectories, but accumulated point-cloud errors may be repeatedly rendered into conditioning.Source: §C.1 · p. 19

D. Additional Qualitative and Video Results

Additional qualitative results cover challenging minute-long camera trajectories, DL3DV-Evaluation, and WorldScore-Static, with further results across broader data. The supplied captions identify these comparisons but do not report their outcomes.

D. Additional Qualitative and Video Results

Related papers

Loading 2609.34722v1…