PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry-Aware Rotary Position Embedding for Consistent Video World Model

Chendong Xiang, Jiajun Liu, Jintao Zhang, Xiao Yang, Zhengwei Fang, Shizun Wang, Zijun Wang, Yingtian Zou, Hang Su, Jun Zhu

arXiv:2602.07854v3cs.CV

TL;DR

Long-horizon camera-conditioned video models struggle with spatial persistence and loop-closure consistency. The paper introduces ViewRope, geometry-aware frame-sparse attention, and ViewBench, reporting substantially improved view consistency with efficient historical-frame selection while identifying weak correspondence across drastic scene transitions as a limitation.

  • Problem

    Existing camera-conditioned video generators often drift or hallucinate details when revisiting previously observed viewpoints, limiting long-term geometric consistency.

  • Method

    ViewRope injects patch-level camera-ray directions into attention, while Geometry-Aware Frame-Sparse Attention selects geometrically relevant historical frames; ViewBench evaluates loop closure and geometric drift.

  • Results

    The approach substantially improves view consistency on ViewBench while remaining efficient, and its best reported embedding ablation achieves 0.0859 training loss.

  • Takeaways & Limitations

    Geometry-aware attention provides a model-native route to long-term spatial persistence without sacrificing memory consistency during long-video generation.

  • Takeaways & Limitations

    The method may struggle with drastic scene transitions where geometric correspondences between views are weak or nonexistent.

Abstract

from arXiv · show

Predictive world models that simulate future observations under explicit camera control are fundamental to interactive AI. Despite rapid advances, current systems lack spatial persistence: they fail to maintain stable scene structures over long trajectories, frequently hallucinating details when cameras revisit previously observed locations. We identify that this geometric drift stems from reliance on screen-space positional embeddings, which conflict with the projective geometry required for 3D consistency. We introduce \textbf{ViewRope}, a geometry-aware encoding that injects camera-ray directions directly into video transformer self-attention layers. By parameterizing attention with relative ray geometry rather than pixel locality, ViewRope provides a model-native inductive bias for retrieving 3D-consistent content across temporal gaps. We further propose \textbf{Geometry-Aware Frame-Sparse Attention}, which exploits these geometric cues to selectively attend to relevant historical frames, improving efficiency without sacrificing memory consistency. We also present \textbf{ViewBench}, a diagnostic suite measuring loop-closure fidelity and geometric drift. Our results demonstrate that ViewRope substantially improves long-term consistency while reducing computational costs.

1. Introduction

Existing camera-conditioned video generators struggle to preserve 3D-consistent scene structure over long trajectories, especially when views revisit earlier locations. The paper addresses this with ray-based attention, geometry-guided frame retrieval, and ViewBench evaluation.

  • Pose-conditioned world models generate future observations under explicit viewpoint trajectories for interactive AI.
  • Current generators often hallucinate details and drift during loop-closure trajectories instead of reconstructing previously observed structures and appearances.
  • Screen-space positional embeddings misalign with projective geometry because the same 3D point can appear at different pixels across time.
  • ViewRope injects patch-level camera-ray directions into self-attention, replacing pixel locality with relative ray geometry for retrieving consistent 3D content.
  • Geometry-Aware Frame-Sparse Attention selects co-visible historical frames, enabling consistent long-video generation with low latency.
  • ViewBench measures view consistency and loop-closure behavior in camera-conditioned video generation models.

2. Related Work

Related work conditions visual models with camera geometry, relative pose encodings, explicit memory, and sparse attention. These approaches improve controllability or efficiency but often depend on global frames, auxiliary memory, or learned selection schemes.

  • Raymaps encode camera parameters as per-pixel embeddings containing ray origins and directions or Plücker coordinates.
  • Relative attention encodings such as CaPE, GTA, and PRoPE model view transformations to ground visual tokens in 3D space.
  • Interactive world models increasingly use causal or autoregressive architectures with KV-caching and distillation for real-time interaction.
  • Explicit memory methods retrieve historical frames, maintain geometry-indexed spatial memory, or store 3D point-cloud memory.
  • WorldPack compresses trajectory history through packing and selective attention without auxiliary modules.
  • Sparse attention reduces quadratic sequence-length cost by attending to subsets of tokens, commonly using learnable selection schemes.

3. Method

The method formalizes long-horizon camera-conditioned generation around loop-closure consistency, then injects viewing-ray geometry into attention and selects relevant historical frames sparsely. Training progressively introduces autoregressive conditioning, ViewRope, and frame sparsity.

  • Problem Formulation: The world model generates future video consistent with a target camera trajectory, represented by rotations, translations, and intrinsics.
  • Problem Formulation: Standard local temporal objectives do not prevent long-horizon geometric drift because screen-space proximity does not match physical correspondence.
  • Problem Formulation: Loop closure requires revisited views to agree photometrically over co-visible regions when their poses are sufficiently similar.
  • ViewRope: ViewRope assigns each token a per-patch rotation derived from camera intrinsics and extrinsics, making attention sensitive to relative view geometry.
  • ViewRope: Rotating query and key subvectors by viewing-direction transforms aligns part of feature space with physical ray directions and captures angular relationships between rays.
  • Geometry-Aware Frame Sparse Attention: Frame-sparse attention uses frame-sized blocks, estimates geometric relevance from sampled tokens, applies causal masking, and selects top-k past blocks.
  • Geometry-Aware Frame Sparse Attention: With fixed k, sparse attention scales linearly with the number of frames; inference maintains a KV cache and repeats relevance-based selection during denoising.
  • Training Schedule: Training progressively uses short-clip teacher forcing, ViewRope, and frame-sparse attention to stabilize autoregressive streaming and long-context retrieval.

4. Experiments

Experiments evaluate ViewRope and ViewBench across camera-motion benchmarks, geometric consistency, sparse attention, qualitative retrieval, efficiency, and channel-allocation choices. ViewRope improves loop closure and long-sequence efficiency while preserving visual quality, with temporal-channel allocation performing best.

  • Experimental Setup: ViewBench evaluates view consistency under full 3-axis rotations, loop-closure trajectories, and diverse photorealistic environments.Its metrics include frame-level visual quality and Loop Closure Error (LCE), where lower LCE indicates better persistent spatial memory.
  • View Consistency Comparison: ViewRope reduces LCE by 4% versus GTA, the strongest baseline, while maintaining comparable or better PSNR and SSIM.Both geometry-aware methods outperform 3D RoPE, and the reported comparison uses the same backbone, training budget, and data for fairness.
  • Comparison with State-of-the-Art Interactive World: ViewRope reduces LCE versus HY-WorldPlay by 6.5% at 30°, 7.9% at 45°, and 11.4% at 75°.The reported advantage increases with rotation magnitude across the evaluated interactive world-model comparisons.
  • Efficiency of Geometry-Aware Sparse Attention: ViewRope with Sparse reduces LCE by 16% versus sliding-window attention on 90° and 180° scenarios.Naive sparse attention and GTA with Sparse suffer loss divergence, whereas ViewRope with Sparse maintains stable convergence; the analysis attributes this to geometrically structured Q/K relevance scoring.
  • Efficiency of Geometry-Aware Sparse Attention: Random frame selection causes 25.2% LCE degradation, while excluding ViewRope-selected frames causes a further 38.1% degradation.The counterfactual comparison supports the importance of the frames selected by the geometry-aware mechanism.
  • Efficiency of Geometry-Aware Sparse Attention: Sparse attention reduces training time from 27.66 s/iter to 22.01 s/iter on 201-frame sequences, achieving approximately 25% acceleration.Qualitative cases also show ViewRope preserving scene structures that sliding-window attention loses, blurs, or replaces with hallucinated details.
  • Ablation Studies: Allocating ViewRope to the lowest-frequency temporal bands achieves the best ablation performance at 0.0859.Replacing the original 3D RoPE components directly degrades performance, while increasing sparse top-k improves visual quality but peaks in geometric consistency at k = 5.

5. Conclusion and Future Work

ViewRope is presented as a geometry-aware positional encoding that improves long-term spatial persistence and consistency, especially during loop closure, while Geometry-Aware Sparse Attention reduces computational costs. The method remains limited when scene transitions provide weak or nonexistent geometric correspondences.

  • ViewRope embeds camera ray directions into attention to maintain long-term spatial persistence and geometric consistency in loop-closure scenarios.Geometry-Aware Sparse Attention selectively attends to geometrically relevant historical frames without sacrificing performance.
  • ViewBench experiments report state-of-the-art consistency with significantly lower computational costs than dense attention baselines.
  • Drastic scene transitions can challenge the method when geometric correspondences between views are weak or nonexistent.The authors identify explicit 3D modeling, implicit representations, distillation, and reinforcement learning for more dynamic or longer sequences as future directions.

6. Impact Statement

The paper positions view-consistent video generation as potentially useful for several interactive-world-model applications while emphasizing responsible development. It also documents a teacher-forcing and frame-sparse-attention pipeline, but the supplied passages do not add technical impact findings.

  • View-consistent video generation could support VR/AR, entertainment, education, and training applications.
  • High-quality video generation raises concerns about deepfake misuse, copyright, and privacy in virtual content creation.The paper calls for careful consideration and responsible development.
  • The implementation uses teacher forcing, sampled attention indices, a teacher-forcing mask, and TopK selection before frame-sparse attention.

A.1. Training Configuration

The appendix describes teacher-forced training and cached autoregressive inference with frame-sparse attention, alongside the model configuration and loop-closure formulation. The formal objective targets consistency between revisited views over mutually visible regions.

  • Training Configuration: Inference maintains a clean KV cache of prior frames, samples relevant indices, selects TopK matches, and caches the current frame for subsequent steps.
  • Training Configuration: The model is based on WAN 2.2 TI2V-5B with 5 billion parameters, 480 × 832 training resolution, and 61 frames per clip.Teacher forcing aligns the model with autoregressive generation and KV-caching behavior.
  • Training Configuration: Training uses a batch size of 64 for 6k steps with AdamW at learning rate 5 × 10^-5, requiring approximately 2 days on 16 NVIDIA A100 GPUs.
  • Training Configuration: Training data combines Context-as-Memory, GF-Minecraft, and ViewBench, with ViewBench contributing approximately 500k frames of complete 3-axis rotation sequences.
  • Training Configuration: Dataset sampling is balanced at a 1:1:1 ratio, and RoPE variants use the same latent channels for fair comparison.ViewRope is embedded in temporal channels 32–44.
  • Loop Closure Formulation: The loop-closure formulation uses pose similarity and a tolerance threshold to identify revisits, then enforces photometric consistency after a projective warp over mutually visible regions.The warp uses camera motion and scene geometry, while the consistency loss can use a robust penalty such as Huber.

C. ViewBench Benchmark

ViewBench is an evaluation suite built from diverse UE5 environments and structured camera trajectories, including rotation-only and rotation-plus-translation motion. Its separate evaluation set measures frame quality and loop-closure memory using LCE.

  • ViewBench Benchmark: ViewBench extends existing dataset comparisons with additional properties documented in its appendix tables.
  • ViewBench Benchmark: The benchmark contains 10 photorealistic UE5 environments spanning indoor, outdoor, urban, and natural settings.The stated diversity covers geometry, lighting, and texture.
  • ViewBench Benchmark: Pure-rotation trajectories cover all 7 axis combinations and use rotate-away–rotate-back loops with magnitudes of 30°, 75°, 90°, and 180°, yielding approximately 7,000 clips.
  • ViewBench Benchmark: Rotation-plus-translation trajectories combine RotateOnly, MoveOnly, MoveAndRotate, and Orbit actions within a compact exploration radius.Action types may also include roll rotation with type-specific probabilities.
  • ViewBench Benchmark: Each frame includes camera pose, Euler angles, position, field of view, and WASD states, with depth-based overlap annotations for attention-recall analysis.
  • ViewBench Benchmark: The training set contains 1,059 sequences and approximately 500k frames, while the separately collected evaluation set contains 600 non-overlapping pure-rotation samples.
  • ViewBench Benchmark: Evaluation compares generated videos with ground truth using PSNR, SSIM, LPIPS, and Loop Closure Error.LCE compares the generated return-pose frame with the ground-truth first frame and isolates memory after an extended camera excursion.

C.6. Complete Baseline Results on ViewBench

ViewBench results cover all rotation magnitudes and show strong performance at moderate angles, while large-angle results expose frame-rate and autoregressive error-accumulation limits. These limitations are described as system-level and separate from ViewRope’s positional encoding.

  • ViewRope consistently outperforms all baselines at 30°, 45°, and 75° rotation magnitudes.
  • At 90°–180°, ViewRope performs below HY-WorldPlay because evaluation motion exceeds the constant angular velocity used during training.
  • Resampling 180° trajectories to 161 frames can produce per-frame angular steps that cause ViewRope to under-rotate to approximately 80°.
  • Teacher-forcing training leaves inference vulnerable to compounded errors over the long autoregressive sequences required by large-angle trajectories.
  • The authors characterize these limitations as system-level and independently addressable from ViewRope.

C.7. Qualitative Comparison

Qualitative loop-closure comparisons show ViewRope preserving scene structure, lighting, and architectural identity across yaw, pitch, and combined rotations. Competing systems exhibit brightness collapse, geometric drift, hallucinated content, or blurry and unrelated return scenes.

  • The qualitative protocol rotates away from the starting viewpoint and then reverses to test closed-loop return fidelity.
  • Case 1: Yaw + Pitch: In an urban yaw-plus-pitch loop, ViewRope preserves scene structure and lighting while M-G 2.0 collapses in brightness and HY-WorldPlay drifts geometrically.
  • Case 2: Pure yaw: In a pure-yaw Asian street loop, ViewRope and HY-WorldPlay recover the original storefronts and layout, whereas M-G 2.0 hallucinates absent yellow trees.
  • Case 3: Pure pitch: In a pure-pitch Roman architecture loop, ViewRope faithfully restores the original arched stone structures while competing returns are unrelated or blurry.

D.1. Ablation of Number of Topk Frames

The top-k frame ablation examines the trade-off between visual quality and geometric consistency when retrieving historical frames for 90° and 180° synthesis.

  • Increasing retrieved frames generally improves PSNR, SSIM, and LPIPS visual-quality metrics by providing richer reference texture.
  • Loop Closure Error reaches its optimum at an intermediate top-k setting rather than increasing monotonically with more retrieved frames.
Loading 2602.07854v3…