Source-linked AI summary
WorldWarp: Propagating 3D Geometry with Asynchronous Video Diffusion
Hanyang Kong, Xingyi Yang, Xiaoxu Zheng, Xinchao Wang
TL;DR
Long-range camera-conditioned video generation needs geometric conditioning that can handle camera trajectories and imperfect 3D-warped priors. WorldWarp combines an online 3D geometric cache with non-causal diffusion that fills blank regions and revises warped content. The authors report state-of-the-art performance in geometric consistency and image quality on challenging view-extrapolation benchmarks.
Problem
Long-range camera-conditioned video lacks effective 3D conditioning: pose encodings can generalize poorly to OOD poses, while explicit spatial priors can contain occlusions and distortions that standard generation methods handle poorly.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
Re-estimating a short-term 3DGS cache from recent generated history avoids irreversible error propagation and supports high-fidelity consistency over long camera paths.
Source: §1 · p. 2For sequences exceeding 1000 frames, minor visual artifacts or geometric inconsistencies can accumulate and eventually degrade visual quality or geometric stability.
Source: §9 · p. 14
Abstract
from arXiv · showhide
Generating long-range, geometrically consistent video presents a fundamental dilemma: while consistency demands strict adherence to 3D geometry in pixel space, state-of-the-art generative models operate most effectively in a camera-conditioned latent space. This disconnect causes current methods to struggle with occluded areas and complex camera trajectories. To bridge this gap, we propose WorldWarp, a framework that couples a 3D structural anchor with a 2D generative refiner. To establish geometric grounding, WorldWarp maintains an online 3D geometric cache built via Gaussian Splatting (3DGS). By explicitly warping historical content into novel views, this cache acts as a structural scaffold, ensuring each new frame respects prior geometry. However, static warping inevitably leaves holes and artifacts due to occlusions. We address this using a Spatio-Temporal Diffusion (ST-Diff) model designed for a "fill-and-revise" objective. Our key innovation is a spatio-temporal varying noise schedule: blank regions receive full noise to trigger generation, while warped regions receive partial noise to enable refinement. By dynamically updating the 3D cache at every step, WorldWarp maintains consistency across video chunks. Consequently, it achieves state-of-the-art fidelity by ensuring that 3D logic guides structure while diffusion logic perfects texture. Project page: \href{https://hyokong.github.io/worldwarp-page/}{https://hyokong.github.io/worldwarp-page/}.
1. Introduction
WorldWarp addresses the challenge of long-range view extrapolation by combining geometrically grounded warped priors with diffusion-based generation. Its online 3D cache is updated from recent generated history, and the authors report state-of-the-art geometric consistency and visual fidelity.
1. Introduction
- Long-range camera-conditioned video needs effective 3D conditioning, but pose encodings can generalize poorly to OOD poses while explicit 3D priors can contain occlusions and distortions.Standard inpainting and video-generation techniques are ill-suited to handling severe disocclusions and warped-prior distortions together.Source: §1 · p. 2
- WorldWarp generates novel-view sequences chunk by chunk using an autoregressive inference pipeline and a non-causal ST-Diff model conditioned on forward-warped images.The warped images provide dense 2D spatial priors, including from future camera positions, for bidirectional attention.Source: §1 · p. 2
- An online 3D geometric cache built with 3DGS is optimized on recent high-fidelity generated history and renders warped priors for the next chunk.This cache provides geometric grounding for generating new content and filling occlusions.Source: §1 · p. 2
- WorldWarp achieves state-of-the-art performance on challenging view-extrapolation benchmarks, with improved geometric consistency and image quality over existing methods.The framework avoids irreversible error propagation by re-estimating a short-term 3DGS cache at each step, grounding generation in recent geometry over long camera paths.Source: §1 · p. 2
2. Related Works
Related work distinguishes novel-view interpolation from extrapolation and reviews alternatives for camera-conditioned video generation. WorldWarp instead uses forward-warped images as a dense geometric prior.
Novel view synthesis
- View extrapolation generates views beyond the original scene, introducing substantial new content, whereas interpolation stays within the training-view distribution.The review also situates extrapolation among approaches leveraging pretrained video diffusion models.Source: §2 · p. 2
Camera pose encoding and 3D explicit spatial priors
- Camera-pose encodings provide viewpoint control but lack scene content, while explicit 3D priors offer geometric grounding yet can propagate estimation errors and incur high computational cost.The paper uses forward-warped images as a dense 2D hint, contrasting them with both approaches.Source: §2 · p. 3
Auto-regressive video diffusion models
- Causal autoregressive strategies cannot use warped images from future camera positions as conditional hints, so the paper adopts a non-autoregressive framework.The related-work discussion also notes that learning effective camera embeddings for autoregressive approaches is non-trivial.Source: §2 · p. 3
3. Preliminaries
The preliminaries motivate dense geometric camera representations and per-frame diffusion noise. Independent noise levels enable non-causal denoising, which fits novel-view synthesis because future views can receive warped geometric priors.
3.1. Camera-Conditioned Video Generation
- Raw camera intrinsics and extrinsics can be suboptimal because their unconstrained numerical values are difficult for a network to correlate with visual content.The section motivates translating abstract parameters into dense, pixel-wise representations with clearer geometric interpretation.Source: §3.1 · p. 3
- Plücker embeddings represent each pixel with a 6D ray vector, forming a dense tensor that is more informative for the diffusion model.The paper presents dense, geometrically grounded priors as a consideration for fine-grained camera control.Source: §3.1 · p. 3
3.2. Diffusion Forcing and Non-Causal Priors
- Diffusion Forcing assigns each video frame an independent noise level, unlike conventional models that use one shared noise level for all frames.The method trains per-frame noise prediction; the displayed loss sums prediction errors over frames.Source: §3.2 · p. 3
- Per-frame noise supports non-causal attention by letting a frame be denoised using an arbitrary partially masked set of other frames.For novel-view synthesis, forward-warping can provide geometry-consistent priors for future frames, enabling bidirectional spatio-temporal diffusion.Source: §3.2 · p. 3
4. Method
WorldWarp uses forward-warped views as geometric priors for a bidirectional diffusion model, then generates novel views in autoregressive chunks. Its training combines warped and ground-truth latents under spatially and temporally varying noise, while inference refreshes an optimized 3D cache at each iteration.
4.1. Spatio-Temporal Diffusion with Warped Priors
- Forward-warped source views provide a geometric prior that lets ST-Diff use bidirectional attention across all target frames.During training, one source frame is warped to every frame viewpoint; the camera-conditioned prior supports dropping causal constraints.Source: §4.1 · p. 3
- The training pipeline composites valid warped latents with ground-truth latents in filled regions, then performs diffusion and compositing in latent space.A downsampled validity mask selects between the warped and ground-truth features.Source: §4.1 · pp. 3–4
- Noise varies independently across frames and between warped and filled regions, using a spatial noise map for each frame.The schedule distinguishes the warped and filled regions within each frame.Source: §4.1 · p. 4
- ST-Diff denoises the noisy composite sequence using its noise maps and conditioning, with a sequence-wide L2 objective toward the ground-truth velocity target.The target is defined as ε_t − z_t.Source: §4.1 · pp. 4–5
4.2. Autoregressive Inference Pipeline
- At each inference iteration, TTT3R estimates poses and a point cloud from the available history, which initializes an optimized 3DGS cache for high-fidelity warping.The online cache is optimized using the history frames and their estimated poses; it supplies higher-quality warped-region features than the fixed-radius training point clouds.Source: §4.2 · p. 5
- Inference initializes warped regions with partial noise and blank regions with pure noise, then denoises a novel-view chunk that becomes the next iteration’s history.ST-Diff also receives the VLM prompt; the objective is to fill occlusions and revise warped regions that may be blurred or distorted.Source: §4.2 · pp. 5–6, Figure 3 · p. 5
- Qualitative comparisons on RealEstate10K and DL3DV show WorldWarp against GenWarp, CameraCtrl, and VMem across diverse camera motion.The caption describes the results as demonstrating spatial and temporal consistency.Source: Figure 4 · p. 6
5. Experiments
WorldWarp is evaluated on RealEstate10K and DL3DV for view quality and camera-pose accuracy, with long-term comparisons, design ablations, and inference latency analysis. It outperforms the reported baselines, while ablations examine the roles of the online 3DGS cache and spatial-temporal noise schedule.
5.1. Implementation Details.
- Experiments use RealEstate10K and DL3DV, evaluating perceptual quality, detail preservation, and geometric alignment; implementation fine-tunes the Wan2.1-T2V-1.3B model on 8 H200 GPUs.The evaluation metrics include FID, PSNR, SSIM, LPIPS, rotation distance, and translation distance.Source: §5.1 · p. 6
5.2. Comparisons on the RealEstate10K Dataset
- 17.13 PSNR and 0.352 LPIPS are the highest reported long-term RealEstate10K values, while Rdist 0.697 and Tdist 0.203 are the lowest pose errors; WorldWarp leads across all 12 metrics.The evaluation covers short-term (50th-frame) and long-term (200th-frame) synthesis.Source: §5.2 · p. 7, Table 1 · p. 7
5.3. Comparisons on the DL3DV Dataset
- 14.53 PSNR, 1.007 Rdist, and 0.412 Tdist are WorldWarp’s long-term DL3DV results, with a lead in all 12 metrics despite complex trajectories degrading every method.The PSNR exceeds DFoT (13.51) and VMem (12.28); the table covers short-term (50th-frame) and long-term (200th-frame) synthesis.Source: §5.3 · p. 7, Table 2 · p. 8
- ST-Diff fills blank areas initialized with full noise while revising distortions and enhancing details in non-blank regions initialized with partial noise.Figure 5 illustrates the ground-truth images, warped conditions, validity masks, and generated frames.Source: Figure 5 · p. 7
5.4. Ablation Study
- 9.22 PSNR for the No Cache baseline and 11.12 versus 17.13 PSNR for RGB point-cloud versus online-optimized 3DGS caching support the online cache’s role in long-term quality.The 3DGS cache takes 500 optimization steps per chunk.Source: §5.4 · pp. 7–8, Table 3 · p. 8
- Full spatial-temporal-varying noise reaches 17.13 PSNR and 0.697 Rdist, while spatial-only noise improves camera accuracy and temporal-only noise improves generation quality.Full-sequence noise yields 9.92 long-term PSNR and 1.574 Rdist; spatial-only noise reaches 1.040 Rdist, and temporal-only noise reaches 13.20 long-term PSNR and 1.209 Rdist.Source: §5.4 · p. 8, Table 3 · p. 8
- 54.5 seconds generates a 49-frame chunk; ST-Diff denoising takes 42.5 seconds, while 3D-related components take 8.5 seconds total.The breakdown reports 50 denoising steps, 5.8 seconds for initial 3D estimation, 2.5 seconds for cache optimization, and 0.2 seconds for warping.Source: §5.4 · pp. 8–9, Table 4 · p. 8
6. Conclusion
WorldWarp is an autoregressive framework for long-range, geometrically consistent novel-view extrapolation. Its ST-Diff model and online 3D cache target imperfect warped priors and irreversible error propagation, and the paper reports state-of-the-art performance.
6. Conclusion
- WorldWarp couples an online 3D geometric cache with ST-Diff, whose spatial-temporally varying noise schedule fills blank regions and revises distorted content.Blank regions start from pure noise, while distorted content starts from a partially noised state.Source: §6 · p. 9
7. Implementation Details
WorldWarp generates video autoregressively in 49-frame chunks with five context-overlap frames and an online optimized 3DGS cache. Its diffusion schedule constrains context frames, preserves valid warped regions with reduced noise, and initializes blank regions with Gaussian noise.
7. Implementation Details
- Generation starts from one source image and proceeds autoregressively in 49-frame chunks with a fixed five-frame context overlap.Camera poses and intrinsics are estimated using TTT3R; longer paths use velocity-based translation and SLERP rotation extrapolation.Source: §7 · p. 13
- The online 3DGS cache is optimized for 500 iterations per chunk to render warped priors.The model is trained end-to-end for 10k steps on 8 NVIDIA H200 GPUs at 480 × 720 resolution.Source: §7 · p. 13
- Five context frames are hard-constrained; valid warped regions start with reduced noise, while blank or occluded regions start with standard Gaussian noise.Target-frame noise strength is τ = 0.8, and reverse diffusion uses 50 denoising steps.Source: §7 · p. 13
8. More Experiment Results
Visualizations show WorldWarp on RealEstate10K and DL3DV, including stylized scenes generated from text prompts. A noise-map analysis describes how noise varies across context and generated tokens during reverse diffusion.
8.1. Visualization Results
- WorldWarp generates stylized sequences from prompts such as “Van Gogh style” and “Studio Ghibli style,” while preserving the underlying 3D geometric consistency.The results are presented as qualitative visualizations.Source: §8.1 · p. 13
8.2. Analysis of Spatially-Adaptive Noise Dynamics
- The noise-map grid shows zero noise for the five context frames, reduced noise in valid 3DGS-projected regions, and high-variance noise in blank regions.Rows represent latent temporal tokens and columns progress from denoising step T = 999 to T = 0; the generated tokens comprise 11 rows.Source: §8.2 · p. 13
9. Limitations
The authors identify two limitations: error accumulation over long autoregressive sequences and dependence on upstream geometric estimates. In challenging scenes, inaccurate depth or poses can produce warps that ST-Diff may not repair.
Error Accumulation in Long-horizon Generation
- Beyond 1,000 frames, accumulated visual artifacts or geometric inconsistencies can degrade visual quality or geometric stability.Each chunk becomes historical context for the next, allowing minor errors to propagate; the authors say perfect fidelity for infinite-length sequences remains unresolved.Source: §9 · pp. 13–14
Dependency on Geometric Priors
- Performance depends heavily on upstream depth and camera-pose estimates, which may be inaccurate in scenes with extreme lighting, transparency, or little texture.Incorrect estimates can yield substantially wrong warps, and ST-Diff may fail to recover high-quality frames when geometric guidance is fundamentally flawed or excessively noisy.Source: §9 · pp. 14–15