Source-linked AI summary
LaVR: Scene Latent Conditioned Generative Video Trajectory Re-Rendering using Large 4D Reconstruction Models
Mingyang Xie, Numair Khan, Tianfu Wang, Naina Dhingra, Seonghyeon Nam, Haitao Yang, Zhuo Hui, Christopher Metzler, Andrea Vedaldi, Hamed Pirsiavash, Lei Luo
TL;DR
Video re-rendering must generate coherent novel views from monocular input despite weaknesses in both explicit geometric reconstruction and unconditioned generation. LaVR conditions a video diffusion model on implicit 4D reconstruction latents and camera poses, and reports stronger geometric consistency and visual quality than both baseline types. The method is limited by transparent moving objects and added computation from CUT3R conditioning.
Problem
Existing video re-rendering methods trade geometric stability against visual flexibility because explicit reconstruction is vulnerable to depth errors, while unconditioned generation lacks spatial awareness.
Method
LaVR conditions a video diffusion model on implicit scene latents from a large 4D reconstruction model together with source camera poses, avoiding explicit depth and point-cloud reconstruction.
Results
LaVR achieves stronger geometric consistency and higher visual quality than existing geometry-conditioned and unconditioned baseline methods.
Takeaways & Limitations
Soft latent geometry lets the pretrained diffusion prior regularize local inconsistencies while retaining coherent structure and parallax along novel trajectories.
Takeaways & Limitations
The method struggles with moving transparent objects and incurs additional computation from CUT3R conditioning.
Abstract
from arXiv · showhide
Given a monocular video, the goal of video re-rendering is to generate views of the scene from a novel camera trajectory. Existing methods face two distinct challenges. Geometrically unconditioned models lack spatial awareness, leading to drift and deformation under viewpoint changes. On the other hand, geometrically-conditioned models depend on estimated depth and explicit reconstruction, making them susceptible to depth inaccuracies and calibration errors. We propose to address these challenges by using the implicit geometric knowledge embedded in the latent space of a large 4D reconstruction model to condition the video generation process. These latents capture scene structure in a continuous space without explicit reconstruction. Therefore, they provide a flexible representation that allows the pretrained diffusion prior to regularize errors more effectively. By jointly conditioning on these latents and source camera poses, we demonstrate that our model achieves state-of-the-art results on the video re-rendering task. Project webpage is https://lavr-4d-scene-rerender.github.io/.
1. Introduction
Video re-rendering must preserve scene geometry and dynamics when synthesizing views along unseen camera paths. LaVR uses implicit geometry latents from a large 4D reconstruction model to combine geometric consistency with diffusion-based visual quality.
- 1. Introduction: Video re-rendering synthesizes dynamic scenes from unseen camera paths while requiring spatial and temporal coherence under arbitrary camera motion.Monocular inputs provide weak geometric supervision, making joint inference of structure and dynamics challenging.
- 1. Introduction: Explicit geometric methods can distort shapes and parallax when depth errors propagate into reconstructed point clouds, whereas unconditioned methods retain realism but lack geometric stability.These methods face complementary weaknesses: rigid reconstruction errors versus insufficient spatial awareness.
- 1. Introduction: LaVR conditions video diffusion on latent 4D scene representations that capture appearance, geometry, and dynamics without explicit reconstruction.The latent representation is extracted from monocular videos and guides novel trajectory generation.
- 1. Introduction: Implicit geometry latents provide softer structural guidance that diffusion priors can regularize against small inconsistencies and depth noise.This design aims to retain informative geometry cues without requiring accurate depth estimation.
- 1. Introduction: LaVR reports stronger geometric consistency and higher visual quality than both geometry-conditioned and unconditioned baselines on quantitative and qualitative evaluations.The approach uses a lightweight adapter to integrate reconstruction-model latents with video latents for efficient diffusion conditioning.
2. Related Work
Prior work spans explicit scene reconstruction, large feed-forward 3D/4D models, and generative camera-controlled video synthesis. These approaches trade geometric enforcement against robustness and visual flexibility under monocular inputs and large camera motion.
- 2. Related Work: Neural scene representations and dynamic reconstruction methods rely on accurate, complete geometry, which is difficult to obtain from monocular video.The literature includes NeRF, Gaussian Splatting, depth-based warping, and neural dynamic representations.
- 2. Related Work: Large 3D/4D reconstruction models use high-capacity architectures and large-scale pretraining to infer scene structure, poses, and time-varying geometry from limited visual input.Systems such as DUST3R, SPANN3R, CUT3R, MONST3R, and MegaSAM provide geometric priors for generative synthesis.
- 2. Related Work: Generative novel-view methods commonly condition video diffusion on camera poses or trajectory signals to control viewpoint changes.ReCamMaster instead synthesizes new trajectories without explicit 4D structure, gaining flexibility at the cost of weaker geometric stability.
- 2. Related Work: Explicit 4D pipelines enforce geometry but remain vulnerable to reconstruction errors, while non-4D-conditioned methods are more robust but struggle with spatial coherence during large camera motion.This trade-off motivates conditioning generation with implicit geometric information.
3. Method
LaVR conditions a video-to-video diffusion model on CUT3R’s latent scene state, camera poses, and text to synthesize a source scene along a target trajectory. A lightweight adapter converts implicit tokens into spatial features compatible with the pretrained DiT while preserving its priors.
- 3. Method: The method synthesizes novel trajectory frames by conditioning video-to-video diffusion on the latent state of a large 4D reconstruction model.The latent state aggregates scene structure and camera motion without explicit depth or point-cloud reconstruction.
- 3.1. Scene State Latents with CUT3R: CUT3R maintains a persistent token state across frames that captures evolving 3D scene understanding and can decode poses, point maps, and depth.The state uses tokens updated by ViT-encoded source frames to preserve temporal changes in scene content and camera pose.
- 3.2. Adapting CUT3R Latents to the DiT Backbone: A lightweight adapter transforms frame-wise CUT3R tokens into geometry-aware spatial features that match the diffusion model’s latent interface.It subsamples states, embeds tokens, applies query-based cross-attention over a target h × w grid, and projects features to shape (T/k, h, w, c).
- 3.2. Adapting CUT3R Latents to the DiT Backbone: The adapted CUT3R features are concatenated with source and noisy output video latents before entering the DiT, avoiding backbone changes while preserving spatiotemporal compatibility.Source and target camera poses are injected through separate lightweight MLP adapters, while text supplies additional scene context.
- 3.3. Training Strategy: Training updates the CUT3R and pose adapters and selected DiT layers with conditional flow matching while freezing the remaining DiT and Video VAE parameters.The model is trained on MultiCamVideo using paired source and target trajectories, with a 3× higher learning rate for the CUT3R adapter.
4. Experiments
Across quantitative and qualitative evaluations, the proposed method provides accurate trajectories, strong consistency, and natural novel views while avoiding artifacts associated with explicit point-cloud or absent geometric conditioning.
- 4.1. Results: Qualitatively, the method produces more natural and geometrically consistent novel views than point-cloud-conditioned and unconditioned baselines.Point-cloud methods exhibit warping, whereas the unconditioned baseline hallucinates implausible content and loses consistency across occlusions.
- 4.1. Results: The method achieves the lowest translation and rotation errors across baselines when reconstructing target camera trajectories.Evaluation uses absolute and relative errors for camera translation and rotation.
- 4.1. Results: The method outperforms TrajectoryCrafter and ReCamMaster on cycle consistency and has consistently lower error than Gen3C.Cycle consistency is evaluated on static scenes from 50 DL3DV videos.
- 4.1. Results: It shows all-round high video quality and achieves the best results on all reported consistency metrics.VBench evaluation covers dynamic video inputs, while Table 1 reports consistency comparisons.
- 4.3. Ablation on Source Pose Conditioning: Latent token conditioning provides the main ablation gain, while source pose conditioning contributes a smaller complementary improvement.The ablation compares latent-token conditioning alone with the addition of source pose conditioning.
- 4.3. Ablation on Source Pose Conditioning: The method currently struggles with moving transparent objects and incurs additional computation from its CUT3R conditioning mechanism.The limitation is linked to CUT3R’s difficulty estimating reliable geometry for such scenes.
- 4.2. Qualitative Results on 4D Reconstruction: The latent 4D condition avoids stretching, missing details, hallucinated objects, and inconsistent reconstructions across novel trajectories.Qualitative comparisons attribute these artifacts to point-cloud or absent geometric conditioning.
5. Conclusion
LaVR conditions a video diffusion model on large 4D reconstruction-model latents to synthesize novel camera trajectories in dynamic scenes. Experiments show stronger geometric consistency and higher visual quality than geometry-conditioned and unconditioned baselines.
- LaVR conditions a video diffusion model on large 4D reconstruction-model latents for novel camera trajectory synthesis in dynamic scenes.The latents provide geometry-aware guidance in a soft form, allowing pretrained diffusion priors to regularize local inconsistencies.
- Experiments on static and dynamic scenes show stronger geometric consistency and higher visual quality than existing geometry-conditioned and unconditioned baselines.