PaperlayerView exact revision on arXiv

Source-linked AI summary

GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control

Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-David, Thomas Müller, Alexander Keller, Sanja Fidler, Jun Gao

arXiv:2503.03751v1cs.CVcs.GR

TL;DR

GEN3C addresses inconsistent videos and imprecise camera control by conditioning video diffusion on a depth-derived 3D cache rendered along user-specified trajectories. It achieves more accurate camera control and state-of-the-art sparse-view novel-view synthesis, while dynamic-video generation still depends on pre-generated object motion.

  • Problem

    Existing video models provide limited 3D modeling, causing temporal inconsistencies, while camera-parameter conditioning often yields imprecise control.

  • Method

    GEN3C builds a spatiotemporal point-cloud cache from depth estimates of seed images or generated frames, renders it using user camera poses, and conditions a video diffusion model on those renderings.

  • Results

    GEN3C achieves more accurate camera control than previous methods and state-of-the-art results in sparse-view novel-view synthesis, including driving and monocular dynamic scenes.

  • Takeaways & Limitations

    The explicit 3D cache supports precise viewpoint control, 3D-consistent high-fidelity video generation, and sparse-view synthesis in challenging settings.

  • Takeaways & Limitations

    For dynamic content, GEN3C relies on a pre-generated video to provide object motion, and generating that video is itself challenging.

Abstract

from arXiv · show

We present GEN3C, a generative video model with precise Camera Control and temporal 3D Consistency. Prior video models already generate realistic videos, but they tend to leverage little 3D information, leading to inconsistencies, such as objects popping in and out of existence. Camera control, if implemented at all, is imprecise, because camera parameters are mere inputs to the neural network which must then infer how the video depends on the camera. In contrast, GEN3C is guided by a 3D cache: point clouds obtained by predicting the pixel-wise depth of seed images or previously generated frames. When generating the next frames, GEN3C is conditioned on the 2D renderings of the 3D cache with the new camera trajectory provided by the user. Crucially, this means that GEN3C neither has to remember what it previously generated nor does it have to infer the image structure from the camera pose. The model, instead, can focus all its generative power on previously unobserved regions, as well as advancing the scene state to the next frame. Our results demonstrate more precise camera control than prior work, as well as state-of-the-art results in sparse-view novel view synthesis, even in challenging settings such as driving scenes and monocular dynamic video. Results are best viewed in videos. Check out our webpage! https://research.nvidia.com/labs/toronto-ai/GEN3C/

1. Introduction

GEN3C addresses video-generation controllability and consistency by explicitly modeling approximate 3D geometry and rendering it along user-specified camera trajectories. The approach supports consistent videos, sparse-view synthesis, scene manipulation, and broader content-creation applications.

  • GEN3C targets controllability and consistency, including user adjustment of camera motion, scene composition, and dynamics across long-generated videos.
  • The method constructs a point-cloud 3D cache from depth estimates of input images or previously generated frames, then renders it according to the user’s camera trajectory.
  • The video model translates imperfect cache renderings into high-quality videos by correcting projection artifacts and filling missing information.
  • GEN3C maintains consistency when the camera revisits regions, whereas previous work produces severe artifacts without explicit history modeling.
  • The explicit 3D cache also enables scene manipulation by modifying the underlying point cloud.
  • Experiments cover single-image, sparse-view, and dense multi-view inputs, with generalization to dynamic scenes and applications including object removal and scene editing.

2. Related Work

Related work spans novel-view synthesis, camera-controllable video generation, and consistent video generation. GEN3C differs by using an explicit depth-based 3D cache to condition video generation and support sparse or dynamic settings.

  • Novel View Synthesis: Novel-view synthesis methods produce realistic images from posed images but commonly require dense inputs and can artifact from extreme viewpoints.
  • Camera-Controllable Video Generation: GEN3C uses a depth-derived 3D cache and renders it along user-provided poses, rather than relying only on learned mappings from camera parameters.
  • Camera-Controllable Video Generation: Camera-controllable video methods often input numerical camera parameters, but precise control and generalization to unseen camera motions remain difficult.
  • Consistent Video Generation: Prior consistent-video work also uses point-cloud conditioning, while GEN3C estimates seed-image depth and reconciles it with a diffusion-based video model.

3. Background: Video Diffusion Models

Video diffusion models learn data distributions through iterative denoising of noisy samples, often operating in a compressed latent space. GEN3C uses a pretrained image-to-video diffusion model within this framework.

  • A diffusion model learns a data distribution through iterative denoising of progressively noised data samples.
  • Training constructs noisy samples as x_τ = α_τx_0 + σ_τϵ, with Gaussian noise and a scheduled diffusion time.
  • The model parameters are optimized with a denoising score-matching objective whose target may be noise, a transformed sample, or the clean sample.
  • After training, repeated application of the denoising model to Gaussian noise produces samples from the learned data distribution.
  • Latent diffusion compresses RGB videos with a pretrained VAE encoder, performs diffusion in latent space, and decodes the result with a VAE decoder.
  • GEN3C uses Stable Video Diffusion and remains compatible with other image-to-video diffusion architectures.

4. Method: 3D-Informed Video Generation

GEN3C builds a spatiotemporal point-cloud cache from depth estimates, renders it along user-specified camera trajectories, and injects those renderings into a fine-tuned video diffusion model. This design aggregates multiple views, fills disoccluded regions, and supports consistent video generation, cache editing, and autoregressive long-video inference.

  • 4.1. Building a Spatiotemporal 3D Cache: GEN3C constructs a spatiotemporal 3D cache as an L × V array of point clouds unprojected from RGB-image depth estimates.The cache dimensions represent temporal length L and number of camera views V.
  • 4.1. Building a Spatiotemporal 3D Cache: The cache is configured for single-image video, static sparse- or dense-view NVS, and dynamic NVS by varying its view and temporal elements.Single-image and static-NVS caches duplicate their available observations across the temporal dimension.
  • 4.2. Rendering the 3D Cache: Point-cloud rendering maps each cache element and camera pose to an RGB image and a disocclusion mask identifying pixels that need filling.The mask flags regions not covered by the rendered point cloud.
  • 4.2. Rendering the 3D Cache: For a user-provided camera sequence, GEN3C renders every cache view into an image-and-mask video, then concatenates each view's renderings over time.These rendered videos provide the visual conditioning used for generation.
  • 4.3. Fusing and Injecting the 3D Cache: GEN3C masks uncovered latent regions by element-wise multiplication and fuses multiple viewpoint features with max pooling while leaving the model architecture unchanged.This avoids adding mask-channel parameters that might not generalize from the pre-trained video diffusion model.
  • 4.4. Training and Inference: The modified video diffusion model is fine-tuned on rendered-cache and RGB-video pairs, then generates videos by iterative denoising conditioned on cache renderings.A pre-trained VAE decodes the final denoised latent into RGB video; a 14-frame video takes around 30 seconds on one A100 GPU.

5. Experiments and Applications

GEN3C is evaluated across single-image, sparse-view, driving, dynamic, and out-of-domain video-generation settings. The experiments show precise camera control, 3D-consistent rendering, missing-region completion, and utility for scene editing, with robustness to noisy depth and extreme viewpoint changes.

  • 5.1. Training Details: Training combines static real-world video for spatial consistency with synthetic multi-view dynamic video for temporal consistency.The setup addresses the lack of paired multi-view dynamic real-world data.
  • 5.2. Single View to Video Generation: GEN3C uses depth-derived 3D caches and rendered camera trajectories to generate videos from single images or sparse views.The same framework supports single-image generation, two-view NVS, and arbitrary numbers of input views during inference.
  • 5.2. Single View to Video Generation: GEN3C outperforms baselines on in-domain and out-of-domain single-view testing while more precisely following the ground-truth camera trajectory.CameraCtrl generalizes poorly out of domain, whereas explicit 3D-cache modeling causes only a small performance drop for GEN3C.
  • 5.4. Novel View Synthesis for Driving Simulation: On driving scenes, GEN3C achieves significantly better FID scores than reconstruction methods, especially for views far from the original trajectory.Qualitatively, GEN3C fills missing regions where reconstruction-based baselines produce severe artifacts.
  • 5.4. Novel View Synthesis for Driving Simulation: GEN3C supports 3D editing by removing cars, changing their trajectories, and generating plausible re-simulation videos.The explicit 3D cache provides the representation used for these scene modifications.
  • 5.5. Monocular Dynamic Novel View Synthesis: For monocular dynamic NVS, GEN3C preserves object details and dynamics while aligning generated videos with user-specified camera motion.It also produces photorealistic out-of-domain results that preserve 3D content, and smoothly bridges disjoint views despite depth or lighting differences.
  • 5.6. Ablation Study: With noisy depth, performance drops negligibly at small noise levels and remains reasonable at a 30% noise ratio.A more powerful video diffusion model further improves quality under extreme camera viewpoint changes.

6. Conclusion

GEN3C combines a 3D cache with user-provided camera trajectories to achieve consistent video generation and precise camera control. It reports stronger camera control than previous methods and state-of-the-art sparse-view novel view synthesis, including challenging dynamic settings.

  • GEN3C constructs a 3D cache from seed images or previously generated videos and renders it into 2D videos along a user-provided camera trajectory.The rendered cache strongly conditions video generation.
  • GEN3C achieves more accurate camera control than previous methods.
  • GEN3C achieves state-of-the-art results in sparse-view novel view synthesis, including driving scenes and monocular dynamic novel view synthesis.
  • For dynamic content, GEN3C relies on a pre-generated video to provide object motion.The authors identify generating that video as a challenge and suggest text conditioning for motion as a future extension.

Appendices

The appendices provide supplementary method details, experiments, and additional qualitative results.

  • Additional details of the method are provided in Appendix A.
  • Additional experiments are provided in Appendix B.
  • Additional qualitative results are provided in Appendix C.

A. Method Details

The method details describe an autoregressive process that updates a point-cloud 3D cache between overlapping video chunks. Depth from generated frames is aligned to the cache, normalized, unprojected, and appended before generating subsequent chunks.

  • Long videos are divided into overlapping chunks of length L with one-frame overlap, and each chunk is generated sequentially in an autoregressive manner.The first chunk follows the main-paper inference pipeline.
  • The last frame of the first chunk is assigned a pixel-wise depth estimate, which is aligned with the existing 3D cache by minimizing reprojection error.The optimization adjusts depth scaling and translation coefficients.
  • The depth-alignment objective minimizes the masked squared L2 difference between transformed estimated depth and target depth.
  • The optimized scale and translation are applied to normalize the depth estimate before unprojection.
  • The normalized depth is unprojected into a 3D point cloud using the frame’s camera parameters and appended to the existing cache.The updated cache conditions the next chunk, and the process repeats for subsequent chunks to support consistent long-video generation.

B. Experimental Details

The experimental appendix describes the evaluation setup, training procedure, inference settings, and baseline configurations.

  • The experiments use AdamW with learning rate 3e-5 and apply 15% dropout to the rendered-cache and first-frame CLIP conditions.
  • Training begins on RE10K and DL3DV at 320 × 576 for 100K iterations, then fine-tunes on all four datasets at 576 × 1024 for another 100K iterations.
  • Inference uses classifier-free guidance with guidance weight 3 and 25 diffusion steps.
  • The appendix provides further details of the compared baselines.
  • GenWarp and MotionCtrl use official Stable Video Diffusion checkpoints, while NVS-Solver uses its official training-free codebase on the testing data.RE10K is the training dataset for GenWarp and MotionCtrl.

B.4. Two-views NVS

The evaluation covers novel-view generation and related video synthesis settings using reconstructed 3D caches rendered along user-specified camera trajectories. These pipelines support driving and monocular dynamic scenes by incorporating depth from video frames over time.

  • Driving Simulation: For driving simulation, depth from each frame is unprojected and concatenated along the temporal dimension to represent dynamic objects.
  • Inference Pipeline: GEN3C renders a 3D cache along a user-provided camera trajectory to generate video from novel viewpoints.The cache is constructed from estimated depth and rendered before generation.
  • Monocular Dynamic NVS: For monocular dynamic scenes, separately predicted frame depths are concatenated temporally before rendering the requested camera trajectory.

C.1. Generalization with mask channel

Directly applying mask values to the latent generalizes better than concatenating a mask channel, reducing artifacts for extreme novel views. Qualitative comparisons also report sharper details and stronger object dynamics than GCD on Kubric4D.

  • Mask incorporation: Direct latent multiplication of mask values generalizes better than concatenating a mask channel and significantly reduces artifacts in extreme novel views.Concatenation adds parameters that do not generalize well to out-of-distribution masks, especially in driving simulations.
  • Mask incorporation: Ground-truth views for novel driving trajectories are unavailable, making out-of-distribution mask generalization particularly important.
  • 3D cache completion: Rendered-depth comparisons show that GEN3C can correct projection artifacts and fill missing regions in the 3D cache.
  • Kubric4D comparison: GEN3C produces sharper video details, more object details, and stronger dynamics than GCD on the Kubric dataset.
Loading 2503.03751v1…