PaperlayerView exact revision on arXiv

Source-linked AI summary

VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control

Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, David B. Lindell, Sergey Tulyakov

arXiv:2407.12781v3cs.CV

TL;DR

Existing text-to-video models lack fine-grained camera control, and prior methods do not support transformer architectures that jointly model spatial and temporal information. VD3D introduces ControlNet-like spatiotemporal Plücker-coordinate conditioning for video transformers and achieves state-of-the-art camera controllability and video quality, while exposing limitations for dynamic scenes and frozen upsamplers.

  • Problem

    Text-to-video models lack explicit 3D camera control, while existing camera-conditioning techniques are incompatible with video transformers that jointly process spatial and temporal information.

  • Method

    VD3D uses a ControlNet-like conditioning mechanism with spatiotemporal camera embeddings based on Plücker coordinates for a video transformer.

  • Results

    VD3D achieves state-of-the-art camera controllability and video quality, including on manually crafted prompts and unseen camera trajectories.

  • Takeaways & Limitations

    The method enables users to specify camera poses for video generation and supports downstream applications such as multi-view text-to-video generation.

  • Takeaways & Limitations

    Dynamic scenes rendered from different viewpoints can show inconsistent motion, and camera conditioning is applied only to the low-resolution model while the upsampler remains frozen.

Abstract

from arXiv · show

Modern text-to-video synthesis models demonstrate coherent, photorealistic generation of complex videos from a text description. However, most existing models lack fine-grained control over camera movement, which is critical for downstream applications related to content creation, visual effects, and 3D vision. Recently, new methods demonstrate the ability to generate videos with controllable camera poses these techniques leverage pre-trained U-Net-based diffusion models that explicitly disentangle spatial and temporal generation. Still, no existing approach enables camera control for new, transformer-based video diffusion models that process spatial and temporal information jointly. Here, we propose to tame video transformers for 3D camera control using a ControlNet-like conditioning mechanism that incorporates spatiotemporal camera embeddings based on Plücker coordinates. The approach demonstrates state-of-the-art performance for controllable video generation after fine-tuning on the RealEstate10K dataset. To the best of our knowledge, our work is the first to enable camera control for transformer-based video diffusion models.

1 INTRODUCTION

Existing text-to-video models provide realistic synthesis but lack explicit 3D camera control, while prior camera-conditioning methods do not extend to jointly spatiotemporal video transformers. VD3D addresses this gap with ControlNet-like spatiotemporal camera embeddings and achieves state-of-the-art controllable video synthesis.

  • Motivation: Text-to-video models produce coherent, physically plausible videos but lack built-in explicit 3D control, making camera specification laborious through prompting and trial and error.They struggle even with simple zoom-in or zoom-out trajectories.
  • Contribution: The method augments 2D video generation with control over camera position and orientation, providing finer-grained control than text prompting for downstream applications.The contribution targets use by artists and other end users.
  • Research gap: U-Net-based camera-control techniques are not applicable to modern video transformers, which jointly process spatial and temporal tokens without standalone temporal layers.The shift toward joint spatiotemporal modeling creates a need for transformer-specific camera conditioning.
  • Method: VD3D introduces a ControlNet-like mechanism using spatiotemporal camera embeddings derived from Plücker coordinates and combined with network inputs through a separately trained cross-attention layer.The method is designed for spatiotemporal transformers rather than decomposed spatial-temporal architectures.
  • Results: VD3D achieves state-of-the-art camera controllability and video quality on manually crafted prompts and unseen camera trajectories, with downstream multi-view text-to-video applications.The evaluation compares against transformer adaptations of previous camera-control methods.

2 RELATED WORK

Related work spans text-to-video, 4D generation, controllable diffusion, and concurrent camera-control methods. Existing camera-control approaches mainly target U-Net architectures, leaving jointly spatiotemporal video transformers insufficiently addressed for text-based dynamic-scene generation.

  • Text-to-video generation: Text-to-video methods commonly augment text-to-image models with temporal layers or train on images and videos to improve video-generation quality.This work builds on that broader 2D video-generation line.
  • 4D generation: 4D-generation methods create dynamic 3D scenes from controllable viewpoints, using text or images while improving visual and motion quality.Their target is controllable viewpoint generation for dynamic scenes.
  • Controllable generation with diffusion models: Diffusion models support conditioning on signals such as text, depth, edges, pose, and camera poses for controllable image, video, and multi-view generation.Prior transformer-based ControlNet work addressed spatial conditioning, whereas this paper targets spatiotemporal camera-pose conditioning.
  • Concurrent 3D camera control methods: Concurrent camera-control methods condition U-Net temporal layers with Plücker coordinates or other embeddings, and some manipulate attention without additional training.These approaches demonstrate promising results primarily for U-Net-based video diffusion models.
  • Concurrent 3D camera control methods: Existing camera-control techniques do not apply to modern video transformers that jointly model spatiotemporal dynamics, while a concurrent transformer method focuses on image-based novel-view synthesis rather than text-based dynamic scenes.This distinction motivates a mechanism tailored to transformer-based text-to-video generation.

3 METHOD

VD3D adapts FIT-based video transformers for camera control by combining pixel-wise Plücker embeddings with ControlNet-like conditioning. The method preserves the transformer’s visual-generation pathway while gradually injecting camera information through cross-attention during diffusion denoising.

  • 3.1 LARGE TEXT-TO-VIDEO TRANSFORMERS: The video generator represents frames as patch tokens and performs joint spatiotemporal computation through FIT blocks that read from and write to latent tokens.The model processes video tokens through recurrent FIT blocks, with self-attention computation performed in a shorter latent-token sequence.
  • 3.2 CAMERA CONTROL FOR SPATIOTEMPORAL TRANSFORMERS: Camera parameters are normalized relative to the first frame to establish a consistent coordinate system across dataset samples.The method recomputes frame rotations and translations relative to the first camera frame before constructing the conditioning representation.
  • 3.2 CAMERA CONTROL FOR SPATIOTEMPORAL TRANSFORMERS: Plücker coordinates convert each pixel’s camera ray into a fine-grained spatiotemporal representation, which is patchified and projected into camera tokens matching the video-token sequence.This representation provides positional information for every pixel across frames, supporting accurate camera-motion conditioning.
  • 3.2 CAMERA CONTROL FOR SPATIOTEMPORAL TRANSFORMERS: VD3D injects camera information through a ControlNet-like mechanism that gradually conditions FIT read cross-attention layers with zero-initialized parameters.The design targets rapid fine-tuning on limited camera-annotated data while preserving visual quality during adaptation.
  • 3.2 CAMERA CONTROL FOR SPATIOTEMPORAL TRANSFORMERS: Compared with MotionCtrl and CameraCtrl adaptations, the proposed conditioning is designed to avoid degraded visual quality or weak camera sensitivity in transformer models.The comparison uses the same camera trajectory input and adapts both baselines to the same transformer-based model.
  • 3.2 CAMERA CONTROL FOR SPATIOTEMPORAL TRANSFORMERS: The method preserves original network behavior at initialization by zero-initializing output convolutions and derives fine-grained 3D camera control for large video diffusion transformers.The resulting approach is named VD3D and is presented as a camera-conditioning method tailored to spatiotemporal transformers.

4 EXPERIMENTS

The experiments compare VD3D with adapted camera-control baselines on video transformers and evaluate camera alignment, visual quality, text alignment, and pose accuracy. Ablations show that Plücker embeddings in spatiotemporal patches and trainable cross-attention are central to effective camera control, while the method also supports image-conditioned multi-view generation.

  • 4.1 ASSESSMENT: The evaluation compares VD3D with adapted MotionCtrl and CameraCtrl baselines using RealEstate10K and MSR-VTT prompts, including in- and out-of-distribution tests.Camera pose accuracy is evaluated with ParticleSfM after normalizing camera scales across scenes.
  • 4.1 ASSESSMENT: Adapting MotionCtrl degrades visual quality and text alignment, whereas CameraCtrl preserves visual quality but shows little agreement with input camera poses.The comparison controls training duration by using the same number of iterations for all models.
  • 4.1 ASSESSMENT: 82% and 78% of participants prefer VD3D's camera alignment over adapted MotionCtrl and CameraCtrl, respectively.All user-study results are statistically significant at p < 0.001, and VD3D also outperforms prior works in camera pose accuracy.
  • 4.2 ABLATIONS: Plücker embeddings provide essential spatial conditioning, and training ControlNet cross-attention is key to learning camera control while preserving the original model weights.Simply adding zero-initialized Plücker features to patches provides almost no camera control; the trainable cross-attention block is confirmed by camera evaluation.
  • 4.2 ABLATIONS: Integrating Plücker embeddings into spatiotemporal patches yields better camera conditioning than adding camera matrices or Plücker features to the context vector.The context-vector alternatives produce poor pose accuracy or higher camera pose errors.
  • 4.2 ABLATIONS: The model generates view-consistent renderings from arbitrary viewpoints when given a real image of an unseen complex scene.This image-to-multiview setting is presented as relevant to downstream 3D reconstruction pipelines.

5 CONCLUSION

The paper adds camera-pose controllability to large video transformers, supporting applications including novel-view synthesis and single-image 3D reconstruction. Its main limitation is scope: dynamic scenes may have inconsistent motion across viewpoints, the upsampler lacks camera conditioning, and generation is limited to 16 frames.

  • 5 CONCLUSION: VD3D enables users to specify the camera poses from which video frames are rendered, extending controllability in large-scale video transformers.The targeted applications include novel-view synthesis, single-image 3D reconstruction, and text-conditioned scene synthesis.
  • 5 CONCLUSION: Dynamic scenes rendered from different camera viewpoints can have inconsistent motion, despite static scenes appearing 3D consistent.The authors identify joint control of scene motion and camera motion as future work.
  • 5 CONCLUSION: The approach conditions only the low-resolution SnapVideo model, keeps its upsampler frozen, and currently generates relatively short 16-frame videos.Jointly training the upsampler could improve camera control but would introduce architectural and computational challenges.

6 ETHICS STATEMENT

The paper uses RealEstate10K under a Creative Commons Attribution 4.0 International License and notes that generative video technologies can be misused. It recommends safeguards including output filtering, watermarking, and access control.

  • 6 ETHICS STATEMENT: Generative video technologies can be misused, so researchers and developers should consider output filtering, watermarking, and access control.The statement frames these safeguards as measures to consider as the methods continue to improve.
  • 6 ETHICS STATEMENT: RealEstate10K is open-sourced under a Creative Commons Attribution 4.0 International License and is used for camera-control development.The dataset is sourced from content using a CC-BY license.

7 REPRODUCIBILITY STATEMENT

The paper describes reproducibility through detailed method and experiment sections, documented implementation specifics, and established training data. It also situates the work among related 3D, 4D, and controllable-generation research.

  • 7 REPRODUCIBILITY STATEMENT: Section 3 covers the method and algorithmic components, Section 4 covers evaluation protocols and comparisons, and Appendix C documents implementation details.The stated goal is comprehensive reproducibility of the camera-control method.
  • 7 REPRODUCIBILITY STATEMENT: The paper summarizes related 3D and extended 4D work in the appendix because of space constraints.The related-work scope includes 3D generation, 4D generation, and controllable generation with diffusion models.

B QUANTITATIVE EVALUATION

The evaluation covers single image-to-multiview generation and established 2D video-generation metrics, using camera-conditioned RealEstate10K evaluation with ground-truth correspondences.

  • Single image-to-multiview generation is evaluated for all models.
  • RealEstate10K ground-truth correspondences enable conditioning on camera trajectories and evaluation with PSNR, SSIM, and LPIPS.Both low-resolution base-model and upsampled high-resolution results are reported.

B.2 QUALITY METRICS

The quality evaluation uses distribution-similarity metrics, whose interpretation requires caution after dataset-specific fine-tuning.

  • FID and FVD measure similarities between dataset distributions rather than directly measuring visual quality.
  • Fine-tuning is expected to degrade FID and FVD relative to the original model when fitting a different data distribution.The authors relate this pattern to the FID increase reported for ControlNet after fine-tuning StableDiffusion.
  • Joint training with 2D video data is proposed as a future strategy to help alleviate this metric degradation.

B.3 BASELINE VARIANTS

Baseline-variant experiments compare alternative camera-feature placements, freezing choices, and trajectory types against the proposed approach.

  • Alternative MotionCtrl and CameraCtrl variants perform worse than the main-paper variants.The variants are presented primarily to document explored implementations.
  • Integrating MotionCtrl camera matrices into cross-attention outputs and CameraCtrl Plücker embeddings into patches produces worse camera control than the proposed model.
  • The proposed method outperforms MotionCtrl independently of whether attention layers are trained or frozen.The comparison uses matched batch size, iteration count, and parameter size.
  • On rotation-only and user-defined trajectories with substantial directional changes, the proposed methods achieve higher accuracy and generalize across variable rotations and translations.The tested trajectories include rotations, translations, zooming, and combinations of these motions.

B.5 EXPERIMENTS WITH VANILLA DIT

The method is also implemented with a vanilla video DiT architecture, where the proposed spatiotemporal Plücker-token conditioning is presented as architecture-agnostic.

  • VD3D is implemented on a pre-trained text-to-video DiT model in the latent space of the CogVideoX autoencoder.Results are reported in Tables 10 and 11.
  • The vanilla DiT version further improves quality and camera accuracy on out-of-distribution MSR-VTT prompts.
  • The proposed spatiotemporal Plücker tokens are aligned with video patch tokens through ControlNet-type conditioning.The authors describe this mechanism as agnostic to the transformer architecture.
  • The base video DiT uses 32 DiT blocks, with text cross-attention, self-attention, and a fourfold-expanded fully connected network in each block.
  • Base DiT training uses AdamW, a 0.0001 learning rate, 0.01 weight decay, 750,000 iterations, cosine scheduling, and bfloat16.
  • Inference uses deterministic rectified flow with 40 steps, plus time shifting for higher-resolution and longer videos.

C TRAINING DETAILS

The paper uses two generator sizes and evaluates camera control, multi-view generation, and quality across multiple trajectory and backbone settings.

  • Compute: Training the 4B model used 64 NVIDIA A100 40GB GPUs across eight nodes for 1.5 days.A smaller-model run used eight A100 GPUs for approximately one day.
  • Model variants: The main model has 4B parameters, while a 700M generator supports ablations and initial explorations.Both variants use batch size 256 for 50,000 optimization steps.
  • Evaluation settings: Multi-view generation is evaluated with reference camera trajectories and single-view RealEstate10K test images.Reconstruction metrics are computed on subsequent frames for both low-resolution and upsampled high-resolution generations.
  • Evaluation settings: Quality metrics are evaluated using text prompts from the RealEstate10K and MSR-VTT test sets.The evaluation includes the main quality setup and additional-baseline variants.
  • Camera-pose evaluation: Camera-pose evaluation measures translation and rotation errors estimated from generated videos using ParticleSfM.Evaluations cover reference trajectories, random rotations, combined translations and rotations, and a vanilla video DiT backbone.
  • Backbone evaluation: The vanilla video DiT quality evaluation uses the CogVideoX latent space and includes prompts from RealEstate10K and MSR-VTT test sets.This extends evaluation beyond the primary backbone setting.
Loading 2407.12781v3…