Source-linked AI summary
CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation
Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, Arash Vahdat
TL;DR
Precise camera-pose control remains limited in video diffusion despite its importance for cinematic and downstream applications. CamCo adapts a pre-trained image-to-video model with Plücker-coordinate conditioning, epipolar attention, and curated dynamic-video training, achieving stronger camera controllability, 3D consistency, visual quality, and object motion than prior methods.
Problem
Video diffusion models offer limited precise control over camera viewpoints, restricting cinematic expression and practical use in augmented reality, filmmaking, and game development.
Method
CamCo adapts a pre-trained image-to-video diffusion model with Plücker-coordinate camera conditioning, epipolar constraint attention, and dynamic-video data curation using Particle-SfM camera annotations.
Results
CamCo demonstrates superior camera controllability, 3D consistency, visual quality, and object motion compared with previous works.
Takeaways & Limitations
CamCo provides camera-controllable image-to-video generation that produces 3D-consistent videos while supporting both camera ego-motion and dynamic subjects.
Takeaways & Limitations
Because training videos share camera intrinsics, CamCo cannot generate complex intrinsic changes such as dolly zoom and currently produces only 14 frames at 256 × 256 resolution.
Abstract
from arXiv · showhide
Recently video diffusion models have emerged as expressive generative tools for high-quality video content creation readily available to general users. However, these models often do not offer precise control over camera poses for video generation, limiting the expression of cinematic language and user control. To address this issue, we introduce CamCo, which allows fine-grained Camera pose Control for image-to-video generation. We equip a pre-trained image-to-video generator with accurately parameterized camera pose input using Plücker coordinates. To enhance 3D consistency in the videos produced, we integrate an epipolar attention module in each attention block that enforces epipolar constraints to the feature maps. Additionally, we fine-tune CamCo on real-world videos with camera poses estimated through structure-from-motion algorithms to better synthesize object motion. Our experiments show that CamCo significantly improves 3D consistency and camera control capabilities compared to previous models while effectively generating plausible object motion. Project page: https://ir1d.github.io/CamCo/
1 Introduction
Camera viewpoint control remains underexplored despite its importance for cinematic expression and practical video applications. CamCo addresses this gap with fine-grained camera conditioning and geometry-aware generation for 3D-consistent videos.
- Motivation: Camera viewpoint control has received little attention compared with other signals such as depth, edges, and human pose.Camera motion supports filmmaking and applications including augmented reality, filmmaking, and game development.
- Limitations of Existing Methods: Existing methods represent camera trajectories with fixed categories or one-dimensional camera values, limiting motion granularity and precision.These approaches use LoRA or adapter layers to condition video generation on camera information.
- CamCo: CamCo adapts a pre-trained image-to-video diffusion model and represents camera intrinsics and extrinsics with pixel-wise Plücker-coordinate embeddings.The dense conditioning signal is designed to improve camera motion control while preserving the base model’s generative capabilities.
- CamCo: CamCo adds epipolar constraint attention modules to enforce geometric consistency across video frames.The framework also curates in-the-wild videos with estimated camera poses to improve generation involving large object motions.
- Results: Experiments report superior 3D consistency, visual quality, and camera controllability compared with previous methods.The contribution summary also characterizes CamCo as a high-quality image-to-video framework for 3D-consistent videos.
2 Related Work
Related work spans diffusion-based video generation, controllable generation, and efforts to extend 2D generative models toward 3D tasks. These approaches establish the broader context for CamCo’s camera-controlled video generation.
- Diffusion-Based Generation: Diffusion models became leading tools for text-to-image synthesis by leveraging large-scale image-text datasets.Researchers adapted these models for video generation because high-quality video-text datasets are scarce.
- Diffusion-Based Generation: Text-to-video methods include training-free pipelines, reusable motion modules, and image-model extensions with temporal layers and curated video data.Examples include Text2Video-Zero, AnimateDiff, VideoLDM, and Stable Video Diffusion.
- Controllable Generation: Controllable generation methods add signals such as depth, edges, semantic maps, and object bounding boxes while preserving the synthesis backbone.ControlNet, T2I-Adapter, and GLIGEN established this control-signal paradigm for text-to-image generation.
- 3D Generation: Limited high-quality 3D scene datasets make it difficult to transfer image and video generation advances directly to 3D generation.Score distillation sampling addresses this challenge by distilling knowledge from 2D diffusion models.
3 Method
CamCo adapts a pre-trained image-to-video diffusion model with dense Plücker-coordinate camera conditioning and epipolar attention for camera-controlled, geometry-consistent videos. It further uses estimated camera poses from dynamic web videos to improve object motion.
- Framework: CamCo integrates camera control and geometry consistency into a pre-trained image-to-video diffusion model.The framework adds camera parameterization, control modules, epipolar attention, and curated motion data.
- Camera Parameterization: Plücker coordinates encode camera intrinsics and extrinsics into pixel-wise ray embeddings for fine-grained conditioning.They represent each pixel ray using its origin and normalized direction, providing dense camera information.
- Camera Control: Adapter layers concatenate Plücker embeddings with network features inside each temporal attention block while preserving the pre-trained generator’s capabilities.A 1 × 1 convolution projects the concatenated features back to the original feature space.
- Epipolar Constraint Attention: Epipolar constraint attention cross-attends target-frame features to features sampled along corresponding epipolar lines in the first frame.This imposes geometric relationships without requiring exact pointwise correspondences, which are unavailable from in-the-wild images.
- Epipolar Constraint Attention: The epipolar attention matrix has complexity O(hwl) instead of O((hw)^2), improving efficiency by one order of magnitude when l ≈ O(h) ≈ O(w).Training updates only the ECA layers while freezing the base video model.
- Motion Data: Dynamic web videos with camera poses estimated by Particle-SfM augment training data to address CamCo’s tendency to generate static scenes.The pipeline samples WebVid clips and estimates camera trajectories from 32 frames because annotated dynamic videos are limited.
4 Experiments
CamCo is evaluated on static and dynamic video generation using camera-following, reconstruction, pose-accuracy, and visual-quality measures. It outperforms baselines and ablated variants, while handling both camera changes and object motion.
- Experimental Setup: Evaluations compare CamCo with Stable Video Diffusion, VideoCrafter, and MotionCtrl on static and dynamic video settings.Static experiments use RealEstate-10k; dynamic experiments use 1,000 videos from the annotated WebVid test split.
- Metrics: Camera pose accuracy is measured from COLMAP-estimated camera-to-world matrices using summed relative rotation and translation errors.COLMAP reconstruction success and sparse-point counts are also reported as indicators of reconstruction quality and 3D consistency.
- Metrics: FID and FVD measure the distance between generated frames or videos and corresponding reference videos, independently of camera-following ability and geometry quality.These metrics evaluate visual quality using deep features extracted from video frames.
- Static Generation: On static videos, CamCo achieves the best camera-following ability and outperforms baselines across visual quality, camera controllability, and geometric consistency metrics.The evaluated input images and camera trajectories were unseen during training.
- Dynamic Generation: On dynamic videos, CamCo produces perspective changes and object motion, while its full model achieves the best FID and FVD values.The uncurated-dataset and static-version variants show inferior results, indicating poorer object motion.
- Ablation Studies: The full model performs best across 2D visual quality and camera-pose metrics, whereas removing epipolar attention or changing camera conditioning degrades control and appearance.Ablated metrics include FID, FVD, COLMAP error, reconstructed points, translation error, and rotation error.
5 Conclusion
CamCo is a camera-controllable framework for generating 3D-consistent videos. Its experiments report superiority over previous works in camera controllability, geometric consistency, and visual quality across images from various domains.
- Conclusion: CamCo combines a pre-trained image-to-video diffusion model with camera conditioning, geometry-constraint blocks, and a data-curation pipeline for object motion.The framework is designed to preserve generative capabilities while improving camera control, geometry consistency, and dynamic motion generation.
- Conclusion: Experiments report superiority over previous works in camera controllability, geometry consistency, and visual quality across images from various domains.
A Additional Details on Epipolar Constraint Attention
The epipolar constraint is defined geometrically through projections between camera views. An epipolar line is the image-plane projection of the 3D line joining a point and another camera’s projection center.
- Epipolar Geometry: An epipolar line is the projection on one camera’s image plane of the line connecting a 3D point with another camera’s projection center.
B.1 COLMAP configuration
The COLMAP configuration assumes shared intrinsics across video frames and uses affine-shape and domain-size-pooling SIFT extraction with guided feature matching. Videos are retried up to five times, while long sequences are sampled into 32 frames for Particle-SfM annotation.
- COLMAP configuration: COLMAP assumes all frames in each video share camera intrinsics and enables affine-shape and domain-size-pooling SIFT extraction.
- COLMAP configuration: Feature matching uses guided matching with a maximum of 65,536 matches, and each video is retried at most five times.
- Particle-SfM preprocessing: For long videos, the pipeline randomly samples a frame stride and uniformly selects 32 frames for Particle-SfM camera annotation.
- Evaluation setup: RealEstate-10k uses the PixelSplat train-test split, while baselines run at their original resolution and open-source checkpoints provide comparison models.
C Implementation Details
CamCo builds on SVD and uses a relative camera system with normalized poses, sampled training strides, and fixed-resolution frame processing. Inference produces 14-frame videos using 25 sampling steps.
- CamCo builds upon SVD, a UNet-based open-source image-to-video diffusion model similar in architecture to VideoLDM.
- Camera poses are made relative to the first frame, which is placed at the world origin and rotated toward the x-axis.
- Training randomly samples sequence strides and normalizes camera extrinsics to unit maximum distance from the world origin.
- Training frames are center-cropped to squares and downsampled to 256 × 256 resolution.
- Inference uses 25 sampling steps to generate 14 frames, with a 14-frame latent-decoder chunk size.
D Limitations and Future Work
CamCo preserves the input image’s camera intrinsic and generates 14 frames at 256 × 256, limiting intrinsic changes, viewpoints, and coverage of large scenes.
- CamCo cannot generate complex camera-intrinsic changes such as dolly zoom because training frames share the same camera intrinsics.
- Its 14-frame, 256 × 256 output supports only limited viewpoints and may be insufficient for large scenes.
- Future work will explore generating longer videos at larger resolutions.
E Broad Impacts
CamCo focuses on image-input video generation and may contain social biases that perpetuate stereotypes or misrepresentations, despite safety measures described by the authors.
- The paper focuses on generating videos from image input.
- CamCo may contain social biases that perpetuate stereotypes or misrepresentations in generated videos, potentially influencing public perceptions and causing societal harm.
- The text states that NSFW content was removed from the T2I base model and that a safety checker is released.