PaperlayerView exact revision on arXiv

Source-linked AI summary

Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control

Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas Guibas, Gordon Wetzstein

arXiv:2405.17414v1cs.CVcs.GR

TL;DR

Generating the same scene across multiple camera trajectories remains challenging despite progress in camera-controlled video generation. CVD combines epipolar cross-view synchronization with camera control to generate multiple videos with improved consistency over baselines.

  • Problem

    Camera-controlled video methods generally handle single trajectories, leaving consistent generation of the same scene across different camera trajectories unresolved.

    Source: §1 · p. 1

  • Method

    CVD adds epipolar-geometry-based cross-view synchronization to a camera-controlled video model, with hybrid training on static multi-view and monocular dynamic data.

    Source: §1 · p. 2, §4.1 · p. 5, §4.2 · p. 6

  • Results

    CVD is reported to achieve strong geometric and semantic consistency and significantly outperform relevant baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CVD generates videos with distinct camera trajectories that share scene content and motion, supporting camera-customized multi-view scene capture.

    Source: §1 · p. 2, §6 · p. 9

    CVD depends on the consistency of its AnimateDiff and CameraCtrl base models, does not resolve within-video inconsistency, and cannot synthesize videos in real time.

    Source: §6.1 · p. 9

Abstract

from arXiv · show

Research on video generation has recently made tremendous progress, enabling high-quality videos to be generated from text prompts or images. Adding control to the video generation process is an important goal moving forward and recent approaches that condition video generation models on camera trajectories make strides towards it. Yet, it remains challenging to generate a video of the same scene from multiple different camera trajectories. Solutions to this multi-video generation problem could enable large-scale 3D scene generation with editable camera trajectories, among other applications. We introduce collaborative video diffusion (CVD) as an important step towards this vision. The CVD framework includes a novel cross-video synchronization module that promotes consistency between corresponding frames of the same video rendered from different camera poses using an epipolar attention mechanism. Trained on top of a state-of-the-art camera-control module for video generation, CVD generates multiple videos rendered from different camera trajectories with significantly better consistency than baselines, as shown in extensive experiments. Project page: https://collaborativevideodiffusion.github.io/.

1 Introduction

Camera control has advanced video generation, but generating the same scene across different camera trajectories remains difficult. CVD combines collaborative video generation, cross-video synchronization, and hybrid training to produce more consistent camera-controlled videos.

1 Introduction

  • Earlier control approaches use signals such as flow, keypoints, or depth, but the cited methods still lack camera control for video generation.Text- or image-conditioned approaches also lack precise control over content and motion.Source: §1 · p. 1
  • Existing camera-control methods condition generation on pose sequences but support single-camera trajectories, leaving multi-video scene content and dynamics inconsistent.Consistent multi-video camera control is sought for applications such as large-scale 3D scene generation, while large-scale dynamic multi-view training data is scarce.Source: §1 · p. 1
  • CVD generates videos with different camera trajectories that share scene content and motion, using epipolar attention and a learnable cross-view synchronization module.The training scheme combines RealEstate10K for camera control with WebVid10M for motion.Source: §1 · p. 2
  • CVD is presented as the first approach to generate multiple videos with consistent content and dynamics while providing camera control, and it significantly outperforms relevant baselines.The contribution summary also reports superior multi-view consistency against all baseline methods.Source: §1 · p. 2

2 Related Work

Related work spans video diffusion, controllable generation, and coherent multi-view imagery. Prior work addresses camera trajectories and multi-view consistency, but cited image methods have limits for large-scale or dynamic scenes.

Video Diffusion Models

  • Video diffusion models build on image-generation systems with temporal modules or noise strategies, while SORA uses space-time diffusion with a transformer for long-video generation.AnimateDiff adds temporal modules to Stable Diffusion; Align-your-latents uses 3D convolutions and factorized space-temporal blocks.Source: §2 · p. 2

Controllable Video Generation

  • Controllable-video methods use signals such as depth, skeletons, images, flow, motion, and camera trajectories, with CameraCtrl encoding camera parameters through Plücker Embeddings.Other cited approaches use motion or camera LoRAs, or concatenate camera parameters as features.Source: §2 · p. 2

Multi-View Image Generation

  • Multi-view image work ranges from pose-conditioned novel views to scene-generation methods, while the figure contrasts separate inconsistent videos with shared-content collaborative generation trained on pairs and extended to more videos.The figure depicts separate generations as potentially differing in geometry, objects, and motion.Source: §2 · p. 2, Figure 1 · p. 3
  • Several cited multi-view methods target objects or scenes, but prior methods are described as limited for large-scale 3D scenes; PoseGuided-Diffusion's static indoor prior does not generalize to in-the-wild or dynamic scenes.Po et al. provide a survey of recent diffusion advances in visual computing.Source: §2 · p. 3

3 Collaborative Video Generation

Collaborative Video Diffusion extends video diffusion to generate multiple structurally consistent videos under separate conditions. It uses pairwise training and cross-view synchronization, then supports arbitrary video counts at inference.

3 Collaborative Video Generation

  • Video diffusion progressively adds noise to video data and trains a model to estimate the backward distribution; inference starts from Gaussian noise and repeatedly denoises toward a video.The loss minimizes noise-prediction error, with noisy video features formed from the original feature and sampled noise.Source: §3 · p. 3
  • CVD introduces an objective for simultaneously generating multiple structurally consistent videos, addressing disparities in content and spatial arrangement that separate video generations can exhibit.Inference-stage extended attention can increase semantic similarity without addressing structural consistency.Source: §3 · p. 3
  • The architecture processes two or more noisy video features and camera trajectories to predict noise for each video, with masked cross-view attention using same-frame features and their fundamental matrix.The image autoencoder is omitted from the architecture illustration.Source: Figure 2 · p. 4
  • The model represents an arbitrary number of videos with separate conditions and targets their joint data distribution.Synchronous multi-view videos from one dynamic 3D scene are given as an example.Source: §3 · p. 4
  • Because large-scale multi-view video data is scarce, CVD trains on consistent video pairs and uses an inference algorithm to generate arbitrary numbers of videos.The model is designed to accommodate any number of input video features.Source: §3 · p. 4

4 Collaborative Video Diffusion with Camera Control

CVD extends camera-controlled video diffusion to generate videos from multiple camera trajectories, using cross-view synchronization to align their geometry. It trains on synchronized video pairs and provides an inference strategy for generating arbitrary numbers of videos.

4 Collaborative Video Diffusion with Camera Control

  • CVD extends CameraCtrl to process multiple noisy video features, training on synchronized pairs whose camera trajectories share the same first pose.The setup assumes corresponding frames are captured simultaneously and sets the first pose of every trajectory to be identical.Source: §4 · p. 4

4.1 Cross-View Synchronization Module

  • The Cross-View Synchronization Module uses epipolar geometry to align corresponding synchronized frames, addressing structural inconsistency that extended attention alone leaves.For the first frames, whose identical camera poses leave the fundamental matrix undefined, the method uses pseudo-epipolar lines.Source: §4.1 · p. 5

4.2 Hybrid Training Strategy from Two Datasets

  • Training uses folded RealEstate10K subsequences for synchronized pairs and homography-augmented WebVid10M clones, with CameraCtrl/ground-truth versus AnimateDiff/pseudo-epipolar configurations.RealEstate10K mostly contains static indoor scenes; WebVid10M supplements it with varied video content but has no camera information.Source: §4.2 · pp. 5–6

4.3 Towards More Videos

  • For M videos, CVD selects feature pairs, predicts pairwise noise, and averages predictions per video feature; exhaustive pairing is O(M^2) versus O(M) for partitioning.Exhaustive pairing covers every pair and can produce more consistent videos, while multi-partitioning trades off between the strategies.Source: §4.3 · p. 6

5 Experiments

CVD outperforms baselines on geometry consistency and performs better than or on par with prior work on semantic matching, cross-video consistency, and frame fidelity. Qualitative results also show consistent multi-view generation for static and dynamic scenes and arbitrary camera trajectories.

5.1 Quantitative Results

  • For per-video geometry evaluation, 1,000 videos were generated from RealEstate10K camera-trajectory pairs and prompts, then assessed using frame correspondences and estimated camera poses.CameraCtrl and MotionCtrl are the comparison baselines; the evaluation measures pose angle errors and correspondence epipolar errors.Source: §5.1 · p. 6
  • CVD achieves the highest scores on all geometry metrics and is better than or on par with prior work on semantic matching, cross-video consistency, and frame fidelity.Table 1 reports geometry consistency; Table 2 reports semantic and fidelity metrics.Source: Table 1 · p. 7, Table 2 · p. 7
  • Across 500 generic-scene video pairs using RealEstate10K trajectories and WebVid10M captions, CVD greatly outperforms the modified CameraCtrl and MotionCtrl baselines on cross-video geometry consistency.The baselines generate a reference video first, then use its first frame to condition a second video.Source: §5.1 · p. 7
  • CLIP-based semantic metrics use WebVid10M, while fidelity metrics use RealEstate10K; CVD surpasses baselines on CLIP metrics and is better than or on par with prior work on fidelity.FID and KID comparisons exclude models with different base models.Source: §5.1 · p. 7, Table 2 · p. 7

5.2.1 Comparison with Baselines

  • Qualitative comparisons show stronger content alignment than CameraCtrl, CameraCtrl with SparseCtrl, MotionCtrl, and MotionCtrl with SVD, including for dynamic content such as lightning and waves.Figure 4 reports consistency across static and dynamic scenes; MotionCtrl with SVD generated no motion beyond simple static-camera zooming in the authors’ tests.Source: §5.2.1 · p. 8, Figure 4 · p. 8

5.2.2 Additional results for arbitrary views generation

  • CVD generates groups of camera-conditioned videos sharing content, structure, and motion, including views moving in four directions and interpolated camera trajectories.Figure 5 shows cameras looking at the same 3D point on the left and trajectory interpolation between the first and fourth rows on the right.Source: §5.2.2 · p. 8, Figure 5 · p. 9

6 Discussion

CVD uses epipolar geometry and multi-view sampling to synchronize content and dynamics across videos with customizable cameras. Its performance depends on its base models, and it does not generate videos in real time.

6 Discussion

  • CVD combines epipolar geometry with multi-view sampling to exchange information across videos and support synchronized multi-camera video generation.The discussion describes integrating dynamic single-view in-the-wild videos during training to maintain diverse motion patterns.Source: §6 · p. 9

6.1 Limitations

  • CVD depends on AnimateDiff and CameraCtrl, does not solve within-video consistency, and cannot synthesize videos in real time because diffusion is computationally intensive.Shape shifting and dynamic inconsistencies from the base models may persist in generated outputs.Source: §6.1 · p. 9

6.2 Broader Impacts

  • The authors warn that the method could be misused to create deceptive content such as deepfakes and advocate continued improvement of deepfake detection.Source: §6.2 · p. 9

A.1 More Analysis on Collaborative Video Generation.

The appendix describes collaborative generation through a score function built from video-pair models and discusses when that estimate can reflect the joint video distribution. It also reports that the full pipeline outperforms ablations on geometric and semantic consistency.

A.1 More Analysis on Collaborative Video Generation.

  • CVD estimates a score for multiple videos by applying a score function trained on video pairs to all videos and combining pairwise score components with a weighted sum.The pair order is irrelevant.Source: §A.1 · p. 13
  • The authors say denoising is more likely to align with the true distribution when each video's prediction from all noisy videos resembles its prediction from a pairwise input.They argue this is reasonable because a few views can often reveal the underlying geometry, and identify it as a key reason pair training can yield consistent multi-view videos.Source: §A.1 · p. 14
  • The full pipeline outperforms ablation settings on both geometric and semantic consistency for generic scenes.Table 3 evaluates prompts from WebVid10M.Source: Table 3 · p. 15

A.2 Ablation Study on CVD

The ablation study evaluates epipolar constraints, mixed training, homography augmentation, and the number of synchronization modules. It reports that these design choices contribute to geometric or semantic consistency.

A.2 Ablation Study on CVD

  • Without epipolar constraints, outputs can remain semantically consistent, but the constraints are essential for geometric consistency.The ablation replaces epipolar attention with normal self-attention in the Cross-View Synchronization Module.Source: §A.2 · p. 15
  • Mixed training and homography augmentation greatly improve all metrics, including semantic consistency.The variants remove mixed training or turn off homography transformations applied to WebVid10M videos.Source: §A.2 · p. 15
  • The ablation also tests training only on RealEstate10K and using one rather than two Cross-View Synchronization Modules.The RealEstate10K-only setting follows CameraCtrl's training setup.Source: §A.2 · p. 15

A.3 Implementing Details

The implementation combines AnimateDiff with CameraCtrl and retains a plug-and-play path for swapping in fine-tuned base models.

A.3 Implementing Details

  • The pipeline is built on AnimateDiff, an open-source text-to-video model.The paper describes it as popular among artists.Source: §A.3 · p. 15
  • CameraCtrl is deployed to provide camera conditioning.Source: §A.3 · p. 15
  • The base model can be swapped for a fine-tuned version, including models adapted with DreamBooth or LoRA.The authors describe this as a plug-and-play property.Source: §A.3 · p. 15

A.3.1 Training

The appendix details training and inference settings, then presents attention-map, identical-trajectory, fine-tuned-model, and additional qualitative results.

A.3.1 Training

  • Training pairs comprise 65,000 RealEstate10K videos and 2,400,000 WebVid10M videos, each represented by two 16-frame videos.WebVid10M clips are duplicated and the second video receives random homography deformations; RealEstate10K clips are split into two videos.Source: §A.3.1 · p. 15
  • Training mixes RealEstate10K and WebVid10M at a 7:3 ratio, freezes backbone parameters, and optimizes newly injected layers for 100k iterations.The training uses eight NVIDIA A100 GPUs, an effective batch size of 8, and takes approximately 30 hours.Source: §A.3.1 · pp. 15–16

A.3.2 Inference

  • For two-, four-, and six-view generation, the inference settings use (R, P) values of (1, 1), (4, 1), and (6, 2), respectively.Inference uses a DDIM scheduler with 25 steps.Source: §A.3.2 · p. 16

A.4 Results of Attention Maps

  • The attention-map visualization shows a pixel attending to a corresponding region in the second image along its epipolar line.Source: §A.4 · p. 16, Figure 6 · p. 17

A.5 Performances with identical camera trajectories

  • For identical camera trajectories, CVD generates identical video contents and reaches an MSE of 0.01, versus 0.07 for CameraCtrl and 0.06 for CameraCtrl+SparseCtrl.Source: §A.5 · pp. 16–17, Figure 7 · p. 17

A.6 Additional results for LoRA fine-tuned models

A.7 More Qualitative Results

  • Figures 9–13 show video pairs generated with different realizations and camera trajectories for each prompt.The appendix directs readers to the supplementary video for better illustrations.Source: §A.7 · p. 17, Figure 9 · p. 19, Figure 10 · p. 20
Loading 2405.17414v1…