PaperlayerView exact revision on arXiv

Source-linked AI summary

Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attention

Dejia Xu, Yifan Jiang, Chen Huang, Liangchen Song, Thorsten Gernoth, Liangliang Cao, Zhangyang Wang, Hao Tang

arXiv:2410.10774v1cs.CV

TL;DR

Video generation has advanced, but 3D consistency and precise camera control remain challenging, particularly for consistent videos of one scene from different camera paths. Cavia combines view-integrated attention with mixed-source training; the authors report stronger geometric consistency and perceptual quality than prior methods.

  • Problem

    Generated videos still face limited 3D consistency and camera controllability, and producing consistent sequences of the same scene from distinct camera paths remains challenging.

    Source: §1 · p. 1

  • Method

    Cavia extends a monocular video generator with cross-view and cross-frame attention and joint training on static, synthetic multi-view dynamic, and monocular dynamic videos.

    Source: §1 · p. 2

  • Results

    Cavia reports superior geometric and perceptual quality to baseline methods in monocular video generation and cross-video consistency.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The framework can generate four views at inference, with improved view consistency and 3D reconstruction of generated frames reported by the authors.

    Source: §1 · p. 2

    Cavia has limited ability to generate large object motion, and its shared-intrinsic camera assumption limits generalization to complex intrinsic changes at inference.

    Source: F LIMITATIONS · p. 22

Abstract

from arXiv · show

In recent years there have been remarkable breakthroughs in image-to-video generation. However, the 3D consistency and camera controllability of generated frames have remained unsolved. Recent studies have attempted to incorporate camera control into the generation process, but their results are often limited to simple trajectories or lack the ability to generate consistent videos from multiple distinct camera paths for the same scene. To address these limitations, we introduce Cavia, a novel framework for camera-controllable, multi-view video generation, capable of converting an input image into multiple spatiotemporally consistent videos. Our framework extends the spatial and temporal attention modules into view-integrated attention modules, improving both viewpoint and temporal consistency. This flexible design allows for joint training with diverse curated data sources, including scene-level static videos, object-level synthetic multi-view dynamic videos, and real-world monocular dynamic videos. To our best knowledge, Cavia is the first of its kind that allows the user to precisely specify camera motion while obtaining object motion. Extensive experiments demonstrate that Cavia surpasses state-of-the-art methods in terms of geometric consistency and perceptual quality. Project Page: https://ir1d.github.io/Cavia/

1 INTRODUCTION

Cavia targets the gap between video-generation progress and limited 3D consistency and camera control, especially when generating consistent videos of the same scene from distinct camera paths. It combines view-integrated attention with diverse training data to generate camera-controllable, multi-view videos.

1 INTRODUCTION

  • Prior camera-control methods still produce imprecise 3D consistency or little object motion, and monocular generators struggle to make consistent videos from distinct camera trajectories.Generating several sequences independently can also yield inconsistent scenes; scarce real-world multi-view video data leaves many approaches limited to near-static scenes or synthetic objects.Source: §1 · p. 1
  • Cavia extends a monocular video generator with cross-view and cross-frame attention to improve consistency across viewpoints and frames.Its joint training uses static videos, rendered synthetic multi-view dynamic videos, and pose-annotated monocular videos to support geometric consistency, object motion, and complex scenes.Source: §1 · p. 2
  • Cavia is evaluated on real and text-to-image-generated images across indoor, outdoor, object-centric, and large-scene cases, with geometric consistency and perceptual quality assessed.The authors report superior qualitative and quantitative performance to previous methods.Source: §1 · p. 2
  • The framework can generate four views at inference, with the authors reporting improved view consistency and 3D reconstruction of generated frames.Source: §1 · p. 2

2 RELATED WORKS

Prior work addresses camera control, multiview image generation, and 4D generation, but the cited approaches leave challenges in dynamic, multi-view video generation for complex scenes. Cavia focuses on that setting with view-integrated attention and mixed training data.

2.1 CAMERA CONTROLLABLE VIDEO DIFFUSION MODELS

  • Camera-control video methods target monocular generation, while CVD and ViVid-Zoo are described as limited to simple motion or fixed-view, object-centric results.Cavia instead targets arbitrary viewpoints and complex scenes using view-integrated attention and mixed training data.Source: §2.1 · pp. 2–3

2.2 MULTI-VIEW IMAGE GENERATION

  • Multiview image-generation approaches improve viewpoint consistency but focus on static scenes or objects, whereas this work introduces object motion into multiview videos with complex scenes.Source: §2.2 · p. 3

2.3 4D GENERATION

  • The reviewed 4D-generation methods face challenges producing realistic results with complex backgrounds, while Cavia emphasizes multi-view, 3D-consistent videos for complex scenes.Source: §2.3 · p. 3

3 METHOD

Cavia extends image-to-video generation to multiple camera-controlled sequences, building on a pretrained video diffusion model. Its cross-frame and cross-view attention mechanisms integrate spatial-temporal and inter-view features to support consistency.

3.1 OVERVIEW

  • Cavia extends image-to-video diffusion by conditioning on camera information and generating V video sequences from the input image and camera paths.Source: §3.1 · p. 3

3.2 CAMERA CONTROLLABLE VIDEO DIFFUSION MODEL

  • Camera conditioning uses Plücker-coordinate ray embeddings concatenated channel-wise with SVD latent inputs to provide precise camera instructions.The coordinates are defined from the normalized ray direction and camera origin; the first convolution is enlarged and new matrices are zero-initialized.Source: §3.2 · p. 4
  • Cross-frame attention inflates SVD's 1D temporal attention into 3D attention over spatial-temporal features, enabling joint modeling when viewpoint changes cause large pixel displacements.The rearrangement preserves feature dimensions and allows inheritance of pretrained SVD weights while incorporating spatial information into attention.Source: §3.2 · pp. 4–5

3.3 CONSISTENT MULTI-VIEW VIDEO DIFFUSION MODEL

  • Cross-view attention rearranges corresponding frames across views so spatial attention exchanges inter-view information, addressing the view inconsistency of independently sampled monocular sequences.Extending token length while retaining other dimensions allows pretrained-weight inheritance, varying view counts in training, and inference-time extrapolation to additional views.Source: §3.3 · p. 5

4 JOINT TRAINING STRATEGY ON CURATED DATA MIXTURES

Cavia jointly trains on static, dynamic multi-view, and monocular video data, using a shared attention design to accommodate varying view counts. The data are curated and reformatted to provide scene geometry, object motion, and complex backgrounds.

4 JOINT TRAINING STRATEGY ON CURATED DATA MIXTURES

  • The view-integrated attention design permits joint use of static, multi-view dynamic, and monocular videos while inheriting module weights.Source: §4 · p. 5

4.1 JOINT TRAINING ON DATA MIXTURES

  • Static-scene videos are reformatted into synchronized V-view sequences with a shared starting frame, and their frame order may be reversed for augmentation.A subsequence of length (F − 1) × V + 1 yields V views with F frames per view.Source: §4.1 · p. 5
  • Animatable Objaverse objects are rendered along smooth trajectories with varied elevation and azimuth, producing multi-view dynamic videos that reduce overfitting to simple camera movements.Source: §4.1 · p. 6
  • Pose-annotated monocular videos are processed as V = 1 samples and combined with multi-view data to contribute object motion and complex backgrounds.Particle-SfM supplies camera poses; noisy or unnatural clips are filtered, and variable view counts are accommodated by the attention modules.Source: §4.1 · p. 6

4.2 DATA CURATION

  • The static-video collection draws on several video datasets and filtered 3D assets, while dynamic-object curation yields 19,000 assets rendered from arbitrary viewpoints and timesteps.The asset sources include Objaverse and Objaverse-XL; dynamic assets exclude objects with excessive or minimal motion.Source: §4.2 · pp. 6–7
  • The monocular-video pipeline filters pose-annotated clips using point-cloud size, aesthetics, OCR, and camera-motion classification, producing 393,000 videos.Videos with failed frame registration, too few or too many points, detected character regions, low aesthetic scores, or static camera motion are excluded.Source: §4.2 · p. 7

5 EXPERIMENTS

Cavia is evaluated on geometric consistency, perceptual quality, and qualitative camera-control and object-motion comparisons. The experiments also point to further ablations and applications in the appendix.

5.1 QUANTITATIVE COMPARISONS

  • The qualitative evaluations use input images and camera trajectories not seen during training.The paper directs readers to supplementary videos for clearer comparisons.Source: §5 · p. 7, Figure 4 · p. 8, Figure 5 · p. 9
  • Cavia significantly outperforms existing baselines on monocular frame-pose consistency measured with COLMAP errors on RealEstate10K.The evaluation samples 1,000 test sequences and reports AUC below rotation and translation thresholds of 5°, 10°, and 20°.Source: §5.1 · p. 7, Table 1 · p. 10
  • Cavia outperforms baselines on 2-view cross-video consistency and achieves the best reported visual quality across monocular and multi-view evaluations.Table 2 reports the 2-view comparison; the visual-quality evaluation uses FID and FVD.Source: §5.1 · pp. 8–10, Table 2 · p. 10

5.2 QUALTITATIVE COMPARISON

  • Qualitatively, Cavia follows camera instructions and generates object motion, while MotionCtrl and CameraCtrl show trajectory errors, distortions, or static scenes.Figure 4 compares camera-control behavior; Figure 5 shows two-view generations with object motion highlighted.Source: Figure 4 · p. 8, Figure 5 · p. 9, §5.2 · p. 10

5.3 ABLATION STUDIES AND APPLICATIONS

  • The appendix presents ablations and explores 3D reconstruction of generated frames and four-view generation.Videos are included in the supplementary material for qualitative comparison.Source: §5.3 · p. 10

6 CONCLUSION

The conclusion presents Cavia as a framework for camera-controllable, consistent multi-view video generation. It attributes its results to cross-frame and cross-view attention and joint training on varied video sources.

6 CONCLUSION

  • Cavia combines cross-frame and cross-view attention with joint training on static 3D scenes and objects, animatable objects, and in-the-wild monocular videos.The paper reports superiority over previous works in geometric consistency and perceptual quality.Source: §6 · p. 10
  • The framework targets consistent multi-view videos with camera control.The conclusion describes Cavia as a framework for this generation task.Source: §6 · p. 10
  • Experiments demonstrate superiority over previous works in geometric consistency and perceptual quality.Source: §6 · p. 10

A ADDITIONAL IMPLEMENTATION DETAILS

Implementation uses separate static and dynamic training stages, with mixed monocular training during the dynamic stage. The model is fine-tuned from SVD 1.0 at 256 × 256 resolution, and inference decodes all 14 frames together.

A ADDITIONAL IMPLEMENTATION DETAILS

B ADDITIONAL DATA CURATION DETAILS

The appendix describes curated static-object, static-scene, dynamic-object, and monocular-video data pipelines. It also identifies module and training-data ablations.

B ADDITIONAL DATA CURATION DETAILS

  • Filtering reduced an initial pool of 800k Objaverse objects to approximately 400k curated static-object instances.Filters remove objects without textures or captions, small or multiple objects, and low-quality objects.Source: B ADDITIONAL DATA CURATION DETAILS · p. 16
  • Static object orbits use 84-frame Blender EEVEE renders with irregular azimuths and varying elevations.The camera is positioned to keep the object visible without clipping.Source: B ADDITIONAL DATA CURATION DETAILS · p. 16
  • Static scenes come from five datasets, with dataset-specific sampling and filtering for training sequences.The listed sources are RealEstate10k, WildRGBD, MVImgNet, CO3Dv2, and DL3DV-10K.Source: B ADDITIONAL DATA CURATION DETAILS · p. 16
  • Dynamic-object filtering retains 19,000 objects, removing those with too little motion or excessive or sudden movement.Rendering adds random frame strides from 1 to 3; larger strides produce faster object motion.Source: B ADDITIONAL DATA CURATION DETAILS · pp. 16–17
  • Table 3 compares module replacements and training-data variants, including the full model, static-only training, and training without monocular videos.Figure 6 identifies ablations on Plücker coordinates and cross-frame attention.Source: Table 3 · p. 17, Figure 6 · p. 18
  • The monocular-video curation pipeline combines camera-pose, text, aesthetic, and motion filtering, yielding 393,000 clips.The total comprises 355,000 clips from InternVid and 38,000 from the Panda-70M subset of OpenVid.Source: B ADDITIONAL DATA CURATION DETAILS · p. 17

C EVALUATION DETAILS

Evaluation details specify the comparison checkpoints, video-quality metrics, and COLMAP configuration; ablation videos are supplied in the supplement.

C EVALUATION DETAILS

  • MotionCtrl and CameraCtrl are compared using the authors’ open-source SVD checkpoints, with FID and FVD measured using clean-fid and common-metrics-on-video-quality.FVD is reported in VideoGPT format.Source: C EVALUATION DETAILS · p. 18
  • COLMAP uses a 65,536-match SIFT limit and a shared-camera setting to support feature matching and align SfM with mostly single-camera videos.The configuration follows DSNeRF and CamCo.Source: C EVALUATION DETAILS · pp. 18–19
  • Qualitative comparisons for cross-view attention and joint training on monocular videos are provided in the supplementary material.Source: Figure 7 · p. 19, Figure 8 · p. 19

D ABLATION STUDIES

The ablations identify contributions from camera conditioning, cross-frame and cross-view attention, and monocular-video joint training; application comparisons show Cavia’s video and reconstruction quality against CVD.

E APPLICATIONS

  • In four-view comparisons, Cavia produces better geometric consistency and perceptual quality than CVD, which tends to generate black border pixels.The figure caption notes that CVD’s borders may relate to homography-warping augmentations.Source: Figure 9 · p. 20

D ABLATION STUDIES

  • The full model achieves the best perceptual quality and geometric consistency in ablations evaluated on RealEstate10K trajectories, with Real10K and General test splits.The Real10K and General test images come from RealEstate10K and InternVid, respectively.Source: D ABLATION STUDIES · pp. 19–20
  • Without cross-frame attention, generated scenes show distortions such as bent walls, while removing Plucker coordinates simplifies camera motion and misses complex viewpoint instructions.Source: D ABLATION STUDIES · p. 20
  • Removing cross-view attention produces videos with different object motions, including inconsistent penguin movement and firewood appearance across views.Source: D ABLATION STUDIES · p. 20
  • Joint training with monocular videos preserves complex backgrounds during object motion, unlike Objaverse overfitting, which tends toward simplified backgrounds.Objaverse training samples use single random-color backgrounds.Source: D ABLATION STUDIES · p. 20

E APPLICATIONS

  • Cavia’s reconstructed 3D Gaussians appear sharper and clearer than CVD’s, whose reconstruction shows floaters and blurry artifacts.Source: Figure 10 · p. 21

E.1 ADVANCING TO FOUR VIEWS AT INFERENCE

Cavia extends its multi-view generation to four views at inference and produces generated frames that support clearer 3D reconstruction than CVD.

E.1 ADVANCING TO FOUR VIEWS AT INFERENCE

  • Cavia produces more consistent and realistic four-view videos than CVD, whose border artifacts include changing wall and window structures across viewpoints.Cavia’s cross-view attention also allows more views to be extrapolated at inference without enumerating viewpoint pairs.Source: §E.1 · p. 21

E.2 3D RECONSTRUCTION OF GENERATED FRAMES

  • Across 16 novel views rendered along an elliptical trajectory, Cavia yields clearer 3D reconstruction with fewer floaters than CVD.CVD’s cupboard regions show floaters, while its wall and TV reconstructions are blurry.Source: §E.2 · p. 22

F LIMITATIONS

Cavia’s motion generation, camera-intrinsic generalization, and scene-scale treatment have stated limitations.

F LIMITATIONS

  • Cavia has limited ability to generate large object motion, mainly because of the SVD base video generator.Source: F LIMITATIONS · p. 22
  • The data pipeline assumes shared camera intrinsics across frames, limiting generalization to complex intrinsic changes at inference.Source: F LIMITATIONS · p. 22
  • Training uses normalized scene scales rather than metric-calibrated scales; the authors identify calibration as future work.They note that calibration depends on a well-generalizable metric depth estimator becoming publicly available.Source: F LIMITATIONS · p. 22
Loading 2410.10774v1…