Source-linked AI summary
CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models
Rundi Wu, Ruiqi Gao, Ben Poole, Alex Trevithick, Changxi Zheng, Jonathan T. Barron, Aleksander Holynski
TL;DR
Dynamic 3D reconstruction from monocular video is limited by partial observations and difficult capture requirements. CAT4D uses a multi-view video diffusion model plus deformable 3D Gaussian optimization to create controllable multi-view videos and reconstruct dynamic scenes, achieving competitive results across reconstruction and generation tasks. Its main boundaries are temporal extrapolation, camera–time disentanglement under occlusion, and physical accuracy of recovered motion.
Problem
Dynamic 3D reconstruction from partial monocular observations is difficult, while synchronized multi-view video required by many methods is nearly impossible for typical users.
Method
CAT4D transforms monocular video into multi-view video with a diffusion model, then optimizes a deformable 3D Gaussian representation for dynamic scene reconstruction.
Results
CAT4D provides disentangled novel-view and novel-time synthesis, improves over a 4D-GS baseline, performs roughly on par with Shape-of-Motion and MoSca, and supports scene-scale 4D generation.
Takeaways & Limitations
The approach enables 4D reconstruction from monocular videos without the multiple priors and external information used by several competing methods, and supports generation from captured or generated videos.
Takeaways & Limitations
The diffusion model struggles with temporal extrapolation and camera–time disentanglement under occlusion, while recovered 3D motion fields may not be physically accurate.
Abstract
from arXiv · showhide
We present CAT4D, a method for creating 4D (dynamic 3D) scenes from monocular video. CAT4D leverages a multi-view video diffusion model trained on a diverse combination of datasets to enable novel view synthesis at any specified camera poses and timestamps. Combined with a novel sampling approach, this model can transform a single monocular video into a multi-view video, enabling robust 4D reconstruction via optimization of a deformable 3D Gaussian representation. We demonstrate competitive performance on novel view synthesis and dynamic scene reconstruction benchmarks, and highlight the creative capabilities for 4D scene generation from real or generated videos. See our project page for results and interactive demos: https://cat-4d.github.io/.
1. Introduction
CAT4D addresses the challenge of reconstructing dynamically changing 3D worlds from partial observations by learning a multi-view video prior and combining it with deformable 3D Gaussian reconstruction. The system supports controllable viewpoints and timestamps, achieves competitive reconstruction results without several external priors, and enables scene-scale 4D generation.
- Reconstructing dynamic 3D worlds from ordinary images and videos remains difficult because captures provide only partial observations.
- 4D reconstruction typically requires synchronized multi-view video, which is nearly impossible for typical users.
- Limited synchronized multi-view data constrains the amount and diversity of training data available for learning data-driven 4D priors.
- CAT4D trains a multi-view video diffusion model on diverse real and synthetic data to transform monocular videos into consistent multi-view videos for dynamic 3D reconstruction.The model synthesizes appearances at specified novel viewpoints and times.
- The model supports novel-view and novel-time synthesis, sparse-view reconstruction with scene motion, and monocular-video reconstruction without multiple external supervisory signals.
- CAT4D also produces scene-scale 4D generations from fixed-viewpoint videos, including environments containing multiple dynamic objects.
- Multi-view video models differ from video, multi-view, and camera-controlled video models by generating all views at all timestamps.
2. Related Work
Prior dynamic reconstruction and 4D-generation methods rely on synchronized captures, external supervision, costly optimization, or restricted camera trajectories. CAT4D instead uses diffusion to produce controllable, mutually consistent multi-view videos for reconstruction.
- Dynamic reconstruction methods commonly use NeRF or 3DGS with deformation fields, while recent systems add segmentation, depth, or long-term 2D-track supervision.
- These methods may require user interaction and cannot reconstruct regions never observed in the input frames.
- Earlier 4D-generation approaches used score distillation with image or video diffusion models, while later work combined video and multi-view image diffusion.
- Recent sampling and feed-forward methods generate multi-view videos for reconstruction but can be limited to predefined camera trajectories or show limited results.
- Camera-controlled video models do not generate mutually consistent multi-view videos, whereas CAT4D trains one model to produce controllable novel viewpoints from monocular video.
3. Method
CAT4D transforms monocular video into multi-view video with a diffusion model, then reconstructs dynamic 3D scenes using deformable 3D Gaussians. Its training data and alternating sampling strategy support controllable camera-time synthesis and nearly consistent multi-view videos.
- CAT4D first converts monocular video into multi-view video, then optimizes a deformable 3D Gaussian representation for dynamic scene reconstruction.
- 3.1. Multi-view Video Diffusion Model: The diffusion model conditions on input images, camera parameters, and times to synthesize target images at specified viewpoints and timestamps.
- 3.1. Multi-view Video Diffusion Model: The model extends CAT3D’s multi-view latent diffusion architecture with time conditioning injected into the U-Net’s residual blocks.
- 3.2. Training Data: Training mixes synthetic 4D data, multi-view images, static-viewpoint monocular videos, and generative augmentations to provide camera and scene-motion supervision.
- 3.3. Sparse-View Bullet-Time 3D Reconstruction: For sparse-view bullet-time reconstruction, CAT4D generates anchored novel views at one target time before fitting a standard 3DGS model with LPIPS loss.
- 3.4. Generating Consistent Multi-view Videos: For 4D reconstruction, the method alternates column-wise multi-view sampling and row-wise temporal sampling to construct a K×L image grid.
- 3.4. Generating Consistent Multi-view Videos: Multi-view sampling uses overlapping sliding windows and pixel-wise medians, while temporal sampling applies an analogous row-wise procedure.
4. Experiments
CAT4D is evaluated on disentangled camera-time control, sparse-view bullet-time reconstruction, 4D reconstruction, sampling strategies, and 4D creation. The experiments compare against prior methods and assess the contribution of alternating sampling and generated multi-view evidence.
- Ablation Study: The sampling ablation compares alternating sampling with multi-view-only, temporal-only, and independent sliding-window variants against NSFF ground truth.
- Separate Control over Camera and Time: The disentangled-control experiment tests fixed or varying viewpoints and times using three input images from the NSFF dataset, comparing CAT4D with 4DiM.
- Sparse-View Bullet-Time 3D Reconstruction: The sparse-view bullet-time experiment evaluates reconstruction of a consistent static scene from dynamic input images, comparing CAT4D with CAT3D variants.
- 4D Reconstruction: The 4D reconstruction evaluation uses monocular videos from DyCheck and compares CAT4D with 4D-GS, Shape-of-Motion, and other dynamic reconstruction methods.
- 4D Creation: CAT4D also creates 4D scenes from monocular videos generated by text-to-video or image-to-video models.
5. Discussion
CAT4D creates 4D scenes from captured or generated monocular videos by converting them into consistent multi-view videos for deformable 3D Gaussian reconstruction. Its limitations include temporal extrapolation, camera–time disentanglement, and physical accuracy of recovered motion.
- Discussion: CAT4D transforms monocular inputs into consistent multi-view videos and reconstructs the scene as deformable 3D Gaussians.The approach supports both captured and generated monocular videos.
- Limitations: The diffusion model struggles with temporal extrapolation beyond input frames and with disentangling camera viewpoint from temporal progression when dynamic objects become occluded.The authors identify these as limitations of the current model.
- Limitations: Recovered 3D motion fields may be plausible from novel viewpoints without being physically accurate.Depth or motion supervision could improve results but would reduce applicability to dense video captures.
A. Method Details
The method fine-tunes a timestamp-conditioned latent diffusion model and uses alternating DDIM sampling to generate novel views across time, followed by deformable Gaussian reconstruction. Camera trajectories are selected from the input video, while timestamp assumptions and reconstruction losses support different applications.
- Diffusion Model: CAT4D initializes from CAT3D and fine-tunes a latent diffusion model with timestamp embeddings for multi-view video generation.The model is trained first with 3 input and 13 target views, then further adapted with 9 input and 8 target views for 4D reconstruction.
- Sampling: Sampling uses DDIM with 25 steps and classifier-free guidance weights s1 = 3.0 and s2 = 4.5.The alternating strategy generates K′ = 128 views per timestamp in about 1 minute on 16 A100 GPUs.
- Camera Trajectory Selection: Novel-view camera trajectories are designed from the input video trajectory to cover viewpoints needed for rendering after reconstruction.Videos with sufficient view coverage use anchor views selected from the input camera trajectory.
- Sparse-View Bullet-Time Reconstruction: For sparse-view bullet-time reconstruction, unknown input timestamps can be approximated by assigning 0 to the bullet-time frame and 1 to the other frames.This simplified assignment was found to work well in practice.
- 4D Reconstruction: The reconstruction pipeline extends 4D-GS with L1, DSSIM, and LPIPS photometric losses plus Gaussian densification and SfM- or MonST3R-based initialization.The loss weights are 0.8, 0.2, and 0.4 respectively.
B. Datasets Details
The training data combines real and synthetic multi-view imagery and video, with augmentations that vary lighting, viewpoints, timestamps, and pseudo-ground-truth observations. These datasets are mixed using specified sampling weights.
- Synthetic Data: Objaverse uses animated assets rendered under four lighting conditions as synchronized eight-frame videos from eight evenly spaced viewpoints.The dataset contains around 42k filtered animated assets.
- Synthetic Data: Kubric contributes 4k randomly generated scenes rendered as synchronized eight-frame videos from eight viewpoints on smooth camera paths.The viewpoints are evenly spaced along the camera path.
- Video Data: Static-view video data is filtered using temporal consistency of four corner patches, then frames are shuffled and camera parameters are standardized.The standardized cameras use a central principal point and focal lengths sampled from [0.8·512, 1.2·512].
- Augmentation: CO3D augmentation uses Lumiere to animate input images, samples one frame from each generated video, and treats those frames as pseudo-ground-truth inputs at other timestamps.This produces around 24k augmented sequences.
- Augmentation: Static-view video augmentation uses CAT3D to generate seven novel views per input image and samples one as pseudo ground truth at another viewpoint.The procedure creates additional viewpoint variation for training.
- Dataset Mixture: The combined datasets include Objaverse, Kubric, Re10K, MVImgNet, CO3D, MQ4K, static-view video data, and two augmented sources with separate mixing weights.The listed weights are 2.5, 2.5, 1.0, 1.0, 1.0, 1.0, 5.0, 1.0, and 1.0 respectively.
C. Baselines Details
The baseline and sampling evaluations define controlled camera trajectories and compare generated multi-view videos against ground truth across viewpoints and timestamps. CAT3D baselines are evaluated with one or three conditioning images.
- CAT3D Baselines: CAT3D-3cond uses CAT4D’s camera trajectory, while CAT3D-1cond uses a default forward-facing spiral path and manually matched scene scale.The comparison evaluates baselines with three versus one input image.
- Sampling Ablation: The sampling-strategy ablation evaluates generated multi-view videos on the NSFF dataset.The evaluation follows the setup used by prior work.
- Model Comparison: The model comparison figure shows input images, fixed-viewpoint varying-time space-time slices, and varying-viewpoint fixed-time video frames.Static backgrounds should form vertical lines while dynamic objects should form smooth curves in the slices.
- Sampling Ablation: The NSFF protocol simulates a moving monocular camera using images from 12 camera viewpoints at different timestamps, totaling 24 inputs.Generated videos are compared with ground truth at all 12 viewpoints.
D. Ablation Study of Training Data
The ablation compares training with all datasets, without augmented datasets, and with synthetic 4D data only. Augmented and real-world data improve visual quality and motion control, while synthetic-only training retains control but produces less natural motion and worse novel views.
- Table 6 compares all training datasets, no augmentation, and synthetic-only training on NSFF camera and time control.Each model is trained for 60k iterations before evaluation.
- Quantitative differences are small because most evaluated pixels belong to static background, whereas visual differences are clearer on in-the-wild data.The qualitative comparison is referenced in Fig. 11.
- Synthetic-only training provides surprisingly good camera and time control but often generates unnatural scene motion and worse novel views.The passage attributes this pattern to a likely generalization issue.
- No-augmentation training often fails in the fixed-viewpoint, varying-time setting because the camera still moves when only scene dynamics should change.The authors suggest imperfect filtering of non-static-viewpoint videos as a possible cause.