Source-linked AI summary
CAT3D: Create Anything in 3D with Multi-View Diffusion Models
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul Srinivasan, Jonathan T. Barron, Ben Poole
TL;DR
High-quality 3D creation remains difficult from limited observations because conventional reconstruction needs hundreds to thousands of images. CAT3D generates consistent novel views with multi-view diffusion and reconstructs them into renderable 3D representations, producing scenes in as little as one minute and outperforming prior work across supported tasks.
Problem
Creating detailed 3D scenes conventionally requires hundreds to thousands of captured photos, motivating reconstruction from fewer observations.
Method
CAT3D generates camera-conditioned novel views with multi-view diffusion, then feeds observed and generated views into a robust 3D reconstruction pipeline.
Results
CAT3D creates photorealistic 3D objects or scenes in as little as one minute and outperforms prior works across multiple supported input settings and tasks.
Takeaways & Limitations
Decoupling the generative prior from 3D extraction provides an efficient, simple, and high-quality approach to 3D generation from varying numbers of inputs.
Takeaways & Limitations
CAT3D performs poorly when test views come from cameras with different intrinsics, and generated views may not all be mutually 3D consistent.
Abstract
from arXiv · showhide
Advances in 3D reconstruction have enabled high-quality 3D capture, but require a user to collect hundreds to thousands of images to create a 3D scene. We present CAT3D, a method for creating anything in 3D by simulating this real-world capture process with a multi-view diffusion model. Given any number of input images and a set of target novel viewpoints, our model generates highly consistent novel views of a scene. These generated views can be used as input to robust 3D reconstruction techniques to produce 3D representations that can be rendered from any viewpoint in real-time. CAT3D can create entire 3D scenes in as little as one minute, and outperforms existing methods for single image and few-view 3D scene creation. See our project page for results and interactive demos at https://cat3d.github.io .
1 Introduction
CAT3D addresses the difficulty of creating high-quality 3D content from limited observations by generating additional consistent views and reconstructing the scene from them.
- 3D content is increasingly needed for interactive applications, but high-quality creation remains scarce because it requires specialized tools, time, and effort.
- Photogrammetry makes 3D creation more accessible, yet detailed scenes still require hundreds to thousands of captured photos.
- Reducing dense captures to a single image, text, or sparse views would make 3D content creation more accessible, but prior solutions are specialized and limited.
- CAT3D reframes observation-limited reconstruction as generating a collection of consistent novel observations from any number of input images.
- CAT3D uses multi-view diffusion and robust reconstruction to create renderable 3D representations from captured or synthesized inputs in as little as one minute.
2 Related Work
Related work uses 2D, camera-conditioned, multi-view, video, or feed-forward priors to reduce the observation burden, with different trade-offs in control, consistency, quality, and efficiency.
- 2D priors: 2D text-to-image priors support text-to-3D generation, but typically require iterative distillation to produce coherent NeRFs.
- 2D priors with camera conditioning: Camera-conditioned priors improve pose control over text-to-image models, but independently model output views and can require time-consuming 3D distillation.
- Multi-view priors: Multi-view priors jointly model multiple views, providing stronger consistency priors for 3D content from partial observations.
- Video priors: Off-the-shelf video diffusion models generate realistic videos but lack exact camera control and struggle with camera-only motion without scene dynamics.
- Feed-forward methods: Feed-forward methods produce 3D representations within a few seconds, but their quality is often worse than approaches built on image-space priors.
3 Method
CAT3D generates many camera-conditioned novel views with a multi-view diffusion model, then applies robust NeRF reconstruction to create 3D representations from limited inputs.
- Two-stage pipeline: CAT3D first generates many novel views consistent with one or more inputs, then runs a robust 3D reconstruction pipeline on them.
- Multi-view diffusion model: The diffusion model takes one or more image–camera-pose pairs and generates multiple target images conditioned on their target camera parameters.
- Model architecture: The architecture resembles video latent diffusion, replacing time embeddings with camera-pose embeddings for each image.
- Model architecture: A single model handles eight total conditioning and target views, training with either 1 or 3 conditional views.
- Camera conditioning: Raymaps encode camera-ray origins and directions at each latent location, using coordinates relative to the first conditional camera pose.
- Novel-view sampling: Target viewpoints are grouped by nearby camera positions and sampled independently to expand a model trained on only eight views into a larger synthetic view set.
- Camera trajectories: Scene reconstruction requires camera trajectories that are sufficiently thorough and dense to fully constrain scene coverage.
- Robust 3D reconstruction: Because generated views are not perfectly 3D consistent, CAT3D modifies NeRF training with robust losses including LPIPS.
4 Experiments
CAT3D achieves strong few-view and single-image 3D creation results, while ablations identify design choices that improve view consistency, reconstruction, and generalization. Its few-view results are state of the art across nearly all settings, with generation reduced from about one hour to a few minutes.
- 4.1 Few-View 3D Reconstruction: CAT3D achieves state-of-the-art performance across nearly all few-view reconstruction settings and metrics.It is evaluated on five real-world datasets covering 3-, 6-, and 9-view reconstruction.
- 4.1 Few-View 3D Reconstruction: Generation time falls from 1 hour for ZeroNVS and ReconFusion to a few minutes with CAT3D.
- 4.2 Single image to 3D: CAT3D produces higher-resolution single-image scene results than ZeroNVS and RealmDreamer while preserving input details for scenes and objects.For objects, its CLIP results are competitive with object-centric baselines, while it also works on whole scenes.
- 4.3 Ablations: Video diffusion architectures with 3D self-attention and raymap camera-pose embeddings generate views consistent enough for 3D reconstruction.The model was evaluated for sample quality and few-view reconstruction across in-domain and out-of-domain datasets.
- 4.3 Ablations: Jointly modeling 5 or 7 output views improves sample metrics and produces more consistent views for improved 3D reconstruction.
- 4.3 Ablations: LPIPS loss improves texture and geometry, while increasing generated views from 80 to 720 improves central-object geometry but can introduce background blur.The blur is attributed to inconsistencies in generated content.
5 Discussion
CAT3D presents a unified pipeline that generates consistent novel views and reconstructs them into efficient, high-quality 3D content. Its main limitations concern camera intrinsics, base-model coverage, view consistency, and manually designed trajectories.
- 5 Discussion: CAT3D unifies multi-view generation and 3D reconstruction for 3D content creation from any number of input images.The generative prior is decoupled from 3D extraction.
- 5 Discussion: CAT3D cannot handle test cases well when input views come from multiple cameras with different intrinsics.Training data used roughly constant camera intrinsics for views of the same scene.
- 5 Discussion: Generation quality declines when scene content is out of distribution for the base text-to-image model.
- 5 Discussion: Large generated view sets may contain mutually inconsistent views because the diffusion model supports relatively few output views.
- 5 Discussion: Manually constructed camera trajectories may be difficult to design for large-scale open-ended 3D environments.Automatically determining trajectories is identified as a future direction.
A Ablations
The ablations identify multi-view architecture, raymap camera conditioning, 3D attention, pretrained initialization, and reconstruction choices that improve performance while exposing compute–quality trade-offs.
- Number of target views: Five target views substantially improve novel-view synthesis and few-view reconstruction over generating a single target view.The multi-view architecture is favored with multiple input views compared with PixelNeRF in the ReconFusion baseline.
- Camera conditioning: Raymap camera conditioning performs better across all metrics than an 8-dimensional camera-parameter encoding.Raymaps are compared against an encoding containing relative position, relative rotation quaternion, and absolute focal length.
- Attention layers: 3D self-attention is critical for connecting multiple views, but applying it at 64 × 64 incurs substantial overhead for marginal gains.CAT3D therefore applies 3D attention only at feature-map sizes of 32 × 32 and smaller.
- Multi-view diffusion model training: Pretrained text-to-image initialization outperforms training from scratch, especially for out-of-domain cases.Performance improves consistently through 1M iterations, while joint fine-tuning improves in-domain synthesis and preserves comparable out-of-domain results.
- 3D reconstruction: LPIPS loss is crucial for high-quality texture and geometry, while more views regularize central-object geometry but can blur background textures.The comparison uses 80 views on one orbital path versus 720 views on nine variably scaled orbital paths.
B Details of Multi-View Diffusion Model
CAT3D adapts a pretrained latent text-to-image diffusion model into a multi-view model, using attention and camera-aware inputs to synthesize view-consistent content.
- Model architecture: CAT3D initializes a latent diffusion model from text-to-image training and adapts it for multi-view generation.The architecture is modified by inflating 2D self-attention into 3D self-attention for connecting views.
- Reconstruction design choices: Figure 6 compares rendered images and depth maps for 720 views on multiple orbital paths, 80 views on one path, and 720 views without LPIPS loss.The comparisons are conducted on a Mip-NeRF 360 scene.
- Training: The model is fine-tuned for 1.4M iterations across 1 conditional + 7 target views and 3 conditional + 5 target views.The first 1M iterations use the former setting; the remaining 0.4M iterations use an equal mixture.
C Details of Generating Novel Views
CAT3D generates dense novel-view sets with parallel diffusion sampling, using scene-dependent camera trajectories and anchor selection to cover the reconstruction domain.
- Sampling: DDIM sampling generates 80 views in 5 seconds on 16 A100 GPUs.The configuration uses 50 sampling steps and classifier-free guidance weight 3.
- Single-image trajectories: Single-image reconstruction uses spiral and orbit trajectories, each containing 80 views.The spiral moves into and out of the scene, while the orbit is used for images with a central object.
- Few-view trajectories: Few-view reconstruction uses dataset-specific trajectories producing 800, 960, 480, 640, or 720 views.The paths are fitted, shifted, scaled, or otherwise adapted for RealEstate10K, LLFF, DTU, CO3D, and Mip-NeRF 360.
- View generation settings: CAT3D uses 1 conditional + 7 target views for anchors and 3 conditional + 5 target views for full-view generation.Groups of target views are selected by their trajectory indices.
- Anchor selection: Anchor views are selected greedily by choosing cameras farthest from already selected views to spread coverage.The strategy is used for single-image conditioning across trajectory-like and dispersed random view sets.
D Details of 3D Reconstruction
The reconstruction pipeline combines synthetic views, trajectory-aware weighting, and perceptual supervision, while balancing scene coverage against generated-view consistency.
- Implementation: The few-view reconstruction implementation uses smaller Zip-NeRF settings and runs synthetic-view generation and reconstruction on 16 A100 GPUs.The smaller view-dependence network and 1000 iterations reduce overfitting and substantially speed training and rendering.
- Pipeline: Generated views are combined with a robust 3D reconstruction pipeline to form renderable scene representations.The pipeline is applied after CAT3D produces a large set of nearly consistent novel views.
- Camera trajectories: Camera trajectories must be dense and thorough enough to constrain the scene without passing through objects or using unusual angles.Trajectory design depends on scene content and is crucial for compelling few-view reconstruction.
- Perceptual loss: LPIPS loss weights are 0.25 for single-image reconstruction and several few-view datasets, and 1.0 for CO3D and Mip-NeRF 360.The stated settings cover RealEstate10K, LLFF, DTU, CO3D, and Mip-NeRF 360.
- Distance-based weighting: Distance-based weighting upweights generated views nearer captured views during later training stages.The schedule uses a Gaussian kernel and linearly anneals b from 0 to 15 for few-view reconstruction.