Source-linked AI summary
Zero-1-to-3: Zero-shot One Image to 3D Object
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, Carl Vondrick
TL;DR
Single-image novel-view synthesis and 3D reconstruction are severely under-constrained because the input reveals only part of an object. Zero-1-to-3 fine-tunes a large diffusion model on synthetic multi-view data to control relative camera transformations, achieving state-of-the-art results and strong generalization, while complex-background scenes remain a challenge.
Problem
Single-image novel-view synthesis and 3D reconstruction are severely under-constrained because a monocular image shows only a partial view of an object.
Method
Zero-1-to-3 fine-tunes Stable Diffusion on synthetic rendered view pairs to control relative camera transformations for novel-view synthesis and reconstruction.
Results
The method achieves state-of-the-art results for zero-shot novel-view synthesis and 3D reconstruction, with strong generalization to out-of-distribution scenes and artistic images.
Takeaways & Limitations
Large diffusion models provide useful learned shape priors for zero-shot single-image view synthesis and 3D reconstruction.
Takeaways & Limitations
Generalization to scenes with complex backgrounds remains an important challenge, with quality degrading on scenes compared with in-distribution object samples.
Abstract
from arXiv · showhide
We introduce Zero-1-to-3, a framework for changing the camera viewpoint of an object given just a single RGB image. To perform novel view synthesis in this under-constrained setting, we capitalize on the geometric priors that large-scale diffusion models learn about natural images. Our conditional diffusion model uses a synthetic dataset to learn controls of the relative camera viewpoint, which allow new images to be generated of the same object under a specified camera transformation. Even though it is trained on a synthetic dataset, our model retains a strong zero-shot generalization ability to out-of-distribution datasets as well as in-the-wild images, including impressionist paintings. Our viewpoint-conditioned diffusion approach can further be used for the task of 3D reconstruction from a single image. Qualitative and quantitative experiments show that our method significantly outperforms state-of-the-art single-view 3D reconstruction and novel view synthesis models by leveraging Internet-scale pre-training.
1 Columbia University 2 Toyota Research Institute
Zero-1-to-3 synthesizes an object image from a specified camera viewpoint, preserving rich details across large relative transformations and generalizing to complex geometry and artistic styles.
- Zero-1-to-3 synthesizes an object image from a specified camera viewpoint given a single RGB image.
- The generated views contain rich details consistent with the input view despite large relative transformations.
- The method achieves strong zero-shot performance on objects with complex geometry and artistic styles.
1. Introduction
The paper motivates single-image 3D understanding as a difficult but important capability and proposes controlling large diffusion models to perform zero-shot novel-view synthesis and reconstruction. It argues that these models learn rich 3D priors from 2D Internet-scale image data and reports state-of-the-art results for both tasks.
- Humans infer 3D shape and appearance from one camera view, supporting object manipulation, navigation, and visual creativity.
- Existing reconstruction methods often rely on expensive 3D annotations or category-specific priors, while newer approaches still require geometry-related training information.
- Zero-1-to-3 fine-tunes large diffusion models to control relative camera rotation and translation for single-image novel-view synthesis and 3D reconstruction.
- The paper's primary contribution is demonstrating rich 3D priors in diffusion models trained only on 2D images.
- The method achieves state-of-the-art results for zero-shot novel-view synthesis and 3D reconstruction from a single RGB image.
2. Related Work
Related work spans large-scale 2D diffusion models, 3D representations, novel-view synthesis, and single-view reconstruction. Zero-1-to-3 instead learns viewpoint control in a diffusion-based image-translation model and extracts geometric information without additional depth data.
- 3D generative models: Large-scale image-text datasets and diffusion architectures enable high-fidelity synthesis, but scaling generative models directly to 3D requires expensive annotated 3D data.
- Viewpoint bias: Text-to-image models exhibit viewpoint bias, with most generated chairs appearing in forward-facing canonical poses.
- 3D generative models: Diffusion models are repurposed for 3D generation through NeRF representations or distillation losses from 2D diffusion models.
- Novel-view synthesis: Zero-1-to-3 models novel-view synthesis as viewpoint-conditioned image-to-image translation and combines the learned model with 3D distillation for single-image reconstruction.
- Single-view object reconstruction: Single-view reconstruction methods based on meshes, voxels, or point clouds are constrained by their 3D data variety and often generalize poorly.
- Single-view object reconstruction: The method extracts rich geometric information directly from pre-trained Stable Diffusion, avoiding the need for additional depth information.
3. Method
Zero-1-to-3 treats novel-view synthesis as viewpoint-conditioned image translation, using a fine-tuned diffusion model to control camera extrinsics from one RGB image. The same viewpoint-conditioned model can supervise neural-field optimization for single-image 3D reconstruction.
- Viewpoint-conditioned synthesis: The model synthesizes a novel object image from a single RGB input conditioned on relative camera rotation R and translation T.The target is a perceptually similar image under the desired camera transformation.
- Viewpoint-conditioned synthesis: Pretrained diffusion models provide broad natural-image coverage, but require explicit viewpoint controls because they do not encode viewpoint correspondences and inherit canonical-pose biases.These limitations hinder extracting 3D knowledge from the pretrained models.
- Learning camera control: Fine-tuning on paired images with relative camera extrinsics teaches a latent diffusion model to control viewpoints while retaining its existing image representation.The conditioning combines the input image and camera extrinsics, and inference uses iterative denoising from Gaussian noise.
- Novel-view synthesis: On GSO and RTMV, synthesized views preserve photorealistic appearance and ground-truth consistency, including high fidelity under large viewpoint changes.The GSO comparison highlights textual and geometric details, while RTMV is out-of-distribution relative to Objaverse.
- Learning camera control: The learned viewpoint mechanism extrapolates beyond objects in the fine-tuning set, enabling zero-shot novel views for object classes without 3D assets or training examples.The model retains photorealistic image generation while adding camera-viewpoint control.
- 3D reconstruction: For single-image 3D reconstruction, the method combines viewpoint-conditioned diffusion supervision with randomly sampled rendered viewpoints, MSE, depth-smoothness, and near-view consistency losses.The reconstruction procedure perturbs rendered views with Gaussian noise and denoises them using conditioning from the input image, camera extrinsics, and timestep.
4. Experiments
The experiments evaluate Zero-1-to-3 on zero-shot novel view synthesis and single-view 3D reconstruction using synthetic and in-the-wild data. Across benchmarks and qualitative examples, the method produces photorealistic novel views and high-fidelity 3D geometry, while modeling uncertainty through diverse samples.
- Tasks and evaluation: The evaluation covers zero-shot novel view synthesis and 3D reconstruction on images from outside the Objaverse dataset, including synthetic scenes and in-the-wild images.The benchmarks include Google Scanned Objects and RTMV, while qualitative tests include daily-object photographs, Internet images, paintings, and generated images.
- Novel view synthesis results: On GSO and out-of-distribution RTMV scenes, the method significantly outperforms the baselines and generates photorealistic images consistent with ground truth.The reported comparisons include DietNeRF, Image Variations, and SJC-I; RTMV contains complex scenes composed of 20 random objects.
- Tasks and evaluation: Novel view synthesis is evaluated with PSNR, SSIM, LPIPS, and FID, while 3D reconstruction is measured using Chamfer Distance and volumetric IoU.The metrics cover multiple aspects of image similarity and 3D reconstruction quality.
- Novel view synthesis results: In-the-wild results show robust synthesis across challenging surface materials and geometry while preserving object type, identity, and low-level details across randomly sampled viewpoints.The figure includes iPhone photographs and an Internet image, with five randomly sampled viewpoints shown without cherry-picking.
- Diversity across samples: Diffusion sampling produces diverse plausible novel views because single-image inputs leave geometry and appearance information unobserved.Multiple conditional samples at a fixed target viewpoint vary in both geometry and appearance.
- 3D reconstruction results: For 3D reconstruction, the method produces high-fidelity meshes consistent with ground truth and improves both Chamfer Distance and volumetric IoU over prior works.MCC often misses geometry on the back of objects, while Point-E's sparse 4,096-point clouds can create holes in reconstructed surfaces.
5. Discussion
Zero-1-to-3 uses Stable Diffusion’s learned geometric priors for zero-shot single-image novel-view synthesis and 3D reconstruction. Its scene-level generalization remains limited, while the framework preserves multi-object composition and lighting details in generated views.
- Discussion: Zero-1-to-3 fine-tunes Stable Diffusion on synthetic data to control camera viewpoints for novel-view synthesis and 3D reconstruction.The method leverages shape priors learned by Stable Diffusion.
- Discussion: The approach preserves multiple-object composition and lighting details when synthesizing novel views from Dall-E-2-generated images.
- From objects to scenes: Scene quality degrades on RTMV compared with in-distribution GSO samples, and complex-background scenes remain an important challenge.
- From scenes to videos: Extending the framework to dynamic scenes could support future work on occlusion reasoning and dynamic object manipulation.
- Combining graphics pipelines with Stable Diffusion: Stable Diffusion may also encode lighting, shading, and texture knowledge that could support graphics tasks such as scene relighting.
A. Coordinate System & Camera Model
The camera model represents object-centered camera locations and relative transformations in spherical coordinates, using polar angle, azimuth angle, and radius.
- Coordinate System & Camera Model: Camera locations and relative transformations are represented with a spherical coordinate system centered on the object.
- Coordinate System & Camera Model: The parameters θ, φ, and r denote the polar angle, azimuth angle, and distance from the object center, respectively.
- Coordinate System & Camera Model: Assets are normalized to fit inside the XYZ unit cube [−0.5, 0.5]3 before dataset creation.
B. Dataset Creation
The finetuning dataset is rendered from Objaverse objects in Blender using randomized viewpoints, lighting, and denoised Cycles rendering at 512×512 resolution.
- Dataset Creation: Each Objaverse object is rendered from 12 randomly sampled views for the finetuning dataset.
- Dataset Creation: Blender’s Cycles engine renders images with 128 samples per ray, followed by denoising.
- Dataset Creation: Training images are rendered at 512×512 resolution, with transparent backgrounds padded using white.
- Dataset Creation: Randomized area lighting is applied during rendering to vary illumination conditions.
C. Finetuning Stable Diffusion
The rendered synthetic dataset is used to fine-tune Stable Diffusion for novel-view synthesis conditioned on an input image and relative camera information.
- Finetuning Stable Diffusion: The rendered dataset fine-tunes a pretrained Stable Diffusion model for novel-view synthesis.
- Finetuning Stable Diffusion: Because the original Stable Diffusion network is not conditioned on multimodal text embeddings, its architecture is modified to accept image conditioning.
- Finetuning Stable Diffusion: The adapted model is further conditioned on relative camera information to control the synthesized viewpoint.
C.1. Training Details
Training and evaluation use a zero-shot, single-view RGB setting with selected baselines, regularized voxel radiance fields, and dataset-specific mesh extraction procedures. Novel-view synthesis is fast, while full 3D reconstruction remains substantially more computationally expensive.
- Training details: Training uses AdamW with a 10^-4 learning rate, and reduces resolution from 512 × 512 to 256 × 256 after high batch variance and slower convergence.The corresponding latent resolution changes from 64 × 64 to 32 × 32.
- 3D reconstruction: VoxelRF regularization combines depth-map smoothness, which helps remove holes, with near-view consistency across nearby randomly sampled views.The implementation also removes SJC’s emptiness and center losses.
- Mesh extraction and evaluation: Mesh extraction smooths and erodes density grids before marching cubes, using thresholds of 8 d̄ for GSO and 4 d̄ for RTMV.Evaluation normalizes shapes within the unit cube, samples 2000 points for Chamfer distance, and computes volumetric IoU at resolution 64^3.
- Evaluation scope: Zero-1-to-3 is compared only with zero-shot methods that accept single-view RGB images and have adaptable official implementations.The comparison includes methods meeting these stated scope and availability criteria.