Source-linked AI summary
Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, Hao Su
TL;DR
Single-image novel-view systems can produce geometrically inconsistent views, limiting their usefulness for coherent 3D content. Zero123++ jointly generates six views while reusing Stable Diffusion through tailored pose, noise, and conditioning schemes, producing consistent high-quality results and supporting depth control. Its future work identifies a remaining gap between high-quality multi-view images and high-quality 3D meshes.
Problem
Existing single-image novel-view methods can generate geometrically inconsistent views, leaving a gap between multi-view images and coherent 3D scenes.
Method
Zero123++ tiles six views into one prediction target and combines fixed camera poses, a linear noise schedule, Reference Attention, and trainable global image conditioning to reuse Stable Diffusion priors.
Results
Zero123++ generates realistic, consistent, highly detailed multi-view images, achieves the best LPIPS score on the validation split, and generalizes to real photos, AI-generated images, and 2D illustrations.
Takeaways & Limitations
The model provides a multi-view diffusion base for image-to-3D pipelines and can be extended with a depth-controlled ControlNet.
Takeaways & Limitations
A remaining gap separates high-quality multi-view images from high-quality 3D meshes, while a two-stage refiner remains future work.
Abstract
from arXiv · showhide
We report Zero123++, an image-conditioned diffusion model for generating 3D-consistent multi-view images from a single input view. To take full advantage of pretrained 2D generative priors, we develop various conditioning and training schemes to minimize the effort of finetuning from off-the-shelf image diffusion models such as Stable Diffusion. Zero123++ excels in producing high-quality, consistent multi-view images from a single image, overcoming common issues like texture degradation and geometric misalignment. Furthermore, we showcase the feasibility of training a ControlNet on Zero123++ for enhanced control over the generation process. The code is available at https://github.com/SUDO-AI-3D/zero123plus.
1. Introduction
Zero123++ addresses inconsistent novel-view generation by modeling multiple views jointly and improving how pretrained Stable Diffusion capabilities are reused. It targets geometric consistency, conditioning, and training-resolution limitations in prior approaches.
- Zero-1-to-3 generates each novel view independently, allowing diffusion sampling to break consistency between views.
- Zero123++ tiles six views surrounding an object into one image, enabling joint modeling of their multi-view distribution.
- The method also addresses Zero-1-to-3’s underuse of Stable Diffusion conditioning mechanisms and reduced training resolution.
2. Improving Consistency and Conditioning
Zero123++ improves multi-view consistency through joint view prediction, fixed camera poses, a linear noise schedule, and local and global image conditioning that reuse Stable Diffusion priors.
- 2.1. Multi-view Generation: Jointly predicting six tiled views models correlations that independent conditional marginals overlook.
- 2.1. Multi-view Generation: Fixed absolute elevations and relative azimuths remove orientation ambiguity without requiring an additional elevation-estimation module.The six poses interleave 30° downward and 20° upward elevations, with azimuths starting at 30° and increasing by 60°.
- 2.2. Consistency and Stability: Noise Schedule: The linear noise schedule supplies more low-SNR denoising steps for global structure, addressing the scaled-linear schedule’s weaker global consistency.
- 2.2. Consistency and Stability: Noise Schedule: A toy finetuning task shows the linear schedule can produce a blank white image, whereas the scaled-linear schedule only slightly whitens it.
- 2.3. Local Condition: Scaled Reference Attention: Reference Attention adds a noisy reference-image branch whose self-attention keys and values provide local conditioning while reusing Stable Diffusion priors.
- 2.3. Local Condition: Scaled Reference Attention: Scaling the reference latent by 5 produces the highest consistency with the conditioning image on ShapeNet Cars.
- 2.4. Global Condition: FlexDiffuse: Global conditioning adds a trainable CLIP image embedding to prompt embeddings through shared token weights.The released models obtain the prompt embedding from an empty prompt.
- 2.4. Global Condition: FlexDiffuse: Without global conditioning, visible regions remain satisfactory but unseen regions deteriorate significantly because global object semantics cannot be inferred.
3. Comparison to the State of the Art
Zero123++ produces consistent, high-quality multi-view images across diverse inputs and achieves the best LPIPS score on the validation split. The text-to-image-to-multi-view pipeline also yields realistic, consistent, and detailed views.
- Zero123++ generates consistent, high-quality multi-view images and generalizes to AI-generated and 2D illustration inputs.The comparison includes Objaverse, a real photograph, an SDXL-generated image, and an anime illustration.
- Zero123++ achieves the best LPIPS score on the validation split among the evaluated models.LPIPS is computed between tiled sets of six generated and ground-truth images.
- The text-to-image-to-multi-view pipeline produces realistic, consistent, and highly detailed multi-view images.The pipeline first generates an image with SDXL from text prompts, then applies Zero123++.
- MVDream shows a shift toward cartoonish, flat textures, while Zero-1-to-3 does not guarantee multi-view consistency in the comparison.
4. Depth ControlNet for Zero123++
The depth-controlled Zero123++ extends the base model with ControlNet for geometry control. It supports either a supplied input view or an input image generated from depth, achieving an LPIPS of 0.086 on validation.
- The depth-controlled model uses ControlNet with normalized linear depth images to control Zero123++ through geometry.The model is trained using depth images corresponding to target RGB images.
- 0.086 LPIPS is achieved by the depth-controlled Zero123++ model on the validation split.
- Depth-controlled Zero123++ can take a single view as input or generate the input image from depth without input colors.The latter option uses vanilla depth-controlled Stable Diffusion to generate the input image.
5. Future Work
The authors identify future work in refinement, scaling training data, and mesh reconstruction. These directions address local detail generation, broader training coverage, and the gap between multi-view images and high-quality 3D meshes.
- A two-stage generate-refine pipeline is proposed to improve local details while addressing consistency limitations of ε-parametrized models.The proposed refiner would use an ε-parametrized SDXL model to leverage stronger priors.
- Further scaling-up would train Zero123++ on a larger dataset such as Objaverse-XL.The current model is trained on the medium-scale Objaverse dataset, which contains around 800k objects.
- Mesh reconstruction remains a future direction because high-quality multi-view images and high-quality 3D meshes are not yet fully bridged.The report presents preliminary Zero123++ mesh-generation results in Fig. 12.