PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model

Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, Hao Su

arXiv:2310.15110v1cs.CVcs.GR

TL;DR

Single-image novel-view systems can produce geometrically inconsistent views, limiting their usefulness for coherent 3D content. Zero123++ jointly generates six views while reusing Stable Diffusion through tailored pose, noise, and conditioning schemes, producing consistent high-quality results and supporting depth control. Its future work identifies a remaining gap between high-quality multi-view images and high-quality 3D meshes.

  • Problem

    Existing single-image novel-view methods can generate geometrically inconsistent views, leaving a gap between multi-view images and coherent 3D scenes.

    Source: §1 · p. 2, §2.1 · p. 2

  • Method

    Zero123++ tiles six views into one prediction target and combines fixed camera poses, a linear noise schedule, Reference Attention, and trainable global image conditioning to reuse Stable Diffusion priors.

    Source: §1 · p. 2, §2.1 · p. 2, §2.4 · p. 4

  • Results

    Zero123++ generates realistic, consistent, highly detailed multi-view images, achieves the best LPIPS score on the validation split, and generalizes to real photos, AI-generated images, and 2D illustrations.

    Source: §3.2 · p. 5, §3.1 · p. 5

  • Takeaways & Limitations

    The model provides a multi-view diffusion base for image-to-3D pipelines and can be extended with a depth-controlled ControlNet.

    Source: §1 · p. 2, §4 · p. 5

    A remaining gap separates high-quality multi-view images from high-quality 3D meshes, while a two-stage refiner remains future work.

    Source: §5 · p. 7

Abstract

from arXiv · show

We report Zero123++, an image-conditioned diffusion model for generating 3D-consistent multi-view images from a single input view. To take full advantage of pretrained 2D generative priors, we develop various conditioning and training schemes to minimize the effort of finetuning from off-the-shelf image diffusion models such as Stable Diffusion. Zero123++ excels in producing high-quality, consistent multi-view images from a single image, overcoming common issues like texture degradation and geometric misalignment. Furthermore, we showcase the feasibility of training a ControlNet on Zero123++ for enhanced control over the generation process. The code is available at https://github.com/SUDO-AI-3D/zero123plus.

1. Introduction

Zero123++ addresses inconsistent novel-view generation by modeling multiple views jointly and improving how pretrained Stable Diffusion capabilities are reused. It targets geometric consistency, conditioning, and training-resolution limitations in prior approaches.

  • Zero-1-to-3 generates each novel view independently, allowing diffusion sampling to break consistency between views.Source: §1 · p. 2
  • Zero123++ tiles six views surrounding an object into one image, enabling joint modeling of their multi-view distribution.Source: §1 · p. 2
  • The method also addresses Zero-1-to-3’s underuse of Stable Diffusion conditioning mechanisms and reduced training resolution.Source: §1 · p. 2

2. Improving Consistency and Conditioning

Zero123++ improves multi-view consistency through joint view prediction, fixed camera poses, a linear noise schedule, and local and global image conditioning that reuse Stable Diffusion priors.

2.1. Multi-view Generation

  • Jointly predicting six tiled views models correlations that independent conditional marginals overlook.Source: §2.1 · p. 2
  • Fixed absolute elevations and relative azimuths remove orientation ambiguity without requiring an additional elevation-estimation module.The six poses interleave 30° downward and 20° upward elevations, with azimuths starting at 30° and increasing by 60°.Source: §2.1 · p. 2, Figure 2 · p. 2

2.2. Consistency and Stability: Noise Schedule

  • The linear noise schedule supplies more low-SNR denoising steps for global structure, addressing the scaled-linear schedule’s weaker global consistency.Source: Figure 3 · p. 2, §2.2 · p. 3
  • A toy finetuning task shows the linear schedule can produce a blank white image, whereas the scaled-linear schedule only slightly whitens it.Source: §2.2 · p. 3, Figure 4 · p. 3

2.3. Local Condition: Scaled Reference Attention

  • Reference Attention adds a noisy reference-image branch whose self-attention keys and values provide local conditioning while reusing Stable Diffusion priors.Source: §2.3 · p. 3, Figure 6 · p. 4
  • Scaling the reference latent by 5 produces the highest consistency with the conditioning image on ShapeNet Cars.Source: §2.3 · pp. 3–4, Figure 7 · p. 4

2.4. Global Condition: FlexDiffuse

  • Global conditioning adds a trainable CLIP image embedding to prompt embeddings through shared token weights.The released models obtain the prompt embedding from an empty prompt.Source: §2.4 · p. 4
  • Without global conditioning, visible regions remain satisfactory but unseen regions deteriorate significantly because global object semantics cannot be inferred.Source: §2.4 · p. 4, Figure 8 · p. 4

3. Comparison to the State of the Art

Zero123++ produces consistent, high-quality multi-view images across diverse inputs and achieves the best LPIPS score on the validation split. The text-to-image-to-multi-view pipeline also yields realistic, consistent, and detailed views.

  • Zero123++ generates consistent, high-quality multi-view images and generalizes to AI-generated and 2D illustration inputs.The comparison includes Objaverse, a real photograph, an SDXL-generated image, and an anime illustration.Source: §3.1 · p. 5
  • Zero123++ achieves the best LPIPS score on the validation split among the evaluated models.LPIPS is computed between tiled sets of six generated and ground-truth images.Source: §3.1 · p. 5, Table 1 · p. 5
  • The text-to-image-to-multi-view pipeline produces realistic, consistent, and highly detailed multi-view images.The pipeline first generates an image with SDXL from text prompts, then applies Zero123++.Source: §3.2 · p. 5
  • MVDream shows a shift toward cartoonish, flat textures, while Zero-1-to-3 does not guarantee multi-view consistency in the comparison.Source: §3.2 · p. 5

4. Depth ControlNet for Zero123++

The depth-controlled Zero123++ extends the base model with ControlNet for geometry control. It supports either a supplied input view or an input image generated from depth, achieving an LPIPS of 0.086 on validation.

  • The depth-controlled model uses ControlNet with normalized linear depth images to control Zero123++ through geometry.The model is trained using depth images corresponding to target RGB images.Source: §4 · p. 5
  • 0.086 LPIPS is achieved by the depth-controlled Zero123++ model on the validation split.Source: §4 · p. 5
  • Depth-controlled Zero123++ can take a single view as input or generate the input image from depth without input colors.The latter option uses vanilla depth-controlled Stable Diffusion to generate the input image.Source: §4 · p. 5

5. Future Work

The authors identify future work in refinement, scaling training data, and mesh reconstruction. These directions address local detail generation, broader training coverage, and the gap between multi-view images and high-quality 3D meshes.

  • A two-stage generate-refine pipeline is proposed to improve local details while addressing consistency limitations of ε-parametrized models.The proposed refiner would use an ε-parametrized SDXL model to leverage stronger priors.Source: §5 · p. 7
  • Further scaling-up would train Zero123++ on a larger dataset such as Objaverse-XL.The current model is trained on the medium-scale Objaverse dataset, which contains around 800k objects.Source: §5 · p. 7
  • Mesh reconstruction remains a future direction because high-quality multi-view images and high-quality 3D meshes are not yet fully bridged.The report presents preliminary Zero123++ mesh-generation results in Fig. 12.Source: §5 · p. 7

Related papers

Loading 2310.15110v1…