PaperlayerView exact revision on arXiv

Source-linked AI summary

Consistent123: Improve Consistency for One Image to 3D Object Synthesis

Haohan Weng, Tianyu Yang, Jianan Wang, Yu Li, Tong Zhang, C. L. Philip Chen, Lei Zhang

arXiv:2310.08092v1cs.CV

TL;DR

Image-to-image diffusion models can synthesize high-quality novel views but do not guarantee consistency, which matters for downstream 3D tasks. Consistent123 synthesizes views simultaneously using attention mechanisms and sampling strategies; the paper reports improved quality in 3D lifting applications.

  • Problem

    Image-to-image diffusion models do not guarantee consistency across synthesized views, and autoregressive approaches face accumulated errors, slow inference, and limited category generalization.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    Consistent123 synthesizes multiple views simultaneously using cross-view and shared self-attention, supports arbitrary sampled view counts, and progressively reduces classifier-free guidance.

    Source: §1 · p. 2

  • Results

    Consistent123 significantly boosts 3D-lifting quality in native reconstruction and image-to-3D generation.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors present Consistent123 as a potential foundation model for downstream 3D-generation research and applications.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large image diffusion models enable novel view synthesis with high quality and excellent zero-shot capability. However, such models based on image-to-image translation have no guarantee of view consistency, limiting the performance for downstream tasks like 3D reconstruction and image-to-3D generation. To empower consistency, we propose Consistent123 to synthesize novel views simultaneously by incorporating additional cross-view attention layers and the shared self-attention mechanism. The proposed attention mechanism improves the interaction across all synthesized views, as well as the alignment between the condition view and novel views. In the sampling stage, such architecture supports simultaneously generating an arbitrary number of views while training at a fixed length. We also introduce a progressive classifier-free guidance strategy to achieve the trade-off between texture and geometry for synthesized object views. Qualitative and quantitative experiments show that Consistent123 outperforms baselines in view consistency by a large margin. Furthermore, we demonstrate a significant improvement of Consistent123 on varying downstream tasks, showing its great potential in the 3D generation field. The project page is available at consistent-123.github.io.

1 INTRODUCTION

Image diffusion offers high-quality, zero-shot novel-view synthesis, but image-to-image translation does not ensure consistency across views. Consistent123 synthesizes views together with cross-view and shared self-attention, and uses sampling strategies to support arbitrary view counts and balance geometry with texture.

1 INTRODUCTION

  • Autoregressive methods improve consistency but accumulate errors, run slowly, struggle with loop closure, and are mostly trained on limited object categories.Consistent multi-view synthesis for arbitrary object categories remains unexplored in the cited discussion.Source: §1 · p. 2
  • Cross-view attention links synthesized views, while shared self-attention shares condition-view keys and values to align novel views without additional trainable parameters.The spatial layers use Zero123 pretrained weights, with only cross-view attention weights trainable to preserve zero-shot capability.Source: §1 · p. 2
  • Consistent123 samples arbitrary numbers of views simultaneously while training at a fixed length, and more sampled views improve consistency through greater overlap and interaction.This contrasts with autoregressive methods, whose sampling quality can worsen as conditioning views increase.Source: §1 · p. 2
  • Progressive classifier-free guidance decreases guidance during denoising to trade geometry for texture, since large guidance favors geometry and low guidance refines texture.The paper reports that these sampling strategies further boost consistency and synthesized-view quality.Source: §1 · p. 2
  • Consistent123 improves downstream 3D reconstruction and image-to-3D generation, with evaluations spanning Objaverse, GSO, and RTMV.The paper evaluates view consistency and similarity with ground truth on these benchmarks.Source: §1 · p. 2

2 RELATED WORK

Prior 3D synthesis approaches trade among view consistency, image quality, generalization, and sampling flexibility. Consistent123 instead synthesizes all views simultaneously for generic 3D object synthesis.

Geometric-aware 3D Object Synthesis

  • Geometric-aware methods preserve view consistency by construction, but many target well-aligned datasets, while multi-class approaches face limited model capacity and 3D data scale.The related work describes NeRF-class methods, tri-planes, and point clouds as examples of geometric-aware synthesis.Source: §2 · pp. 2–3

Geometric-free 3D Object Synthesis

  • Geometric-free methods use image-to-image translation; Zero123 adapts pretrained diffusion for camera control, while other approaches add autoregressive conditioning or distillation.The cited examples include Watson et al. (2023), Zhou & Tulsiani (2023), Tseng et al. (2023), and Liu et al. (2023b).Source: §2 · p. 3
  • Post-processing can improve view consistency at the expense of sampling flexibility, limiting downstream applications.The related work connects this trade-off to methods that condition on previous views or optimize the final 3D representation.Source: §2 · p. 3
  • Consistent123 synthesizes all views simultaneously for generic 3D objects, unlike MVDiffusion, which targets text-to-scene synthesis and uses a different attention mechanism.The passage identifies MVDiffusion as a concurrent work with a similar simultaneous-view approach.Source: §2 · p. 3

3 METHODS

Consistent123 modifies pose-guided image-to-image diffusion to generate and align multiple novel views together. Its sampling techniques support variable-length view generation and schedule guidance to balance geometry and texture.

3.1 POSE-GUIDED IMAGE-TO-IMAGE DIFFUSION MODEL

  • Pose-guided image-to-image diffusion conditions denoising on an image and relative pose, but geometry-free models still lack guaranteed cross-view consistency.The denoising U-Net predicts noise from the noisy target and condition embedding.Source: §3.1 · p. 3

3.2 VIEW CONSISTENCY VIA LATENT ALIGNMENT

  • Consistent123 denoises target views jointly and inserts trainable cross-view attention after each U-Net self-attention layer, while retaining frozen pretrained Zero123 weights.The multi-view denoising process is conditioned on the input image and sequence of relative poses.Source: §3.2 · p. 4
  • Each novel view queries condition-image keys and values, transferring spatial layout while aligning outputs without extra trainable parameters.Sharing the condition-image keys and values also reduces part of the self-attention computation.Source: §3.2 · p. 4

3.3 SAMPLING TECHNIQUES FOR NOVEL VIEW SYNTHESIS

  • Sampling length is unrestricted by training length: models trained on 8 views can generate over 64 simultaneously, with more sampled views improving consistency.For extremely large view counts, such as over 256, the authors suggest generating a manageable set first and then predicting nearby views in another round.Source: §3.3 · p. 5
  • Progressive classifier-free guidance decreases guidance scale during denoising: high early values support geometry and artifact reduction, while lower late values refine texture.The method shifts from outlining the object to refining details as denoising proceeds.Source: §3.3 · p. 5
  • Among the tested guidance-scale reduction functions, the concave function performed best empirically.The tested functions were linear, concave, and convex.Source: §3.3 · p. 5

4 EXPERIMENT RESULTS

Consistent123 improves view consistency across in- and out-of-distribution benchmarks, while ablations identify cross-view attention as its most important component. It also improves novel-view quality and downstream 3D lifting results.

4.1 EXPERIMENT SETUP

  • The evaluation uses Objaverse, GSO, and RTMV, and measures consistency with a NeRF-based score plus PSNR, SSIM, and LPIPS against ground-truth views.Objaverse is used for finetuning; the test benchmarks include Objaverse, GSO, and RTMV.Source: §4.1 · pp. 5–6

4.2 CONSISTENCY EVALUATION

  • Consistent123 significantly improves view consistency over baselines on both in-distribution and out-of-distribution tests, without losing image quality.The evaluation samples 64 views per object; stochastic conditioning improves consistency at the expense of image quality.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 ABLATION STUDY

  • Cross-view attention is the most important ablated component; shared self-attention aligns condition and synthesized views, while progressive guidance reduces artifacts and slightly improves consistency.Each ablation removes one component while keeping the others unchanged.Source: §4.3 · p. 7, Table 2 · p. 7
  • Increasing the number of sampled views improves view consistency and quality, with greater overlap and interaction across views offered as the explanation.Arbitrary-length sampling is supported despite training at a fixed view count.Source: §4.3 · pp. 7–8, Figure 6 · p. 8
  • The concave PCFG reduction performs best, and applying shared self-attention only in U-Net decoder layers is slightly better.The concave schedule builds layout quickly before refining details with more steps.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 NOVEL VIEW SYNTHESIS EVALUATION

  • Consistent123 outperforms most baselines on ground-truth novel-view similarity without sacrificing synthesized-view quality.The reported evaluation-view count differs slightly from Zero123, producing small differences in final metrics.Source: §4.4 · p. 8, Table 4 · p. 9
  • Consistent123 significantly boosts 3D lifting results in NeuS, Dreamfusion, and One-2-3-45 compared with Zero123.The figure compares Zero123 in the first row with Consistent123 in the second for each object.Source: Figure 7 · p. 9

5 LIFT CONSISTENT MULTI-VIEWS TO 3D

Consistent123’s synthesized views improve downstream 3D reconstruction and image-to-3D generation compared with Zero123-based views.

Native 3D Reconstruction

  • Consistent123 views greatly improve both geometry and texture in native Neus reconstruction compared with Zero123.The paper illustrates this comparison in Figure 7(a).Source: §5 · p. 9

Image-to-3D Generation

  • Zero123-based image-to-3D generation fails especially on unseen sides, while Consistent123-backed generation produces relatively good-quality objects.The comparison includes DreamFusion and One-2-3-45 lifting methods.Source: §5 · p. 9
  • Consistent123 improves image-to-3D generation quality over Zero123-based results in the reported lifting comparisons.The authors note further room to improve lifting quality despite the consistency boost.Source: §5 · p. 9

6 CONCLUSION

Consistent123 synthesizes consistent multiple views using attention-based alignment, with sampling strategies for arbitrary view counts and better view quality and consistency. The authors report improved quality in native 3D reconstruction and image-to-3D generation.

6 CONCLUSION

  • Consistent123 aligns synthesized views through cross-view attention and shared self-attention.The model is presented as an image-to-3D model for synthesizing consistent multiple views.Source: §6 · p. 9
  • Two sampling strategies support arbitrary view counts and improve view quality and consistency.Source: §6 · p. 9
  • Consistent123 boosts 3D lifting quality in native reconstruction and image-to-3D generation tasks.The paper presents it as a potential foundation model for these downstream tasks.Source: §6 · p. 9

A.1 MORE QUALITATIVE RESULTS

The appendix provides additional qualitative results across object datasets, in-the-wild and text-generated images, and downstream 3D tasks.

A.1 MORE QUALITATIVE RESULTS

A.2 EVALUATION WITH DIFFERENT NUMBERS OF SAMPLING VIEWS

The appendix evaluates GSO consistency across sampling-view counts beyond the 64-view setting used in prior experiments. Consistent123 outperforms Zero123 across the varying evaluation counts.

A.2 EVALUATION WITH DIFFERENT NUMBERS OF SAMPLING VIEWS

  • Consistent123 outperforms Zero123 by a large margin on GSO consistency across varying numbers of sampling views.Figure 14 presents this comparison.Source: Figure 14 · p. 17, §A.2 · p. 17
  • Figure 14 reports consistency scores for different sampling-view counts on GSO.Source: Figure 14 · p. 17
  • The earlier consistency experiments used 64 sampling views; the appendix reports additional GSO evaluation counts.Source: §A.2 · p. 17

A.3 IMPLEMENTATION DETAILS FOR IMAGE-TO-3D GENERATION

The section describes implementation adjustments for adapting Consistent123 to DreamFusion and One-2-3-45 image-to-3D pipelines.

A.3 IMPLEMENTATION DETAILS FOR IMAGE-TO-3D GENERATION

  • Replacing Zero123 with Consistent123 boosts 3D-lifting quality, with method-specific modifications to better use Consistent123.Source: §A.3 · p. 17
  • DreamFusion’s noise-scale lower bound starts near its upper bound and gradually decreases toward 0.02 to help initialize texture.The authors report that vanilla DreamFusion’s uniform 0.02–0.98 schedule caused texture loss on unseen object sides.Source: §A.3 · p. 17
  • Consistent123 generates all required One-2-3-45 views concurrently in one stage, further improving quality and speed.Vanilla One-2-3-45 first synthesizes 8 views, then predicts 4 nearby views for each.Source: §A.3 · p. 17

A.4 DIFFERENT TRAINING RENDERINGS OF OBJAVERSE

The section compares Objaverse training renderings: circular perturbation improves view consistency while maintaining similarity with ground truth.

A.4 DIFFERENT TRAINING RENDERINGS OF OBJAVERSE

  • The selected Objaverse training set uses poses sampled circularly with perturbation.Source: §A.4 · pp. 17–18
  • Circular-perturbation renderings improve view consistency over spherical sampling, the training rendering choice used by Zero123.Source: Table 5 · p. 17, §A.4 · p. 18
  • Circular perturbation achieves similar ground-truth comparison performance to spherical sampling.Source: §A.4 · p. 18, Table 6 · p. 18
Loading 2310.08092v1…