PaperlayerView exact revision on arXiv

Source-linked AI summary

Consistent123: One Image to Highly Consistent 3D Asset Using Case-Aware Diffusion Priors

Yukang Lin, Haonan Han, Chaoqun Gong, Zunnan Xu, Yachao Zhang, Xiu Li

arXiv:2309.17261v2cs.CV

TL;DR

Single-image 3D reconstruction is ill-posed, and fixed guidance strategies struggle to generalize across objects and maintain consistency. Consistent123 adapts guidance in two stages, and the authors report stronger consistency and geometric and textural quality than purely 3D, purely 2D, and fused-prior methods.

  • Problem

    Single-view reconstruction is ill-posed, and case-agnostic strategies struggle to adapt to arbitrary objects and produce highly consistent results.

    Source: §1 · p. 2

  • Method

    Consistent123 first uses 3D structural priors with CLIP-based adaptive boundary detection, then progressively shifts guidance toward 2D texture priors through dynamic prior fusion.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    Consistent123 demonstrates stronger 3D consistency than purely 3D, purely 2D, and fused-prior methods, with superior geometric and textural quality and fewer multi-face issues.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The method adapts guidance to reconstruction progress and different inputs, producing coherent 3D assets.

    Source: §1 · p. 2

    Reconstruction quality is affected by the input viewpoint during stage 1, and stage-2 output quality depends on the asset description; brief or ambiguous descriptions can introduce inaccurate details.

    Source: §5 · p. 10

Abstract

from arXiv · show

Reconstructing 3D objects from a single image guided by pretrained diffusion models has demonstrated promising outcomes. However, due to utilizing the case-agnostic rigid strategy, their generalization ability to arbitrary cases and the 3D consistency of reconstruction are still poor. In this work, we propose Consistent123, a case-aware two-stage method for highly consistent 3D asset reconstruction from one image with both 2D and 3D diffusion priors. In the first stage, Consistent123 utilizes only 3D structural priors for sufficient geometry exploitation, with a CLIP-based case-aware adaptive detection mechanism embedded within this process. In the second stage, 2D texture priors are introduced and progressively take on a dominant guiding role, delicately sculpting the details of the 3D model. Consistent123 aligns more closely with the evolving trends in guidance requirements, adaptively providing adequate 3D geometric initialization and suitable 2D texture refinement for different objects. Consistent123 can obtain highly 3D-consistent reconstruction and exhibits strong generalization ability across various objects. Qualitative and quantitative experiments show that our method significantly outperforms state-of-the-art image-to-3D methods. See https://Consistent123.github.io for a more comprehensive exploration of our generated 3D assets.

1 INTRODUCTION

Single-image 3D reconstruction remains ill-posed, while prior approaches rely on object-specific 3D knowledge or fixed strategies; Consistent123 adapts its 3D-to-2D guidance and reports improved consistency and quality.

1 INTRODUCTION

  • Single-view reconstruction is ill-posed, and existing approaches either depend on costly, domain-specific 3D knowledge or use fixed strategies that struggle across arbitrary objects.The introduction also notes that 2D-prior methods can suffer from multi-face results, while 3D-prior methods can struggle with consistency.Source: §1 · p. 2
  • The proposed guidance schedule reflects an object- and progress-aware view: establish overall structure first, then refine geometry and texture details.The authors identify case-aware early optimization driven solely by 3D structural priors as supporting eventual reconstruction fidelity and consistency.Source: §1 · p. 2
  • Consistent123 first optimizes 3D content with 3D priors, then gradually reduces their influence while 2D priors guide texture refinement.The stage transition is triggered when the changing rate of similarity between fixed-perspective samples and textual information falls below a threshold.Source: §1 · pp. 2–3
  • A CLIP-based adaptive detection mechanism identifies when 3D optimization ends, avoiding manual adjustment of the prior ratio across objects with diverse geometry and texture.The method periodically samples 3D content from fixed perspectives and measures its similarity with textual information.Source: §1 · pp. 2–3
  • Consistent123 demonstrates stronger 3D consistency than purely 3D, purely 2D, and fused-prior methods, with superior geometric and textural quality and fewer multi-face issues.The introduction reports these qualitative comparisons without providing metric values.Source: §1 · p. 3

2 RELATED WORK

Prior work uses 2D visual and diffusion models for 3D generation, while single-image reconstruction methods combine learned priors, generated views, or 3D structure in different ways.

2.1 TEXT-TO-3D GENERATION

2.2 SINGLE IMAGE 3D RECONSTRUCTION

  • Single-image 3D reconstruction uses 3D-aware GANs or diffusion models for novel-view generation, while other methods optimize 3D objects directly with 2D diffusion priors and techniques such as textual inversion or rough 3D initialization.The described approaches include GAN inversion, synthetic-data-trained 3D diffusion, camera-conditioned Stable Diffusion, and fixed-weight combinations of rough 3D priors with textual inversion.Source: §2.2 · pp. 3–4

3 METHODOLOGY

Consistent123 reconstructs a reference view, initializes novel-view geometry with 3D guidance, then adaptively introduces 2D texture guidance. Its CLIP-based boundary check and changing balance of priors are designed to support consistent geometry and texture refinement.

3 METHODOLOGY

  • Consistent123 organizes reconstruction around a reference view and two novel-view stages, with the output described as 3D-consistent and high in texture quality.The novel-view stages are detailed in Sections 3.2 and 3.3.Source: §3 · p. 4

3.1 REFERENCE VIEW RECONSTRUCTION

  • Reference-view reconstruction combines RGB, mask, and depth losses, with controllable weights for each supervision.The RGB and mask losses use MSE, while the depth loss uses normalized negative Pearson correlation.Source: §3.1 · pp. 4–5

3.2 OPTIMIZATION BOUNDARY JUDGEMENT

  • Stage 1 uses only 3D structural priors; periodic CLIP checks track changing similarity, and a rate below threshold δ marks the structure as recovered.Checks occur every 20 iterations in the reported experiments.Source: §3.2 · p. 5

3.3 DYNAMIC PRIOR

  • Because 2D-only guidance can produce unfaithful novel views, Dynamic Prior incrementally combines 3D and 2D guidance, decreasing the structural loss weight and increasing the texture loss weight as timestep t increases.The paper calls the unfaithful-detail issue “Over Imagination” and reports more consistent, exquisite outputs than single-prior or fixed-ratio alternatives.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments compare Consistent123 with prior methods across RealFusion15, C10, and GSO, and examine its two-stage optimization and dynamic priors. The reported results favor Consistent123 on consistency and reconstruction quality, including a user-study advantage.

4.1 IMPLEMENTATION DETAILS

  • A single-image reconstruction takes about 30 minutes for 10,000 iterations on one NVIDIA A100 GPU, with roughly 16 minutes for stage one and 14 for stage two.The experiments use Stable Diffusion 2.1 and Zero-1-to-3 as the 2D and 3D priors, respectively.Source: §4.1 · p. 6

4.2 COMPARISON WITH STATE-OF-THE-ART

  • Qualitative comparisons show RealFusion producing flat results, Make-it-3D suffering multi-face errors, and Zero-1-to-3 and Magic123 showing weak side-view texture consistency.Consistent123 is reported to maintain semantic, texture, and geometry consistency across views.Source: §4.2 · pp. 7–8, Figure 4 · p. 7
  • Consistent123 gains 11.2% in RealFusion15 CLIP-Similarity over the original SOTA; on C10 it improves PSNR by 2.972 and LPIPS by 0.066 over the previous top model, and leads across evaluation metrics.On GSO, it achieves the highest CD and IoU and excels in multi-view PSNR and LPIPS.Source: §4.2 · p. 8, Table 1 · p. 8, Table 2 · p. 8

4.3 ABLATION STUDY OF TWO STAGE OPTIMIZATION

  • Without boundary judgment, 3D-only initialization yields unrealistic novel-view colors, while dynamic-prior-only optimization produces facial inconsistency and multi-face errors; boundary judgment improves texture and structure.Figure 5 illustrates the ablation of two-stage optimization.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 ABLATION STUDY OF DYNAMIC PRIOR

  • Exponential dynamic-prior variation achieves higher CLIP-Similarity on most tested categories and is reported to mitigate multi-head errors, improving reconstruction quality and consistency.It injects 2D priors more quickly after the initial stage has established basic geometry, while retaining some 3D-prior influence.Source: §4.4 · p. 9, Table 3 · p. 9

4.5 USER STUDY

  • In a perceptual study conducted because ground-truth 3D models were unavailable, Consistent123 showed a 65.7% advantage over alternatives.Participants selected the result that best represented the depicted object's texture and structure.Source: §4.5 · p. 9

5 CONCLUSION AND DISCUSSION

Consistent123 combines 3D-prior geometry optimization with gradually introduced 2D-prior texture guidance. The authors report improved structural and textural consistency on benchmark and curated datasets, while noting viewpoint and description-related limitations.

Conclusion

  • The two-stage framework prioritizes 3D-prior geometry optimization, then gradually incorporates 2D-prior guidance for texture detail.A pretrained image-text model discriminates among multi-view samples between the stages before dynamic-prior optimization.Source: §5 · p. 9
  • The authors report enhanced structural and textural 3D consistency on existing benchmark datasets and datasets they curated.Source: §5 · p. 9

Limitations

  • Stage 1 reconstruction quality is affected by input viewpoint, while stage 2 output quality depends on asset-description detail.Finer descriptions improve consistency; brief or ambiguous descriptions can lead to inaccurate details through Stable Diffusion's “Over Imagination” issue.Source: §5 · p. 10

A.1 ADDITIONAL RESULTS

The appendix presents additional Consistent123 assets and shows one reconstruction from eight perspectives to illustrate 3D consistency.

Additional results

  • Figure 7 shows a Consistent123 reconstruction from eight perspectives to demonstrate its 3D consistency.The model is the one discussed in Section 4.2.Source: §A.1 · p. 14, Figure 7 · p. 14
  • Corresponding videos are available on the project website.Source: §A.1 · p. 14
  • Figure 8 presents 3D assets obtained by Consistent123.Source: Figure 8 · p. 15

A.2 METRICS FOR GSO DATASET

The GSO evaluation measures mesh geometry and rendered-view reconstruction quality using four metrics, with meshes normalized before comparison.

Metrics for GSO dataset

  • GSO evaluation measures Chamfer Distance, volumetric IoU, multi-view PSNR, and multi-view LPIPS.Source: §A.2 · p. 15
  • Chamfer Distance compares sampled vertices from the real and reconstructed meshes, using 2,000 sampled points in the experiments.A smaller distance indicates a better reconstruction.Source: §A.2 · p. 15
  • Volumetric IoU uses voxelized meshes, while multi-view PSNR and LPIPS compare rendered images against ground-truth views from 18 viewpoints.The passage specifies azimuth and elevation angles for those viewpoints.Source: §A.2 · p. 16

A.3 ABLATION STUDY OF CONCEPT INJECTION

The concept-injection ablation compares textual inversion with no textual inversion across categories in RealFusion15 and C10. The authors report little performance gain for textual inversion across most categories and omit it as the default because prompt tuning takes about two hours per case.

Ablation study of concept injection

  • Figure 9 compares 2D-prior results with and without textual inversion across 10 categories in RealFusion15 and C10.Source: Figure 9 · p. 16
  • Textual inversion does not significantly improve performance across most categories, so it is excluded from the default pipeline.It remains available as an optional selection.Source: §A.3 · p. 16
  • Prompt tuning for textual inversion takes approximately 2 hours per case on a single NVIDIA A100 GPU.Source: §A.3 · p. 16

A.4 ABLATION STUDY OF HYPERPARAMETERS.

The hyperparameter ablations examine reference-view reconstruction-loss weights and boundary-judgment settings on RealFusion15 and C10. They specify five settings for each ablation.

A.4 ABLATION STUDY OF HYPERPARAMETERS.

  • The reference-view loss ablation varies λrgb, λmask, and λdepth on RealFusion15 and C10.Table 4 presents the quantitative results for five parameter settings.Source: §A.4 · p. 16
  • The default loss weights are λrgb = 1000, λmask = 500, and λdepth = 10, among five tested settings.The alternatives include settings with λdepth = 0, 500, 1000, and 400.Source: §A.4 · p. 16
  • For boundary judgment, five settings ablate threshold δ and sliding-window size L: (L=1, δ=0.000025), (L=20, δ=0.000025), (L=5, δ=0.0001), (L=5, δ=0.001), and default (L=5, δ=0.00025).Quantitative results for the settings are shown in Table 5.Source: §A.4 · p. 16

A.5 MORE INTERESTING EXPLORATION.

The exploration reports that the default two-stage strategy achieves the best results across the tested metrics, while a user study found a substantial preference for Consistent123. The authors also identify complex examples that remain failure cases and visualize weight changes across three objects.

A.5 MORE INTERESTING EXPLORATION.

  • The default strategy achieves the best results on all metrics; removing boundary judgment, fixing the second-stage 2D weight, or using 2D guidance from the start performs worse.The experiments attribute the results to boundary judgment, gradual introduction of 2D prior, and 3D initialization; Table 6 covers the optimization-strategy ablation.Source: §A.5 · p. 17, Table 6 · p. 17

A.6 USER STUDY

  • Consistent123 leads previous methods in a preference study of 784 feedbacks from 56 users comparing five outputs across 14 cases.The cases came from RealFusion15 and C10; participants ranked outputs against four other methods.Source: §A.6 · p. 17

A.7 ADDITIONAL COMPARISON RESULTS

  • The back-view chair, drum kit, and green potted plant are failure cases, and existing methods also handle them poorly.Figure 10 is a qualitative comparison against state-of-the-art methods on NeRF4.Source: §A.7 · p. 18, Figure 10 · p. 18

A.8 VISUALIZATION OF THE WEIGHT CHANGE

  • Figure 11 visualizes weight changes for randomly selected cactus, dragon, and fish cases from RealFusion15.The passage identifies the cases but does not state a comparative outcome.Source: Figure 11 · p. 18

Related papers

Loading 2309.17261v2…