Source-linked AI summary
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, Hao Su
TL;DR
Single-image 3D generation remains limited by scarce 3D training data and inconsistent multi-view synthesis. One-2-3-45++ finetunes 2D diffusion for consistent multi-view generation, lifts those views with multi-view-conditioned 3D diffusion, and produces high-quality textured meshes in under a minute with strong input-image fidelity.
Problem
Limited 3D training data restricts generalization to unseen categories, while existing multi-view generation can produce inconsistent views.
Method
One-2-3-45++ combines tiled consistent multi-view generation from a finetuned 2D diffusion model with multi-view-conditioned 3D reconstruction and lightweight texture optimization.
Results
Under 60 seconds, One-2-3-45++ generates realistic textured meshes with high fidelity to the input image and outperforms baselines in user preference and CLIP similarity.
Takeaways & Limitations
The method provides precise fine-grained control while producing high-quality single-image 3D meshes rapidly.
Abstract
from arXiv · showhide
Recent advancements in open-world 3D object generation have been remarkable, with image-to-3D methods offering superior fine-grained control over their text-to-3D counterparts. However, most existing models fall short in simultaneously providing rapid generation speeds and high fidelity to input images - two features essential for practical applications. In this paper, we present One-2-3-45++, an innovative method that transforms a single image into a detailed 3D textured mesh in approximately one minute. Our approach aims to fully harness the extensive knowledge embedded in 2D diffusion models and priors from valuable yet limited 3D data. This is achieved by initially finetuning a 2D diffusion model for consistent multi-view image generation, followed by elevating these images to 3D with the aid of multi-view conditioned 3D native diffusion models. Extensive experimental evaluations demonstrate that our method can produce high-quality, diverse 3D assets that closely mirror the original input image. Our project webpage: https://sudo-ai-3d.github.io/One2345plus_page.
1 UC San Diego 2 Zhejiang University 3 Tsinghua University 4 UCLA 5 Stanford University
The paper presents One-2-3-45++ as transforming a single RGB image into a high-fidelity textured mesh in under one minute.
- One-2-3-45++ transforms a single RGB image of any object into a high-fidelity textured mesh in under one minute.
- The generated meshes closely mirror the original input image.
- The figure shows the input image, textured mesh, and normal map.
1. Introduction
Single-image 3D generation is important but constrained by limited 3D data, slow optimization-based methods, and inconsistent multi-view predictions. One-2-3-45++ addresses these issues with consistent multi-view generation, 3D diffusion, and lightweight texture optimization.
- Generating 3D shapes from a single image or text prompt is a long-standing computer-vision problem essential for numerous applications.
- Limited publicly available 3D training data restricts generalization across unseen categories in open-world scenarios.
- Per-shape optimization methods use 2D priors but can require tens of minutes or longer for each input.
- One-2-3-45 predicts 3D shapes feed-forward, but inconsistent Zero123 multi-view predictions compromise reconstruction quality.
- One-2-3-45++ jointly generates six tiled views, reconstructs meshes with multi-view-conditioned 3D diffusion, and efficiently refines texture using those views as supervision.
- Under 60 seconds, the method generates realistic textured meshes with fine-grained control and fidelity to the input image.
2. Related Work
Related work spans 3D-native generation, 2D-prior optimization, novel-view synthesis, and sparse-view reconstruction. One-2-3-45++ combines these directions while using 3D diffusion to generate content directly.
- 3D-native models generate point clouds, voxels, meshes, parametric models, and implicit fields, but limited 3D data restricts category coverage.
- Per-shape optimization methods draw on 2D generative priors to optimize a separate 3D representation for each input.
- One-2-3-45 uses pretrained 2D diffusion to synthesize views and reconstructs a textured mesh in 45 seconds, but its views lack 3D consistency.
- One-2-3-45++ produces an initial textured mesh within 20 seconds and a refined mesh in roughly one minute.
- Generalizable NeRF methods unproject and aggregate 2D features into 3D space from sparse source views.
- Diffusion-assisted sparse-view methods often frame reconstruction as novel-view synthesis and require additional 3D distillation or processing.
3. Method
One-2-3-45++ generates consistent multi-view images from one input, lifts them into 3D with conditioned diffusion, extracts a mesh, and refines its texture.
- 3.1. Consistent Multi-View Generation: The method fine-tunes a 2D diffusion model to generate six tiled views simultaneously, allowing views to interact during diffusion.
- 3.1. Consistent Multi-View Generation: Predetermined absolute elevations and relative azimuths remove the need to infer the input image’s elevation during reconstruction.
- 3.2. 3D Diffusion with Multi-View Condition: The 3D reconstruction module learns plausible shapes conditioned on generated multi-view images using expressive 3D-native diffusion models.
- 3.2. 3D Diffusion with Multi-View Condition: Two diffusion networks separately produce a full occupancy volume and a high-resolution sparse volume.
- 3.2. 3D Diffusion with Multi-View Condition: Local patch features from each view are projected into 3D, aggregated with a shared-weight MLP and max pooling, and assembled into conditional feature volume C.
- 3.2. 3D Diffusion with Multi-View Condition: During inference, coarse occupancy is denoised, subdivided into a sparse volume, and denoised again to predict SDF and color before Marching Cubes extracts the mesh.
- 3.2. 3D Diffusion with Multi-View Condition: Texture refinement fixes mesh geometry while optimizing a TensoRF color field against rendered multi-view images with an L2 loss.
4. Experiments
Experiments compare One-2-3-45++ with optimization-based and feed-forward methods for image-to-3D and text-to-3D generation, using objective metrics, qualitative results, and user studies. Ablations analyze the roles of consistent multi-view generation, multi-view-conditioned 3D diffusion, and texture refinement.
- 4.1. Comparison on Image to 3D: One-2-3-45++ surpasses all image-to-3D baselines in F-Score and CLIP similarity.
- 4.1. Comparison on Image to 3D: 92% of the time, users preferred One-2-3-45++ over One-2-3-45 in the image-to-3D comparison.The study collected 2,385 evaluated pairs from 53 participants.
- 4.3. Analyses: Replacing consistent multi-view generation with Zero123XL causes a noticeable performance decline in the overall pipeline ablation.The ablation evaluates multi-view generation, reconstruction, and texture refinement modules on the complete GSO dataset.
- 4.3. Analyses: Multi-view local conditions and global input-view CLIP features both improve the 3D diffusion module, while predicted-image training conditions reduce performance.The method constructs a 3D feature volume from multi-view local features and uses ground-truth renderings to train the diffusion module.
- 4.3. Analyses: One-2-3-45++ surpasses Zero123, Zero123 XL, Wonder3D, and SyncDreamer in PSNR, LPIPS, and foreground mask IoU.The comparison uses the complete GSO dataset; the reported metrics do not assess 3D consistency across views.
5. Conclusion
One-2-3-45++ transforms a single image of any object into a high-quality 3D textured mesh, typically in under 60 seconds. The meshes show high fidelity to the original input image, while future work may improve geometry robustness and detail with additional 2D diffusion guidance.
- One-2-3-45++ transforms a single image of any object into a 3D textured mesh, typically in under 60 seconds.
- The generated meshes exhibit high fidelity to the original input image.
- Additional guiding conditions from 2D diffusion models could improve geometry robustness and detail alongside RGB images.