One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization

Minghua Liu; Chao Xu; Haian Jin; Linghao Chen; Mukund Varma T; Zexiang Xu; Hao Su

Source-linked AI summary

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization

Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, Hao Su

arXiv:2306.16928v1Jun 29, 2023cs.CVcs.AIcs.RO

TL;DR

Single-image 3D reconstruction is underconstrained and existing diffusion-guided optimization methods are slow, inconsistent, and geometrically weak. One-2-3-45 generates Zero123 multi-view predictions, estimates camera elevation, and lifts them into a 360-degree textured mesh through SDF-based feed-forward reconstruction. It reports better geometry, 3D consistency, input-image adherence, quality, and efficiency, including reconstruction in 45 seconds.

Problem
Single-image 3D reconstruction must infer invisible regions from insufficient evidence, while existing optimization-based methods are slow, memory-intensive, 3D-inconsistent, and poor at producing high-quality geometry.
Method
The method uses Zero123 for camera-conditioned multi-view synthesis, estimates the input elevation, and reconstructs a 360-degree mesh with an SDF-based module trained for inconsistent predictions.
Results
45 seconds: One-2-3-45 reconstructs single-image 3D meshes in a feed-forward pass and outperforms existing methods in reported mesh quality and efficiency.
Takeaways & Limitations
The approach produces better geometry, more consistent 3D meshes, and closer adherence to the input image while also extending to text-to-3D.
Takeaways & Limitations
Multi-view fusion remains challenging because inconsistent Zero123 predictions can cause fusion failures.

Abstract

from arXiv · show

Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.

1 Introduction

Single-image 3D reconstruction is difficult because unseen regions are underdetermined, while existing diffusion-guided optimization methods are slow, memory-intensive, inconsistent, and geometrically weak. One-2-3-45 addresses these issues with feed-forward multi-view synthesis, pose estimation, and SDF-based reconstruction.

Motivation: Single-image reconstruction must infer invisible regions from insufficient evidence, yielding multiple plausible 3D solutions and poor generalization from category-specific priors.The task supports applications including robotic manipulation, navigation, 3D content creation, and AR/VR.
Limitations of Existing Methods: Optimization-based methods commonly require tens of minutes per shape, substantial memory, and tens of thousands of rendering and prior-model inference iterations.Their memory demands increase with image resolution because the 2D prior model requires full images.
Limitations of Existing Methods: These methods can produce 3D-inconsistent shapes and good RGB renderings but poor extracted meshes because each iteration views only one perspective and often uses density fields.Examples include two-faced or Janus-like shapes.
Approach: One-2-3-45 combines Zero123 multi-view synthesis with pose estimation and an SDF-based reconstruction module to produce a 360-degree textured mesh in one feed-forward pass.Zero123 generates view-conditioned predictions, while the estimated elevation supplies camera poses for reconstruction from inconsistent multi-view outputs.
Results: 45 seconds: the method reconstructs any object from one image without costly optimization, with better geometry, greater 3D consistency, closer input-image adherence, and improved quality and efficiency.The paper evaluates the approach on synthetic data and real images.

2 Related Work

Related work includes 2D-prior-guided 3D generation, 3D-prior-based reconstruction, and generalizable neural reconstruction. These approaches differ in whether they optimize per scene, learn from 3D data, or infer 3D structure from multiple source views in a feed-forward pass.

3D Generation Guided by 2D Prior Models: 2D generative and vision-language models use Internet-scale image pretraining to provide visual and 3D-world priors for 3D generative tasks.DreamField, DreamFusion, and Magic3D exemplify per-shape optimization guided by CLIP or diffusion models.
3D Generation Guided by 3D Priors: Earlier methods learned 3D priors from synthetic data or real scans and generated voxels, point clouds, polygon meshes, or parametric models.These methods combine 2D image encoders with representation-specific 3D generators.
Image-to-3D with 2D Priors: Single-image 3D methods extend diffusion-based generation through textual inversion or additional similarity losses to condition models on the input image.These approaches adapt text-conditioned diffusion systems for image-to-3D reconstruction.
Generalizable Neural Reconstruction: Generalizable neural reconstruction methods learn across scenes, unproject 2D features into 3D, and generate implicit fields from a few source views in a feed-forward pass.Some aggregate features with MLPs or transformers, while others construct explicit 3D feature or cost volumes.
Generalizable Neural Reconstruction: One-2-3-45 instantiates this reconstruction direction by feeding posed, Zero123-generated multi-view images into an SDF-based neural surface reconstruction module.Its pipeline combines multi-view synthesis, pose estimation, and 360-degree mesh reconstruction.

3 Method

The method combines Zero123’s view-conditioned image synthesis with a cost-volume-based SDF reconstruction module and specialized training and pose-estimation strategies. These components address inconsistent predicted views and unknown input elevation while enabling feed-forward 360° mesh reconstruction.

3.1 Zero123: View-Conditioned 2D Diffusion: Zero123 synthesizes object views conditioned on relative camera transformations using a Stable Diffusion model fine-tuned on paired multi-view images.Its canonical setup assumes centered objects and spherical cameras, with transformations parameterized by elevation, azimuth, and radius differences.
3.2 Reconstruction from Predicted Views: Traditional NeRF- and SDF-based reconstruction methods fail to produce high-quality meshes from inconsistent Zero123 predictions, even when given ground-truth camera poses.The experiment generates 32 uniformly sampled predicted views and feeds them to TensoRF and NeuS.
3.3 Neural Surface Reconstruction from Imperfect Multi-View Predictions: The reconstruction module extracts 2D features, builds a projected-feature variance cost volume, and predicts geometry and color through sparse 3D convolutional and neural SDF components.It accepts posed source images and outputs a textured mesh in a single feed-forward pass, with SDF-based rendering for RGB and depth.
3.3 Neural Surface Reconstruction from Imperfect Multi-View Predictions: Two-stage source-view selection and groundtruth-prediction mixed training use nearby Zero123 predictions with ground-truth RGB and depth supervision to learn consistent 360° reconstruction.During training, each ground-truth view produces four nearby predictions, and the module receives all 4 × n predictions while supervision includes RGB and depth.
3.4 Camera Pose Estimation: Because changing all source elevations can distort the shape, the elevation module selects the candidate angle with the smallest reprojection error and uses it to construct all source-view poses.The candidates are evaluated in a coarse-to-fine search using four nearby views predicted from the input image.

4 Experiments

Experiments evaluate One-2-3-45 qualitatively, quantitatively, and through ablations across synthetic, real, and text-conditioned inputs. The results report strong geometry, input adherence, 3D consistency, and runtime, while showing the importance of the reconstruction training strategies and elevation estimation.

Qualitative examples cover both synthetic and real images, showing input images alongside textured and textureless reconstructed meshes.
One-2-3-45 is compared with Point-E, Shap-E, Zero123, 3DFuse, and RealFusion using textured and textureless outputs or volume renderings.3DFuse and RealFusion do not natively export textured meshes, so their volume renderings are shown instead.
The method outperforms all baselines on F-Score, surpasses all but Shap-E on CLIP similarity, and runs faster than optimization-based methods while matching 3D native diffusion models.Evaluation uses 20 randomly selected shapes from each of Objaverse and GSO, with runtime measured on an A100 GPU.
The reconstruction module takes approximately 5 seconds, while Zero123 predictions take roughly 1 second per image on an A100 GPU.
Ablation Study: Ablations show that two-stage source-view selection, nearby views, and depth loss are important for consistent reconstruction and fine-grained geometry.Using 32 uniformly posed predictions can fail completely, while omitting nearby views loses local correspondence and omitting depth loss degrades reconstruction.
Ablation Study: Elevation errors distort reconstruction, whereas predicted elevations can match ground-truth results; predicted elevation errors have a 5.4-degree median and 9.7-degree average.
Ablation Study: The method is not very sensitive to the number of source views when the reconstruction module is retrained for the corresponding setting.
Ablation Study: Direct 360-degree reconstruction avoids the multi-view fusion difficulties caused by inconsistent Zero123 predictions.

5 Conclusion

The paper presents a feed-forward method that reconstructs high-quality 360° meshes from single images, with superior geometry, 3D consistency, and input-image adherence compared with existing zero-shot approaches.

The method reconstructs a high-quality 360° mesh of any object from a single image.
Compared with existing zero-shot approaches, the results exhibit superior geometry, enhanced 3D consistency, and closer adherence to the input image.
A single forward pass avoids time-consuming optimization and significantly reduces processing time.
The method can be extended to support text-to-3D.

6 Appendix

The appendix provides additional comparisons, implementation details, elevation-estimation procedures, and failure cases spanning object reconstruction, real-world images, and text-to-3D.

6.1 Qualitative Comparison: Qualitative comparisons cover Objaverse and GoogleScannedObjects test shapes that were unseen during reconstruction-module training.
6.2 Real-World Images: Real-world comparisons with Shap-E show that the method adheres more closely to the input image.
6.3 Text-to-3D: The text-to-3D examples show stronger fine-grained detail and compositionality than the compared approaches.
6.3 Text-to-3D: For text-to-3D, DALL-E 2 first generates an image from text, which the method then uplifts into a 3D textured mesh.
6.4 Elevation Estimation: Elevation estimation uses four nearby Zero123 views and selects the angle with the smallest reprojection error through coarse-to-fine search.
6.4 Elevation Estimation: The reprojection error is computed from triangulated keypoints across image triplets and their projected locations.
6.5 Training and Evaluation: Training uses a reconstruction loss combining RGB, depth, Eikonal, and sparsity terms, with weights λ0 = 1, λ1 = 0.1, and λ2 = 0.02.
6.6 Failure Cases and Limitations: The method struggles when Zero123 produces inconsistent views, especially for insufficiently informative or ambiguous input images.