Source-linked AI summary
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, Ying Shan
TL;DR
InstantMesh addresses limited 3D data and annotation by combining multi-view diffusion with sparse-view mesh reconstruction from a single image. It generates high-quality 3D assets efficiently and outperforms recent image-to-3D baselines qualitatively and quantitatively, while remaining limited on tiny and thin structures.
Problem
Limited scale and poor annotations in 3D datasets hinder reproducing large-scale 2D generation advances for single-image 3D asset creation.
Method
InstantMesh generates 3D-consistent multi-view images, reconstructs a mesh with an LRM-based sparse-view model, and applies differentiable iso-surface extraction with geometric supervision.
Results
InstantMesh outperforms recent image-to-3D baselines qualitatively and quantitatively, achieving state-of-the-art performance on public datasets.
Takeaways & Limitations
Mesh-based representation and geometric supervision improve training efficiency and reconstruction quality while enabling high-quality 3D asset generation within seconds.
Takeaways & Limitations
FlexiCubes is less effective than NeRF at modeling tiny and thin structures, despite improving mesh smoothness and reducing artifacts.
Abstract
from arXiv · showhide
We present InstantMesh, a feed-forward framework for instant 3D mesh generation from a single image, featuring state-of-the-art generation quality and significant training scalability. By synergizing the strengths of an off-the-shelf multiview diffusion model and a sparse-view reconstruction model based on the LRM architecture, InstantMesh is able to create diverse 3D assets within 10 seconds. To enhance the training efficiency and exploit more geometric supervisions, e.g, depths and normals, we integrate a differentiable iso-surface extraction module into our framework and directly optimize on the mesh representation. Experimental results on public datasets demonstrate that InstantMesh significantly outperforms other latest image-to-3D baselines, both qualitatively and quantitatively. We release all the code, weights, and demo of InstantMesh, with the intention that it can make substantial contributions to the community of 3D generative AI and empower both researchers and content creators.
1. Introduction
InstantMesh addresses the difficulty of scaling 3D generation from single images by combining multi-view diffusion with sparse-view reconstruction and direct mesh optimization.
- Limited 3D dataset scale and poor annotations make it difficult to reproduce 2D diffusion advances for 3D generation.
- Earlier methods distilled 2D diffusion priors into 3D through per-scene optimization, but text-guided SDS often produced multi-face Janus artifacts.
- 2D distillation methods offer strong zero-shot generation but are time-consuming, whereas LRMs map image tokens to 3D representations for faster creation.
- Triplane-based LRM rendering is memory-intensive and limits training scale and the use of high-resolution depth and normal supervision.
- InstantMesh combines multi-view diffusion, sparse-view reconstruction, differentiable iso-surface extraction, and direct mesh supervision in a feed-forward framework.The authors report that the process can be accomplished in seconds and is designed to scale to large datasets.
2. Related Work
Prior image-to-3D research progressed from single-view reconstruction and 2D-prior distillation toward multi-view diffusion and feed-forward large reconstruction models.
- Image-to-3D: Image-to-3D methods have modeled point clouds, meshes, SDF grids, and neural fields, but limited training data hinder open-world generalization.
- Diffusion-based generation: SDS enabled zero-shot text-to-3D synthesis, while Zero123 and later multi-view methods improved consistency by conditioning generation on camera views.
- InstantMesh: InstantMesh’s overview uses six synthesized views followed by transformer-based sparse-view reconstruction to produce a mesh in around 10 seconds.
- Large Reconstruction Models: Large Reconstruction Models use transformers to map image tokens to implicit 3D triplanes for feed-forward image-to-3D creation.
3. InstantMesh
InstantMesh combines multi-view diffusion with a sparse-view transformer reconstruction model, then uses mesh-based training and geometric supervision to improve efficiency and quality.
- Architecture: InstantMesh feeds 3D-consistent multi-view images from a diffusion model into a sparse-view reconstruction model that predicts a high-quality mesh.
- Multi-view diffusion model: The reconstruction model accepts free-viewpoint images, allowing different multi-view generators to support text-to-3D and image-to-3D creation.
- Multi-view diffusion model: Zero123++ generates six views with a tailored viewpoint distribution covering both upper and lower object regions.
- Multi-view diffusion model: White-background fine-tuning stitches six target images into a 3 × 2 grid and trains the UNet for 1000 steps at a learning rate of 1.0 × 10^-5.
- Data preparation: Training renders 512 × 512 images, depths, and normals from 32 random viewpoints per Objaverse object, using a filtered high-quality subset.
- Mesh reconstruction: The second training stage switches to meshes, integrating FlexiCubes to extract surfaces from triplane implicit fields and enable geometric supervision.
- Mesh reconstruction: FlexiCubes replaces the density MLP’s output with SDF prediction and adds MLPs for deformation and weights needed for mesh extraction.
- Mesh reconstruction: Density-to-SDF initialization reverses field direction and places the initial iso-surface at the SDF zero level-set, improving training stability and convergence speed.
4. Experiments
InstantMesh is evaluated against image-to-3D baselines on public datasets using both visual and geometric metrics, alongside qualitative mesh comparisons and NeRF/Mesh variants.
- Evaluation Setup: The evaluation uses GSO and Omni3D sets, including 300 GSO objects and 130 Omni3D objects.The experiments assess orbiting and benchmark views.
- Evaluation Setup: The study evaluates 2D visual quality through rendered novel views and 3D geometric quality through shape metrics.Novel views are compared with ground-truth views, while the reported tables cover multiple evaluation sets.
- Quantitative Results: InstantMesh significantly outperforms baselines on SSIM and LPIPS, indicating the strongest perceptual novel-view quality.Its PSNR is slightly below the best baseline because multi-view diffusion generates plausible rather than pixel-faithful novel views.
- Quantitative Results: InstantMesh significantly outperforms baselines on CD and FS, indicating higher fidelity in generated shapes.The qualitative results likewise describe its geometries as the most reliable among the compared methods.
- Qualitative Results: Figure 3 compares textured meshes at fixed elevations, with InstantMesh showing more plausible geometry and appearance than the baselines.Other methods use elevation 0° because they reconstruct in view space, whereas InstantMesh uses 20°.
- Variant Comparison: The NeRF variant achieves slightly better metrics than the Mesh variant, but the metric drop for Mesh is marginal and its rendering is more efficient.Mesh uses explicit depth and normal supervision, producing smoother surfaces than meshes extracted from NeRF density fields.
5. Conclusion
InstantMesh combines a transformer-based sparse-view reconstruction model with multi-view diffusion to generate high-quality 3D assets. Its mesh representation and geometric supervision improve training efficiency and reconstruction quality, while several limitations remain.
- Conclusion: InstantMesh is an open-source feed-forward framework that generates high-quality 3D assets from multi-view diffusion outputs.Its reconstruction component is a transformer-based sparse-view large reconstruction model.
- Conclusion: Mesh-based representation and additional geometric supervision significantly boost training efficiency and reconstruction quality.The conclusion also notes improvements to data preparation and training strategy.
- Limitations: The 64 × 64 triplane decoder may bottleneck high-definition 3D modeling.This limitation follows the LRM and Instant3D design.
- Limitations: Generation quality is influenced by multi-view inconsistency in the diffusion model, while FlexiCubes is less effective for tiny and thin structures than NeRF.The authors identify more advanced diffusion architectures as a possible future direction.