Source-linked AI summary
Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image
Kailu Wu, Fangfu Liu, Zhihan Cai, Runjie Yan, Hanyang Wang, Yating Hu, Yueqi Duan, Kaisheng Ma
TL;DR
Single-view image-to-3D is difficult because existing approaches can be slow, inconsistent, or limited in texture and geometric detail. Unique3D combines multi-view and normal diffusion, progressive upscaling, and ISOMER mesh reconstruction to generate detailed meshes efficiently. Experiments report improved fidelity, geometric detail, resolution, and generalizability, while the method remains limited for skewed or non-perspective inputs and lacks texture-map support.
Problem
Single-image 3D generation remains challenging because prior methods trade optimization efficiency against consistency, texture detail, and complex geometry.
Method
Unique3D combines multi-view and normal diffusion, progressive multi-view upscaling, and ISOMER reconstruction from RGB images and normal maps.
Results
Experiments report that Unique3D outperforms existing methods in fidelity, geometric detail, resolution, and generalizability across diverse images.
Takeaways & Limitations
Unique3D provides an efficient framework for generating high-fidelity, consistent, textured 3D meshes from single-view images.
Takeaways & Limitations
The multi-view prediction model may perform less satisfactorily on skewed or non-perspective inputs, and geometric coloring does not support texture maps.
Abstract
from arXiv · showhide
In this work, we introduce Unique3D, a novel image-to-3D framework for efficiently generating high-quality 3D meshes from single-view images, featuring state-of-the-art generation fidelity and strong generalizability. Previous methods based on Score Distillation Sampling (SDS) can produce diversified 3D results by distilling 3D knowledge from large 2D diffusion models, but they usually suffer from long per-case optimization time with inconsistent issues. Recent works address the problem and generate better 3D results either by finetuning a multi-view diffusion model or training a fast feed-forward model. However, they still lack intricate textures and complex geometries due to inconsistency and limited generated resolution. To simultaneously achieve high fidelity, consistency, and efficiency in single image-to-3D, we propose a novel framework Unique3D that includes a multi-view diffusion model with a corresponding normal diffusion model to generate multi-view images with their normal maps, a multi-level upscale process to progressively improve the resolution of generated orthographic multi-views, as well as an instant and consistent mesh reconstruction algorithm called ISOMER, which fully integrates the color and geometric priors into mesh results. Extensive experiments demonstrate that our Unique3D significantly outperforms other image-to-3D baselines in terms of geometric and textural details.
1 Introduction
Single-image 3D generation is important but ill-posed, while existing methods trade efficiency, consistency, fidelity, or generalizability. Unique3D addresses these challenges with multi-view and normal diffusion, progressive upscaling, and ISOMER reconstruction.
- Single-view 3D generation supports applications including gaming, architecture, art, and animation but remains ambiguous because one image underdetermines 3D geometry.
- SDS-based methods produce diverse results but often require hours of per-case optimization and suffer from poor geometry and inconsistent views.
- Unique3D generates orthographic multi-view images and normal maps, progressively upscales them, and reconstructs meshes with ISOMER.
- Experiments on diverse wild images report higher fidelity, geometric detail, resolution, and generalizability than existing methods.
- ISOMER reconstructs 3D meshes from RGB images and normal maps, targeting intricate geometric details and textures.
2 Related Work
Prior approaches either optimize 3D representations slowly or use efficient feed-forward and multi-view pipelines that remain limited in quality, texture, consistency, or geometry. These limitations motivate efficient mesh reconstruction from high-quality multi-view information.
- Traditional mesh reconstruction uses complex photogrammetry stages, while newer learned methods reduce effort but retain computational costs and difficulty with complex geometry.
- SDS-based 2D lifting distills 3D knowledge from 2D diffusion models but suffers from long optimization times and multi-face inconsistencies.
- Multi-view diffusion methods improve efficiency and consistency by generating views for reconstruction, yet prior feed-forward results can lack texture detail and 3D consistency.
3 Method
Unique3D generates high-resolution, consistent multi-view color images and normal maps from one image, then reconstructs a detailed mesh through ISOMER. Its pipeline combines progressive upscaling, direct mesh optimization, and explicit handling of multi-view inconsistency.
- Multi-view Generation: Unique3D generates four orthographic multi-view images and corresponding normal maps, then progressively lifts both to high resolution.The framework uses separate multi-view and normal diffusion models before mesh reconstruction.
- Multi-view Generation: A channel-wise noise offset improves generated images with uniform backgrounds by reducing the discrepancy between sampling noise and training samples.
- ISOMER Reconstruction: ISOMER reconstructs meshes through topology initialization, coarse-to-fine optimization, and explicit treatment of cross-view inconsistency, completing reconstruction in no more than 10 seconds.The algorithm directly reconstructs meshes from high-resolution multi-view images.
- ISOMER Reconstruction: The initial mesh integrates pseudo-normal maps after random rotations, estimates depth, builds front and back meshes, joins them with Poisson reconstruction, and simplifies them by 2000 faces.Randomized integration is used because diffusion-generated pseudo-normal maps are not guaranteed to form an irrotational normal field.
- ISOMER Reconstruction: Coarse-to-fine optimization uses mask and normal losses with differentiable rendering, while mesh operations maintain uniform face distribution and reasonable edge lengths.The mask loss regulates contours, and the normal loss optimizes visible-area normal directions.
- Consistency Refinement: Expansion regularization addresses surface collapse under limited-view normal supervision by moving vertices slightly along their normals during optimization.
- Consistency Refinement: ExplicitTarget assigns each vertex a weighted target from supervised views, using squared cosine weights because projected area and prediction accuracy correlate with viewing angle.This design avoids directly minimizing discrepancies across all inconsistent views, which can create wave-pattern flaws.
4 Experiments
Experiments compare Unique3D with existing methods, assess its reconstruction components, and test robustness across views, resolutions, and challenging inputs. Results report higher visual and geometric quality, while ablations identify ExplicitTarget, expansion regularization, and high-resolution maps as important design choices.
- Quantitative Comparison: Unique3D is evaluated on GSO using visual metrics PSNR, SSIM, LPIPS, and Clip-Similarity, plus geometric metrics Chamfer Distance, Volume IoU, and F-score.Generated meshes are normalized to a shared bounding box, and visual quality is assessed across 24 rendered views.
- Quantitative Comparison: Replacing Wonder3D’s reconstruction method with ISOMER is reported to improve quality and speed while enhancing consistency.The comparison is conducted using the same quantitative evaluation framework on GSO.
- Ablation Study: ExplicitTarget improves geometry in challenging cases and prevents obvious texture artifacts caused by inconsistent multi-view coloring.Without ExplicitTarget, mesh defects and coloring artifacts are observed in the ablations.
- Ablation Study: Expansion regularization avoids some surface collapses during reconstruction under limited-view normal supervision.The method moves vertices slightly along their normals during optimization.
- Robustness and Resolution: Multi-level super-resolution preserves the generated views’ general structure while improving detail resolution, and high-resolution maps are necessary for high-resolution meshes.Additional tests show strong performance on randomly rotated objects and complex samples containing text or human photographs.
5 Conclusion
Unique3D is presented as an efficient, high-quality single-image 3D mesh framework, with detailed textured outputs generated within 30 seconds. Its multi-view prediction remains less satisfactory for skewed or non-perspective inputs, and geometric coloring does not support texture maps.
- Unique3D generates detailed and textured meshes from single-view images within 30 seconds.
- The multi-view prediction model may produce less satisfactory predictions for skewed or non-perspective inputs.
- The geometric coloring algorithm currently does not support texture maps.
A More Results
Additional single-image generation results are provided to illustrate the method's outputs.
- Additional generation results from single images are shown in Figure 9.
B Network Architecture and Training Details
The implementation combines multi-view diffusion, high-resolution multi-view upscaling, normal prediction, and mesh color completion. Training details specify the main architectural choices and optimization settings for these components.
- Multi-view Image Generation: The multi-view image generator uses view-index class embeddings and jointly forwards four perspectives with concatenated self-attention features for consistency.
- Multi-view Image Upscale: The multi-view upscale network fine-tunes ControlNet-Tile with Stable Diffusion 1.5, empty text, and IP-Adapter guidance to enhance multi-view details.
- Normal Prediction Diffusion: The normal prediction diffusion model maps RGB images to normal maps and adds a reference U-Net for pixel-wise reference attention.
- Color Completion: Invisible-region colors are completed by propagating surrounding colors through a simple multi-step algorithm.
- Color Completion: Additional propagation iterations smooth color gradients and reduce stark demarcation lines across connected mesh components.
- ExplicitTarget: ExplicitTarget computes vertex colors from visible views using cosine-angle-squared weights between vertex normals and view directions.
D ExplicitTarget algorithm
ExplicitTarget uses visible-view signals to assign colors to mesh vertices, while the surrounding mesh optimization operations address defects and refine triangulation.
- ExplicitTarget: ExplicitTarget assigns each vertex a weighted sum of supervised color signals from the visible views.
- ExplicitTarget: The weighting uses projected nearby-surface area and normal-prediction confidence, with the reported normal accuracy relationship having Pearson correlation coefficient -0.304.
- Mesh Optimizations: Edge collapse merges triangles to eliminate narrow defects and help prevent topological artifacts.
- Mesh Optimizations: Edge split divides overly long edges to refine the mesh and keep local edge lengths near the optimal scale.
- Mesh Optimizations: Edge flip changes mesh connectivity by altering triangulation to improve mesh quality.
F Ablation Study on Mesh Initialization
The ablation compares Unique3D’s fast mesh initialization with sphere initialization. Sphere-based direct optimization can fail when the target requires a different topology, such as a hollow structure.
- Mesh Initialization: The study compares the proposed fast initialization method against sphere initialization.Sphere initialization is described as a common practice in mesh-based reconstruction.
- Mesh Initialization: Sphere-based direct optimization cannot modify topology when the target structure differs intrinsically.The paper illustrates this failure with hollow structures that cannot be achieved from the initial topology.
- Mesh Initialization: A hollow structure is an example where topology-aware initialization matters for successful mesh reconstruction.The first-row example shows optimization failing to create a hollow form because the topologies differ.
G User Study
The user study evaluates rendered 360-degree videos across consistency, subject fidelity, prompt fidelity, and overall quality. Unique3D is significantly preferred by users across these aspects.
- User Study: 30 volunteers rated 3D consistency, subject fidelity, prompt fidelity, and overall quality on a 1–10 scale.Higher scores indicate better performance.
- User Study: The study uses rendered 360-degree videos of subject-driven 3D models and presents five randomly selected method samples to each volunteer.This setup evaluates generated models through rotating rendered views.
- User Study: Unique3D is significantly preferred by users across all four evaluated aspects.The reported aspects are 3D consistency, subject fidelity, prompt fidelity, and overall quality.
H Social Impact
Unique3D may broaden access to 3D content creation for creative industries. Its easier generation of high-quality models also raises concerns about misuse and effects on traditional 3D modeling work.
- Positive Impacts: The framework could make single-image 3D model creation easier for artists and designers across gaming, film, and education.The paper associates this accessibility with increased innovation and creative applications.
- Negative Impacts: Easier generation of high-quality 3D models raises concerns about deepfakes and job displacement for traditional 3D modelers.The paper also identifies intellectual-property and privacy challenges from irresponsible use.