Source-linked AI summary
CraftsMan3D: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refiner
Weiyu Li, Jiarui Liu, Hongyu Yan, Rui Chen, Yixun Liang, Xuelin Chen, Ping Tan, Xiaoxiao Long
TL;DR
Existing 3D generation methods struggle with optimization time, geometric quality, and user editing. CraftsMan combines 3D-native diffusion with a normal-based geometry refiner and mesh-to-SDF preprocessing. The authors report high-quality assets and interactive refinement, while noting that model controllability warrants further investigation.
Problem
Existing 3D generative methods struggle to produce ready-to-use results, including high-fidelity detail and support for user editing.
Source: §1 · p. 1Method
CraftsMan preprocesses meshes for SDF training, generates coarse geometry with 3D-native diffusion, and refines it using normal maps, including user-selected edit regions.
Source: §1 · p. 2, §3.3 · p. 5Results
The authors report superior-quality 3D assets compared with existing methods and a normal-based refiner that enhances geometry within 20 seconds.
Source: §1 · p. 2Takeaways & Limitations
The refiner supports interactive manipulation of generated geometry to better align it with users’ envisioned designs.
Source: §1 · p. 2The controllability of the Latent Set Diffusion model warrants further investigation.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We present a novel generative 3D modeling system, coined CraftsMan, which can generate high-fidelity 3D geometries with highly varied shapes, regular mesh topologies, and detailed surfaces, and, notably, allows for refining the geometry in an interactive manner. Despite the significant advancements in 3D generation, existing methods still struggle with lengthy optimization processes, irregular mesh topologies, noisy surfaces, and difficulties in accommodating user edits, consequently impeding their widespread adoption and implementation in 3D modeling software. Our work is inspired by the craftsman, who usually roughs out the holistic figure of the work first and elaborates the surface details subsequently. Specifically, we employ a 3D native diffusion model, which operates on latent space learned from latent set-based 3D representations, to generate coarse geometries with regular mesh topology in seconds. In particular, this process takes as input a text prompt or a reference image and leverages a powerful multi-view (MV) diffusion model to generate multiple views of the coarse geometry, which are fed into our MV-conditioned 3D diffusion model for generating the 3D geometry, significantly improving robustness and generalizability. Following that, a normal-based geometry refiner is used to significantly enhance the surface details. This refinement can be performed automatically, or interactively with user-supplied edits. Extensive experiments demonstrate that our method achieves high efficacy in producing superior-quality 3D assets compared to existing methods. HomePage: https://craftsman3d.github.io/, Code: https://github.com/wyysf-98/CraftsMan
1. Introduction
Existing 3D generation methods face trade-offs in efficiency, geometric quality, and support for user editing. CraftsMan combines 3D-native generation with interactive refinement, alongside a mesh-to-SDF preprocessing pipeline.
1. Introduction
- Existing methods struggle with lengthy optimization, irregular or noisy geometry, and accommodating user edits.SDS-based methods can require time-consuming, unstable optimization; multi-view methods can leave artifacts and struggle with complex structures; native 3D models have difficulty producing high-fidelity detail.Source: §1 · p. 2
- CraftsMan combines mesh-to-SDF preprocessing, 3D-native diffusion, and interactive mesh refinement to generate high-fidelity geometries with varied shapes, regular topology, and detailed surfaces.The refiner supports interactive manipulation to better align generated geometry with users’ envisioned designs.Source: §1 · p. 2
- The mesh-to-SDF preprocessing pipeline integrates visibility checks with winding-number analysis to improve watertight conversion success and use existing 3D data.Source: §1 · p. 2
- The 3D-native diffusion model generates coarse geometry, which a normal-based refiner enhances with surface detail using Poisson Normal Blending and Relative Laplacian Smoothing regularization.Source: §1 · p. 2
- The authors report that the 3D Native DiT produces superior-quality 3D assets compared with existing methods, while the refiner supports interactive manipulation.The stated goal of interactive manipulation is to better align coarse geometry with users’ envisioned designs.Source: §1 · p. 2
2. Related work
This section reviews 3D generation using 2D supervision and 3D-native generative models. It describes limitations in optimization cost, reconstruction quality, geometric detail, and training data conversion.
2.1. 3D Generation using 2D Supervision
- 2D-supervised methods lift image-generation priors to 3D, but category-limited views can impair generalization, while per-shape optimization can take dozens of minutes and require substantial computation.Source: §2.1 · p. 2
- These methods generate multiview images efficiently for 3D reconstruction, but dependence on 2D image quality and indirect 3D-distribution modeling can hinder faithful geometry recovery; self-occlusion, complex lighting, and view inconsistency remain challenging.Figure 2 shows the approach producing accurate complex geometric structures, including those self-occluded in input images, compared with SOTA reconstruction-based models.Source: §2.1 · p. 3, Figure 2 · p. 3
2.2. 3D Native Generative Models
- Native 3D generative models use point clouds, meshes, or implicit functions; direct 3D-data training is challenging due to high memory and computational costs, while latent-space diffusion models may lack geometric detail, over-smooth surfaces, and train unstably.Source: §2.2 · p. 3
3. Method
CraftsMan follows a coarse-to-refined workflow: it preprocesses meshes, generates geometry with a multi-view-conditioned latent-set diffusion model, then refines normals for detail and edits.
3.2. Multi-view guided 3D generation model
- The latent-set DiT is conditioned on generated multi-view images, with camera-aware features, to generate coarse geometry from text or image inputs.The input prompt or image first goes through a multi-view diffusion model; camera parameters modulate feature extraction through adaptive layer normalization.Source: Figure 4 · p. 4, §3.2 · p. 4
3.1. Data Preprocessing.
- Watertight conversion success rises from 60% to 80% using visibility checks plus winding-number tests for occluded grid cells.The method uses 50 sphere cameras, a 256- or 512-resolution grid, and a default winding-number threshold of 0.75.Source: §3.1 · p. 4
3.2. Multi-view guided 3D generation model
- The Perceiver-based shape VAE maps surface points and normals into latent sets, then decodes a TSDF field with a 1/256 threshold.The threshold replaces the occupancy field for stable optimization and better performance.Source: §3.2 · p. 4
3.3. Normal-based Geometry Refinement
- Rendered coarse-mesh normals are detail-enhanced with ControlNet-Tile and training-free cross-view attention; users can mask regions for interactive edits.The mask marks the target regions to update during refinement.Source: §3.3 · pp. 5–6, Figure 6 · p. 5
- Poisson blending removes low-frequency normal changes that distort global shape, while relative Laplacian smoothing penalizes deviations from the coarse mesh’s initial Laplacian coordinates.The initial coarse mesh provides the prior for smoothing rather than forcing Laplacian coordinates to zero.Source: §3.3 · p. 6
4. Experiments
Experiments evaluate generation and refinement qualitatively and quantitatively, and use ablations to examine the effects of key design choices. The reported comparisons favor CraftsMan on complex-geometry generation and mesh refinement.
4.1. Implementation Details
- The shape auto-encoder has 185M parameters and the DiT latent set diffusion model has 500 million parameters; the latter is trained on 32 A800 GPUs.Training uses ground-truth multi-view images.Source: §4.1 · p. 6
4.2. Evaluation of Mesh Generation
- On multi-view generation, CraftsMan produces regular-topology coarse geometry for further detailing, while Shap-E, Michelangelo, and InstantMesh show different shape or detail shortcomings; CraftsMan also has better image-mesh consistency than Direct3D.The reported shortcomings include noisy or incomplete shapes, over-smoothing or ambiguity, and adhesive geometry lacking detail.Source: §4.2 · p. 7
- On GSO, CraftsMan surpasses generation-based methods and has comparable results to InstantMesh in shorter time; on a complex Objaverse subset, it outperforms InstantMesh.The GSO evaluation uses front-view images and reports Chamfer Distance and Volume IoU; the Objaverse subset includes self-occlusion.Source: §4.2 · p. 7
4.3. Evaluation of Mesh Refinement
- CraftsMan's mesh refinement achieves higher CLIP similarity and faster refinement than Fantasia3D and Latent-NeRF.The quantitative comparison uses 20 Objaverse objects with the same text descriptions; qualitative results report clear, coherent details without compromising the original mesh's structural integrity.Source: Table 3 · p. 8, §4.3 · p. 8
4.4. Ablation Study
- Camera-pose injection improves view differentiation and shape generation; the ablation reports the best Chamfer Distance and IoU performance for this approach.Without camera-pose information, the model is prone to generating geometries with incorrect orientations.Source: §4.4 · p. 8
- Poisson Fusion corrects the global bias from normal diffusion, while Relative Laplace regularization counters shape shrinkage and yields a more reasonable result.Without both regularizers, the ablation shows an oddly sharp head; Poisson Fusion alone corrects the bias but the shape still shrinks.Source: §4.4 · p. 8, Figure 10 · p. 8
5. Conclusion and Discussion
CraftsMan3D generates high-fidelity 3D meshes in 30 seconds by first creating coarse geometry and then refining its surface details. The authors identify Latent Set Diffusion controllability as an open issue and mesh texture generation as future work.
5. Conclusion and Discussion
- CraftsMan3D creates high-fidelity 3D meshes in 30 seconds through coarse geometry generation followed by surface-detail refinement.The conclusion describes the workflow as mimicking a craftsman's modeling process.Source: §5 · p. 8
- The controllability of the Latent Set Diffusion model warrants further investigation.Source: §5 · p. 8
- Generating textures for 3D meshes is identified as a promising direction for future research.Source: §5 · p. 8