Source-linked AI summary
MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers
Yiwen Chen, Tong He, Di Huang, Weicai Ye, Sijin Chen, Jiaxiang Tang, Xin Chen, Zhongang Cai, Lei Yang, Gang Yu, Guosheng Lin, Chi Zhang
TL;DR
Automatically produced 3D assets are difficult to use in mesh-based industry pipelines because existing extraction methods produce dense meshes with inferior topology. MeshAnything treats extraction as shape-conditioned AM generation, combining a VQ-VAE with an autoregressive transformer. It produces substantially fewer-face meshes with precision comparable to previous methods, while remaining limited by a maximum face count and lower stability than reconstruction-based extraction.
Problem
Existing mesh extraction produces dense meshes that ignore geometric features, resulting in inferior topology and inefficiencies for storage, rendering, simulation, and post-processing.
Method
MeshAnything learns a mesh vocabulary with a VQ-VAE and uses a shape-conditioned decoder-only transformer to autoregressively generate Artist-Created Meshes aligned with input shape information.
Results
MeshAnything generates Artist-Created Meshes with significantly fewer faces and refined topology while achieving precision close to or comparable with previous methods.
Takeaways & Limitations
Because it converts diverse 3D representations into Artist-Created Meshes, MeshAnything can integrate with varied 3D asset-production pipelines for 3D industry applications.
Takeaways & Limitations
MeshAnything cannot exceed its maximum face-count limit for large scenes or particularly complex objects and is less stable than reconstruction-based mesh extraction.
Abstract
from arXiv · showhide
Recently, 3D assets created via reconstruction and generation have matched the quality of manually crafted assets, highlighting their potential for replacement. However, this potential is largely unrealized because these assets always need to be converted to meshes for 3D industry applications, and the meshes produced by current mesh extraction methods are significantly inferior to Artist-Created Meshes (AMs), i.e., meshes created by human artists. Specifically, current mesh extraction methods rely on dense faces and ignore geometric features, leading to inefficiencies, complicated post-processing, and lower representation quality. To address these issues, we introduce MeshAnything, a model that treats mesh extraction as a generation problem, producing AMs aligned with specified shapes. By converting 3D assets in any 3D representation into AMs, MeshAnything can be integrated with various 3D asset production methods, thereby enhancing their application across the 3D industry. The architecture of MeshAnything comprises a VQ-VAE and a shape-conditioned decoder-only transformer. We first learn a mesh vocabulary using the VQ-VAE, then train the shape-conditioned decoder-only transformer on this vocabulary for shape-conditioned autoregressive mesh generation. Our extensive experiments show that our method generates AMs with hundreds of times fewer faces, significantly improving storage, rendering, and simulation efficiencies, while achieving precision comparable to previous methods.
1 INTRODUCTION
Automatically produced 3D assets can match manually created assets in shape and color, but their use in mesh-based industry pipelines is limited by dense, low-quality mesh extraction. MeshAnything instead generates artist-created meshes conditioned on shape, using a VQ-VAE, autoregressive transformer, and noise-resistant decoding.
- Motivation: Automatically produced 3D assets can achieve shape and color quality comparable to manually created assets, but their replacement potential remains largely unrealized in mesh-based pipelines.The industry relies on meshes for efficiency and controllability, while many production methods use alternative 3D representations.
- Motivation: Current extraction methods use dense faces while ignoring geometric characteristics, producing poorer topology, inefficient storage, rendering, simulation, and harder post-processing.The resulting meshes can contain several orders of magnitude more faces than artist-created meshes.
- Approach: MeshAnything formulates mesh extraction as generation, producing Artist-Created Meshes aligned with given 3D assets and suitable for industry pipelines.The proposed Shape-Conditioned AM Generation setting aims to mimic human-created mesh shape and topology quality.
- Approach: The model learns a mesh vocabulary with a VQ-VAE and trains a shape-conditioned decoder-only transformer for autoregressive mesh generation.Point-cloud shape features are projected into mesh-token space and placed at the beginning of mesh-token sequences.
- Results: MeshAnything generates meshes with significantly fewer faces and refined topology while achieving precision close to or comparable with previous methods.Experiments evaluate the model as a shape-conditioned AM-generation approach for integrating diverse 3D asset-production methods.
- Approach: A noise-resistant decoder injects shape conditions and is fine-tuned with noised token sequences to reduce the training–inference domain gap.This process is intended to make decoding robust to poor-quality sequences produced by the transformer.
2 RELATED WORKS
Prior mesh-extraction and mesh-generation methods generally produce dense meshes with inefficient topology or lack shape conditioning. MeshAnything differs by generating artist-created meshes through shape-conditioned generation.
- Mesh extraction: Mesh-extraction research includes isosurface extraction and gradient-based mesh optimization, with Marching Cubes a widely used isosurface method.Isosurface methods extract polygonal meshes representing scalar-function level sets, while gradient-based methods optimize neural networks under loss functions.
- Mesh generation: Data-driven and neural mesh-generation approaches still ignore shape characteristics or produce dense, low-quality topology that limits industry use.The related approaches therefore retain the same issues as earlier mesh-extraction methods.
- Mesh generation: MeshAnything formulates mesh extraction as generation and aims to mimic human artists by producing Artist-Created Meshes with hundreds of times fewer faces.This contrasts with approaches that do not produce efficient topology.
- Artist-created mesh generation: Existing AM-generation methods lack shape conditioning, so they must learn both complex 3D shape distributions and topological structures.MeshGPT combines a VQ-VAE with an autoregressive transformer but is limited to several ShapeNet categories.
3 SHAPE-CONDITIONED AM GENERATION
Shape-Conditioned AM Generation formulates mesh extraction as estimating the conditional distribution p(M|S), where meshes align with supplied 3D shape information. This conditioning avoids jointly learning complex shape and topology distributions, reducing training difficulty and enabling integration with diverse 3D pipelines.
- Training analysis: Shape-conditioned mesh generation has significantly lower training and validation PPL than unconditional and image-conditioned generation.The paper attributes the lower training burden to avoiding direct learning of the complex 3D shape distribution.
- Formal formulation: Shape-Conditioned AM Generation estimates p(M|S), producing Artist-Created Meshes aligned with input 3D shape information.S may be represented by voxels or point clouds and can come from reconstruction, generation, or scanning pipelines.
- Applications: The framework can connect 3D asset production methods that output shape information to Artist-Created Mesh generation.This includes pipelines based on reconstruction, generation, and scanning, using representations such as voxels or point clouds.
- Formal formulation: Existing AM generation estimates p(M|C), requiring models to learn both underlying shape and complex mesh topology.C can represent images, text, or empty sets for unconditional generation.
- Formal formulation: The conditional formulation targets shape-conditioned mesh generation rather than jointly modeling shape and topology distributions.The paper derives this setting using the chain rule and the stronger, more direct relationship between S and M.
4 METHOD
MeshAnything pairs shape conditions with Artist-Created Meshes, tokenizes meshes with a VQ-VAE, and generates mesh tokens autoregressively from point-cloud features. A shape-aware noise-resistant decoder improves reconstruction from imperfect generated tokens, while the transformer produces aligned meshes at inference.
- Shape condition strategy: The method requires paired shape conditions and Artist-Created Meshes, with shape conditions designed for extraction, augmentation, and efficient encoding.Point clouds are selected as an explicit, flexible representation, while high-resolution voxels are more expensive and less precise for this role.
- Shape condition strategy: Training point clouds are intentionally made coarser to reduce the domain gap between Artist-Created Meshes and automatically produced 3D assets.The pipeline extracts signed distance functions from AMs and converts them into coarse meshes before sampling point clouds.
- VQ-VAE with noise-resistant decoder: A VQ-VAE converts triangle-face sequences into quantized geometric tokens, which then serve as the vocabulary for autoregressive mesh generation.The VQ-VAE encodes face features, quantizes them with a codebook, and decodes reconstructed vertex coordinates; its encoder-decoder becomes a tokenizer and detokenizer.
- VQ-VAE with noise-resistant decoder: The decoder is fine-tuned with shape information and random Gumbel noise so it can refine meshes from imperfect transformer-generated token sequences.The experiments reported in Tables 3 and 4 show improved decoder noise resistance and mesh generation quality.
- Shape-conditioned autoregressive transformer: Point-cloud features are projected into the mesh-token space and prepended to mesh tokens as conditioning for the decoder-only transformer.At inference, the transformer generates mesh tokens from the point-cloud tokens, and the noise-resistant decoder reconstructs the final AM.
- Shape-conditioned autoregressive transformer: Table 1 compares MeshAnything with baseline AM generation and mesh extraction methods using human preference votes.The caption reports better alignment with human preferences against the baseline AM method and the most votes against mesh extraction baselines.
5 EXPERIMENTS
Experiments evaluate MeshAnything on diverse datasets, representations, and baselines. The results show strong mesh quality, topology, and user preference, with shape conditioning improving over prior generation methods.
- Data Selection: The combined dataset contains 51k Objaverse meshes and 5k ShapeNet meshes after filtering meshes above 800 faces and manually removing low-quality examples.Ten percent was held out for evaluation, with the remaining 90% used for training.
- Experimental Setup: MeshAnything generates meshes from text-to-3D, image-to-3D, NeRF, and Gaussian Splatting representations.Rodin provides text-to-3D and image-to-3D assets, while reconstruction pipelines provide NeRF and Gaussian Splatting models.
- Comparisons: MeshAnything is compared with both shape-conditioned mesh-generation and point-cloud-to-mesh extraction methods.The experiments include retrained mesh-generation baselines and a point-cloud extraction baseline, including Blender remeshing for simplified topology.
- User Study: 41 users provided 1,230 valid comparisons, and both user studies favored MeshAnything over mesh-generation and mesh-extraction baselines.The studies evaluated shape quality and topology quality using 30 results from each method.
- Quantitative Experiments: MeshAnything significantly outperforms prior mesh-generation methods across shape and topology quality, supporting shape-conditioned mesh generation as a more suitable setting.The retrained MeshGPT and MeshAnything differ only in whether shape conditioning is used.
6 CONCLUSION
The paper introduces Shape-Conditioned Artist-Created Mesh Generation and the MeshAnything model for converting diverse 3D assets into artist-created meshes. A noise-resistant decoder and extensive experiments support the method’s quality and potential for 3D industry applications.
- Conclusion: MeshAnything formulates improved mesh extraction and generation as Shape-Conditioned Artist-Created Mesh Generation.The model generates artist-created meshes that adhere to given 3D assets.
- Conclusion: MeshAnything converts assets from any 3D representation into artist-created meshes for integration with diverse 3D asset production methods.The stated goal is to facilitate application of these production methods in the 3D industry.
- Conclusion: The noise-resistant decoder improves generation quality by handling low-quality token sequences produced by autoregressive transformers.This decoder is introduced as an architectural component of MeshAnything.
- Conclusion: Extensive experiments demonstrate superior performance and highlight MeshAnything’s potential to scale toward 3D industry applications.The conclusion positions this performance as an advantage over previous methods.
A APPENDIX
Appendix results examine controllability and robustness of MeshAnything. They show integration with multiple production pipelines and the role of the noise-resistant decoder in improving mesh quality.
- Additional Qualitative Results: Additional qualitative results show highly controllable mesh generation when MeshAnything is integrated with various 3D production pipelines.The figure presents the method across multiple pipeline conditions.
- Qualitative Comparisons: MeshAnything can generate meshes with better topology and fewer faces than ground truth while preserving similar shapes.The comparison is reported for one qualitative setting in Figure 6.
- Qualitative Comparisons: A different generated topology with a similar shape is presented as evidence that the method does not simply overfit the ground-truth mesh structure.The result is interpreted as efficient topology construction conditioned on shape.
- Noise-Resistant Decoder: The noise-resistant decoder uses shape conditions to resist low-quality token sequences and produce higher-quality meshes.The ablation contrasts it with the ordinary VQ-VAE decoder under flawed autoregressive sequences.
A.1 METRICS
The appendix defines separate evaluation procedures for mesh extraction and mesh generation. These procedures measure geometric fidelity, topology, complexity, quality, and diversity under the reported decoder ablations.
- Evaluation Procedure: Mesh quality is evaluated by sampling 100K points from ground-truth and predicted mesh faces and computing reconstruction metrics.The sampling procedure supports assessment of multiple reconstruction properties.
- Decoder Ablation: Table 3 evaluates reconstruction performance under different noise levels with and without the noise-resistant decoder.The table directs readers to the appendix for metric explanations.
- Decoder Ablation: Table 4 reports the noise-resistant decoder ablation for mesh-generation quality.This isolates the decoder’s contribution within the mesh-generation evaluation.
- Mesh Extraction Metrics: Mesh-extraction evaluation reports CD, ECD, NC, number of vertices, and number of faces.CD measures overall reconstruction quality, ECD sharp-edge preservation, and NC surface-normal quality.
- Mesh Generation Metrics: Mesh-generation evaluation uses COV for diversity, MMD for nearest-neighbor distance, and 1-NNA for joint quality and diversity assessment.The passages note that COV and MMD each have specific limitations in evaluating output quality.
A.2 EXPERIMENTS
Experiments evaluate MeshAnything across diverse 3D representations, input-quality conditions, decoder designs, and mesh-extraction baselines. The results show efficient topology, robustness to moderate point-cloud noise, and improved reconstruction and shape-alignment performance from the noise-resistant conditional decoder.
- Additional Qualitative Experiments: MeshAnything does not simply overfit: conditioned on disrupted ground-truth shapes, it generates meshes with topology comparable to manually created meshes.Ground-truth meshes are converted into dense meshes with Marching Cubes before serving as conditions, disrupting their original face structure.
- Comparison with mesh extraction baselines: Hundreds of times more faces are required by mesh-extraction baselines to achieve results comparable to MeshAnything.Remesh methods also require substantially more vertices at similar face counts, indicating lower topology efficiency.
- Comparison with mesh extraction baselines: MeshAnything lags reconstruction-based methods in Chamfer Distance and Normal Consistency because generative failure cases reduce robustness.Shape-alignment metrics do not fully reflect the method’s topological advantages, and MeshAnything can produce fewer faces than ground truth.
- Ablations on Noise-Resistant Conditional Decoder: The noise-resistant decoder with shape conditioning achieves better reconstruction as input noise increases.The shape condition helps the decoder identify and correct imperfections in input token sequences.
- Ablations on Noise-Resistant Conditional Decoder: The noise-resistant decoder also improves shape alignment when generating meshes from dense meshes derived from corrupted ground-truth meshes.This evaluates the decoder during inference rather than only during reconstruction.
- Experiments on the Impact of Input Point Cloud Quality on Generated Results: MeshAnything tolerates moderate point-cloud noise, with a noticeable performance decrease only at the largest tested noise level.The model also tolerates generated point clouds, supporting integration with 3D generation models.
A.3 LIMITATIONS
MeshAnything has a maximum face-count limit and is therefore unable to convert large scenes and particularly complex objects. Its generative formulation is also less stable than reconstruction-based mesh extraction methods.
- MeshAnything cannot convert large scenes or particularly complex objects because it cannot generate meshes beyond its maximum face-count limit.
- MeshAnything is less stable than reconstruction-based mesh extraction methods because of its generative nature.
A.4 SOCIAL IMPACT
MeshAnything could reduce the labor cost of obtaining Artist-Created Meshes and support industries such as gaming, film, and the metaverse. The paper also notes a potential risk of criminal misuse.
- Automatically generating Artist-Created Meshes could significantly reduce labor costs in the 3D industry.
- Lower-cost access to Artist-Created Meshes could also enable potential criminal activities.