Source-linked AI summary
MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers
Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, Matthias Nießner
TL;DR
MeshGPT addresses automated triangle-mesh generation by directly producing compact mesh representations rather than relying on less suitable tokenizations or alternative representations. It learns geometric embeddings and autoregressively generates triangle sequences, improving mesh-generation quality over state-of-the-art methods, including a 9% increase in shape coverage and a 30-point improvement in FID scores.
Problem
Automated triangle-mesh generation requires representations that preserve geometric patterns and neighboring-triangle information without the excessive sequence lengths caused by coordinate-based tokens.
Method
MeshGPT learns a vocabulary of quantized geometric triangle embeddings with graph convolutions, then trains a GPT-based transformer for autoregressive next-index prediction.
Results
9% increase in shape coverage and a 30-point improvement in FID scores demonstrate improved 3D mesh-generation quality across multiple ShapeNet categories.
Takeaways & Limitations
MeshGPT directly generates clean, coherent, compact meshes whose triangulation patterns more closely follow real artist-created meshes.
Takeaways & Limitations
Autoregressive sampling takes 30 to 90 seconds, and the tokenization may be less effective for scene-scale generation.
Abstract
from arXiv · showhide
We introduce MeshGPT, a new approach for generating triangle meshes that reflects the compactness typical of artist-created meshes, in contrast to dense triangle meshes extracted by iso-surfacing methods from neural fields. Inspired by recent advances in powerful large language models, we adopt a sequence-based approach to autoregressively generate triangle meshes as sequences of triangles. We first learn a vocabulary of latent quantized embeddings, using graph convolutions, which inform these embeddings of the local mesh geometry and topology. These embeddings are sequenced and decoded into triangles by a decoder, ensuring that they can effectively reconstruct the mesh. A transformer is then trained on this learned vocabulary to predict the index of the next embedding given previous embeddings. Once trained, our model can be autoregressively sampled to generate new triangle meshes, directly generating compact meshes with sharp edges, more closely imitating the efficient triangulation patterns of human-crafted meshes. MeshGPT demonstrates a notable improvement over state of the art mesh generation methods, with a 9% increase in shape coverage and a 30-point enhancement in FID scores across various categories.
1. Introduction
MeshGPT generates triangle meshes directly as sequences of triangles, targeting compact representations that preserve sharp details and curved boundaries. It addresses dense, artifact-prone meshes produced when other 3D representations require post-processing conversion.
- Triangle meshes provide coherent, controllable, compact surfaces that integrate directly into modern rendering pipelines.
- Existing voxel, point-cloud, and neural-field methods often require mesh conversion before downstream use.
- Post-processing neural fields with iso-surfacing can produce dense, over-tessellated meshes with oversmoothing and bumpy artifacts.
- MeshGPT proposes generating meshes directly as triangle sequences using latent geometric tokens and a GPT-inspired decoder-only transformer.
- Generated examples are compact while representing both sharp details and curved boundaries.
- 9% average shape coverage increase and 30-point FID improvement were reported across multiple ShapeNet categories.
2. Related Work
Prior 3D generation methods use voxels, point clouds, neural fields, patches, or partitions, but each has limitations for producing practical, detailed meshes. MeshGPT is positioned against these approaches as a direct mesh-generation method.
- Voxel-based 3D Shape Generation: Voxel methods suffer from gridlike artifacts and high memory requirements, limiting fine-detail and complex-geometry capture.
- Point Cloud Generation: Point clouds represent surface samples but are not ideal for downstream applications requiring 3D meshes.
- Neural Implicit Fields: Neural implicit-field methods support arbitrary topologies and resolutions but require conversion into meshes.
- 3D Mesh Generation: Direct mesh generation remains underexplored despite learning-based approaches using mesh structures, patches, and binary space partitioning.
- 3D Mesh Generation: Compared with PolyGen, MeshGPT uses one decoder-only network and learned triangle tokens instead of separate vertex-and-face sequence generators.
3. Method
MeshGPT learns compact geometric tokens from mesh neighborhoods and uses them to autoregressively generate triangle sequences. Its encoder-decoder tokenization captures local structure while shortening sequences for transformer modeling.
- 3.1. Learning Quantized Triangle Embeddings: MeshGPT learns a triangle vocabulary by extracting geometric features with graph convolutions, quantizing them, and decoding them through a 1D ResNet.
- 3.2. Mesh Generation with Transformers: A GPT-style decoder-only transformer predicts each next codebook index from preceding quantized embeddings, with start/end tokens and positional encodings.
- 3.1. Learning Quantized Triangle Embeddings: Direct coordinate tokenization creates 9N-length sequences and misses neighboring-triangle geometry and mesh-distribution priors.
- 3.1. Learning Quantized Triangle Embeddings: Graph convolutions treat mesh faces as nodes connected to neighboring faces, fusing neighborhood information into geometrically enriched embeddings.
- 3.1. Learning Quantized Triangle Embeddings: Residual quantization represents features with stacked codebook indices, reducing the representation to effectively D codes per face.
- 3.1. Learning Quantized Triangle Embeddings: The decoder predicts discretized coordinate distributions rather than continuous coordinates, reducing floating-face artifacts and improving reconstruction resemblance.
- 3.2. Mesh Generation with Transformers: Using |T| = DN tokens yields shorter sequences than naive 9N tokenization when D < 9.
4. Experiments
Experiments on ShapeNet evaluate MeshGPT using standard shape, visual, compactness, user-preference, novelty, completion, and ablation analyses. MeshGPT produces compact, detailed meshes, supports diverse and novel outputs, and benefits from its learned tokens, sequence compression, encoder features, and pretraining.
- Dataset and Metrics: MeshGPT is trained across all 55 ShapeNetV2 categories and fine-tuned on Chair, Table, Bench, and Lamp for reported evaluation.Training includes random shifts, scaling, and planar decimation augmentations.
- Quantitative Results: MeshGPT outperforms baselines on shape quality, visual quality, and compactness metrics in unconditional mesh generation.The comparison includes Polygen, BSPNet, AtlasNet, and GET3D, with GET3D also evaluated after QEM simplification to 400 faces.
- Qualitative Results: Qualitative comparisons show compact meshes with sharp geometric details and high geometric fidelity, whereas baselines often miss details, over-triangulate, or produce overly simple shapes.The comparisons cover Chair, Table, Bench, and Lamp meshes.
- User Study: 68% of users preferred MeshGPT over GET3D for shape quality, while 73% preferred its triangulation quality; it was also significantly preferred over AtlasNet, Polygen, and BSPNet.The user study involved 49 participants and 784 total responses.
- Novelty and Completion: MeshGPT generates both training-like and novel realistic-looking Chair shapes, and it can produce multiple possible completions for partial shapes.Novelty analysis used 500 generated shapes and Chamfer Distance distributions.
- Ablations: Ablations show that learned geometric tokens, sequence compression, per-vertex quantization, graph-encoder features, and large-scale pretraining each improve performance.Removing sequence compression causes context forgetting and repeated structures, while removing pretraining leads to overfitting and suboptimal performance.
- Limitations: Autoregressive sampling takes 30 to 90 seconds, and the learned tokenization may be less effective for scene-scale generation.The current model is limited to GPT2-medium because of computational resources, whereas larger models may benefit from more data and compute.
5. Conclusion
MeshGPT directly generates triangle meshes through a learned geometric vocabulary and autoregressive transformer. The resulting meshes are described as clean, coherent, compact, and closer to real-data triangulation patterns.
- MeshGPT outputs meshes directly as triangles rather than relying on a separate mesh-conversion stage.
- The method learns geometric embeddings over meshes and trains a transformer to autoregressively predict triangle sequences.
- Generated meshes are described as clean, coherent, compact, and more closely aligned with triangulation patterns in real data.
- The supplementary document adds implementation details, user-study information, and further qualitative and quantitative analyses.
A. Data
The experiments use ShapeNetV2 shapes processed by planar decimation and augmented during training. Decimation is selected using a Hausdorff-distance threshold, while scaling and jitter-shift diversify training shapes.
- All experiments use the ShapeNetV2 dataset, with planar decimation applied to each shape using Blender.
- For each original shape, the selected decimated version has Hausdorff distance closest to, but below, a preset threshold.
- Training applies independent axis scaling from 0.75 to 1.25, followed by normalization of the longest side to unit length.
- Jitter-shift augmentation ranges from −0.1 to 0.1 while maintaining meshes within the unit bounding box around the origin.
B.1. Architecture
The architecture encodes mesh-face geometry and topology with graph convolutions, quantizes vertex-associated features, reconstructs triangles with a sequence decoder, and models token sequences with a GPT-style transformer.
- Encoder: SAGEConv layers process a face graph whose node features include triangle coordinates, area, edge angles, and normals.
- Decoder: The decoder predicts nine discrete triangle coordinates across a 128^3 discretized space, and reconstruction combines reconstruction and commitment losses.
- Quantization: Residual vector quantization recursively approximates encoded features using ordered codebook embeddings in a coarse-to-fine manner.
- Per-vertex quantization: Face features are split among three vertices, with shared-vertex features averaged before residual quantization.
- Transformer: A GPT-2 medium transformer predicts the next codebook embedding in the mesh sequence using self-attention.
- Baseline: PolyGen is reimplemented with matched GPT2-medium architecture, pretraining, finetuning, and train-time augmentations for comparison.
C. User Study Details
The user study compares meshes from MeshGPT and baselines through pairwise preferences on shape quality and triangulation quality. The study uses randomly selected mesh pairs from Chair and Table categories.
- A Django-based web application presents the questionnaire interface for the user study.
- The study randomly selects 16 mesh pairs from each baseline and MeshGPT across the Chair and Table categories.
- Half of the pairs assess shape quality, while the other half assess triangulation quality.
- Participants choose the preferred sample for each stated comparison.
D. Shape Novelty Analysis
The analysis compares generated shapes with training-set shapes using nearest neighbors identified by Chamfer Distance, after normalizing both sets for scale and position.
- Top-3 nearest neighbors are identified for each generated shape using Chamfer Distance.The comparison covers all samples used in the main paper.
E. Additional Results
Additional results report unconditional generations, evaluation procedures, user-study setup, and encoder-decoder ablations, including comparisons of quantization and architecture choices.
- The user study presents random ground-truth shapes, then asks participants to rate shape quality and triangulation preference between two methods.
- Unconditional generations are shown across different ShapeNet categories.
- Encoder-decoder ablations measure triangle accuracy and test-set cross-entropy loss for choices involving quantization, encoders, graph convolutions, and decoders.Triangle accuracy requires all 9 face coordinates to be correctly predicted.
- Per-vertex tokenization repeats tokens when faces share vertices, producing sequences that are easier for the transformer to learn than wholly unique per-face sequences.