Source-linked AI summary
PolyGen: An Autoregressive Generative Model of 3D Meshes
Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, Peter W. Battaglia
TL;DR
Existing learning-based 3D synthesis often avoids directly modeling meshes because their unordered elements and discrete face structures challenge deep-learning architectures. PolyGen instead models vertices and faces autoregressively with Transformers and pointer networks, producing coherent, diverse, directly usable mesh samples while supporting class, image, and voxel conditioning.
Problem
Meshes are efficient and widely used for 3D geometry, but their unordered elements and discrete face structures make them challenging for deep-learning architectures, leading prior approaches to use alternative representations.
Method
PolyGen models mesh generation as autoregressive sequence modeling, generating vertices first and faces conditioned on those vertices with Transformers and pointer networks.
Results
PolyGen creates diverse, realistic geometry directly usable in graphics applications and supports conditional generation from object classes, images, and voxels.
Takeaways & Limitations
Direct probabilistic mesh modeling provides a unified approach for 3D content creation, computer vision, and robotics, while n-gons reduce mesh size and triangulation variability.
Takeaways & Limitations
The model uses 8-bit vertex quantization, while 14 bits or higher is typical for lossy mesh compression; extending to higher-resolution meshes remains future work.
Abstract
from arXiv · showhide
Polygon meshes are an efficient representation of 3D geometry, and are of central importance in computer graphics, robotics and games development. Existing learning-based approaches have avoided the challenges of working with 3D meshes, instead using alternative object representations that are more compatible with neural architectures and training approaches. We present an approach which models the mesh directly, predicting mesh vertices and faces sequentially using a Transformer-based architecture. Our model can condition on a range of inputs, including object classes, voxels, and images, and because the model is probabilistic it can produce samples that capture uncertainty in ambiguous scenarios. We show that the model is capable of producing high-quality, usable meshes, and establish log-likelihood benchmarks for the mesh-modelling task. We also evaluate the conditional models on surface reconstruction metrics against alternative methods, and demonstrate competitive performance despite not training directly on this task.
1. Introduction
PolyGen addresses the difficulty of learning directly from unordered mesh elements and discrete face structures by autoregressively modeling vertices and faces with Transformers. It generates diverse, realistic geometry that is directly usable in graphics applications.
- Motivation: Meshes are efficient, widely used 3D representations supporting graphics, games, film, virtual reality, computer vision, and robotics.They enable rapid creation of virtual-world objects and support planning and interaction in 3D space.
- Challenge: Meshes challenge deep-learning architectures because their elements are unordered and their face structures are discrete.Existing approaches therefore often use voxels, point clouds, occupancy functions, or surfaces instead.
- Challenge: Alternative representations leave mesh reconstruction to postprocessing, which can produce results of varying quality.PolyGen instead autoregressively estimates a joint distribution over mesh vertices and faces.
- Approach: PolyGen uses a vertex Transformer followed by a face model conditioned on the generated vertices.The vertex model predicts vertex coordinates sequentially, while the face model predicts vertex indices from the vertices and current face sequence.
- Evaluation: The model evaluates log-likelihood and predictive accuracy, supports class, image, and voxel conditioning, and produces diverse, realistic geometry usable in graphics applications.It also compares generated-sample statistics with real data and conditional models with existing mesh-generation methods.
2. PolyGen
PolyGen models polygon meshes directly by autoregressively generating vertices and then faces with Transformer-based components. It uses n-gon representations, probabilistic sampling, and conditioning mechanisms to generate compact meshes while addressing variable-length geometry and computational constraints.
- Mesh Modeling: PolyGen factors mesh generation into autoregressive vertex generation followed by face generation conditioned on the sampled vertices.Both stages can optionally condition on context such as object class, images, or voxels.
- n-gon Meshes: n-gons represent variable-size polygons, reducing mesh size and removing inconsistent triangulation choices across examples.Large flat surfaces can be represented with a single polygon, while triangle meshes remain a supported subset.
- Vertex Model: The vertex model orders vertices deterministically, flattens their coordinates into a sequence, and predicts the next coordinate or stopping token autoregressively.Vertices are ordered by z-coordinate, then y and x for ties, and the model is trained by maximizing observed-data log-probability.
- Vertex Model: A Transformer decoder captures long-range mesh dependencies, while 8-bit quantization models coordinates discretely with a trade-off between fidelity and mesh size.The paper notes that 14-bit or higher resolution is typical for lossy mesh compression and identifies higher-resolution meshes as future work.
- Face Model: The face model uses pointer networks to predict variable-length sequences of vertex indices, with additional tokens marking polygon and face termination.Its categorical output is defined over the input vertex indices plus end-polygon and stopping tokens.
- Masking Invalid Predictions: Masking invalid predictions improves evaluation log-likelihood, but applying masking during training slightly worsens performance, so training is performed without masking.The reported mechanism reallocates probability mass from invalid regions to valid values during evaluation.
3. Experiments
The experiments evaluate PolyGen’s modelling quality, sample distributions, conditional generation, and reconstruction performance. Results show strong unconditional modelling and competitive multi-sample reconstruction, with conditioning benefits concentrated in vertex prediction.
- Evaluation setup: Log-likelihood is the primary evaluation metric, complemented by predictive accuracy, generated-mesh statistics, and conditional chamfer distance.The study also compares unconditional models with uniform and Draco baselines and evaluates conditional models against existing mesh-generation methods.
- Unconditional modelling: Discrete vertex embeddings improve bits per vertex from 2.56 to 2.46, while data augmentation improves performance by 1.64 bits per vertex on average.Cross-attention harms face-model performance, whereas restricting predictions to valid values yields only a minor improvement.
- Unconditional modelling: 4.26 bits per vertex with 85% vertex and 90% face predictive accuracy is achieved by the best unconditional model.This model substantially outperforms uniform and Draco baselines.
- Sample statistics: PolyGen samples match the ShapeNet test distribution across mesh statistics, and nucleus sampling with top-p = 0.9 improves alignment for several statistics.The evaluated statistics include vertex and face counts, node degree, average face area, and average edge length.
- Conditional modelling: Voxel conditioning improves vertex-model performance most, followed by images and class labels, while conditional face models perform slightly worse than the best unconditional model.Global average pooling hurts vertex models but slightly benefits face models.
4. Related Work
PolyGen relates to generative models of point clouds, voxels, implicit functions, polygons, and graphs, while directly modelling both mesh geometry and connectivity.
- 3D generative representations: Prior 3D generative approaches model point clouds, voxels, signed distance functions, or other implicit functions rather than meshes directly.These representations differ in whether geometry is ordered, unordered, volumetric, or functional.
- Point-cloud models: PolyGen’s vertex model resembles PointGrow but handles variable-length vertex sequences with a deeper self-attention architecture.PointGrow operates on fixed-length point clouds and uses a comparatively shallow bespoke architecture.
- Sequence modelling: PolyGen borrows sequence-to-sequence training, Transformers, pointer networks, and autoregressive raw-data modelling ideas from natural-language and signal-generation research.These architectural connections motivate its sequential treatment of mesh data.
- Polygon models: Unlike Polygon-RNN, PolyGen operates in 3D and explicitly models the connectivity of multiple polygons.Polygon-RNN autoregressively outputs 2D coordinate sequences that implicitly define segmented regions.
- Graph models: Graph generative models focus on connectivity, whereas PolyGen models both vertex attributes and their incorporation into connectivity modelling.Meshes can be viewed as attributed graphs.
5. Conclusion
The paper presents PolyGen as an autoregressive Transformer-and-pointer-network model that generates coherent, diverse 3D mesh samples. It positions direct mesh generation as useful for computer vision, robotics, and 3D content creation.
- Conclusion: PolyGen poses mesh generation as autoregressive sequence modelling and combines Transformers with pointer networks for variable-length mesh sequences.The model is designed to generate both mesh geometry and polygon connectivity.
- Conclusion: The authors report coherent and diverse mesh samples and identify applications in computer vision, robotics, and 3D content creation.The conclusion frames these applications as a potential range enabled by the model.
- Data augmentation: The training pipeline creates 50 augmentations per input mesh while varying lighting, camera, and material properties during rendering.These augmentations and randomized rendering conditions support the experimental data pipeline.
A. Data Augmentation
The data-augmentation pipeline expands each ShapeNet mesh into 50 training variants using geometric transformations and variable planar decimation, with normalization applied throughout.
- Pipeline: Each ShapeNet mesh produces 50 augmented versions for training after normalization by its bounding-box diagonal.The mesh is renormalized after each augmentation, and augmentation occurs before vertex quantization.
- Axis scaling: Independent axis scaling samples sx, sy, and sz uniformly from [0.75, 1.25].The three coordinate axes receive separate scale factors.
- Piecewise linear warping: Piecewise-linear warping uses five sub-intervals with gradients sampled from a log-normal distribution of variance 0.5.For x and y coordinates, symmetry about zero is preserved by reflecting the warp.
- Planar mesh decimation: Planar mesh decimation creates n-gon meshes by merging adjacent faces whose surface-angle difference exceeds a sampled tolerance.Different tolerances produce meshes with different sizes and connectivity.
B. Rendering
The rendering setup uses Blender to generate image-conditioning data from meshes, while randomizing scene appearance and camera parameters to introduce visual variability.
- Blender renders 3D meshes for training image-conditional models.
- Meshes are placed at the origin and scaled so their bounding-box long diagonal is 1m.
- Lighting combines a fixed 20W area light with randomly sampled 15W point lights and independently sampled positions.
- Camera distance, elevation, rotation, focal length, and filter size are randomly sampled for rendered views.
- Because ShapeNet materials and textures were inconsistent, the authors randomly generated materials for each mesh texture group.
- A noise shader and color ramp interpolate each sampled input color toward white before assigning diffuse face materials.
C. Transformer blocks
The model uses Transformer-based vertex and face architectures, with residual-path normalization and conditioning mechanisms suited to global and high-dimensional inputs.
- Transformer blocks move layer normalization inside the residual path, with residual and intermediate representations processed by masked multihead attention and fully connected layers.
- The vertex model is a masked Transformer decoder that predicts coordinate locations and a stopping token from flattened vertex coordinates.
- The face model embeds vertices and face tokens, gathers embeddings by vertex index, and decodes successive vertex indices with masked attention.
- Class embeddings are projected and added to intermediate representations after self-attention, while image and voxel inputs use encoder-produced context sequences for cross-attention.
- Image and voxel encoders use pre-activation ResNets with two-dimensional and three-dimensional convolutions, respectively.
- The reconstruction baseline uses the same encoders, projects feature maps to 1024 dimensions, applies global average pooling, and decodes with a four-layer MLP.
- AtlasNet evaluation generates meshes by passing 2D triangulated meshes through patch models and sampling points on the resulting surfaces.
E. Alternative Vertex Models
Alternative vertex models trade computational efficiency against likelihood and performance: shorter-sequence variants train faster, while richer decoders improve likelihood or recover base-model performance.
- A coordinate-concatenation variant shortens self-attention sequences threefold and substantially improves training time, but significantly reduces performance.
- The coordinate-concatenation variant models each vertex with a mixture of 40 discretized logistics, replacing sequential coordinate processing.
- A MADE-style masked MLP removes the parametric distribution assumption and improves test data likelihood while increasing computation cost.
- Unlike the base model, the MADE decoder directly accesses only coordinate components within one vertex and relies on the torso for earlier vertices’ information.
- A six-layer Transformer decoder lets the model attend directly to all generated coordinates, producing performance closer to the base model than the MADE version.
- A two-layer Transformer embeds each coordinate triplet, allowing a ten-layer torso while achieving test likelihood similar to the base model.
F. Masking Invalid Predictions
The model masks predictions that violate hard constraints observed in the mesh data and redistributes the masked probability mass across valid alternatives.
- Invalid predictions are identified using hard data constraints and masked during model evaluation.
- The stopping token can occur only after an x-coordinate, while z-, y-, and x-coordinates obey hierarchical ordering constraints.
- New face tokens cannot repeat, and the first vertex index of each new face must not precede the previous face’s first index.
- Within a face, vertex indices must exceed the first index and remain unique.
- A new face’s first index cannot exceed the lowest unreferenced vertex index.
G. Draco Compression Settings
The comparison with Draco uses specific compression and quantization settings, but the resulting scores are only approximate because the quantization schemes differ.
- PolyGen is compared with Draco using its highest compression setting and both 8-bit quantized and unquantized position settings.These settings support comparison with the model’s 8-bit mesh representations.
- Draco’s reported scores are not directly comparable because its quantization differs from PolyGen’s uniform quantization.The authors therefore treat the results as a ballpark estimate of compression.
H. Unconditional Samples
PolyGen’s unconditional samples generally follow learned shape classes, but their distribution reflects ShapeNet imbalance and includes several recognizable failure modes.
- PolyGen’s unconditional samples mostly produce objects consistent with a shape class.The samples are generated with nucleus sampling at top-p = 0.9.
- Tables, chairs, and sofas constitute a large proportion of samples, reflecting ShapeNet’s class imbalance.Many other classes are underrepresented in the dataset.
- Some samples contain disconnected components, premature stopping, incomplete objects, or no recognizable class form.These are identified as failure modes in the random unconditional collection.