Source-linked AI summary
DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning
Ruowen Zhao, Junliang Ye, Zhengyi Wang, Guangce Liu, Yiwen Chen, Yikai Wang, Jun Zhu
TL;DR
Auto-regressive mesh generators face long token sequences, limited face counts, geometric defects, and limited human-preference alignment. DeepMesh addresses these issues with efficient tokenization and data strategies, then applies DPO-based reinforcement learning; it reports artist-like meshes with up to 30k faces and strong performance across metrics, while remaining limited by conditioning quality, data scale, and model size.
Problem
Auto-regressive mesh generation is limited by long token sequences, unstable training data, missing human-preference alignment, and incomplete or defective meshes.
Method
DeepMesh combines improved high-resolution mesh tokenization and pre-training strategies with DPO post-training based on human evaluations and 3D geometry metrics.
Results
DeepMesh generates diverse, high-quality artist-like meshes with up to 30k faces at quantization resolution 512 and outperforms existing methods across various metrics.
Takeaways & Limitations
The framework supports artist-like meshes with geometric complexity, intricate details, and refined topology under point-cloud and image-conditioned generation.
Takeaways & Limitations
Generation quality is constrained by low-level point-cloud features, limited 3D data, and the use of only a 1B-parameter model.
Abstract
from arXiv · showhide
Triangle meshes play a crucial role in 3D applications for efficient manipulation and rendering. While auto-regressive methods generate structured meshes by predicting discrete vertex tokens, they are often constrained by limited face counts and mesh incompleteness. To address these challenges, we propose DeepMesh, a framework that optimizes mesh generation through two key innovations: (1) an efficient pre-training strategy incorporating a novel tokenization algorithm, along with improvements in data curation and processing, and (2) the introduction of Reinforcement Learning (RL) into 3D mesh generation to achieve human preference alignment via Direct Preference Optimization (DPO). We design a scoring standard that combines human evaluation with 3D metrics to collect preference pairs for DPO, ensuring both visual appeal and geometric accuracy. Conditioned on point clouds and images, DeepMesh generates meshes with intricate details and precise topology, outperforming state-of-the-art methods in both precision and quality. Project page: https://zhaorw02.github.io/DeepMesh/
1. Introduction
DeepMesh addresses long-sequence pre-training, unstable data quality, and missing human-preference alignment in auto-regressive mesh generation. Its tokenization, data strategies, and DPO-based post-training support higher-resolution artist-like meshes.
- Challenges: Auto-regressive mesh generation faces long token sequences, unstable training from poor-quality meshes, absent preference alignment, and geometric defects.Reported defects include holes, missing parts, and redundant structures.
- Pre-training: 72% sequence-length reduction preserves geometry details and lowers training computation cost.The framework also uses data packaging and curation to improve loading, balancing, training stability, and data quality.
- Preference Alignment: DeepMesh introduces DPO for 3D auto-regressive models, using human evaluations and 3D geometry metrics to construct preference-labeled training pairs.The model is subsequently fine-tuned with reinforcement learning on these samples.
- Results: The framework generates diverse, high-quality artist-like meshes with up to 30k faces at quantization resolution 512.This extends generation beyond the limited face counts described among existing challenges.
- Contributions: The contributions combine efficient high-resolution mesh tokenization and pre-training strategies with DPO-based human-feedback alignment.These components target efficient training and artist-like mesh generation.
2. Related Work
Prior 3D generation methods often extract dense meshes without modeling topology, while auto-regressive approaches improve structure but remain limited by tokenization scalability. DeepMesh combines an auto-regressive transformer with improved tokenization and preference alignment.
- Prior 3D Generation: SDS, multi-view, and reconstruction methods generally generate 3D assets first and then extract dense meshes.The related work describes these pipelines as unable to model mesh topology directly.
- Auto-Regressive Mesh Generation: MeshGPT and subsequent auto-regressive methods predict structured mesh representations, but their tokenization schemes face compression and vocabulary-size trade-offs.These limitations restrict scalability to high-resolution meshes.
- DeepMesh: DeepMesh is an auto-regressive transformer with self-attention and cross-attention layers, pre-trained on discrete mesh tokens.The overview also includes improved tokenization and post-training with DPO using 5,000 preference pairs.
- Preference Alignment: RLHF uses a reward model and reinforcement learning, whereas DPO avoids the reward-model stage and is presented as easier to perform.The passage identifies 3D mesh generation as an application area not yet covered by prior DPO-based work.
3. Method
DeepMesh combines compressed mesh tokenization, curated and efficiently processed training data, an attention-based architecture, and DPO preference alignment. These components target high-resolution, geometrically complete, visually appealing, and diverse mesh generation.
- Tokenization Algorithm: Existing tokenization repeats shared vertices across connected faces, creating redundant and computationally costly sequences.Prior compression methods still produce relatively long sequences.
- Tokenization Algorithm: Localized face traversal, coordinate sorting, quantization, and XYZ flattening form DeepMesh’s compressed token sequence.The localized traversal reduces long-range dependencies between face tokens.
- Tokenization Algorithm: Approximately 72% compression reduces sequence length for high-poly datasets and is accompanied by a smaller vocabulary size.The passage reports improved training efficiency from these designs.
- Data Curation: The training dataset is approximately 500k meshes with an average face count of 8k, and curation filters poor geometry and chaotic structures.Curation targets irregular topology, excessive fragmentation, and extreme geometric complexity.
- Training Efficiency: Truncated training partitions long sequences into fixed-size context windows and shifts them with a sliding-window mechanism.This strategy addresses the long sequences produced by high-poly meshes.
- Model Architecture: DeepMesh uses an auto-regressive transformer with self-attention, cross-attention, and feed-forward layers, plus a jointly trained point-cloud perceiver encoder.The Hourglass Transformer saves 50% memory while maintaining performance.
- Direct Preference Optimization: DPO post-training aligns outputs with human preferences using preference pairs selected through geometric integrity and visual-appeal criteria.Chamfer Distance evaluates geometric similarity, while volunteers judge wireframes and surface details.
- Generation Results: Qualitative comparisons report better generated geometry and fine-grained detail preservation than baselines, with more faces in DeepMesh meshes.The comparison concerns point-cloud-conditioned generation.
4. Experiments
DeepMesh is evaluated on conditioned mesh generation, diversity, visual quality, geometry similarity, and tokenization efficiency. Across these experiments, it produces detailed, preferred meshes and achieves strong computational efficiency.
- Conditioned generation: DeepMesh generates aesthetically appealing artist-like meshes with more faces and fewer holes or missing components than MeshAnythingv2 and BPT in point-cloud-conditioned generation.The comparison attributes these gains to high-resolution tokenization, DPO preference alignment, and truncated training.
- Diversity: DeepMesh produces distinct meshes that remain consistent with the same input point cloud, supporting diverse design options.The diversity evaluation repeatedly conditions generation on identical point clouds.
- Quantitative results: DeepMesh outperforms the baselines in geometry similarity, and user-study results find its generated meshes most preferred.The evaluation uses Chamfer and Hausdorff distances alongside subjective visual-appeal assessment.
- Tokenization efficiency: The tokenization method balances a low compression ratio and small vocabulary size while achieving the highest computational efficiency among compared methods.Training-efficiency comparisons use 80 meshes for each tested face-count category.
- DPO ablation: DPO produces more visually appealing outputs than pre-training alone while maintaining excellent geometry.The ablation compares post-trained and pre-trained outputs using visual inspection and quantitative evaluation.
5. Conclusion
DeepMesh combines connectivity-aware mesh tokenization with reinforcement-learning-based preference alignment to generate detailed artist-like meshes. Its tokenization organizes local patches and hierarchical coordinate blocks, while experiments report faster training across face-count categories.
- Conclusion: DeepMesh generates artist-like meshes with reinforcement learning, combining geometric detail with refined topology and visual quality.The conclusion attributes these capabilities to improved tokenization, pre-training strategies, and DPO alignment.
- Tokenization algorithm: The tokenization algorithm traverses connected local patches to reduce redundancy and preserve edge-sharing relationships between adjacent faces.Each patch groups a center vertex with adjacent vertices ordered by connectivity.
- Coordinate representation: Vertices are normalized, quantized at resolution 512, flattened in XYZ order, and represented using hierarchical coordinate blocks.The hierarchy uses block volumes A = 4, B = 8, and C = 16.
- Sequence reduction: Adjacent vertices with identical hierarchical offsets are merged to shorten the coordinate sequence.This merging exploits shared block offsets among neighboring vertices after coordinate sorting.
- Training efficiency: The method achieves the fastest training time across 10K, 20K, 30K, and 40K face-count categories.The comparison integrates tokenization methods into the same model architecture and trains on 80 meshes per category.
B.1. Training Data Filtering Pipeline
The training-data filtering pipeline addresses irregular topology, fragmentation, and excessive geometric complexity through staged geometric, model-based, and aesthetic filtering. The resulting dataset is smaller and centered on higher-quality meshes.
- Data-quality challenges: Training data contains unstructured topology, fragmented surfaces, and overly complex structures such as tangled hair geometry.These quality variations motivate the filtering pipeline.
- Filtering pipeline: The filtering pipeline combines mesh-area screening, low-cost baseline pretraining, loss-based candidate selection, and aesthetic scoring.The four stages progressively remove fragmented or poorly performing samples while preserving visually strong complex meshes.
- Filtering pipeline: The aesthetic stage retains the top 20% of candidate meshes scored by a pretrained assessment model.This is intended to avoid mistakenly removing high-quality but complex meshes.
- Filtered dataset: The filtered dataset decreases from 800k meshes to approximately 500k, with an average face count of 8k.The reduction follows removal of poor-quality data.
- Preference data: Preference-pair point clouds come from both the training dataset and a manually selected high-quality test dataset.The mixed sources are used to provide diversity for human-preference learning.
B.3. More Training Details
DeepMesh uses small- and large-scale models trained with efficiency-oriented transformer and memory-saving techniques. Tokenization efficiency is evaluated under controlled single-GPU comparisons across multiple mesh face counts.
- Training setup: Both small-scale and large-scale DeepMesh models are trained for 100k iteration steps to ensure convergence.FlashAttention and Zero2 are used to reduce GPU memory usage.
- Model architecture: DeepMesh adopts Hourglass Transformers with a shortening factor of 3 and linear downsampling and upsampling layers.The hyperparameter settings are maintained from prior work.
- Efficiency evaluation: Tokenization methods are compared within the same model while other parameters remain unchanged, using a single GPU and dynamically adjusted batch sizes.The evaluation covers 80 meshes in each 10K, 20K, 30K, and 40K face-count category.
C.2. Data Curation
Data curation removes low-quality mesh samples that contribute to irregular loss values, producing more stable pre-training.
- Frequent early loss spikes suggest that some training samples disrupt learning through irregular loss values.The authors attribute these inconsistencies to poor mesh structures.
- Filtering low-quality samples mitigates inconsistencies caused by poor mesh structures.
- After curation, the training loss curve improves and pre-training becomes more stable.
D. Limitations and Future Work
DeepMesh’s quality is limited by point-cloud conditioning, limited 3D training data, and the use of a 1B model under constrained computational resources.
- Point-cloud conditioning limits recovery of fine-grained details from the original meshes.
- DeepMesh is trained on a limited number of 3D data, constraining the diversity of generated results.
- The framework uses only a 1B model because of limited computational resources.
- Future work could improve the point-cloud encoder, add salient point sampling, incorporate more datasets, and use a larger model.
E. More Results
Additional visualizations include more generated results and high-resolution renderings that expose finer mesh details, while Figure 11 shows the effect of data curation on training stability.
- Figures 12 and 13 provide additional visualization results generated by DeepMesh.
- Figures 14, 15, and 16 present selected cases as high-resolution renderings for inspecting finer details.
- After data curation, pre-training becomes significantly more stable than before curation, which shows frequent loss spikes.