PaperlayerView exact revision on arXiv

Source-linked AI summary

Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scale

Zekun Hao, David W. Romero, Tsung-Yi Lin, Ming-Yu Liu

arXiv:2412.09548v1cs.GRcs.CV

TL;DR

Direct mesh generation remains limited in face count and coordinate resolution, despite the need to represent detailed realistic objects. MESHTRON combines a hierarchical autoregressive model with truncated-sequence training, sliding-window inference, and order-preserving sampling. It generates meshes up to 64K faces at 1024-level coordinate resolution while reporting lower training memory and higher throughput than existing methods.

  • Problem

    Prior direct mesh generators reach at most 1.6K faces, while realistic shapes often require more than 10K faces and artist-crafted meshes average 32K faces.

    Source: §1 · p. 2

  • Method

    MESHTRON is a point-cloud-conditioned autoregressive mesh generator that uses an Hourglass Transformer, truncated-sequence training with sliding-window inference, and order-preserving sampling.

    Source: §1 · pp. 2–3

  • Results

    MESHTRON generates meshes up to 64K faces at 1024-level vertex resolution, with over 50% lower training memory and 2.5× higher throughput than existing methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    MESHTRON generates high-quality artist-like meshes directly from point-cloud inputs, with control over mesh density and tessellation style.

    Source: §6 · p. 9

    Its low-level point-cloud conditioning makes MESHTRON struggle to add significant detail to degraded 3D shapes.

    Source: §5 · p. 9

Abstract

from arXiv · show

Meshes are fundamental representations of 3D surfaces. However, creating high-quality meshes is a labor-intensive task that requires significant time and expertise in 3D modeling. While a delicate object often requires over $10^4$ faces to be accurately modeled, recent attempts at generating artist-like meshes are limited to $1.6$K faces and heavy discretization of vertex coordinates. Hence, scaling both the maximum face count and vertex coordinate resolution is crucial to producing high-quality meshes of realistic, complex 3D objects. We present Meshtron, a novel autoregressive mesh generation model able to generate meshes with up to 64K faces at 1024-level coordinate resolution --over an order of magnitude higher face count and $8{\times}$ higher coordinate resolution than current state-of-the-art methods. Meshtron's scalability is driven by four key components: (1) an hourglass neural architecture, (2) truncated sequence training, (3) sliding window inference, (4) a robust sampling strategy that enforces the order of mesh sequences. This results in over $50{\%}$ less training memory, $2.5{\times}$ faster throughput, and better consistency than existing works. Meshtron generates meshes of detailed, complex 3D objects at unprecedented levels of resolution and fidelity, closely resembling those created by professional artists, and opening the door to more realistic generation of detailed 3D assets for animation, gaming, and virtual environments.

1 INTRODUCTION

Artist-created meshes encode topology and construction details needed for manipulation and editing, but AI-generated meshes still have a quality gap. Direct mesh generators remain too limited in scale for many realistic objects; MESHTRON addresses this with a scalable autoregressive model and reports large efficiency gains.

1 INTRODUCTION

  • Earlier mesh-generation pipelines often convert other 3D representations into meshes, but the resulting topology can be overly dense, oversmoothed, bumpy, and low in geometric detail.Meshes matter in part because their topology captures intrinsic properties and construction details, such as symmetry and edge flow that follows natural curvature.Source: §1 · p. 2
  • More than 10K faces are common in detailed realistic shapes, while prior direct mesh generators reach at most 1.6K faces and miss most of the artist-mesh distribution.The curated dataset has a mean of 32K faces and a median of 10K.Source: §1 · p. 2
  • MESHTRON combines a hierarchical Hourglass Transformer, truncated-sequence training with sliding-window inference, and order-preserving sampling to scale mesh generation.The Hourglass design uses mesh-sequence structure at coordinate, vertex, and face levels; the sampling strategy maintains realistic sequence structure.Source: §1 · pp. 2–3
  • Compared with iso-surfacing methods DMTet and FlexiCubes, MESHTRON is described as producing high-geometric-detail meshes with well-structured tessellation closer to artist-created meshes.The caption describes iso-surfaced meshes as often having dense tessellation, bumpy artifacts, oversmoothing, and insufficient geometric detail.Source: Figure 2 · p. 3
  • MESHTRON reports over 50% lower training memory and 2.5× higher throughput than existing methods, while generating meshes up to 64K faces at 1024-level vertex resolution.The model is a 1.1B-parameter autoregressive model trained with distributed data parallelism.Source: §1 · p. 3
  • A dataset of 1M artist-crafted meshes has a mean face count of 32K, and meshes with more faces tend to have smaller faces, motivating 1024-level rather than 128-level coordinate quantization.The figure caption states that this higher quantization is used to capture the finer details.Source: Figure 3 · p. 4

2 PRELIMINARIES – AUTOREGRESSIVE MESH GENERATION

The paper formulates mesh generation as sequence generation, representing meshes through faces, vertices, or coordinates. It orders mesh elements consistently and quantizes coordinates into discrete bins for autoregressive prediction.

2 PRELIMINARIES – AUTOREGRESSIVE MESH GENERATION

  • A mesh of N faces can be represented at face, vertex, or coordinate level, with sequence lengths N, 3N, and 3nN, respectively.For triangular faces, n=3.Source: §2 · p. 4
  • Autoregressive generation predicts each coordinate from the previously generated coordinates and represents the mesh probability as their joint probability.The passage gives the conditional form p(c_i|c_<i).Source: §2 · p. 4
  • The ordering convention sorts vertices in yzx order, sorts vertices within each face lexicographically, then sorts faces by their sorted vertex values.Source: §2 · p. 4
  • The sequence uses start-of-sequence and end-of-sequence tokens in groups of nine, while padding tokens equalize batched sequence lengths.Groups of nine preserve face- and vertex-level structure for the Hourglass model.Source: §2 · p. 4
  • Coordinates are quantized into discrete bins, with 1024 levels chosen to represent complex detailed meshes.The paper notes that higher quantization levels improve representation detail and accuracy while increasing generation complexity.Source: §2 · p. 4
  • The Hourglass Transformer’s shortened-stage tokens align with mesh vertices and faces, providing an inductive bias for mesh modeling.Figure 4 depicts two shortening stages, each with a factor of 3×.Source: Figure 4 · p. 5

3 MESHTRON– HIGH-FIDELITY MESH GENERATION AT SCALE

MESHTRON targets the quadratic cost of long mesh sequences by aligning its model with mesh structure and combining hierarchical processing, truncated training, and constrained sampling. Global cross-attention and a rolling cache support full-sequence generation from local training segments.

3.1 HIERARCHICAL MESH MODELING WITH HOURGLASS TRANSFORMERS

  • Mesh tokens form vertex and triangle hierarchies, and token difficulty varies within triangles because adjacent triangles share vertices.Earlier triangle tokens are easier to predict; the last vertex is less constrained and has greater uncertainty.Source: §3.1 · p. 5, Figure 5 · p. 6
  • The Hourglass Transformer uses causality-preserving shortening and upsampling, with static routing that sends only selected tokens through deeper stacks.MESHTRON uses two shortening layers, each reducing sequence length by a factor of three, to align stages with mesh token groups.Source: §3.1 · pp. 5–6

3.2 TRAINING ON TRUNCATED SEQUENCES AND INFERENCE WITH SLIDING-WINDOW

  • Because adjacent triangles are local in the ordered sequence, MESHTRON trains on fixed-length segments and generates with a rolling KV-cache.The cache gives linear complexity without recomputing previous contexts; cached embeddings can carry information from beyond the training window, but this discrepancy does not negatively affect performance.Source: §3.2 · p. 6, Figure 6 · p. 7

3.3 GLOBAL CONDITIONING ON TRUNCATED SEQUENCES WITH CROSS-ATTENTION

  • Cross-attention exposes every mesh segment to global conditioning, letting truncated-sequence training combine local and global information.This avoids the problem of prepending a condition, which would either expose it to only a few segments or require complex concatenation strategies.Source: §3.3 · p. 6

3.4 ROBUST MESH GENERATION WITH MESH SEQUENCE ORDERING ENFORCEMENT

  • Ordering constraints enforce lexicographic coordinate order within and across faces and restrict end-of-sequence tokens to face starts.The constraints prevent 32% of invalid predictions at 1024-level quantization and 27% at 128-level quantization.Source: §3.4 · p. 7

4 EXPERIMENTS

Experiments assess Hourglass efficiency, truncated-sequence training with sliding-window inference, and scaling to 64K faces. MESHTRON handles complex and out-of-distribution meshes better than MeshAnythingV2, while the reported experiments also find efficiency gains and benefits from truncated training.

4 EXPERIMENTS

  • The experiments validate components at 500M parameters on meshes up to 4096 faces and 128-level quantization before a 1.1B-parameter, 64K-face study.Training includes point-cloud augmentations intended to improve generalization to imperfect, non-artist meshes.Source: §4 · p. 7

4.1 HOURGLASS VS PLAIN TRANSFORMER

  • HG-8-8-8 matches or surpasses the plain Transformer at equal training iterations, saving 40% of memory and running almost twice as fast in training and inference.HG-4-8-12 achieves the best reconstruction accuracy at equal wall-clock duration, is 2.5× faster, and uses less than half the plain model’s memory; RoPE performs better than learnable positional embeddings.Source: §4.1 · p. 8, Table 1 · p. 8

4.2 GENERATING FULL MESH SEQUENCES WITH MODELS TRAINED ON TRUNCATED DATA

  • Models trained on sequences truncated to 1024 faces outperform full-sequence-trained models, while using over 50% less memory; omitting sliding-window attention degrades performance.Validation perplexity rises rapidly beyond the training length without sliding-window attention.Source: §4.2 · p. 8, Figure 7 · p. 9

4.3 SCALING MESHTRON TO 64K FACES

  • MESHTRON handles complex shapes better than MeshAnythingV2, uses 8× higher vertex-coordinate quantization for smoother meshes, and faithfully reproduces out-of-distribution non-artist inputs.The full-scale study uses a 1.1B-parameter model and meshes up to 64K faces with 1024-level coordinate quantization; evaluation includes artist-created, scanned, and generated meshes.Source: §4.3 · pp. 8–9

5 LIMITATIONS AND FUTURE WORK

MESHTRON remains limited by inference speed, low-level point-cloud conditioning, occasional long-sequence generation failures, and the scarcity of high-quality 3D training data.

5 LIMITATIONS AND FUTURE WORK

  • The largest model generates at 140 tokens per second, so producing large meshes still takes considerable time.The authors suggest efficient autoregressive models, speculative decoding, and advanced inference systems as possible acceleration directions.Source: §5 · p. 9
  • Low-level point-cloud conditioning limits the model’s ability to add significant detail to degraded 3D shapes.The authors propose higher-level signals such as text or high-resolution surface-normal maps as possible improvements.Source: §5 · p. 9
  • Autoregressive generation can still fail occasionally on long sequences, producing missing parts, holes, or non-termination.The authors describe robustness for long-sequence generation as underexplored.Source: §5 · p. 9
  • MESHTRON relies on large amounts of high-quality 3D training data, which remain scarce relative to other modalities.Using image or video data to support mesh generation is identified as a possible direction.Source: §5 · p. 9

6 CONCLUSION

MESHTRON generates point-cloud-conditioned, artist-like meshes with up to 64K faces and 1024-level coordinate resolution. The paper situates this approach among mesh-generation and efficient-Transformer work, and describes how its Hourglass design preserves causality.

6 CONCLUSION

  • MESHTRON generates artist-quality meshes with up to 64K faces at 1024-level coordinate resolution, with control over density and tessellation style.The conclusion describes the model as directly generating meshes from point-cloud inputs and highlights retopologization as a potential application.Source: §6 · p. 9, Figure 8 · p. 10, Figure 9 · p. 10

A EXTENDED RELATED WORK

  • Related 3D-generation work uses representations including NeRFs, signed distance fields, occupancy fields, Gaussian Splatting, and multichart representations.Many such methods require a further mesh-extraction step for downstream applications.Source: A EXTENDED RELATED WORK · p. 16
  • Prior mesh-generation approaches include direct learned tessellation, autoregressive models, diffusion, binary space partitioning, and CAD-command generation.MeshAnythingV2 is described as supporting meshes up to 1.6K faces; other related work uses iso-surfacing or alternative representations.Source: A EXTENDED RELATED WORK · p. 16
  • Efficient Transformer work includes sparse and sliding-window attention, while Hourglass reduces complexity through sequence shortening and causal upsampling.The paper describes the Hourglass approach as comparatively under-explored for reducing model complexity.Source: A EXTENDED RELATED WORK · pp. 16–17

B CAUSALITY PRESERVATION IN HOURGLASS TRANSFORMERS

C.1 MODEL ARCHITECTURE

MESHTRON uses a point-cloud encoder with an autoregressive Hourglass Transformer, with model depth and input point count varying by experiment.

C.1 MODEL ARCHITECTURE

  • The full-scale model uses a 12-layer point-cloud encoder with 16,384 input points and an HG-4-8-12 Hourglass configuration.Small-scale models use an 8-layer encoder and 8,192 input points.Source: §C.1 · p. 17
  • Small-scale Hourglass models keep layer and channel counts fixed while inserting shortening and upsampling layers.The passage also notes that further architectural details appear in Table C.1.Source: §C.1 · p. 17
  • Training uses a cosine learning-rate schedule from 1e−4 to 1e−5, 5K warm-up iterations, and weight decay of 1e−2.The implementation uses FlashAttention-2 and Transformer Engine.Source: §C.1 · p. 17

C.2 DATA HANDLING

The data pipeline filters the training meshes, triangulates quads while retaining their original proportion as conditioning, and prepares point-cloud inputs with surface-based sampling and augmentation.

C.2 DATA HANDLING

  • The curated training dataset contains 700K meshes with fewer than 64K faces, after filtering for geometric quality and removing scans, reconstructions, decimated meshes, and CAD meshes.Filtering uses rendered-image review, metadata keywords, and geometric heuristics.Source: §C.2 · p. 17
  • Meshes with quads are triangulated for training, while their original quad percentage is retained as conditioning to control generation of triangulated quad meshes.Source: §C.2 · p. 17
  • Point clouds are sampled from mesh surfaces using 20 icosahedron viewpoints, depth-map filtering, and farthest-point subsampling.The approach filters points from inner structures and is described as improving generalization to non-artist meshes that often lack internal structure.Source: §C.2 · p. 18
  • Adding noise to the conditioning point cloud lets MESHTRON trade full faithfulness, including retained imperfections, for creative detail enhancement and topology refinement.Source: Figure 11 · p. 18

C.3 EVALUATION PROTOCOL

For reconstruction evaluation, the ground-truth mesh face count supplies the conditioning input, and generation stops after exceeding twice that count.

C.3 EVALUATION PROTOCOL

  • Reconstruction evaluation conditions on the ground-truth mesh face count.Source: §C.3 · p. 18
  • Generation halts when the generated face count exceeds twice the specified face-count value.Source: §C.3 · p. 18

C.4 TRAINING ON LARGE MESH SEQUENCES BENEFIT SHORT MESH SEQUENCE GENERATION

The section tests whether training on longer mesh sequences helps generate shorter meshes and reports a small validation-perplexity improvement when training data extends to 8K faces.

C.4 TRAINING ON LARGE MESH SEQUENCES BENEFIT SHORT MESH SEQUENCE GENERATION

  • The experiment asks whether longer-sequence training can help smaller-mesh generation, despite earlier methods being capped at 800 or 1.6K faces.Source: §C.4 · p. 19
  • Training on data up to 8K faces, including sequences longer than the 4K-face target, improved performance on shorter sequences.The result suggests cropped-sequence training can help when generating shorter meshes.Source: §C.4 · p. 19
  • Mean validation perplexity on meshes up to 4K faces was 1.0668 for the model trained up to 8K faces versus 1.0671 for the model trained up to 4K faces.Lower perplexity is better; both models were evaluated on the same validation set of meshes up to 4K faces.Source: Figure 12 · p. 19, §C.4 · p. 19

C.5 MESHTRON’S ROBUSTNESS OVER POINT CLOUD DENSITIES

MESHTRON maintains high-quality mesh generation across variations in conditioning point-cloud density, including densities not observed during training. Higher densities can add detail, while lower densities may limit detail but still produce coherent, well-structured outputs.

C.5 MESHTRON’S ROBUSTNESS OVER POINT CLOUD DENSITIES

  • MESHTRON maintains high-quality generations when conditioning point-cloud density varies, including at point counts not observed during training.The figure description also reports robustness to variations in point count, including densities beyond those seen in training.Source: §C.5 · p. 19, Figure 14 · p. 20
  • Higher point densities can yield finer geometric detail, especially for complex geometries, while lower densities may limit detail but still produce coherent, well-structured outputs.The reported detail increase also occurs when the point-cloud density exceeds those observed during training.Source: §C.5 · p. 19
  • With rolling KV-cache, MESHTRON maintains a constant generation rate regardless of sequence length; without it, inference speed decreases rapidly as attention span grows.Source: Figure 13 · p. 19
  • Figure 14 reports robustness to changing conditioning point counts and notes that higher densities can add detail, particularly for complex geometries.Source: Figure 14 · p. 20
  • MESHTRON generates meshes up to 64K faces, while MeshGPT and MeshAnything generations are capped at 800 faces by design.The comparison describes MeshGPT and MeshAnything as two-stage models using a pretrained VQ-VAE tokenizer and an autoregressive model.Source: Figure 15 · p. 20

Related papers

Loading 2412.09548v1…