PaperlayerView exact revision on arXiv

Source-linked AI summary

EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generation

Jiaxiang Tang, Zhaoshuo Li, Zekun Hao, Xian Liu, Gang Zeng, Ming-Yu Liu, Qinsheng Zhang

arXiv:2409.18114v1cs.CV

TL;DR

Auto-regressive mesh generation faces limits in face count, surface resolution, and generalization. EdgeRunner combines compact mesh tokenization with an auto-regressive auto-encoder that maps variable-length meshes to fixed-length latents. The authors report high-quality, diverse generation and improved generalization across point-cloud and image-conditioned tasks.

  • Problem

    Prior auto-regressive mesh methods face limits in face count and vertex resolution, while often struggling to generalize beyond training domains for single-view image conditioning.

    Source: §1 · p. 2

  • Method

    EdgeRunner combines lossless, compact mesh tokenization with an auto-regressive auto-encoder that compresses variable-length triangular meshes into fixed-length latent codes for latent diffusion.

    Source: §1 · p. 3

  • Results

    The authors report high-quality, diverse artistic mesh generation from point clouds and single-view images, with improved generalization and robustness over previous methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The ArAE latent space is presented as a basis for image-to-mesh generation and a direction for scaling auto-regressive models toward practical mesh generation.

    Source: §5 · p. 11

    Generation success decreases and time increases as sequence length grows, and precise control is difficult because many plausible face topologies can represent the same surface.

    Source: §A.2.4 · p. 21

Abstract

from arXiv · show

Current auto-regressive mesh generation methods suffer from issues such as incompleteness, insufficient detail, and poor generalization. In this paper, we propose an Auto-regressive Auto-encoder (ArAE) model capable of generating high-quality 3D meshes with up to 4,000 faces at a spatial resolution of $512^3$. We introduce a novel mesh tokenization algorithm that efficiently compresses triangular meshes into 1D token sequences, significantly enhancing training efficiency. Furthermore, our model compresses variable-length triangular meshes into a fixed-length latent space, enabling training latent diffusion models for better generalization. Extensive experiments demonstrate the superior quality, diversity, and generalization capabilities of our model in both point cloud and image-conditioned mesh generation tasks.

1 INTRODUCTION

EdgeRunner targets limitations in auto-regressive mesh generation—face-count and resolution ceilings, and weak generalization—while preserving mesh topology. It combines compact mesh tokenization with a fixed-length latent model and reports high-quality, diverse generation across point-cloud and image-conditioned tasks.

1 INTRODUCTION

  • The authors identify mesh generation as a way to reduce the time and effort of 3D asset creation and make it more accessible to non-experts.They cite potential applications in digital gaming, virtual reality, and filmmaking.Source: §1 · p. 2
  • Prior auto-regressive methods often generate fewer than 1,600 faces, use 128^3 vertex grids, and struggle to generalize from single-view image conditioning.These limits constrain complex-object representation and surface accuracy.Source: §1 · p. 2
  • EdgeRunner’s EdgeBreaker-adapted tokenization supports lossless face compression, prevents flipped faces, and reduces long-range dependencies to facilitate learning.The paper reports that its tokenization reduces sequence length by 50%, improving training efficiency.Source: §1 · pp. 2–3
  • Its Auto-regressive Auto-encoder combines a lightweight encoder and auto-regressive decoder to compress variable-length triangular meshes into fixed-length latent codes.This fixed-length latent space can be used to train latent diffusion models conditioned on other modalities.Source: §1 · p. 3
  • EdgeRunner reports high-quality, diverse artistic meshes from point clouds and single-view images, with improved generalization and robustness over previous methods.The model supports up to 4,000 faces and vertex discretization at 512^3 resolution.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans optimization, feed-forward reconstruction, 3D diffusion, and auto-regressive mesh generation. These approaches differ in representation and conditioning, while mesh-generating methods seek to preserve topology and extend face-count and generalization limits.

2.1 OPTIMIZATION-BASED 3D GENERATION

  • Early 3D generation methods relied on SDS-based optimization due to limited 3D data; other approaches predict multi-view images for reconstruction, while Unique3D combines high-resolution multi-view diffusion and efficient mesh reconstruction for high-quality, fast image-to-3D generation.Source: §2.1 · p. 3

2.2 FEED-FORWARD 3D GENERATION

  • Feed-forward 3D methods grew with large-scale datasets; LRM demonstrated scalable triplane-NeRF regression and fast asset generation, but resulting meshes often lack quality and diversity.Later work explored multi-view inputs and more efficient 3D representations.Source: §2.2 · p. 3

2.3 DIFFUSION-BASED 3D GENERATION

  • 3D latent diffusion methods compress 3D representations for more efficient diffusion training, while CLAY demonstrated scaling and generalization across diverse input conditions.The paper’s pipeline uses ArAE to compress variable-length meshes into fixed-length latents for diffusion conditioned on inputs such as single-view images.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

2.4 AUTO-REGRESSIVE MESH GENERATION

  • Auto-regressive mesh methods directly model topology, but prior approaches faced face-count and generalization limits; EdgeRunner aims to extend face capacity and control target face count at inference.MeshAnythingV2 raised the maximum face count to 1,600, while earlier approaches were limited to fewer than 1,000 faces.Source: §2.4 · p. 4

3 EDGERUNNER

EdgeRunner uses compact traversal-based tokenization and an auto-regressive auto-encoder to represent variable-length meshes in fixed-length latent codes. The section also describes conditioning latent diffusion on single-view images.

3.1 COMPACT MESH TOKENIZATION

  • EdgeBreaker-based traversal shares edges between adjacent triangles, so each next triangle can be represented with one additional vertex instead of three.The sequence uses B to start a traversal and N to mark the next twin traversal direction.Source: §3.1 · p. 5
  • The tokenizer averages 4 to 5 tokens per face, achieving approximately 50% compression versus the 9 tokens used in previous works, while maintaining consistent face orientation within each sub-mesh.Each token relies only on a short context of previous tokens.Source: §3.1 · p. 6

3.2 AUTO-REGRESSIVE AUTO-ENCODER

  • ArAE uses point-cloud geometry as encoder input and tokenized mesh sequences as decoder output to compress variable-length meshes into fixed-length latent codes.An L2 penalty constrains the latent-code range to make subsequent diffusion-model training easier.Source: §3.2 · pp. 6–7
  • Face-count tokens provide coarse control over generated mesh complexity, with buckets for ≤1,000, 1,000–2,000, 2,000–4,000, and >4,000 faces.Training sometimes replaces the count token with an unconditional token, so the model also learns generation without a specified face count.Source: §3.2 · p. 7

3.3 IMAGE-CONDITIONED LATENT DIFFUSION

  • The image-conditioned latent diffusion model uses a diffusion transformer, CLIP image features, cross-attention for image conditioning, and AdaLN for timestep information.It is trained with the DDPM framework and mean square error loss.Source: §3.3 · p. 7

4 EXPERIMENTS

The experiments report qualitative gains in point-cloud- and image-conditioned mesh generation, along with face-count control. They also examine tokenizer efficiency, inference speed, quantization resolution, and image-conditioning strategies.

4.1 QUALITATIVE RESULTS

  • On unseen test samples, point-cloud conditioning yields more stable, aesthetically pleasing meshes and better loop closure for watertight surfaces.The comparison uses dense meshes generated by other image-to-3D methods as test samples; the model uses only point coordinates and is trained from scratch.Source: §4.1 · p. 7, Figure 5 · p. 8
  • Image-conditioned meshes better capture input-image semantics, and the model generalizes to 2D-style or realistic-lighting images despite training on simple-shaded mesh renders.The image-generation comparison is with an optimization-based approach; Figure 6 shows generated surfaces without wireframes for Unique3D outputs.Source: §4.1 · pp. 7–8, Figure 6 · p. 9
  • Users can specify a face-count condition to generate meshes in different face-count ranges from the same point-cloud input.This provides coarse-grained control over mesh detail.Source: §4.1 · p. 8, Figure 7 · p. 10

4.2 QUANTITATIVE RESULTS

  • The tokenizer allows movement to either adjacent side, reducing subsequences while retaining a compression ratio comparable to AMT.It uses four tokens for a subsequent face versus AMT’s three, and the table reports a three-times-faster implementation.Source: §4.2 · p. 9, Table 1 · p. 10
  • On an A100 with flash-attention and KV cache, generation runs at about 100 tokens per second, taking about 45 seconds for 1,000 faces and 3 minutes for 4,000.A 2,000-face mesh takes about 90 seconds.Source: §4.2 · p. 9

4.3 ABLATION STUDIES

  • Increasing quantization resolution from 128^3 to 512^3 improved vertex-position accuracy and surface smoothness, while direct image-conditioned autoregressive decoding converged less easily and generalized worse.The final model uses 512^3; the alternative uses CLIP vision features as prepended condition tokens, whereas the proposed approach uses latent diffusion to map complex conditions to a fixed-length latent space.Source: §4.3 · p. 10, Figure 8 · p. 10

5 CONCLUSION

The conclusion presents ArAE’s fixed-length latent representation and mesh tokenization as approaches to scalable autoregressive mesh generation, and demonstrates the latent space’s potential for image-to-mesh generation.

5 CONCLUSION

  • ArAE compresses variable-length triangular meshes into fixed-length latent codes, and its tokenization supports longer context lengths and higher resolution for practical mesh generation.The paper also demonstrates the latent space’s potential for image-to-mesh generation.Source: §5 · p. 11

A.1.1 MESH TOKENIZER

The mesh tokenizer traverses discretized triangular meshes using half-edge notation and emits a token sequence. Its modifications use absolute vertex coordinates, avoid long-range dependencies at bifurcations, and simplify face-type tokens.

A.1.1 MESH TOKENIZER

  • The algorithm takes discretized vertices and triangle faces, outputs a token array, and tracks whether vertices and faces have been visited during traversal.A vertex’s three coordinates and a face’s half-edges are used in the notation.Source: §A.1.1 · pp. 17–18
  • Unlike original EdgeBreaker, the tokenizer writes absolute vertex coordinates rather than residual coordinates relative to the previous triangle.The paper says relative encoding is unnecessary for discrete token sequences and discretization reduces relative-coordinate accuracy.Source: §A.1.1 · p. 18
  • At bifurcations, the method duplicates necessary vertices before visiting the second branch, restarting a sub-mesh sequence to avoid long-range dependencies.The paper notes this slightly worsens compression rate.Source: §A.1.1 · p. 18
  • The modified tokenizer uses only L, R, and B face types, replacing C with L for training and merging E into B.The paper says holes and handles can be represented using degenerated faces that can be cleaned during post-processing.Source: §A.1.1 · p. 18

A.1.2 AUTO-REGRESSIVE AUTO-ENCODER

The ArAE uses a point-cloud encoder and autoregressive decoder to represent meshes with fixed-length latents and variable-length token sequences. Its training and inference use processed Objaverse data, mesh augmentations, and rules that constrain valid token sequences.

Datasets

  • The training set contains approximately 112K meshes with fewer than 4,000 faces, and approximately 44K higher-quality Objaverse meshes are selected for finetuning.Meshes are normalized to a unit cube after close vertices are merged and collapsed faces removed; vertices are quantized at 512^3 resolution.Source: §A.1.2 · p. 18

Mesh tokenization

  • The modified EdgeBreaker traversal is compared with the original, and the tokenization is described as reducing potential long-range dependencies between tokens.The traversal comparison is shown in Figure 10; Figure 11 illustrates the dependency reduction.Source: Figure 11 · p. 19, Figure 10 · p. 19

Training and Inference

  • Training uses 64 A100 GPUs for approximately one week, with mesh scaling, rotation, and random edge-collapse decimation augmentations; inference uses top-k 10 sampling and token-validity rules.At each iteration, decimation has a 50% chance of reducing the mesh to no more than one-quarter of its original face count; the next-token rules are detailed in the supplied continuation.Source: §A.1.2 · pp. 18–19

A.1.3 LATENT DIFFUSION

The latent diffusion model trains on Objaverse meshes paired with rendered images and uses a DiT conditioned on image features. Its architecture integrates condition and timestep information, and inference uses DDIM sampling.

Datasets

  • The image-conditioned training set comprises approximately 75K Objaverse meshes, each paired with RGB renders from 21 selected camera poses.The renders come from 38 available poses, of which 21 empirically selected views align with the front face of most objects.Source: §A.1.3 · p. 19

Architecture

  • The DiT uses 24 layers of self-attention and cross-attention, with CLIP-H image features supplied to cross-attention and timestep features supplied through AdaLN.The model has approximately 0.5 billion parameters, 1,024 hidden dimensions, and 16 attention heads.Source: §A.1.3 · p. 19
  • The ArAE latent code is used by the diffusion model to support conditioning on single-view images.Figure 12 depicts the ArAE encoder producing a fixed-length latent code and the diffusion model using it for more difficult conditions such as single-view images.Source: Figure 12 · p. 20

Training and Inference

  • DiT training uses min-SNR weighting of 5.0, while inference uses DDIM with 100 denoising steps and classifier-free guidance scale 7.5.Training uses 1,000 DDPM timesteps and randomly zeroes 10% of image conditions for unconditional training.Source: §A.1.3 · p. 20

A.2 MORE RESULTS

The additional results report user preference across evaluated mesh-quality aspects, correct face orientation, and valid sequence generation. The authors also identify capacity, sequence-length, and control limitations.

A.2.1 FACE ORIENTATION

  • EdgeRunner and MeshAnythingV2 ensure correct face orientation, while MeshAnything fails to do so.The comparison uses rendering with back-face culling.Source: §A.2.1 · p. 20

A.2.2 GENERATION PROGRESS

  • The model learns to generate valid sequences that the modified EdgeBreaker algorithm can detokenize.Figure 14 visualizes generation progress by simultaneous detokenization.Source: §A.2.2 · p. 21, Figure 14 · p. 21

A.2.3 USER STUDY

  • Across eight test cases, users preferred the method across all evaluated mesh-quality aspects on a 1-to-5 rating scale.The study showed renderings of both mesh geometry and wireframe; Table 2 reports the point-conditioned mesh-generation study.Source: §A.2.3 · p. 21, Table 2 · p. 21

A.2.4 LIMITATIONS

  • Meshes above the current capacity challenge the method; within its 4,000-face limit, longer sequences lower success rates and increase generation time, while precise topology control remains difficult.The authors attribute the control difficulty to the many plausible face topologies for the same surface.Source: §A.2.4 · p. 21, Figure 15 · p. 21

Related papers

Loading 2409.18114v1…