PaperlayerView exact revision on arXiv

Source-linked AI summary

MeshXL: Neural Coordinate Field for Generative 3D Foundation Models

Sijin Chen, Xin Chen, Anqi Pang, Xianfang Zeng, Wei Cheng, Yijun Fu, Fukun Yin, Yanru Wang, Zhibin Wang, Chi Zhang, Jingyi Yu, Gang Yu, Bin Fu, Tao Chen

arXiv:2405.20853v2cs.CV

TL;DR

High-quality mesh generation is difficult because meshes are unstructured and require accurate vertex locations and connectivity. MeshXL combines NeurCF coordinate embeddings and predefined ordering to generate meshes autoregressively, with reported gains from larger models and training data and strong results on multiple ShapeNet categories.

  • Problem

    High-quality mesh generation is challenging because meshes are unstructured graphs requiring accurate vertex locations and connectivity, while prior methods face redundancy, pipeline dependence, or reconstruction errors.

    Source: §1 · p. 2

  • Method

    MeshXL uses NeurCF, an explicit coordinate representation with implicit neural embeddings, and predefined ordering to train autoregressive transformers directly on mesh data.

    Source: §1 · p. 2, §5 · p. 5

  • Results

    MeshXL generates higher-quality meshes with larger models and large-scale pre-training data and surpasses prior arts on multiple ShapeNet categories.

    Source: §1 · p. 2, §8 · p. 13

  • Takeaways & Limitations

    NeurCF supports direct, end-to-end mesh modeling and MeshXL shows potential as a base model for conditional 3D asset generation.

    Source: §4 · p. 5, §8 · p. 13

    Sampling an 800-faced mesh requires generating 7,200 tokens, making autoregressive inference relatively slow.

    Source: §8 · p. 13

Abstract

from arXiv · show

The polygon mesh representation of 3D data exhibits great flexibility, fast rendering speed, and storage efficiency, which is widely preferred in various applications. However, given its unstructured graph representation, the direct generation of high-fidelity 3D meshes is challenging. Fortunately, with a pre-defined ordering strategy, 3D meshes can be represented as sequences, and the generation process can be seamlessly treated as an auto-regressive problem. In this paper, we validate the Neural Coordinate Field (NeurCF), an explicit coordinate representation with implicit neural embeddings, is a simple-yet-effective representation for large-scale sequential mesh modeling. After that, we present MeshXL, a family of generative pre-trained auto-regressive models, which addresses the process of 3D mesh generation with modern large language model approaches. Extensive experiments show that MeshXL is able to generate high-quality 3D meshes, and can also serve as foundation models for various down-stream applications.

1 Introduction

Direct high-quality mesh generation is difficult because meshes are unstructured graphs that require accurate vertex locations and connectivity. MeshXL uses NeurCF and a predefined ordering to model meshes autoregressively, with experiments reporting improved quality at larger scales and results surpassing prior methods on multiple ShapeNet categories.

1 Introduction

  • Direct high-quality mesh generation is challenging because meshes are unstructured graphs requiring accurate vertex locations and connectivity.Meshes offer flexibility and accuracy for sharp edges and flat or curved surfaces.Source: §1 · p. 2
  • Prior approaches use intermediate representations and remeshing, separate vertex and connectivity decoders, VQVAE reconstruction, or discrete diffusion for mesh generation.These methods include point-cloud, SDF, and multi-view-image pipelines, as well as PolyGen, MeshGPT, and PolyDiff.Source: §1 · p. 2
  • Point-cloud and voxel methods redundantly sample flat surfaces, reconstruction methods depend on multi-view generation quality, and VQVAE-based methods accumulate reconstruction errors.These limitations motivate direct mesh modeling.Source: §1 · p. 2
  • NeurCF represents explicit 3D coordinates with implicit neural embeddings, while predefined ordering turns mesh generation into an autoregressive task for direct, high-fidelity mesh generation.The approach trains directly on large-scale mesh data without intermediate 3D representations.Source: §1 · p. 2
  • MeshXL surpasses prior arts on multiple ShapeNet categories and generates higher-quality meshes with larger models and large-scale pre-training data.The training collection contains over 2.5 million meshes.Source: §1 · p. 2
  • The paper presents MeshXL as a family of generative pretrained transformers and identifies image- or text-conditioned mesh generation as downstream uses.The contributions also report delicate meshes compatible with existing texturing methods.Source: §1 · p. 2

2 Related Work

The related work reviews common 3D representations, approaches to 3D generation, and the development of 3D foundation models. It situates MeshXL among autoregressive methods and frames its focus as end-to-end mesh generation and scaling.

3D Representations

  • Point clouds represent spatial positions, meshes encode structure as graphs and polygons, 3D Gaussians represent discrete distributions, and NeRFs use learnable volumetric functions.The paper investigates NeurCF, an explicit spatial representation with implicit neural embeddings, for scalable 3D asset generation.Source: §2 · pp. 2–3

3D Generation

  • 3D generation work spans GANs, diffusion, multi-view reconstruction, and autoregressive methods that generate tokens or predict mesh vertices and connectivity.MeshXL explores explicit sequential modeling for high-fidelity mesh generation.Source: §2 · p. 3

3D Foundation Models

  • Large-scale 3D datasets support foundation models for understanding, reconstruction, and generation, with reported capabilities including generalization and instruction following.The paper tests mesh generation scaling and whether MeshXL can serve as a foundation model for downstream tasks.Source: §2 · p. 3

3 Data

MeshXL’s data collection combines four mesh datasets and applies filtering, decimation, and annotation procedures. The resulting collection contains approximately 2.5 million meshes for training and validation.

Data Sources

  • The collection uses about 51k ShapeNet V2 models split 9:1 by category, more than 800k Objaverse objects split 99:1, and selected Github and Thingiverse portions of Objaverse-XL split 99:1.Only a small proportion of 3D-FUTURE furniture meshes are used because their delicate designs contain many faces.Source: §3 · p. 3

Data collection and filtering

  • Meshes with fewer than 800 faces are retained if they have UV maps, are center-aligned, and occupy at least 10% of the rendered frame; meshes with 800–20,000 faces may be planar-decimated.The filtering and preprocessing pipeline leaves approximately 2.5 million data items.Source: §3 · p. 4

Planar Decimation Pipeline

  • Planar decimation quality is assessed using either a lower Hausdorff distance or similar rendered views.Source: §3 · p. 4

Collecting mesh-text pairs

  • For mesh-text pairs, each mesh is rendered from 12 views, annotated with CogVLM, and processed with Mistral-7B-Instruct to extract category and geometry information.Each mesh receives a category and 3 to 5 geometry descriptors.Source: §3 · p. 4

Collecting mesh-image pairs

  • For mesh-image pairs, the pipeline combines rendered multi-view images and depth with Stable Diffusion-augmented images, then manually cleans the generated images.CogVLM-produced sentences serve as prompts for the fine-tuned image-generation model.Source: §3 · p. 4

Data Statistics

  • The combined collection contains approximately 2.5 million 3D meshes from ShapeNet, 3D-FUTURE, Objaverse, and Objaverse-XL.The four sources are used to build the training and evaluation collection.Source: §3 · pp. 3–4, Table 1 · p. 4

4 Neural Coordinate Field

NeurCF encodes discretized mesh coordinates with learnable neural embeddings, and a predefined ordering turns the mesh into a sequence for autoregressive modeling. The representation is direct, end-to-end trainable, and designed to store flat surfaces efficiently.

Neural Coordinate Field

  • NeurCF encodes each discretized 3D coordinate by applying a learnable embedding to its x, y, and z coordinates.The coordinate embeddings are shared among axes, and face embeddings are composed from the embeddings of their vertices.Source: §4 · p. 4

Ordering

  • The ordering procedure normalizes coordinates into a unit cube, discretizes them, cyclically orders vertices within faces by z-y-x, then orders faces by their permuted coordinates.The ordering is intended to preserve face normal direction and yields a sequence of encoded faces.Source: §4 · p. 4

A Sequential Mesh Representation

  • A mesh can be reshaped into n·k·3 coordinate tokens, with hybrid polygon meshes supported through separate triangle and quadrilateral tokens and begin/end markers.An n-faced triangular mesh uses 9n tokens.Source: §4 · pp. 4–5

Comparisons

  • NeurCF directly represents meshes with trainable coordinate embeddings and is storage-efficient relative to voxel fields and point clouds because graph structures model flat surfaces naturally.The figure describes using NeurCF and a predefined ordering for autoregressive mesh generation.Source: §4 · p. 5, Figure 2 · p. 5

5 Method

MeshXL frames mesh generation as autoregressive sequence modeling with decoder-only transformers, then extends the model with image or text conditioning.

Architecture

  • MeshXL uses decoder-only transformers based on OPT, fine-tuned for next-coordinate prediction with newly initialized coordinate and position embeddings.The mesh is represented as a sequence, making generation autoregressive.Source: §5 · p. 5

Generative Pre-Training

  • MeshXL is trained with the standard next-token prediction loss over the mesh sequence.Each sequence receives a beginning-of-sequence token and an end-of-sequence token; inference uses top-k and top-p sampling for diverse outputs.Source: §5 · p. 5

X-to-Mesh Generation

  • For image or text conditioning, BERT or ViT encodes the input, and a Q-Former compresses its features into 32 learnable prefix tokens.At inference, MeshXL predicts mesh tokens after the fixed-length prefix.Source: §5 · p. 5

Scaling Up

  • As parameter count grows, MeshXL achieves lower validation perplexity after training from scratch on 150 billion tokens.The evaluated model sizes are 125M, 350M, and 1.3B.Source: §5 · p. 6, Figure 3 · p. 6

6 Experiments

The experiments evaluate MeshXL on unconditional and conditional mesh generation, scaling, completion, texturing, and visual quality. They report stronger ShapeNet results than a similarly sized MeshGPT model and consistent human preference over baselines.

6.1 Data, Metrics, and Implementation Details

  • Experiments use 2.5 million meshes, standard mesh-generation metrics, and MeshXL base models of 125M, 350M, and 1.3B parameters.Meshes are augmented by rotations and axis-wise scaling; implementation uses 128 A100 GPUs.Source: §6.1 · p. 6, Table 2 · p. 7

6.2 Evaluations and Comparisons

  • Mesh-XL (350M) out-performs similarly sized MeshGPT on ShapeNet, with higher COV, lower MMD, and 1-NNA closer to 50%.The comparison uses 1,000 samples per category after fine-tuning the pre-trained base model.Source: §6.2 · p. 7, Table 3 · p. 7
  • In 434 valid responses, participants consistently preferred MeshXL meshes over baselines across quality, artistic, and triangulation dimensions.Participants scored chair meshes from 0 to 5; ground-truth ShapeNet geometries were also scored as a baseline.Source: §6.2 · pp. 7–8, Table 4 · p. 8

6.3 Ablation Studies

  • With 50% of a mesh as input, MeshXL (1.3B) produces diverse completions of the remaining 50%.The illustrated completion examples are chairs and tables.Source: §6.3 · p. 8, Figure 4 · p. 8
  • MeshXL generates high-quality meshes from corresponding images or text as additional inputs.Figure 5 presents conditional generation examples.Source: §6.3 · p. 8, Figure 5 · p. 9
  • As MeshXL model size grows, Objaverse samples have 1-NNA closer to 50%, larger COV, and smaller JSD, indicating improving diversity and quality.The evaluated base models include 125M, 350M, and 1.3B parameters.Source: Table 2 · p. 7, §6.3 · p. 9, Table 5 · p. 9
  • Meshes produced by MeshXL can be textured with Paint3D and fit existing texturing methods to produce high-quality 3D assets.Paint3D is described as a coarse-to-fine texture-generation pipeline.Source: §6.3 · p. 9, Figure 6 · p. 9

6.4 Visualizations

  • Visualizations show diverse, high-quality meshes with both sharp edges and smooth surfaces across ShapeNet and Objaverse examples.The category-specific comparisons also show generated mesh normal vectors.Source: §6.4 · p. 10, Figure 7 · p. 10

7 Discussions

MeshXL directly generates meshes auto-regressively with decoder-only transformers, unlike PolyGen’s separate vertex and face transformers or MeshGPT’s VQVAE-token reconstruction pipeline. Its autoregressive design can also extend to causal language models beyond decoder-only transformers.

7 Discussions

  • Unlike PolyGen’s separate vertex and face transformers, MeshXL directly generates meshes auto-regressively with decoder-only transformers.PolyGen first generates vertices, then predicts connectivity among them.Source: §7 · p. 10
  • Unlike MeshGPT’s VQVAE-token reconstruction pipeline, MeshXL learns coordinate representations and outputs meshes directly end to end.MeshGPT first quantizes meshes into discrete tokens, then trains a transformer to generate them.Source: §7 · p. 10
  • MeshXL’s autoregressive design can extend to causal language models such as Mamba, RWKV, and xLSTM, not only decoder-only transformers.Source: §7 · p. 10

8 Limitations, Future Work, and Conclusions

MeshXL’s main limitation is slow autoregressive inference; the paper concludes that NeurCF supports high-fidelity mesh generation and that MeshXL’s performance improves with larger training sets and more parameters.

Limitations and Future Work

  • Generating 7,200 tokens for an 800-faced mesh makes MeshXL sampling relatively slow.The authors identify inference time as MeshXL’s main drawback and point to RNN-related methods and multiple-token prediction as possible ways to reduce inference cost.Source: §8 · p. 13

Conclusions

  • MeshXL performs better with larger-scale training data and increased model parameters.Source: §8 · p. 13
  • The authors present NeurCF as an explicit-coordinate representation with implicit neural embeddings and MeshXL as a family of pretrained models for high-fidelity mesh generation.Source: §8 · p. 13
  • MeshXL shows potential as a base model for conditional 3D asset generation.Source: §8 · p. 13

Related papers

Loading 2405.20853v2…