PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Mesh Generation via Compressive Tokenization

Haohan Weng, Zibo Zhao, Biwen Lei, Xianghui Yang, Jian Liu, Zeqiang Lai, Zhuo Chen, Yuhong Liu, Jie Jiang, Chunchao Guo, Tong Zhang, Shenghua Gao, C. L. Philip Chen

arXiv:2411.07025v1cs.GRcs.CV

TL;DR

Mesh tokenization methods produce long sequences that limit learning from high-poly meshes, despite the value of detailed mesh data. BPT combines block-wise indexing and patch aggregation to compress sequences by around 75%, and the paper reports that this supports scaled-data mesh generation with improved performance.

  • Problem

    Existing mesh tokenization methods produce relatively long sequences, limiting generative models' ability to learn from high-poly meshes.

    Source: §1 · p. 2

  • Method

    BPT compresses mesh sequences through block-wise indexing and patch aggregation, and is used in a foundation mesh generative model conditioned on point clouds and images.

    Source: §1 · p. 2

  • Results

    Approximately 75% sequence-length reduction enables use of meshes with over 8k faces, allowing scaled training data that the paper says enhances generation performance.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The resulting model supports image- and point-cloud-conditioned generation of 3D assets described as product-ready.

    Source: §1 · p. 2

    The current 500M-parameter model is described as insufficient, and additional scaling experiments remain future work.

    Source: §7 · p. 8

Abstract

from arXiv · show

We propose a compressive yet effective mesh representation, Blocked and Patchified Tokenization (BPT), facilitating the generation of meshes exceeding 8k faces. BPT compresses mesh sequences by employing block-wise indexing and patch aggregation, reducing their length by approximately 75\% compared to the original sequences. This compression milestone unlocks the potential to utilize mesh data with significantly more faces, thereby enhancing detail richness and improving generation robustness. Empowered with the BPT, we have built a foundation mesh generative model training on scaled mesh data to support flexible control for point clouds and images. Our model demonstrates the capability to generate meshes with intricate details and accurate topology, achieving SoTA performance on mesh generation and reaching the level for direct product usage.

1. Introduction

Mesh creation with functional topology is labor-intensive, and existing generative methods remain limited to low-poly meshes. BPT compresses mesh sequences to support high-poly training, while the resulting foundation model supports conditional generation from images and point clouds.

1. Introduction

  • Increasing the maximum mesh face count improves generation performance, and BPT enables use of meshes exceeding 8k faces.Figure 2 measures generation performance by Hausdorff distance between input point clouds and generated meshes; lower is better.Source: Figure 2 · p. 2
  • Mesh crafting with functional topologies demands substantial design effort, while prior autoregressive methods have shown promising results on low-poly generation.Existing tokenization methods still produce long sequences that limit generative models' ability to learn from high-poly meshes.Source: §1 · pp. 1–2
  • Around 75% shorter than vanilla mesh sequences, BPT achieves the paper's stated state-of-the-art compression ratio across existing tokenization methods.It combines block-wise coordinate indexing with aggregation of nearby faces into patches.Source: §1 · p. 2
  • The foundation model supports conditional mesh generation from point clouds and images, with the stated aim of enabling product-ready 3D assets.The paper describes the model as usable by unprofessional users.Source: §1 · p. 2

2. Scaling Data with Compressive Tokenization

The section motivates expanding training data to include high-poly meshes, which existing mesh-generative models struggle to use. It describes BPT's indexing and patch aggregation and reports improved performance as training face counts increase.

2. Scaling Data with Compressive Tokenization

  • BPT converts Cartesian coordinates to block-wise indexes and aggregates nearby faces into patches to compress the mesh sequence.Figure 3 describes these as vertex-level indexing and face-level aggregation.Source: Figure 3 · p. 3
  • BPT extends the training scope to meshes exceeding 8k faces, addressing the limited detail of predominantly low-poly training data.The passage describes high-poly meshes as a source of rich detail.Source: §2 · pp. 2–3
  • Training the same model on meshes with different maximum face counts showed a positive correlation between face count and generative-model performance; as mesh complexity increased, performance and robustness significantly improved.Source: §2 · p. 3

3. Blocked and Patchified Tokenization

BPT compresses mesh sequences through block-wise vertex indexing and face-patch aggregation, reducing sequence length while preserving topology. Its component mechanisms reduce token repetition and improve sequence locality.

3. Blocked and Patchified Tokenization

  • Approximately 75% compression while preserving mesh topology is achieved by BPT, which combines block-wise indexing and patch aggregation.Source: §3 · p. 3, §3.4 · p. 4

3.1. Preliminary

  • Meshes are represented as ordered triangular faces whose vertices have discretized 3D coordinates.Source: §3.1 · p. 3

3.2. Block-wise Indexing

  • Block-wise indexing represents each vertex with at most two tokens, yielding around 50% compression so far.The method partitions coordinates into blocks and indexes positions within each block; adjacent vertices in the same block can share its index.Source: §3.2 · pp. 3–4

3.3. Patchified Aggregation

  • Patch aggregation reduces center-vertex occurrences from an average of six to one and makes the sequence another 50% shorter.Faces connected to a selected center vertex are aggregated into a patch; other vertices are repeated by connected patches rather than connected faces in most cases.Source: §3.3 · p. 4

3.4. Properties of BPT

  • BPT combines reduced vertex repetition with local patches, and achieves the best AVD across context lengths t among the compared tokenizations.The paper reports that shorter sequences provide training and inference efficiency; AVD@t measures average vertex distance from the previous t vertices, with lower values better.Source: §3.4 · pp. 4–5, Figure 4 · p. 5

4. Mesh Generation based on BPT

The model uses an autoregressive Transformer with cross-attention to condition mesh generation on different modalities. Point clouds condition generation directly, while images are mapped to point-cloud latents for mesh generation.

4. Mesh Generation based on BPT

  • A standard autoregressive Transformer models token sequences, using cross-attention for conditioning on different inputs.Source: §4 · p. 5
  • Point-cloud conditioning uses a pretrained point-cloud encoder, with 4096 randomly selected points sampled as the condition for each iteration.The model samples 50k points from the mesh before selecting the 4096 condition points.Source: §4 · p. 5
  • For image-conditioned generation, a diffusion model maps image features to a point-cloud condition that feeds the trained point-cloud conditional mesh model.The diffusion model uses DiT and image features from a pretrained DINO encoder.Source: §4 · p. 5

5. Experiments

Experiments evaluate mesh generation conditioned on point clouds and images, alongside ablations of tokenization and training data. The results report strong conditional generation and show that training-mesh complexity and sequence truncation affect generation performance and robustness.

5.1. Experiment Settings

  • Training uses a mixture of public and licensed datasets, filtering poor-topology objects and fine-tuning large-scale pretraining on high-quality meshes.The two stages use around 1.5M meshes followed by 0.3M high-quality meshes; image comparisons are qualitative because comparison samples for Edgerunner are unavailable.Source: §5.1 · pp. 5–6

5.2. Point-cloud to Mesh Generation

  • The point-cloud-conditioned outputs recover dense-mesh details while maintaining high-quality topology.Meshes are generated from point clouds sampled from dense meshes.Source: Figure 5 · p. 6
  • The model achieves state-of-the-art Hausdorff and Chamfer distances against MeshAnything and MeshAnythingV2 in point-cloud conditional generation.The comparison uses around 500 dense meshes for testing; Table 2 presents the quantitative comparison.Source: §5.2 · p. 6, Table 2 · p. 6

5.3. Image to Mesh Generation

  • Against Edgerunner gallery examples, the model's image-conditioned meshes show more detail and follow the input images more faithfully.The comparison uses Edgerunner results downloaded from its gallery.Source: §5.3 · p. 7, Figure 6 · p. 7

5.4. Ablation Studies

  • Block size 8 and offset size 16 achieve the best generation performance in the ablation, balancing compression against vocabulary size.The experiment fixes the product of block and offset sizes at 128; overly small blocks make the prediction vocabulary unaffordable.Source: §5.4 · p. 7, Table 3 · p. 7
  • Training on truncated meshes with 4k or 8k context windows reduces robustness and makes incomplete meshes more likely.The study uses sliding-window attention at inference.Source: Figure 7 · p. 7, §5.4 · p. 8
  • Increasing the training meshes' maximum face count from 1600 through 3200 and 4800 to 8000 boosts mesh-generation performance.The study identifies scaling mesh data with more faces as crucial for generation performance.Source: §5.4 · p. 8, Figure 8 · p. 8

6. Related Works

Related work spans neural 3D representations and native mesh generation. The paper positions its approach among methods that optimize mesh representations to improve scalability and robustness.

3D generation with neural representation

  • Neural 3D methods use SDF grids, neural fields, diffusion models, or large reconstruction models, but converting their outputs to meshes can be non-trivial and produce dense, over-smooth meshes.Large Reconstruction Models train Transformer backbones on large-scale datasets to generate neural 3D representations.Source: §6 · p. 8

Native mesh generation

  • Native mesh methods range from patch, ellipsoid, graph, and binary-space-partition representations to autoregressive models, with recent work targeting more generic, scalable mesh generation.The paper groups its approach with concurrent work focused on optimizing mesh representations for scalability and robustness.Source: §6 · p. 8

7. Conclusion

The conclusion presents BPT as a mesh-tokenization improvement that compresses sequences and supports a foundational model conditioned on point clouds and images. It identifies the current model size as insufficient and proposes further scaling and architectural exploration.

7. Conclusion

  • BPT reduces mesh-token sequence length by approximately 75%, supporting meshes with over 8k faces and scaled training data.The paper links this scaling to enhanced generation performance.Source: §7 · p. 8
  • The current 500M-parameter model is still insufficient; future work will scale BPT experiments and explore other sequence-modeling architectures.The paper aims to better utilize mesh inductive bias through those architectures.Source: §7 · p. 8

A. Data Curation

Data curation filters meshes by tokenized sequence length to fit the Transformer context window and uses two-stage training to balance data scale with topology quality.

A. Data Curation

  • Filtering at the 9600-token context window uses almost all meshes under 5k faces and around 58% of those above 5k faces.Tokenized length can vary among meshes with the same face count because of patch aggregation and block compression.Source: A. Data Curation · p. 12, Figure 9 · p. 12
  • Two-stage training first uses around 1.5M large-scale meshes, then fine-tunes on 0.3M high-quality meshes without simple geometry.The stages trade off generalizability and topology quality; low-poly CAD meshes may help learn geometry priors but can have topology unlike human-crafted meshes.Source: A. Data Curation · p. 12

B. Model Architecture

The model follows Michelangelo’s architecture, using an auto-regressive transformer for point-cloud-conditioned mesh generation and an additional diffusion model for image-conditioned point-cloud features.

B. Model Architecture

C. Additional Results

The method generates appropriate topology from dense meshes, while remesh algorithms fail to capture geometry and produce poor topology. The architecture supports image- and point-cloud-conditioned mesh generation.

C. Additional Results

  • Image-conditioned mesh generation uses a diffusion model to generate point-cloud features for the mesh-generation transformer.Source: Figure 10 · p. 13
  • Point-cloud-conditioned mesh generation uses an auto-regressive transformer with cross-attention to point-cloud features.Source: Figure 10 · p. 13
  • The method generates appropriate topology from dense meshes, while remesh algorithms fail to capture geometry and produce poor topology.The generated meshes are described as product-ready.Source: C. Additional Results · p. 12, Figure 11 · p. 13

Related papers

Loading 2411.07025v1…