PaperlayerView exact revision on arXiv

Source-linked AI summary

OptimusMesh: Compact Autoregressive Mesh Generation from Point Clouds via Sparse Latent Pivots

Mazhar Iqbal, Naoya Chiba, Xuanmeng Sha, Tomohiro Mashita, Yuki Uranishi

arXiv:2610.01148v1cs.CV

TL;DR

Compact point-cloud mesh generation is challenging because mesh topology varies and extraction-based approaches can yield dense or over-smoothed meshes. OptimusMesh uses shared sparse pivots to condition autoregressive vertex and face generation. It produces 25.7%–94.1% fewer faces than compared recent autoregressive methods while maintaining competitive geometric fidelity and distributional quality.

  • Problem

    Meshes must be compact for memory, bandwidth, and real-time rendering, yet variable topology challenges generative models and extraction-based pipelines can yield dense or over-smoothed outputs requiring simplification.

    Source: §1 · p. 2

  • Method

    OptimusMesh compresses 2,048 oriented input points into 16 shared latent pivots that condition autoregressive vertex generation followed by face prediction.

    Source: §1 · p. 2

  • Results

    25.7%–94.1% fewer faces than compared recent autoregressive methods, while maintaining competitive geometric fidelity and distributional quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments support sparse geometric conditioning for direct compact point-cloud mesh generation.

    Source: §5 · p. 8

    Under the frozen quantitative-evaluation decoding configuration, early or late termination, redundant triangles, and thin-structure effects from quantization and vertex deduplication remain failure cases.

    Source: §11 · p. 13

Abstract

from arXiv · show

Generating compact and geometrically faithful 3D meshes directly from point clouds remains a fundamental challenge. Point clouds are unordered and sparse, whereas meshes exhibit irregular structure and varying topology. As a result, many existing approaches rely on implicit representations followed by surface extraction or reconstruction. Although effective, these pipelines can produce dense or over-smoothed meshes, often requiring computationally expensive post-processing and simplification. We present OptimusMesh, a framework for direct compact triangle mesh generation from point clouds using sparse latent pivot conditioning. Our key idea is to compress $2{,}048$ oriented input points into only $16$ sparse latent pivots, reducing the geometric conditioning set by $128\times$. These pivots provide a compact structural representation shared across a two-stage autoregressive framework that first generates mesh vertices and then predicts triangular faces conditioned on the generated vertices and the same pivots. Compared with the evaluated recent point-cloud-conditioned autoregressive methods, which use $257$ decoder-conditioning tokens, OptimusMesh uses only $16$, yielding a $16.1\times$ shorter conditioning sequence. Experiments show that OptimusMesh produces the most compact outputs among the compared recent autoregressive methods, using $25.7\%$--$94.1\%$ fewer faces while maintaining competitive geometric fidelity and distributional quality.

1. Introduction

OptimusMesh targets the less-explored efficiency of geometric conditioning for direct point-cloud mesh generation: 16 shared pivots guide autoregressive vertex then face prediction. It produces fewer faces than compared recent autoregressive methods while maintaining competitive geometric fidelity and distributional quality.

1. Introduction

  • Variable mesh topology challenges generative models, while extraction pipelines can yield dense or over-smoothed meshes; OptimusMesh shares sparse pivots across both generation stages.Its 16 pivots condition vertex generation and then face prediction alongside the generated vertices.Source: §1 · p. 2
  • 25.7%–94.1% fewer faces than compared recent autoregressive methods, while maintaining competitive geometric fidelity and distributional quality.The framework generates vertices followed by triangular faces, without implicit surface extraction or costly post-hoc simplification.Source: §1 · p. 2

2.1. 3D Shape Representations and Mesh Extraction

  • Implicit and other neural 3D representations commonly require surface extraction to produce explicit meshes, which can leave dense outputs or require further simplification.Meshes also need to remain compact for memory, bandwidth, and real-time rendering constraints.Source: §2.1 · p. 2, §1 · p. 2

2.2. Point Cloud to Mesh Reconstruction

  • BPA and PSR are sampling-sensitive, SAP [32] uses surface extraction, and Point2Mesh [12] optimizes each shape.OptimusMesh instead directly decodes compact meshes from point clouds.Source: §2.2 · p. 2

2.3. Autoregressive Mesh Generation

  • Prior autoregressive work improves mesh tokenization, topology modeling, and output sequencing; OptimusMesh instead targets input-conditioning efficiency.It compresses 2,048 oriented input points into 16 pivots, yielding a 16.1× shorter decoder-conditioning sequence than methods using 257 conditioning tokens.Source: §2.3 · p. 3

3. Methodology

OptimusMesh compresses an input point cloud into shared sparse pivots, then autoregressively generates mesh vertices and triangular connectivity. The pivots condition both stages of this direct mesh-generation pipeline.

3.1. Overview

  • The two-stage pipeline first generates vertices from sparse pivots, then predicts triangular connectivity from the generated vertices and those same pivots.It produces compact meshes directly, without dense surface extraction or post-processing.Source: §3.1 · p. 3

3.2. Sparse Latent Pivot Encoder

  • The SLIDE-style encoder reduces 2,048 oriented points to 16 latent pivots, each with a 3D coordinate and 45 learned feature channels.Its four Set Abstraction modules extract multi-scale features; a feature-transfer module maps them to the pivots, and the encoder is frozen during decoder training.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3. Autoregressive Vertex Model

  • The vertex model autoregressively predicts quantized coordinate tokens conditioned on pivot features supplied as cross-attention memory.Meshes are normalized to [−0.5, 0.5]^3, coordinates are quantized into 256 bins per axis, and generation ends with EOS.Source: §3.3 · p. 4

3.4. Autoregressive Face Model

  • The face model serializes each triangle as NEW TRI followed by three vertex indices and terminates the sequence with STOP.Vertex coordinates are encoded into contextual token embeddings, and a causal decoder predicts face tokens using pointer logits over the vertex-token table.Source: §3.4 · p. 5
  • At inference, slot-specific grammar masks restrict choices at triangle boundaries and vertex positions, and prevent repeated vertex indices within a triangle.PAD and padded vertex entries are masked during training; PAD is used only for batching.Source: §3.4 · p. 5
  • Face decoding combines the generated vertices with the same pivots, which provide global structural guidance alongside local vertex geometry.Training uses teacher forcing with ground-truth vertices and pivots; inference instead uses generated vertices and the same pivots.Source: §3.4 · p. 5

4. Experiments

Experiments compare OptimusMesh with reconstruction and autoregressive methods on ShapeNet, then show qualitative results on ShapeNet and Toys4K and ablate conditioning density. OptimusMesh combines compact outputs and short conditioning with strong geometric results; the ablation favors 16 pivots.

4.1. Experimental Setup

  • The mixed training set contains 35,907 meshes, with ShapeNet as the primary benchmark and distributional quality, paired fidelity, compactness, and efficiency evaluated.Targets are capped at 800 faces and 900 vertices; geometric conditioning uses 2,048 oriented surface points.Source: §4.1 · p. 6

4.3. Qualitative Results

4.2. Quantitative Results

  • On ShapeNet, OptimusMesh averages 141 vertices and 249 faces, has the lowest 1-NNA deviation, and beats reconstruction methods on MMD-CD and COV-CD.Source: Table 1 · p. 7, §4.2 · p. 7
  • OptimusMesh achieves the best CD-L1, HD, and NC among compared mesh generators, with the most compact outputs and shortest conditioning sequence.MeshRipple achieves the highest F1@2%.Source: Table 2 · p. 7, §4.2 · p. 7
  • OptimusMesh uses 16 decoder-conditioning tokens versus 257, a 16.1× shorter sequence, and is second fastest behind FastMesh.Source: §4.2 · p. 7, Table 3 · p. 8

4.4. Ablation Study

  • K=16 performs best across reported metrics with the shortest conditioning; K=32 does not improve reconstruction, while dense conditioning nearly doubles inference time.Source: Table 4 · p. 8, §4.4 · p. 8

5. Conclusion

OptimusMesh directly generates compact triangle meshes from point clouds through sparse-pivot conditioning and autoregressive vertex and face generation. It produces the most compact outputs and shortest conditioning sequence among compared methods while maintaining competitive geometric fidelity.

5. Conclusion

  • OptimusMesh produces the most compact outputs and shortest conditioning sequence among compared methods while maintaining competitive geometric fidelity.The method compresses 2,048 oriented input points into K=16 latent pivots and directly generates compact triangle meshes with bounded complexity.Source: §5 · p. 8

6. Additional Qualitative Results

Additional qualitative results show OptimusMesh output meshes alongside the generated vertex sets used before face prediction.

6. Additional Qualitative Results

  • Figure 6 pairs example output meshes with their corresponding vertex sets before face prediction.The figure illustrates conversion of predicted vertices into final triangle meshes.Source: §6 · p. 11

7. Dataset and Preprocessing Details

The training data combine ShapeNet, Objaverse, and Toys4K, with separate corpora for encoder and decoder training. Preprocessing prepares oriented point-cloud inputs and compact triangular mesh targets, with deterministic data splits described for Objaverse and Toys4K.

7.1. Dataset Composition

  • The encoder corpus is larger than the decoder corpus because decoder training additionally requires compact triangular targets meeting complexity constraints.The source-level train, validation, and test assignments are preserved through preprocessing.Source: §7.1 · p. 11

7.2. ShapeNet

  • ShapeNet uses 13 categories, with oriented point clouds containing surface positions and normals.Decoder targets are triangulated and planar-decimated; targets have at most 800 faces and 900 vertices.Source: §7.2 · p. 11

7.3. Objaverse

  • Objaverse retains naturally compact GLB assets with 50–800 faces and at most 900 vertices, without simplification.Objects are centered, scaled, and assigned deterministically to approximately 85%/5%/10% training, validation, and test partitions.Source: §7.3 · p. 11

7.4. Toys4K

  • Toys4K compact targets have 20–800 faces and at most 900 vertices, with selection constrained by bidirectional nearest-surface error.The 95th-percentile error must be at most 0.035 and maximum error at most 0.12; splits are approximately 85%/5%/10%.Source: §7.4 · p. 11

7.5. Point-Cloud and Decoder Processing

  • Across datasets, each object supplies 2,048 oriented surface points, and the frozen encoder produces K=16 latent pivots for decoder training.Decoder meshes are quantized and filtered; the vertex model supports at most 900 vertices, while the face model supports at most 800 triangles.Source: §7.5 · p. 11

8. Data Augmentation and Regularization

Stochastic augmentations are applied only during training, with distinct procedures for the sparse-pivot encoder and the vertex and face decoders.

8. Data Augmentation and Regularization

  • All stochastic augmentations are applied only during training.This bounds the augmentation procedures to training rather than validation or test processing.Source: §8 · p. 11

8.1. Sparse-Pivot Encoder

  • Encoder training uses anisotropic scaling, reflection, and translation, while farthest-point sampling initializes the 16 pivot locations.Scaling is up to 1.2, reflection has probability 0.5, and translation magnitude is 0.1.Source: §8.1 · p. 11

8.2. Vertex Decoder

  • Vertex training uses random compact-target selection, translation, token dropout, and corruption of 10% of teacher-forcing coordinate tokens.Corruption uses nearby quantized values with Gaussian noise for 80% of corrupted tokens and random coordinate symbols for 20%; mixed-data sampling follows a 50/40/10 ShapeNet/Objaverse/Toys4K ratio.Source: §8.2 · pp. 11–12

8.3. Face Decoder

  • Face training drops valid vertex-conditioning entries with probability 0.02 and uses Transformer dropout of 0.2, with canonical vertex and face ordering.The model uses AdamW with weight decay 0.01, warm-up followed by cosine decay, gradient clipping, and deterministic validation and test processing.Source: §8.3 · p. 12

9. Inference Sampling and Decoding

Inference uses constrained autoregressive sampling to generate vertices and then faces, followed by direct mesh export. A single frozen decoding configuration is used across reported experiments.

9. Inference Sampling and Decoding

  • Vertex generation uses temperature 0.8 and top-p 0.95, with EOS allowed only at complete vertex boundaries and a limit of 900 vertices.Before face decoding, vertices are clipped, quantized, deduplicated, ordered by (z, y, x), and dequantized.Source: §9 · p. 12
  • Face sampling uses temperature 0.55 and top-p 0.85, permits only valid choices at triangle and vertex-selection positions, and stops after at least 20 triangles or at 800 triangles.Repeated vertex indices are masked; STOP is enabled after the minimum triangle count.Source: §9 · p. 12
  • A ramped STOP bias and edge-consistency and triangle-area biases are applied during sampling, with one frozen decoding configuration and no per-object parameter or seed selection.The STOP bias is 0.18 with power 2.0; edge-consistency and triangle-area biases are 0.015 and 0.006.Source: §9 · pp. 12–13
  • Mesh export converts generated tokens directly into indexed triangles and removes degenerate or duplicate faces, invalid indices, non-finite values, and unreferenced vertices.No mesh simplification, surface reconstruction, test-time optimization, or ground-truth-based alignment is applied.Source: §9 · p. 13

10. Analysis of Sparse Latent Pivots

OptimusMesh compresses each input into 16 sparse latent pivots that share geometric conditioning across both decoders. Their locations span structurally distinct regions, but they are not a reconstructed point cloud.

10. Analysis of Sparse Latent Pivots

  • Each 2,048-point input becomes 16 sparse latent pivots, each with a 3D position and 45 learned feature channels.The same pivots condition both decoders, reducing the geometric conditioning set by 128×.Source: §10 · p. 13
  • The same 16 pivots condition both decoders, providing a shared compact geometric interface.This shared conditioning reduces the geometric conditioning set by 128×.Source: §10 · p. 13
  • Pivot locations remain distributed across structurally distinct regions rather than collapsing to a small area, but should not be read as a reconstructed point cloud.Each pivot also carries learned geometric features aggregated from the input.Source: §10 · p. 13

11. Decoding Failure Modes

Observed decoding failures include incomplete or redundant geometry and distorted local structure. Thin structures may also be affected by quantization and vertex deduplication.

11. Decoding Failure Modes

  • Representative failures include early termination with incomplete geometry, late termination with redundant triangles, and distorted local geometry.The examples use the same frozen decoding configuration as the quantitative evaluation.Source: §11 · p. 13, Figure 8 · p. 13
  • Thin structures may be affected by 8-bit quantization and vertex deduplication.The reported examples use the same frozen decoding configuration as the quantitative evaluation.Source: §11 · p. 13
  • The failure-case figure presents representative examples of decoding problems.The caption identifies early termination, incomplete connectivity, and distorted local geometry.Source: Figure 8 · p. 13
Loading 2610.01148v1…