PaperlayerView exact revision on arXiv

Source-linked AI summary

LoST: Level of Semantics Tokenization for 3D Shapes

Niladri Shekhar Dutt, Zifan Shi, Paul Guerrero, Chun-Hao Paul Huang, Duygu Ceylan, Niloy J. Mitra, Xuelin Chen

arXiv:2603.17995v1cs.CVcs.GRcs.LG

TL;DR

Optimal tokenization of 3D shapes for autoregressive generation remains an open question. LoST orders tokens by semantic salience and achieves state-of-the-art reconstruction, while its tokens support autoregressive models that significantly outperform state-of-the-art models using only 128 tokens at training and inference.

  • Problem

    Optimal tokenization of 3D shapes for autoregressive generation remains an open question, with recent methods using hierarchical or multi-resolution spatial encodings.

    Source: §1 · p. 2

  • Method

    LoST orders shape tokens by semantic salience so short prefixes decode to plausible complete shapes, while RIDA aligns triplane latents with DINO semantics.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    Using only 128 tokens at training and inference, LoST-based autoregressive models significantly outperform state-of-the-art models, while LoST achieves state-of-the-art geometric and semantic reconstruction.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LoST tokens also support semantic shape retrieval beyond autoregressive 3D generation.

    Source: §1 · p. 2

    LoST’s tokenizer and losses are currently instantiated on VAE triplane latents.

    Source: §5 · p. 8

Abstract

from arXiv · show

Tokenization is a fundamental technique in the generative modeling of various modalities. In particular, it plays a critical role in autoregressive (AR) models, which have recently emerged as a compelling option for 3D generation. However, optimal tokenization of 3D shapes remains an open question. State-of-the-art (SOTA) methods primarily rely on geometric level-of-detail (LoD) hierarchies, originally designed for rendering and compression. These spatial hierarchies are often token-inefficient and lack semantic coherence for AR modeling. We propose Level-of-Semantics Tokenization (LoST), which orders tokens by semantic salience, such that early prefixes decode into complete, plausible shapes that possess principal semantics, while subsequent tokens refine instance-specific geometric and semantic details. To train LoST, we introduce Relational Inter-Distance Alignment (RIDA), a novel 3D semantic alignment loss that aligns the relational structure of the 3D shape latent space with that of the semantic DINO feature space. Experiments show that LoST achieves SOTA reconstruction, surpassing previous LoD-based 3D shape tokenizers by large margins on both geometric and semantic reconstruction metrics. Moreover, LoST achieves efficient, high-quality AR 3D generation and enables downstream tasks like semantic retrieval, while using only 0.1%-10% of the tokens needed by prior AR models.

1. Introduction

LoST orders 3D shape tokens by semantic salience so short prefixes produce plausible, complete shapes and later tokens add instance-specific detail. It introduces RIDA for semantic alignment in 3D latent space and reports improved reconstruction and efficient autoregressive generation.

Introduction

  • Classical geometric LoD hierarchies, designed for rendering and compression, use many coarse-scale tokens and yield rough early decodings that make prefixes unusable.These limitations undermine sample efficiency and prevent early prefixes from resembling the final shape geometrically and semantically.Source: §1 · p. 2
  • LoST orders tokens by semantic salience so every prefix decodes to a complete, plausible shape with principal semantics, while longer prefixes refine instance-specific geometry and semantics.The introduction contrasts this behavior with early OctGPT [34] and VertexRegen [43] decodings, which it describes as geometrically and semantically implausible.Source: §1 · p. 2
  • A ViT-based shape encoder compresses triplane features into tokens, while a prefix decoder reconstructs latent features from any prefix using nested token dropout and causal masking [1] [35].A generative decoder enables plausible shapes at extreme compression rates [1] [35].Source: §1 · p. 2
  • RIDA trains a 3D semantics extractor to align inter-sample distances in triplane latent space with DINO semantic distances, rather than directly regressing DINO features from renderings.The approach is inspired by relational knowledge distillation [23].Source: §1 · p. 2
  • LoST sets SOTA reconstruction on geometric and semantic metrics, while its AR models outperform SOTA models using 128 tokens and only 0.1%–10% of prior 3D AR models’ token counts.The tokens also support semantic shape retrieval.Source: §1 · p. 2

2. Related Work

Prior 3D tokenizers shorten or structure representations in ways that still leave costs, fidelity, or prefix semantics unresolved. LoST draws on semantic token ordering to target prefixes that remain semantically relevant and geometrically refined.

3D Tokenization with Flat Element Streams

  • Flat mesh-element streams incur quadratic attention costs and exposure bias, while early prefixes often fail to decode into recognizable or semantically close shapes.PolyGen [21] autoregresses vertices and faces; MeshGPT [28] and MeshXL [5] use triangle tokens, while Llama-Mesh unifies 3D generation and understanding but retains similar problems.Source: §2 · pp. 2–3

Learned 3D Latent Token Sequences

  • Compact learned 3D latent sequences improve global coherence, but often require heavy upsamplers or diffusion for fidelity and do not guarantee semantically linked, complete prefixes.ShapeLLM-Omni [38] autoregressively predicts 3D VAE latent tokens but produces coarse voxels whose final refinement depends on diffusion synthesis.Source: §2 · p. 3

3D Tokenization with Geometric LoD

  • Geometric LoD encodings provide spatially coherent coarse-to-fine refinements, but allocate capacity to cells or edges rather than category-defining semantics, leaving short prefixes overly coarse and incomplete.Prior approaches include progressive meshes, octrees, VertexRegen [43], and OctGPT [34].Source: §2 · p. 3

Hierarchical Image and Video Tokenization

  • Image and video tokenizers demonstrate coarse-to-fine efficiency, controllability, and semantic ordering; LoST targets any-prefix decoding that is semantically relevant and geometrically refined.Relevant precedents include VQGAN [9], MaskGIT [4], MAGVIT-v2 [39], Matryoshka representation [17], FlexTok [1], and Semanticist [35].Source: §2 · p. 3

3. Method

LoST orders tokens by semantic salience so early prefixes capture plausible shapes and principal semantics, while later tokens refine instance-specific details. Its method combines a register-token encoder, a conditional diffusion decoder, RIDA-based semantic guidance, and continuous-token autoregressive modeling.

3. Method

  • LoST targets token sequences whose early prefixes produce plausible shapes with principal semantics and whose later tokens add instance-specific geometric and semantic detail.The paper presents Figure 2 as an overview of the method.Source: §3 · p. 3

3.1. LoST Encoder

  • The encoder uses ViT attention to summarize triplane tokens into learnable register tokens, with causal masking and nested dropout encouraging a hierarchy that front-loads principal semantics.Register tokens attend to the original triplane tokens, which are then discarded; training retains a randomly sized prefix, sampled from powers of 2.Source: §3.1 · p. 3

3.2. LoST Decoder

  • A conditional DiT generates the full latent sequence from any register-token prefix, shifting from semantically plausible generation toward reconstruction as the prefix grows.The model predicts added noise from noisy shape tokens while cross-attending to the retained prefix; deterministic decoding from very short prefixes is described as producing blurry, coarse reconstructions.Source: §3.2 · p. 4

3.3. Semantic Guidance for Learning LoS Tokens

  • Directly aligning 3D generator representations with 2D DINO features performs poorly, while aligning decoded-triplane multiview renders is computationally prohibitive.The paper attributes the mismatch to differences in spatial layout and representation dimensionality.Source: §3.3 · p. 4
  • RIDA pretrains a 3D semantic extractor by mapping triplane latents into a student feature space whose relational distances match DINO’s teacher space.DINO features are extracted from the generated images; a transformer encoder maps triplane encodings to a spatial student grid and pooled embedding, while teacher-guided mining identifies relationships to transfer.Source: §3.3 · pp. 4–5
  • The semantic extractor combines global relational contrast with inter-instance rank and spatial-structure distillation to transfer global and part-level relationships.The experiment weights these components as λglobal = 1.0, λrank = 1.0, and λspatial = 0.5.Source: §3.3 · p. 5
  • After RIDA pretraining, the semantic extractor guides the diffusion generator with a cosine-similarity loss, combined with the geometric denoising loss.The semantic loss compares the generator’s predicted latent with the ground-truth latent.Source: §3.3 · p. 5

3.4. LoST-GPT

  • LoST-GPT autoregressively models continuous register tokens with a GPT-style Transformer and diffusion loss, using prepended OpenCLIP embeddings for conditional generation.At each position, the Transformer predicts a conditioning vector and a small MLP-based diffusion head maps it to the token distribution.Source: §3.4 · p. 5

4. Experiments

LoST is evaluated for tokenization and autoregressive generation, with reported gains in reconstruction and generation quality while using substantially fewer tokens.

4. Experiments

  • LoST is trained on 300k Direct3D-generated shapes, created by lifting Flux.1-synthesized images from prompts generated with Gemini 2.5 Pro.The dataset was generated instead of using Objaverse to reduce preprocessing overhead and maintain compatibility with Direct3D.Source: §4 · p. 5

4.1. Tokenization Evaluation

  • Tokenizer reconstruction is tested on 1k unseen, complex shapes generated with independent Step1X-3D, comparing LoST with OctGPT and VertexRegen using CD, DINO similarity, and FID.The test shapes use a different representation and architecture from LoST’s training setup; semantic consistency uses DINO cosine similarity and FID on 2D renderings.Source: §4.1 · pp. 6–7
  • LoST significantly outperforms OctGPT and VertexRegen on DINO, FID, and CD, especially at low token budgets, using 0.1%-10% of their tokens; even 1-4 tokens often surpass them.The comparison uses the baselines’ recommended token hierarchy levels.Source: §4.1 · pp. 6–7, Table 1 · p. 7
  • A single LoST token decodes a complete, recognizable shape, while later tokens refine instance-specific details; simple shapes may need only 1-4 tokens.As token count increases, generation variance decreases as the task shifts from plausible generation to high-fidelity reconstruction.Source: Figure 3 · p. 6, §4.1 · p. 7

4.2. Autoregressive Generation Evaluation

  • LoST-GPT achieves the lowest FID and highest DINO semantic alignment among image-to-3D competitors, setting a state-of-the-art result while training on only 128 tokens.ShapeLLM-Omni uses a two-stage process with voxel prediction and Trellis refinement; DINO is omitted for text-to-3D OctGPT and Llama-Mesh.Source: §4.2 · pp. 7–8, Table 2 · p. 8

5. Conclusion

LoST orders 3D tokens by semantic salience so short prefixes decode complete, category-plausible shapes, while RIDA aligns triplane latents with DINO semantics without decoding and rendering. The conclusion also identifies limitations in representation scope, decoding cost, early-prefix quality, and generation length.

5. Conclusion

  • LoST orders tokens by semantic salience for complete, category-plausible short-prefix decodes, while RIDA couples triplane latents to DINO semantics without expensive decoding and rendering.The paper argues geometric level-of-detail is a poor organizing principle for next-token prediction.Source: §5 · p. 8
  • The tokenizer and losses currently use VAE triplane latents, while producing final latents with a diffusion decoder adds computational requirements over pure autoregressive decoding.Extending LoST to representations such as Gaussian Splats is proposed as a next step.Source: §5 · p. 8
  • Few-token decodes can still show artifacts versus full-length decodes; proposed future work includes topology-aware regularizers and part-consistency constraints.The paper notes that similar artifacts occur in 2D semantics-first tokenizers.Source: §5 · p. 8
  • Although LoST produces variable-length codes, the current autoregressive generator uses a fixed target length; an EOS token and complexity-aware stopping are proposed extensions.The proposed stopping strategy would use shorter sequences for simple shapes and longer ones for complex shapes.Source: §5 · p. 8

Supplementary Material · 6. Additional Qualitative Results

Supplemental visualizations show that LoST produces high-fidelity reconstructions and that its autoregressive model can generate complete, plausible 3D shapes from few tokens.

6. Additional Qualitative Results

  • The supplemental gallery shows high-fidelity reconstructions that visually surpass recent baselines and complete, plausible AR-generated shapes conditioned on few tokens, consistent with Table 2’s quantitative results.Source: §6 · p. 11

7. Shape Retrieval using RIDA

Shape retrieval evaluates whether RIDA transfers DINOv2 semantic topology into 3D triplanes, using DINOv2 similarity to define semantic neighbors. RIDA outperforms raw triplane latents on both in-distribution and out-of-distribution data.

7. Shape Retrieval using RIDA

  • The evaluation uses DINO similarity as ground truth on held-out in-distribution samples and Step1X-3D-generated OOD shapes.Retrieval quality is measured by Recall@K, mAP@K, and Jaccard Index.Source: §7 · p. 11
  • Direct regression of DINO features stagnates at an early validation-loss plateau and fails to capture discriminative semantic features.It uses the same transformer backbone as RIDA but replaces its relational objectives with spatial MSE regression.Source: §7 · p. 11
  • On the OOD set, RIDA raises mAP@3 from 17.47% to 28.28% over the geometric baseline.The evaluation set contains shapes generated by the unseen Step1X-3D model.Source: Table 3 · p. 11, §7 · p. 11
  • In-distribution, RIDA achieves 44.28% mAP versus 16.42% for triplanes and 22.90% for feature regression.The in-distribution results are critical for training LoST.Source: Table 3 · p. 11, §7 · p. 11

8. Ablation on RIDA

RIDA improves semantic reconstruction most in low-token regimes while preserving geometric quality, and it stabilizes training and accelerates decoder convergence.

8. Ablation on RIDA

  • RIDA regularizes the 3D latent space toward semantic embeddings, improving decoded-shape semantic consistency and stabilizing training under nested dropout.The diffusion decoder already tends toward plausible geometry, particularly at lower guidance scales.Source: §8 · p. 11
  • RIDA consistently improves DINO and DINOv2 semantic reconstruction, with the largest gains at low token budgets where short prefixes must encode meaningful structure.The reported Chamfer Distance remains similar with and without RIDA.Source: §8 · pp. 11–12, Table 4 · p. 12
  • RIDA accelerates diffusion-decoder convergence by approximately 40% compared with extended training without RIDA.Source: §8 · p. 12
  • A direct-regression baseline trained to predict DINO features fails to learn semantics accurately.Qualitative ablation results are provided on the supplemental webpage.Source: §8 · p. 12

9. Details about RIDA

RIDA transfers semantic relations across modalities by matching normalized similarity patterns rather than raw similarities, while separately distilling within-shape token affinities to preserve local geometric structure.

Inter-Instance Rank Distillation

  • RIDA’s inter-instance rank distillation z-scores each anchor’s student and teacher similarity rows, then minimizes their MSE because raw cross-modal cosine similarities fail under modality-specific mean and scale differences.This addresses the loss of continuous relational structure under hard-threshold contrastive separation and makes the comparison invariant to per-anchor affine transformations.Source: §9 · p. 12
  • Because z-scored-vector MSE is proportional to maximizing row-wise Pearson correlation, rank distillation optimizes relative neighborhood rankings as the semantic relation to transfer.The objective factors out each anchor’s mean and standard deviation.Source: §9 · p. 12

Spatial Structure Distillation

  • Spatial structure distillation applies row-wise softmax to student and teacher intra-instance token-affinity matrices, then minimizes per-token KL divergence to preserve relative affinities and local geometry.The affinity matrices are computed within each instance rather than directly matching token features with L2 distance.Source: §9 · p. 12

Final Objective

  • The final semantic pretraining objective weights global, rank, and spatial components by 1.0, 1.0, and 0.5, respectively.The resulting student encoder provides a semantically structured 3D latent space for the generative task.Source: §9 · p. 12

10. Extending LoST to other 3D Representations

LoST and RIDA are designed to be representation-agnostic, and the paper demonstrates LoST in the TRELLIS latent space as a flexible tokenizer for variable-length 3D generation. The section also describes its TRELLIS adaptation and evaluation scope.

10. Extending LoST to other 3D Representations

  • The TRELLIS adaptation reshapes the 16^3 voxel grid with feature dimension 8 into a 64^2 2D grid and uses 16-dimensional register tokens.It operates on TRELLIS Stage-1 latents, preserves the original architecture, and decodes variable-length representations through TRELLIS Stage-2 to recover high-frequency details.Source: §10 · p. 13
  • In TRELLIS Stage-1 latents, LoST demonstrates generalization beyond Direct3D triplanes and supports variable-length 3D generation.Figure 7 presents qualitative results; the prose also reports quantitative tokenization results in Table 5, but the supplied passages do not include its values.Source: Figure 7 · p. 13, §10 · p. 13
  • Table 5 evaluates tokenization on 128 high-quality watertight Objaverse assets filtered via Step1X-3D, using untextured renderings focused on geometry.The prose identifies the dataset as Objaverse; the supplied table caption says these results are consistent with the evaluation-set results in Table 1, but gives no metric values.Source: Table 5 · p. 13, §10 · p. 13
  • ShapeLLM-Omni uses 32-dimensional representation tokens, increasing to 3584 dimensions due to LLM usage.Table 6 compares total token-dimension cost against other autoregressive methods, but the supplied caption gives no other values.Source: Table 6 · p. 13

11. Further Implementation Details

The implementation details specify how LoST is trained and evaluated: training begins without nested dropout, while perceptual metrics are averaged across four rendered views per shape.

Evaluation

  • Perceptual metrics are averaged across four orthogonal, detailed-shading Blender renders per shape.Evaluation uses four orthogonal views for each shape.Source: §11 · p. 13

Tokenizer Training Details

  • LoST trains for 50 initial epochs without nested dropout, retaining causal masking, using bf16, EMA, and an effective batch size of 256 across 8 GPUs.The authors did not explore learned positional encodings or RoPE, which they note could potentially improve performance.Source: §11 · p. 13

Related papers

Loading 2603.17995v1…