LoST: Level of Semantics Tokenization for 3D Shapes

Niladri Shekhar Dutt; Zifan Shi; Paul Guerrero; Chun-Hao Paul Huang; Duygu Ceylan; Niloy J. Mitra; Xuelin Chen

Source-linked AI summary

LoST: Level of Semantics Tokenization for 3D Shapes

Niladri Shekhar Dutt, Zifan Shi, Paul Guerrero, Chun-Hao Paul Huang, Duygu Ceylan, Niloy J. Mitra, Xuelin Chen

arXiv:2603.17995v1Mar 18, 2026cs.CVcs.GRcs.LG

TL;DR

3D autoregressive generation lacks a tokenization strategy that is both efficient and semantically coherent. LoST orders tokens by semantic salience and uses RIDA to align 3D latent relations with DINO semantics. It reports state-of-the-art reconstruction and efficient autoregressive generation, while retaining limitations in representation, decoding cost, early-prefix artifacts, and sequence length.

Problem
Optimal 3D shape tokenization remains open, while geometric LoD hierarchies can be token-inefficient and lack semantic coherence for autoregressive modeling.
Method
LoST learns semantically ordered shape tokens, and RIDA aligns relational structure in 3D triplane latents with DINO feature relationships for semantic supervision.
Results
LoST achieves state-of-the-art reconstruction and autoregressive 3D generation, using 128 tokens and supporting semantic shape retrieval.
Takeaways & Limitations
Short LoST prefixes decode complete, plausible, category-level shapes, while later tokens refine instance-specific geometry and semantics.
Takeaways & Limitations
LoST is instantiated on VAE triplane latents, uses a diffusion decoder that increases computational requirements, and can show artifacts in few-token decodes.

Abstract

from arXiv · show

Tokenization is a fundamental technique in the generative modeling of various modalities. In particular, it plays a critical role in autoregressive (AR) models, which have recently emerged as a compelling option for 3D generation. However, optimal tokenization of 3D shapes remains an open question. State-of-the-art (SOTA) methods primarily rely on geometric level-of-detail (LoD) hierarchies, originally designed for rendering and compression. These spatial hierarchies are often token-inefficient and lack semantic coherence for AR modeling. We propose Level-of-Semantics Tokenization (LoST), which orders tokens by semantic salience, such that early prefixes decode into complete, plausible shapes that possess principal semantics, while subsequent tokens refine instance-specific geometric and semantic details. To train LoST, we introduce Relational Inter-Distance Alignment (RIDA), a novel 3D semantic alignment loss that aligns the relational structure of the 3D shape latent space with that of the semantic DINO feature space. Experiments show that LoST achieves SOTA reconstruction, surpassing previous LoD-based 3D shape tokenizers by large margins on both geometric and semantic reconstruction metrics. Moreover, LoST achieves efficient, high-quality AR 3D generation and enables downstream tasks like semantic retrieval, while using only 0.1%-10% of the tokens needed by prior AR models.

1. Introduction

3D autoregressive generation still lacks an optimal tokenization: geometric hierarchies can be token-inefficient and produce unusable early prefixes. LoST orders tokens by semantic salience and trains semantic supervision to make prefixes plausible and progressively detailed.

3D shape tokenization remains unresolved despite its critical impact on autoregressive generation and analysis.
Geometric LoD hierarchies create coarse-scale token bloat and unusable early decodes that lack geometric and semantic fidelity.
LoST orders tokens by semantic salience so short prefixes decode plausible shapes with principal semantics, while later tokens refine instance-specific details.
LoST trains a ViT-based encoder and prefix decoder with nested token dropout, causal masking, and a generative decoder for plausible extreme-compression reconstructions.
RIDA supplies 3D semantic alignment by addressing the lack of direct semantic supervision for level-of-semantics representations.
LoST reports SOTA reconstruction and autoregressive performance, using 128 tokens while also supporting semantic shape retrieval.

2. Related Work

Prior 3D tokenizers use flat streams, learned latent sequences, or geometric LoD structures, but their prefixes often lack recognizable semantic coherence. Related image and video tokenization motivates a 3D any-prefix decoder that remains semantically relevant and geometrically refined.

3D Tokenization with Flat Element Streams: Flat mesh-element streams are long and irregular, and their prefixes seldom decode to recognizable or semantically close shapes.
Learned 3D Latent Token Sequences: Learned 3D latent sequences improve global coherence but typically rely on coarse fields, heavy refinement, and provide no guarantee of semantically linked prefixes.
3D Tokenization with Geometric LoD: Geometric LoD methods provide spatial coherence through coarse-to-fine refinements, but allocate capacity to geometric elements rather than semantic organization.
Hierarchical Image and Video Tokenization: Image and video tokenization methods demonstrate efficiency and controllability through discrete or coarse-to-fine representations.
The related-work goal is an any-prefix decoder whose outputs remain both semantically relevant and geometrically refined.

3. Method

LoST learns a semantically ordered continuous token sequence from VAE triplane latents, using prefix training and generative decoding for complete shapes. RIDA transfers relational structure from DINO features into a 3D semantic space that guides reconstruction and token hierarchy learning.

LoST targets prefixes that capture principal semantics early and progressively add instance-specific geometric and semantic details.
LoST Encoder: A ViT encoder converts 3D VAE triplane features into a one-dimensional token sequence.
LoST Encoder: Causal masking and nested dropout train earlier tokens to encode principal semantics while later tokens add finer details.
LoST Overview: LoST maps shape latents to semantically ordered tokens, while a conditional generative DiT decoder reconstructs the complete latent from any prefix.
Semantic Guidance: Directly aligning 3D representations with DINO features performs poorly, and aligning rendered multi-view features is computationally prohibitive.
RIDA: RIDA aligns relative distances between triplane latents and DINO features by learning a mapped student feature space for semantic guidance.
RIDA: The semantic extractor uses teacher-mined relationships and objectives for global contrast, inter-instance ranking, and spatial structure distillation.
RIDA: Spatial structure distillation transfers part-level relationships, and the combined objective produces a semantically structured 3D latent space.

4. Experiments

Experiments evaluate LoST's reconstruction, autoregressive generation, and semantic retrieval using compact token sequences and comparisons with established baselines. Results show strong reconstruction and generation quality, with early prefixes preserving recognizable semantics while later tokens add detail.

Experimental Setup: 300k shapes were generated through a prompt-to-image-to-3D pipeline for LoST training.The dataset was created using Gemini 2.5 Pro prompts, Flux.1 image synthesis, and Direct3D image-to-3D generation.
Tokenizer Reconstruction: LoST reconstruction is evaluated against OctGPT and VertexRegen using Chamfer Distance, FID, and DINO similarity.These metrics measure geometric accuracy and semantic consistency of reconstructed shapes.
Tokenizer Reconstruction: 0.1%-10% of baseline token counts suffice for LoST to achieve better reconstruction and alignment, often with only 1-4 tokens.LoST is especially effective at low token budgets, where its prefixes decode plausible complete shapes rather than abstract scaffolds.
Tokenizer Reconstruction: A single LoST token decodes a complete recognizable shape, while later tokens refine instance-specific semantic and geometric details.Qualitative examples include progression from a generic mountain to one with an embedded face.
Semantic Retrieval: RIDA-mapped triplane features retrieve shapes with semantic alignment similar to DINO rather than only geometric similarity.The retrieval example uses a submarine shaped like a fish, where geometric and semantic similarity can diverge.
Autoregressive Generation: LoST-GPT outperforms competing autoregressive methods on image-to-3D generation, achieving the lowest FID and highest DINO semantic alignment score.The comparison includes ShapeLLM-Omni, OctGPT, and Llama-Mesh; LoST-GPT uses 128 training tokens.

5. Conclusion

LoST replaces geometric level-of-detail token ordering with semantic salience, while RIDA aligns 3D triplane latents with DINO semantics without decoding and rendering. The method has clear scope and efficiency trade-offs, including VAE-triplane dependence, diffusion-decoder cost, few-token artifacts, and fixed AR lengths.

LoST orders tokens by semantic salience so short prefixes decode complete, category-plausible shapes before later tokens refine geometric and semantic details.
The tokenizer and losses are instantiated on VAE triplane latents, leaving support for representations such as Gaussian Splats as future work.
The diffusion decoder increases computational requirements compared with pure autoregressive decoding.
Few-token decodes can still exhibit artifacts, and the current AR generator uses a fixed target length despite variable-length codes.

6. Additional Qualitative Results

Qualitative visualizations show high-fidelity reconstructions and complete, plausible 3D shapes generated from only a few tokens. These findings are reported as consistent with the quantitative results in Table 2.

The supplemental gallery illustrates high-fidelity reconstructions that visually surpass recent baselines.
The autoregressive model generates complete and plausible 3D shapes when conditioned on a few tokens.
The qualitative findings are consistent with the strong quantitative performance reported in Table 2.

7. Shape Retrieval using RIDA

Shape retrieval evaluates whether RIDA transfers DINOv2 semantic neighborhood structure to 3D triplane features across in-distribution and out-of-distribution data. RIDA outperforms geometric and direct-regression baselines, including on the challenging OOD set.

DINO similarity defines semantic neighbors, while Recall@K, mAP@K, and Jaccard Index measure retrieval and ranking quality.
RIDA is evaluated against raw triplane latents and direct regression on in-distribution and out-of-distribution shape sets.
RIDA outperforms the geometric baseline on both validation and evaluation sets, confirming stronger semantic alignment.
28.28% mAP@3 on the OOD evaluation set exceeds the 17.47% baseline, indicating robustness to low-level geometric variation.
44.28% in-distribution mAP exceeds the triplane baseline’s 16.42% and feature-regression baseline’s 22.90%.

8. Ablation on RIDA

RIDA improves semantic reconstruction, especially when token budgets are low, by aligning 3D representations with semantic priors. Its gains do not reduce Chamfer geometry performance, and it accelerates diffusion-decoder convergence, while direct regression fails to learn semantics accurately.

RIDA explicitly aligns 3D triplane representations with semantic priors, augmenting the diffusion decoder’s generative capacity.
Low-token regimes show the largest semantic gains because dense geometric encoding is unavailable.
RIDA leaves Chamfer Distance similar while improving DINO and DINOv2 semantic reconstruction scores.
RIDA accelerates diffusion-decoder convergence by approximately 40% compared with extended training without RIDA.
Direct regression trained to predict DINO features fails to accurately learn semantics.

9. Details about RIDA

RIDA distills semantic and spatial relationships into a 3D latent space by aligning relational structures rather than raw cross-modal similarities. Its objective combines global, rank, and spatial components to produce semantically structured representations for generation.

Inter-Instance Rank Distillation: RIDA uses cosine-similarity relationships to transfer continuous relational structure from teacher features into the student 3D latent space.The approach extends relational knowledge distillation beyond hard positive-negative separation.
Inter-Instance Rank Distillation: Rank distillation standardizes each anchor’s similarity row and minimizes mean squared error between the resulting distribution-invariant vectors.Z-scoring removes modality-specific means and scales while preserving relative neighborhood structure.
Inter-Instance Rank Distillation: The rank objective is proportional to maximizing Pearson correlation, so it optimizes relative neighborhood ranking rather than absolute similarities.This design addresses the mismatch between student and teacher similarity distributions.
Spatial Structure Distillation: Spatial distillation matches teacher and student intra-instance token-affinity distributions using row-wise softmax and KL divergence.This preserves local geometric structure by transferring relative affinity patterns between tokens.
Objective: The final semantic pretraining loss is a weighted sum of global, rank, and spatial components, with λglobal = 1.0, λrank = 1.0, and λspatial = 0.5.The resulting encoder supplies a semantically structured 3D latent space for the generative task.

10. Extending LoST to other 3D Representations

LoST and RIDA are presented as representation-agnostic: applying them to TRELLIS Stage-1 latents yields variable-length representations decoded by TRELLIS Stage-2. Additional evaluations cover Objaverse assets and token-dimension costs across autoregressive methods.

10. Extending LoST to other 3D Representations: LoST generalizes beyond Direct3D triplanes by operating on TRELLIS Stage-1 latents and decoding them through TRELLIS Stage-2.The TRELLIS adaptation reshapes the 16^3 voxel grid with feature dimension 8 into a 64^2 2D grid while retaining the original architecture.
10. Extending LoST to other 3D Representations: Figure 7 presents LoST in TRELLIS’s latent space as a representation-agnostic tokenizer for variable-length 3D generation.The figure describes the flexibility of applying LoST across 3D representations.
10. Extending LoST to other 3D Representations: Objaverse evaluation uses 128 high-quality watertight assets, with metrics computed on untextured renderings to focus on geometry.The results are described as consistent with the evaluation set reported in Table 1.
10. Extending LoST to other 3D Representations: ShapeLLM-Omni’s 32-dimensional tokens expand to 3584 dimensions when used by its language model, affecting total token dimension cost comparisons.Table 6 compares total token dimension cost across autoregressive methods.

11. Further Implementation Details

The implementation evaluates rendered views with Blender and uses training procedures intended to stabilize tokenizer optimization. Dataset prompts are generated in batches through a specified Gemini 2.5 Pro template emphasizing diverse visual 3D objects and scenes.

11. Further Implementation Details: Evaluation renders four orthogonal views per shape in Blender with detailed shading and averages perceptual metrics across views.The evaluation protocol computes view-level metrics before reporting their average.
11. Further Implementation Details: Tokenizer training begins with 50 epochs without nested dropout, while causal masking remains active throughout training.Mixed-precision bf16 training and Exponential Moving Average updates are used to stabilize optimization.
11. Further Implementation Details: The authors did not explore learned positional encodings or RoPE, although they state these could potentially improve performance.This is an implementation choice and a stated avenue for possible gains.
11. Further Implementation Details: The prompt-generation constraints require hyper-varied, maximally diverse collections spanning media, history, imagination, everyday artifacts, relics, and treasures.Examples include a sushi bento box, a wind-up clock mechanism, and an antique brass telescope.