PaperlayerView exact revision on arXiv

Source-linked AI summary

ZipTok3D: High-Fidelity 3D Tokenization with Compact Token Prefixes

Mingda Lin, Weijie Wang, Zeyu Zhang, Bowen Cui, Yefei He, Haoyu Zhao, Yuanyu He, Donny Y. Chen, Feng Chen, Bohan Zhuang

arXiv:2609.01740v1cs.CV

TL;DR

Extremely short 3D token sequences can lose reconstruction fidelity because existing tokenizers do not sufficiently preserve geometry in a few tokens. ZipTok3D learns reconstructive global-token prefixes with nested dropout and progressively decodes them using a shared Transformer block. It matches 32-token COD-VAE quality with one token on ShapeNet and four on TRELLIS, while a post-hoc adaptive-budget diagnostic does not establish deployable early-exit cost.

  • Problem

    Existing 3D tokenizers suffer sharp reconstruction degradation at extremely low token budgets, motivating faithful reconstruction from very short sequences.

  • Method

    ZipTok3D jointly learns progressively informative global-token prefixes with nested dropout and iteratively decodes each prefix using a parameter-shared Transformer block.

  • Results

    32× and 8× shorter token sequences achieve comparable reconstruction to 32-token COD-VAE using one token on ShapeNet and four on TRELLIS, respectively.

  • Takeaways & Limitations

    Prefix organization and iterative refinement provide complementary benefits in the few-token regime.

  • Takeaways & Limitations

    No evaluated token-budget candidate satisfies the strict all-metric criterion for 304 ShapeNet and 846 TRELLIS objects, so oracle average budgets do not represent deployable early-exit cost.

Abstract

from arXiv · show

Compact token sequences are essential for efficient 3D generation. However, existing 3D tokenizers typically organize latent representations either over spatial regions or as fixed-size sets of global tokens, both suffering sharp reconstruction degradation when compressed to extremely low token budgets. In this paper, we present ZipTok3D, a 3D tokenizer designed for high-fidelity reconstruction from extremely short token sequences. Its key idea is to organize object geometry into progressively informative global-token prefixes and unfold these compact representations through iterative decoding. Specifically, nested dropout randomly truncates the latent sequence after encoding during training and requires each retained prefix to reconstruct the complete object, thereby prioritizing essential geometric information in the leading tokens. The decoder then repeatedly applies a parameter-shared Transformer block to recover fine-grained geometry from each prefix without a separate generative sampling stage. With the same token dimension, ZipTok3D achieves reconstruction quality comparable to the 32-token COD-VAE baseline using only one token on ShapeNet and four on TRELLIS, yielding $32\times$ and $8\times$ shorter token sequences, respectively.

1 Introduction

ZipTok3D addresses the sharp fidelity loss of extremely low-token 3D reconstruction by learning ordered global-token prefixes and iteratively decoding them. It achieves comparable reconstruction to 32-token COD-VAE with one token on ShapeNet and four on TRELLIS.

  • Motivation: Existing global 3D tokenizers suffer sharp reconstruction degradation when compressed to only a few tokens.This creates a tension between short latent sequences and geometric fidelity.
  • Approach: Nested prefix training requires each retained prefix to reconstruct the complete object, concentrating essential geometry in leading tokens.Subsequent tokens can encode residual details.
  • Approach: Iterative decoding repeatedly applies a parameter-shared Transformer block to unfold compact prefix information into detailed 3D representations.The method uses intermediate reconstruction supervision and no separate generative sampling stage.
  • Results: 32× and 8× fewer tokens achieve comparable reconstruction to 32-token COD-VAE on ShapeNet and TRELLIS, respectively.ZipTok3D uses one token on ShapeNet and four on TRELLIS.

2 Related Work

Prior 3D representations organize information spatially, globally, hierarchically, or adaptively, while shared recurrent refinement progressively recovers detail. ZipTok3D combines global reconstructive prefixes with direct shared iterative decoding rather than generative completion.

  • Spatial representations: Spatially organized methods attach latent features to points, grids, voxels, or hierarchical structures to preserve spatial support.Examples include LION, 3DILG, OctFusion, XCube, TRELLIS, LATTICE, and O-Voxel.
  • Global representations: Global methods compress object-wide geometry into compact token sets for triplane or neural-field reconstruction.VecSet and COD-VAE are representative global approaches.
  • Generation: Compact codes support diverse decoders and generative models, including flow matching, autoregressive modeling, diffusion distillation, and block-wise diffusion.These methods target efficient 3D generation using compact representations.
  • Flexible-length representations: Flexible-length tokenizers use nested dropout, causal prefixes, adaptive allocation, or coarse-to-fine organization to support multiple sequence lengths.Related methods include image, video, octree, supervoxel, and semantic token approaches.
  • Iterative refinement: Shared recurrent refinement increases effective decoding depth without step-specific parameters and has been applied to image, video, and 3D detail recovery.Prior 3D work includes cascaded refinement and recurrent point-cloud completion.

3 Method

ZipTok3D encodes sampled surface points into an ordered latent sequence, retains a prefix, and repeatedly refines a compact triplane state conditioned on that fixed prefix. Nested dropout and intermediate supervision make short prefixes reconstructive at multiple budgets.

  • 3.1 Overview: The encoder maps sampled surface points X into M latent vectors Z with shared width d.The latent sequence is Z = Eϕ(X), with maximum length M = 128.
  • 3.2 Nested Prefix Tokenization: Nested dropout retains a prefix Z:K and masks the suffix during selection and refinement, so reconstruction depends only on the retained K tokens.At inference, the suffix is removed and the same checkpoint operates at trained token budgets.
  • 3.3 Iterative Global-to-Spatial Refinement: The selection block separates learnable triplane tokens into a selected state H0_K and redundant tokens, conditioned on the retained prefix.The operation is denoted (H0_K, R_K) = Sω(T0, Z:K).
  • 3.2 Nested Prefix Tokenization: Short-prefix training optimizes a nested family of prefixes so leading tokens preserve object-wide geometry while longer prefixes add residual detail.This explicitly organizes information across token positions rather than optimizing only the complete latent sequence.
  • 3.3 Iterative Global-to-Spatial Refinement: The same parameter-shared Transformer block repeatedly updates the triplane state for L refinement steps while keeping θ and Z:K fixed.Weight sharing increases effective decoding depth without step-specific parameters.
  • 3.3 Iterative Global-to-Spatial Refinement: The updated and redundant tokens are restored to a complete triplane representation, which an occupancy MLP queries to produce occupancy probabilities.The process progressively unfolds geometric information without an additional latent sampling stage.
  • 3.4 Training Objective: Intermediate supervision trains a randomly sampled refinement state alongside the final output, preventing earlier recurrent states from remaining unconstrained.The objective also includes query weighting, self-distillation, and auxiliary losses inherited from COD-VAE.

4 Experiments

Experiments evaluate ZipTok3D reconstruction, class-conditioned generation, efficiency, and ablations on ShapeNet and TRELLIS under common protocols. Results show that extremely short prefixes retain reconstruction quality while reducing latent sequence length.

  • Experimental setup: ShapeNet and TRELLIS provide reconstruction benchmarks with common splits and evaluation protocols for comparing ZipTok3D against 3DILG, VecSet, and COD-VAE.ShapeNet uses the COD-VAE split, while TRELLIS uses a watertight-processed split partitioned with seed 42.
  • Reconstruction results: 32× shorter: ZipTok3D with one token on ShapeNet matches COD-VAE-32 on rounded CD and F1 and remains within 0.3 IoU points.COD-VAE degrades sharply at two tokens, distinguishing ZipTok3D's result from simply shrinking a fixed latent set.
  • Reconstruction results: One- and two-token prefixes preserve global part layouts and thin structures across ShapeNet and TRELLIS qualitative reconstructions.Examples include bench legs, flower stems, and bridge towers; both settings use five refinement steps.
  • Class-conditioned generation: ZipTok3D-2 uses a 16× shorter stage-2 latent sequence than COD-VAE-32 while remaining close on all three distribution metrics.It also provides higher sampling throughput, similar full throughput and peak memory, and faster full-pipeline performance than VecSet baselines.
  • Ablations: Ablations on ShapeNet at K = 2 compare iterative refinement, intermediate supervision, decoder depth, parameter count, latency, IoU, and F1.The study contrasts five-pass shared refinement with a single application of the six-layer block and with final-output-only supervision.

4.3 Ablation Study

The ablations show that nested prefix training and shared iterative refinement are the primary contributors to reconstruction quality, while intermediate supervision adds a smaller complementary benefit. Prefix length and refinement depth trade off: longer prefixes help most with shallow decoding, whereas refinement matters most for very short prefixes.

  • Component Ablation: Prefix only improves all three reconstruction metrics over COD-VAE at matched 12 × 1 architecture and decoder cost.This supports the contribution of nested prefix training in the original decoder.
  • Component Ablation: Repeated application of the shared block provides the main additional improvement without adding decoder parameters, although it increases latency.Intermediate supervision yields a modest further improvement without changing inference cost.
  • Effect of Prefix Length and Refinement Depth: Longer prefixes provide the largest benefit under a single decoding pass, while additional refinement produces the largest gains at K = 1 and K = 4.The two factors play complementary roles, and both datasets approach saturation by L = 5.
  • Effect of Prefix Length and Refinement Depth: At K = 1, refinement progresses from partial global structure at L = 1 to coherent major structure at L = 3 and sharper local details at L = 5.The qualitative progression is consistent with diminishing quantitative gains near L = 5.

5 Conclusion

ZipTok3D targets high-fidelity reconstruction from extremely short latent sequences by combining nested prefix training with parameter-shared iterative refinement. The method approaches 32-token COD-VAE quality with one token on ShapeNet and similar performance with four tokens on TRELLIS.

  • ZipTok3D is designed for high-fidelity reconstruction from extremely short latent sequences.
  • Nested prefix training preserves object-wide geometry in leading tokens, while parameter-shared iterative refinement unfolds compact information into detailed spatial representations.The decoder adds refinement depth without step-specific parameters.
  • One ZipTok3D token approaches 32-token COD-VAE reconstruction quality on ShapeNet, while four tokens yield similar performance on TRELLIS, reducing token counts by 32× and 8×.

S1 Prefix-Length and Refinement-Depth Analysis

The supplementary analysis confirms an asymmetric interaction between prefix length and refinement depth across query and surface metrics. Shallow decoding benefits most from more tokens, while short prefixes benefit most from additional refinement, with gains largely saturating by five passes.

  • Evaluation Range: The complete sweep evaluates K ∈ {1, 2, 4, 8, 16, 32, 64, 128} and L ∈ {1, 2, 3, 4, 5, 6} using one checkpoint per dataset.The decoder receives the exact K × 512 prefix without suffix padding.
  • Surface-Metric Interaction: Increasing K yields the largest gains when refinement is shallow, whereas increasing L is most beneficial when the prefix is short.This asymmetric interaction appears for mesh F1@0.02 and mesh CD on both datasets.
  • Joint Sweeps: Across query IoU, mesh F1@0.02, and mesh CD, reconstruction is most sensitive to prefix length under shallow decoding and refinement benefits concentrate at small K.Differences between L = 5 and L = 6 are small at the main-paper operating points.
  • Per-Object Refinement Behavior: Table A counts an object as improved when IoU or F1 strictly increases or CD strictly decreases, excluding ties and regressions from the improvement count.Values are rounded to one decimal place.
  • Per-Object Refinement Behavior: Increasing L from 1 to 3 improves a large majority of objects on every metric across both datasets.Increasing L from 3 to 5 still helps more than half of objects, but consistency is lower and more metric-dependent.

S2 Experimental Setup Details

The experiments use ShapeNet and TRELLIS reconstruction benchmarks with defined preprocessing, splits, baselines, token budgets, and training protocols. ZipTok3D is compared using exact prefixes, while baseline methods use their stated fixed or native latent lengths.

  • Datasets: ShapeNet uses a 55-category split with 48,597 training, 2,592 validation, and 1,283 disjoint test objects.Reconstruction is evaluated on the complete test set.
  • Datasets: TRELLIS uses 500,777 training assets and a separate 2,613-object test split produced after preprocessing and deterministic partitioning.The resulting test split includes ABO, Objaverse-XL GitHub, and Objaverse-XL Sketchfab assets.
  • Baselines: Comparisons include 3DILG, VecSet, and COD-VAE under fixed, shape-independent token budgets.ShapeNet comparisons use the common 1,283-object test split and COD-VAE metric implementation.
  • Evaluation Protocol: ZipTok3D is evaluated with exact prefixes of 1, 2, and 4 tokens, while reconstruction baselines use their specified token budgets.The generation comparison uses native baseline stage-2 lengths and ZipTok3D’s exact two-token stage-2 representation.
  • Training: Stage-1 tokenizer training samples 2,048 surface points and uses AdamW with FP16 mixed precision and gradient clipping.ShapeNet trains for 1,000 epochs, whereas TRELLIS trains for 300 epochs at a constant learning rate.

S3 Architecture and Objective Specifications

ZipTok3D combines nested prefix training with iterative decoding to reconstruct complete 3D geometry from progressively longer latent prefixes. Its architecture uses uncertainty-based token selection, shared refinement, restoration, and auxiliary supervision across reconstruction stages.

  • Encoder: The encoder produces a maximum-length latent bank of 128 vectors with width 512 from 2,048 input surface points.The point encoder forms 512 patches and uses progressive self-attention blocks.
  • Selection and refinement: An uncertainty head selects the highest-scoring 25% of 768 triplane tokens, retaining 192 tokens for recurrent refinement while bypassing the remaining 576.Selection occurs once before recurrent refinement, and discrete top-k indices are retained for spatial restoration.
  • Selection and refinement: A parameter-shared six-layer Transformer repeatedly processes selected tokens with the retained prefix, while the prefix and bypassed tokens remain fixed across passes.The shared block has width 512 and eight attention heads, with no pass-specific parameters.
  • Restoration: Restored triplane tokens are mapped to residual patches whose uncertainty modulation updates the initial triplane prediction.The 576 bypassed tokens are combined at their original spatial locations after scattering the refined selected tokens.
  • Stage-2 prefix VAE: The prefix VAE optimizes ordered budgets B = {1, 2, 4, 8, 16} with causal reconstruction, while generation decoding uses five refinement passes.The stage-1 tokenizer is frozen during prefix VAE optimization, and the six-pass decoder supervises prefix VAE training.

S4 Evaluation Protocols

Evaluation measures deterministic reconstruction, class-conditioned generation, and efficiency under specified common protocols. The reported trade-off is that shorter ZipTok3D prefixes do not reduce dense occupancy-query or shared-refinement computation, lowering end-to-end throughput versus single-pass COD-VAE variants.

  • Reconstruction: Reconstruction uses query IoU over 500,000 volume queries plus mesh CD and F1 from dense 128^3 occupancy-grid reconstructions.Meshes are extracted with marching cubes and compared using sampled reconstructed and reference surfaces.
  • Generation: Generation evaluates airplane, car, chair, table, and rifle with 2,000 shapes per category using MMD-CD, COV-CD, and 1-NNA-CD.Reference sets come from the 1,283-object ShapeNet test split, with fixed generated-to-reference ratios for MMD-CD and COV-CD.
  • Efficiency: Efficiency measurements use trained checkpoints on one NVIDIA H20 GPU with batch size 16 and FP32 inference, excluding loading, transfers, marching cubes, and file output.CUDA-event timings are collected after warmup, with peak allocated CUDA memory measured separately.
  • Efficiency: ZipTok3D uses five refinement passes in the full reconstruction efficiency protocol.The table reports token count, full-shape throughput, peak memory, and the corresponding representation–compute trade-off.
  • Efficiency: Shorter prefixes reduce neither dense occupancy queries nor shared refinement calls, so ZipTok3D has lower end-to-end throughput than single-pass COD-VAE variants despite shorter latent sequences.Peak memory remains similar because the common encoder and dense-field query dominate allocation.
  • Statistical analysis: TRELLIS comparisons use 20,000 paired bootstrap resamples of 2,613 aligned objects, with effects defined as ZipTok3D minus COD-VAE.The resulting percentiles quantify variation over evaluation objects conditional on trained models, not variation across independent training runs.

S5 Post-Hoc Adaptive-Budget Diagnostic

The post-hoc diagnostic searches token and refinement budgets for operating points matching or exceeding COD-VAE-32 on all three rounded reconstruction metrics. Its oracle budget distributions describe only shapes with successful candidates, not deployable early-exit costs.

  • Search design: The diagnostic searches K ∈ {1, 2, 4, 8, 16, 32} and L ∈ {1, ..., 6}, prioritizing shorter representations before shallower decoding.It uses ground-truth reconstruction metrics and therefore measures adaptive potential rather than a deployable budget predictor.
  • Search criterion: Candidates are accepted only when all three rounded per-shape metrics are no worse than COD-VAE-32.Rounding uses one decimal for IoU/F1 and three decimals for CD; both methods are compared on the same successful subset.
  • Reported statistics: Table C reports found counts, rates, average K, and average L for the successful subset, with IoU and F1 expressed as percentages.The COD-VAE rows are subset results rather than full-split results.
  • Scope boundary: 304 ShapeNet and 846 TRELLIS objects have no successful candidate in the evaluated grid and are excluded from oracle averages.All fixed-budget evaluations include these objects, so the reported average budgets are not expected costs for a deployable early-exit policy.
  • Budget distribution: 68.3% of the ShapeNet subset and 71.6% of the TRELLIS subset first succeed at three or four refinement passes.Figure C normalizes prefix-length and refinement-depth distributions within each dataset’s successful subset.

S6 Additional Qualitative Comparisons

Additional qualitative comparisons examine ShapeNet and TRELLIS objects with thin components, openings, curved surfaces, repeated structures, and multi-level layouts. The reported reconstructions preserve principal empty regions, separated components, topology, and repeated architectural structures.

  • ShapeNet examples: The qualitative examples include thin components, large openings, curved surfaces, and repeated architectural elements.These examples test reconstruction of varied geometric structures rather than a single object type.
  • ShapeNet examples: The open shelf, chair, and table retain principal empty regions, while the vessel and aircraft preserve separated components and overall topology.The passage locates remaining differences around small protrusions and thin supports.
  • TRELLIS examples: TRELLIS reconstructions recover multi-level building layouts and repeated roof or fortification structures.The comparison emphasizes preservation of large-scale architectural organization.
  • Comparison setup: Figure D compares ZipTok3D at K = 4 and L = 5 with a baseline using 32 COD-VAE tokens across ShapeNet and TRELLIS objects.The comparison covers the top five ShapeNet rows and bottom two TRELLIS rows.
Loading 2609.01740v1…