PaperlayerView exact revision on arXiv

Source-linked AI summary

LATTICE: Democratize High-Fidelity 3D Generation at Scale

Zeqiang Lai, Yunfei Zhao, Zibo Zhao, Haolin Liu, Qingxiang Lin, Jingwei Huang, Chunchao Guo, Xiangyu Yue

arXiv:2512.03052v1cs.GRcs.CV

TL;DR

LATTICE addresses the difficulty of scalable, high-fidelity 3D generation by introducing VoxSet, a semi-structured, position-aware latent representation and a two-stage generation framework. It supports flexible decoding, low-cost training, test-time scaling, and strong performance in 3D asset generation.

  • Problem

    3D asset generation remains difficult because existing representations and latent compression schemes struggle with structural complexity, scalability, fidelity, and efficient training.

  • Method

    LATTICE introduces VoxSet, which anchors latent tokens to a coarse voxel grid, and uses them in a two-stage pipeline for generating sparse structure followed by detailed geometry.

  • Results

    LATTICE achieves superior performance in high-quality 3D mesh generation while supporting test-time scaling, low-cost training, and flexible high-resolution decoding.

  • Takeaways & Limitations

    LATTICE offers a scalable framework for automated, high-fidelity 3D content creation and provides evidence that localizable latent structure can support efficient generation.

Abstract

from arXiv · show

We present LATTICE, a new framework for high-fidelity 3D asset generation that bridges the quality and scalability gap between 3D and 2D generative models. While 2D image synthesis benefits from fixed spatial grids and well-established transformer architectures, 3D generation remains fundamentally more challenging due to the need to predict both spatial structure and detailed geometric surfaces from scratch. These challenges are exacerbated by the computational complexity of existing 3D representations and the lack of structured and scalable 3D asset encoding schemes. To address this, we propose VoxSet, a semi-structured representation that compresses 3D assets into a compact set of latent vectors anchored to a coarse voxel grid, enabling efficient and position-aware generation. VoxSet retains the simplicity and compression advantages of prior VecSet methods while introducing explicit structure into the latent space, allowing positional embeddings to guide generation and enabling strong token-level test-time scaling. Built upon this representation, LATTICE adopts a two-stage pipeline: first generating a sparse voxelized geometry anchor, then producing detailed geometry using a rectified flow transformer. Our method is simple at its core, but supports arbitrary resolution decoding, low-cost training, and flexible inference schemes, achieving state-of-the-art performance on various aspects, and offering a significant step toward scalable, high-quality 3D asset creation.

1. Introduction

LATTICE addresses the quality and scalability gap in 3D generation by adding localizable structure to compact latent representations. Its VoxSet representation and two-stage pipeline support scalable, high-fidelity asset generation with flexible training and inference.

  • Motivation: 3D generation lags behind 2D because it must jointly infer spatial structure and detailed content, expanding the search space and complicating optimization.Existing representations also impose computational and scalability challenges.
  • Method: LATTICE introduces VoxSet, a semi-structured representation that preserves VecSet efficiency while anchoring latent vectors to a regular voxel grid.This structure enables positional embeddings, arbitrary-resolution decoding, and progressive token scaling.
  • Method: The two-stage pipeline first generates a sparse voxelized geometry anchor, then generates detailed geometry VoxSets at arbitrary token resolutions with a rectified-flow transformer.The coarse structure can come from an off-the-shelf geometry model.
  • Results: LATTICE reports superiority over previous state-of-the-art models, with improvements in geometry smoothness and detail preservation.The supplied introduction summarizes overall performance but provides no numerical benchmark values.
  • Results: The approach supports test-time scaling, low-cost training, and a simple pure-Transformer architecture without complex or sparse components.The base 2-billion-parameter model is reported to train in under 24 hours on 64 GPUs, while test-time token counts scale beyond training counts.

2. Related Works

Prior 3D generation research spans diverse explicit and implicit representations, latent compression schemes, and increasingly capable diffusion systems. However, computational burden and difficulty producing highly detailed meshes remain unresolved challenges.

  • 3D Representations: 3D assets use diverse representations including voxels, point clouds, SDFs, meshes, neural radiance fields, and Gaussian splatting.These explicit and implicit forms serve different application contexts.
  • Latent Representations: Latent representations have become important because even flexible 3D forms impose substantial modeling and computational burdens on diffusion models.The related work identifies latent efficiency as a central research focus.
  • Latent Representations: VecSet, Triplane, and Sparse Voxel methods compress 3D assets differently, trading compactness against spatial locality and detail preservation.Sparse voxel methods are associated with fine-detail preservation, while VecSet methods emphasize efficiency.
  • 3D Generation: Recent native 3D diffusion and feedforward methods have improved generation quality, but several approaches still struggle to produce highly detailed meshes.Sparse-voxel-based methods show promising results, motivating further work on scalable localizable guidance.

3. Scalable 3D Generative Modeling

LATTICE combines VoxSet’s structured, scalable latent representation with a two-stage coarse-to-fine pipeline for detailed 3D generation. Its architecture supports arbitrary-resolution decoding, positional conditioning, token-level test-time scaling, and flexible applications.

  • VoxSet Representation: VoxSet compresses 3D assets into compact latent tokens while anchoring them to a structured voxel grid for scalable, position-aware modeling.Its VAE encodes point clouds and reconstructs SDFs, while voxel queries provide test-time-available spatial anchors.
  • VoxSet Representation: Voxel queries replace surface point queries with active-voxel centers, preserving positional locality while making query positions obtainable during test-time generation.The coarse voxel grid reduces the training-test gap because voxel centers are less tied to particular surfaces.
  • LATTICE Pipeline: Coarse geometry guides detailed generation in LATTICE’s two-stage pipeline, using sparse voxel structure before producing geometry latents.The first stage voxelizes coarse geometry from pretrained 3D generators; the second generates sparse voxel latents anchored at voxel centers.
  • VoxSet Representation: VoxSet supports arbitrary-resolution decoding through query jittering and voxel-query sampling above a minimum resolution.This design enables progressive scaling of token counts without retraining the representation for each resolution.
  • Detailed Geometry Generation: Rotary positional embeddings are added to noisy latent tokens in the rectified-flow transformer to exploit VoxSet’s spatial structure and improve convergence.The transformer generates VoxSet latents while image conditioning uses DINOv2-Giant embeddings.
  • Scaling and Applications: Models trained with 6144 tokens can scale at test time to 12288, 24576, and more voxel queries, while larger models show stable scaling.The reported scaling behavior is illustrated in Figure 6, which covers both model/training and test-time effects.
  • Scaling and Applications: LATTICE’s flexible design extends beyond image-conditioned generation to mesh refinement, part refinement, and mesh editing.The supplied passages describe mesh refinement and part refinement applications, while latent and voxel-query manipulation supports editing.

4. Experiments

LATTICE is evaluated on geometry reconstruction, image-to-geometry generation, ablations, and human preference. Across these evaluations, it reports strong reconstruction and generation performance, with voxel queries and VoxSet improving detail and artifacts.

  • 4.1. Reconstruction: The reconstruction evaluation uses Chamfer Distance and F-score, while the generation evaluation uses ULIP and Uni3D similarity metrics.The reconstruction benchmark contains challenging, detailed assets and compares representative VecSet- and voxel-based methods.
  • 4.2. Generation.: LATTICE achieves the best reported image-to-geometry generation performance among the compared methods.The comparison uses ULIP and Uni3D text-mesh and image-mesh similarities, with visual comparisons against open-source and commercial models.
  • 4.1. Reconstruction: Voxel queries produce fewer artifacts, while the VoxSet VAE introduces more detail through better reconstruction capability.The ablation compares different query types and VAEs after training at 1024 and 3072 tokens.
  • 4.1. Reconstruction: The Query Jitter VAE outperforms fixed-resolution VAEs and offers greater flexibility across varying resolutions when used with voxel queries.The original point-query VAE degrades substantially when tested with voxel queries.
  • 4.3. Evaluation: A user study finds that LATTICE significantly outperforms four commercial models across human preference comparisons.Figure 11 reports win rates across Overall, Subject, and Scene categories.

5. Conclusion

The conclusion presents LATTICE as a VoxSet-based framework for high-fidelity 3D asset generation. It emphasizes localizable position information, flexible encoding, low-cost training, and test-time scaling.

  • 5. Conclusion: LATTICE introduces VoxSet, a semi-structured latent representation for scalable, high-fidelity 3D asset generation.The conclusion associates the representation with improved computational complexity, scalability, and fidelity for diffusion generation.
  • 5. Conclusion: LATTICE combines localizable position information with flexible encoding, low-cost training, and strong test-time scaling.The conclusion highlights detailed, smooth, and sharp generated meshes as part of the framework’s reported capabilities.

A.1. Scaling Behavior on Different Architectures

The scaling analysis finds that explicit localizable guidance changes how 3D architectures benefit from larger models and longer token sequences. VoxSet continues improving with parameter and token scaling, unlike VecSet’s limited gains.

  • A.1. Scaling Behavior on Different Architectures: Larger VoxSet models produce increasingly detailed, sharper, smoother, and more regular structures, whereas VecSet models show little improvement with more parameters.The comparison uses VecSet models from 0.6B to 3B parameters and VoxSet models from 0.6B to 4.5B.
  • A.1. Scaling Behavior on Different Architectures: Proper conditioning is identified as a key factor in unlocking the benefits of model scaling.The analysis states that larger models help when conditions correspond clearly to outputs, while limited 3D data may also explain differing scaling behavior.
  • A.1. Scaling Behavior on Different Architectures: VoxSet improves from N to 2N and from 2N to 4N shape tokens at inference, while VecSet shows early saturation.The model can be evaluated with more tokens than used during training, and higher token counts produce richer details.
  • A.1. Scaling Behavior on Different Architectures: Arbitrary-resolution autoencoding supports token scaling because random query sampling trains the VAE across token lengths.The same mechanism applies across the voxel grid or object surface, allowing encoding at different token counts.

A.2. Representation Centric to Generation Centric

LATTICE reframes the representation question around generation-time localizability rather than structure or locality alone. VoxSet combines set-based global information with voxel-anchored guidance that remains identifiable at test time.

  • A.2. Representation Centric to Generation Centric: VoxSet models benefit from increased token counts, while VecSet models exhibit limited gains and early saturation.The comparison illustrates the generation-time scaling difference between the two representations.
  • A.2. Representation Centric to Generation Centric: VoxSet combines the global information of set-based representations with structure that is usable during generation.The paper argues that localizability, rather than locality alone, is the key property for generation performance.
  • A.2. Representation Centric to Generation Centric: VecSet contains structural information, but its structure cannot be identified during test time; VoxSet is proposed to address this limitation.This distinction separates representation structure from the availability of usable guidance during generation.

B. Implementation Details

LATTICE is trained with models spanning 0.6B to 4.5B parameters, multi-stage token scaling, and distributed optimization. Its data pipeline filters assets and prepares watertight geometry, point clouds, and signed-distance fields.

  • Training Setup: Models span 0.6B, 1.9B, and 4.5B parameters, with all sizes trained from scratch rather than progressively scaled.The training setup evaluates scaling across medium, XL, and XXL models.
  • Training Setup: Multi-stage token scaling uses constant learning rates within stages while decreasing the base learning rate from 1 × 10−4 to 1 × 10−6.
  • Training Setup: Training uses flow matching, classifier-free guidance, and DeepSpeed partitioning to support large distributed GPU runs.The reported maximum batch size reaches 2048.
  • Data Preparation: Data preparation filters unsuitable assets, then performs watertighting, point-cloud sampling, and signed-distance-field extraction.The filtering removes AI-generated, scanned, and extremely complex assets among other criteria.

C. Post-Training

LATTICE adds high-quality finetuning and decoding or sampling acceleration after training. Evaluation includes a randomized multi-participant user study that aggregates visual-quality rankings across methods.

  • High-Quality Finetuning: High-quality finetuning updates all DiT parameters on roughly 15k filtered samples to improve detail generation.Samples are selected using face count, sharp-edge count, and reconstruction quality.
  • Model Acceleration: FlashVDM accelerates geometry-VAE decoding, while guidance and step distillation reduce diffusion sampling cost.The models are described as inherent few-step generators under strong structure guidance from RoPE.
  • Evaluation: A user study uses approximately 500 randomized questions, with three independent participants ranking every compared method by visual quality or fidelity.Rankings are aggregated across participants and questions to compute each method’s winning rate.
  • Additional Results: Figure 14 provides additional image-to-geometry results without cherry-picking examples.

E. More Results

Figure 14 presents additional visual results for LATTICE’s image-to-geometry generation. The examples are reported without cherry-picking.

  • More Results: Figure 14 shows additional results for LATTICE’s image-to-geometry generation.
  • More Results: The displayed examples are presented without cherry-picking.
  • More Results: The figure extends the paper’s visual evidence beyond the primary examples.
Loading 2512.03052v1…