Source-linked AI summary
Structured 3D Latents for Scalable and Versatile 3D Generation
Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, Jiaolong Yang
TL;DR
3D generative models still lag behind 2D systems in quality and face difficulties supporting diverse output representations. The paper introduces SLAT, which combines sparse 3D structure with multiview visual features, and uses rectified-flow models to generate high-quality assets in multiple formats. The resulting system supports text or image conditioning, flexible decoding, and local editing, while retaining a two-stage pipeline and image-to-3D lighting limitations.
Problem
Existing 3D generative models lag behind 2D image-generation systems in quality and struggle to accommodate diverse representations and detailed appearance.
Method
SLAT combines sparse surface-intersecting voxel structures with local latents fused from dense multiview vision features, and TRELLIS generates them with two-stage rectified-flow transformers.
Results
TRELLIS generates high-quality 3D assets from text or image conditions, significantly surpasses previous methods, and decodes into diverse output formats.
Takeaways & Limitations
SLAT provides a representation-agnostic basis for high-quality, versatile, and editable 3D asset generation across downstream output formats.
Takeaways & Limitations
The two-stage SLAT generation pipeline can be less efficient than single-stage methods, and image-to-3D outputs retain baked-in lighting effects.
Abstract
from arXiv · showhide
We introduce a novel 3D generation method for versatile and high-quality 3D asset creation. The cornerstone is a unified Structured LATent (SLAT) representation which allows decoding to different output formats, such as Radiance Fields, 3D Gaussians, and meshes. This is achieved by integrating a sparsely-populated 3D grid with dense multiview visual features extracted from a powerful vision foundation model, comprehensively capturing both structural (geometry) and textural (appearance) information while maintaining flexibility during decoding. We employ rectified flow transformers tailored for SLAT as our 3D generation models and train models with up to 2 billion parameters on a large 3D asset dataset of 500K diverse objects. Our model generates high-quality results with text or image conditions, significantly surpassing existing methods, including recent ones at similar scales. We showcase flexible output format selection and local 3D editing capabilities which were not offered by previous models. Code, model, and data will be released.
1. Introduction
The paper introduces SLAT, a unified latent space designed for high-quality 3D generation across representations by combining sparse 3D structure with detailed visual features. TRELLIS builds on SLAT to generate versatile, editable assets from text or image prompts.
- Motivation: Existing 3D generative models still trail 2D image-generation systems in quality and struggle to support diverse representations effectively.Meshes, point clouds, Radiance Fields, and 3D Gaussians serve different applications, while geometry-focused representations often model appearance less effectively.
- Structured LATents: SLAT combines active surface-intersecting voxels with local latents derived from densely rendered multiview image features.The sparse voxels provide coarse structure, while vision-foundation-model features encode finer geometry and appearance details.
- Generation models: TRELLIS uses a two-stage rectified-flow transformer pipeline to generate sparse SLAT structure and then latent vectors for non-empty cells.Models scale to up to 2 billion parameters and are trained on a large 3D asset dataset.
- Capabilities: The method accepts text or image prompts and decodes SLAT into Radiance Fields, 3D Gaussians, meshes, and other 3D representations.This output flexibility is intended to accommodate different downstream requirements.
- Capabilities: The method supports tuning-free local 3D editing and requires no 3D fitting for training objects.The authors also release the code, models, and data for reproduction and downstream applications.
2. Related Works
Prior 3D generation research spans GANs, diffusion, autoregressive, compact-latent, and 2D-assisted approaches. These methods improve coverage and quality but retain challenges in scalability, efficiency, detailed appearance modeling, or direct 3D generation.
- Direct 3D generation: GAN-based methods modeled 3D distributions but faced challenges scaling to more diverse scenarios.
- Direct 3D generation: Diffusion and GPT-style autoregressive models expanded generation across point clouds, voxel grids, triplanes, 3D Gaussians, and meshes, while raw-space efficiency remained difficult.
- Latent-space generation: Compact-latent methods improved efficiency but often focused on shape, required separate texturing, or struggled with highly detailed appearance from surface representations.
- 2D-assisted 3D creation: 2D-assisted methods leveraged stronger image-model generalization to create 3D assets through distillation and related techniques.
- Rectified flow models: Rectified flow models have shown effectiveness for large-scale image and video generation and are applied here to 3D generation at scale.
3. Methodology
The method uses SLAT, a sparse voxel-indexed latent representation that captures geometry and appearance while supporting decoding into multiple 3D formats. A two-stage rectified-flow pipeline generates the sparse structure and its local latents from conditions, while specialized encoders and decoders support reconstruction and tuning-free editing.
- Structured latent representation: SLAT represents each asset with active surface-intersecting voxels and local latents that capture coarse structure plus fine shape and appearance details.The active-voxel positions define the sparse structure, while attached latent vectors encode local information.
- Structured latent encoding: Dense multiview DINOv2 features are projected onto active voxels and averaged to provide detailed local structure and appearance information for encoding.Features are extracted from randomly sampled spherical camera views and aggregated at 64^3 resolution.
- Encoding and decoding: A transformer-based sparse VAE encodes voxelized features into SLAT and decodes them through format-specific networks for 3D Gaussians, Radiance Fields, and meshes.The encoders and decoders use sparse-token processing with positional encodings and shifted-window attention; the decoders have representation-specific output layers and reconstruction losses.
- Encoding and decoding: The method trains Gaussian reconstruction end-to-end, then freezes the encoder and trains separate decoders for other formats, with the structured latents reconstructing those formats faithfully.The Gaussian decoder predicts local Gaussian parameters, while mesh decoding uses FlexiCubes and Radiance Field decoding uses local radiance volumes.
- Structured latent generation: Rectified-flow generation proceeds in two stages: a transformer generates the sparse structure, followed by another transformer generating local latents attached to its non-empty cells.The sparse structure is compressed into a low-resolution continuous feature grid for efficient generation, decoded back to active voxels, and then used to condition latent generation.
4. Experiments
The experiments evaluate SLAT reconstruction fidelity, text/image-to-3D generation quality, and design choices. Results show strong performance across representations and metrics, with versatile generation, editing, and scaling behavior.
- Reconstruction fidelity: Our method outperforms all baselines across every evaluated reconstruction metric and surpasses CLAY on geometry quality.The evaluation covers appearance, shape accuracy, and surface details across diverse output formats.
- Generation quality: The method produces vivid appearances, fine geometric details, prompt-aligned content, and faithful adherence to input-image details.Examples include translucent objects, sharp edges, and complex structures.
- Qualitative comparisons: Our method outperforms previous approaches in visual comparisons, with more vivid appearances, finer geometries, and more precise text and image alignment.Compared methods exhibit structural distortion, featureless outputs, or implausible geometries.
- Quantitative comparisons: The method significantly surpasses previous methods across all evaluated FD, KD, and CLIP metrics on Toys4k.FD and KD use Inception-v3, DINOv2, and PointNet++ features, while CLIP measures prompt consistency.
- Ablation studies: A 643 SLAT resolution improves performance over 32^3, rectified flow improves quality and prompt alignment at either generation stage, and larger models improve performance.The model-size trend holds on both the training distribution and Toys4k.
- Editing applications: The method generates structure-consistent asset variations and supports tuning-free local edits, including region removal, addition, and replacement guided by text or images.The demonstrated edits include adding a river and bridge to an island scene.
5. Conclusion
The paper presents SLAT as a unified latent representation for versatile, high-quality 3D generation and introduces a two-stage rectified-flow pipeline for generating it at scale.
- SLAT encodes geometry and appearance in localized latents anchored on a sparse 3D grid, using dense multiview image features from a vision foundation model.
- SLAT supports decoding into multiple 3D output formats, enabling versatile asset generation.
- TRELLIS uses rectified flow transformers in a two-stage pipeline that first generates sparse structure and then latent vectors for non-empty cells.
- Extensive experiments report superiority in 3D generation quality, versatility, and editability, with potential applications in digital production.
(Supplementary Material)
The supplementary material lists the network configurations used in the paper and defines the abbreviations used in the table.
- Table 6 summarizes the network configurations used throughout the paper.
- SW denotes Shifted Window, MSA and MCA denote Multihead Self-Attention and Multihead Cross-Attention, and Sp. Conv. denotes Sparse Convolution.
A.1. Network Architectures
The architectures primarily use transformers with specialized modules, including sparse-structure VAEs, shifted-window attention, normalization, and sparse convolutional resolution changes.
- The networks primarily consist of transformers augmented with specialized modules; ES and DS form the sparse-structure VAE.
- The sparse-structure VAE uses 3D convolutions and residual downsampling or upsampling blocks to convert active-voxel grids into continuous latents.
- 3D shifted-window attention partitions the 64^3 space into 8^3 windows, while alternating shifted windows make adjacent layers overlap.
- RMSNorm is applied to queries and keys to mitigate training instability caused by exploding norms in multi-head attention.
- Sparse convolutional downsamplers and upsamplers alter sparse-tensor resolution for mesh SDF grids and efficient SLAT generation.
A.2. Training Details
The supplementary training details describe representation-specific decoders, mesh and radiance-field objectives, timestep ablations, and the curated datasets used for training and evaluation.
- The SLAT VAE provides decoders for 3D Gaussians, Radiance Fields, and meshes.
- For 3D Gaussians, each active voxel predicts 32 Gaussians, with regularization used because neural prediction makes original density-control schemes inapplicable.
- Radiance Fields use four orthogonal vectors per active voxel to represent a CP-decomposed local 8^3 radiance volume with rank R = 16.
- Meshes are extracted from 0-level iso-surfaces after sparse structures are upsampled from 64^3 to 256^3; predicted vertex attributes are averaged across associated voxels.
- The mesh objective combines geometry, color, and regularization losses, with consistency and signed-distance terms supporting stable and plausible extraction.
- Table 7 evaluates timestep sampling distributions for the rectified-flow stages, and logitNorm(1, 1) is selected as the default because it fits better on Toys4k.
- The training data is curated and filtered from open-source datasets, while Toys4k is reserved for testing because prior works did not use it for training.
B.2. Data Curation Pipeline
The data curation pipeline filters 3D assets using multiview aesthetic scores and creates detailed, aligned captions for text-conditioned generation.
- Asset quality filtering: Four uniformly distributed rendered views are used to assess each 3D asset’s average aesthetic score.The score helps identify objects with poor visual quality, including minimal texturing or overly simplistic geometry.
- Asset quality filtering: Low aesthetic scores typically identify objects with undesirable visual quality, such as minimal texturing or overly simplistic geometry.
- Caption generation: Existing 3D captions often have poor object alignment or insufficient detail, limiting their usefulness for high-quality text-to-3D generation.
- Dataset overview: The training and evaluation datasets are summarized alongside aesthetic-score distributions and examples from Objaverse-XL.
- Caption generation: GPT4o first generates detailed raw descriptions, distills them into captions typically under 40 words, and produces varying-length prompts.
C.1. Evaluation Protocol
The evaluation protocol measures reconstruction and generation quality across appearance, geometry, prompt alignment, and human preference, using controlled rendering and user-study procedures.
- Reconstruction experiments: Reconstruction fidelity is evaluated on 500 randomly sampled instances from the filtered Toys4k dataset of 3,229 assets.
- Reconstruction experiments: Appearance fidelity uses PSNR and LPIPS, while geometry accuracy uses Chamfer Distance, F-score, PSNR-N, and LPIPS-N.
- Reconstruction experiments: Geometry point clouds are generated from 100 uniformly sampled depth-map views, with 100K points randomly selected for evaluation.
- Generation experiments: Generation quality is assessed on 1,250 Toys4k instances and 5,000 training-set instances using FD, KD, and CLIP metrics.
- Generation experiments: Prompt alignment is measured by averaging CLIP cosine similarities across eight rendered views per asset.
- User study: The user study compares rotating videos of uncurated candidate assets, randomizes trials and candidate order, and collects 2,701 responses from 104 participants.
D. More Results
Additional experiments show the method generating assets from AI-generated and real-world prompts, comparing favorably with prior approaches and supporting editing and scene construction.
- Additional generation results: The method generates assets from AI-generated text prompts, AI-generated images, and challenging real-world images from SA-1B.Outputs are shown as 3D Gaussians, Radiance Fields, and meshes.
- Comparisons: Additional comparisons evaluate the method against alternative 3D generation approaches.
- Comparisons: Compared with Rodin Gen-1, the method exhibits more detailed geometry structures on complex cases while using only open-source training datasets.
- Editing and applications: Additional results demonstrate flexible editing and manipulation of generated 3D assets.
- Editing and applications: Generated assets are used to visualize complex scenes, illustrating potential production applications.
E. Limitations and Future works
The method has efficiency and image-to-3D lighting limitations, while supplementary results cover varied outputs, comparisons, asset variations, editing, and scene construction.
- Limitations: The two-stage generation pipeline can be less efficient than end-to-end methods that generate complete 3D assets in one stage.
- Limitations: Image-to-3D generation retains lighting effects from the reference image, producing baked-in shading and highlights.
- Additional results: Supplementary figures show additional Gaussian, Radiance Field, and mesh results from AI-generated and real-world prompts.
- Comparisons: Further visual comparisons include prior methods and Rodin Gen-1, with prompts and output examples shown across the figures.
- Applications: Additional figures illustrate asset variations, local roof replacement, and complex scenes assembled from generated assets.