Source-linked AI summary
XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies
Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, Francis Williams
TL;DR
High-resolution and large-scale 3D generation remains difficult because existing representations and single-level models do not scale effectively. XCube uses hierarchical latent diffusion over sparse voxel hierarchies, achieving state-of-the-art object and scene generation, including millions of voxels at 1024^3 resolution in under 30 seconds. Its text-to-3D capability remains limited for complex prompts.
Problem
Existing single-level 3D latent diffusion models and representations have limited scalability for high-resolution objects and large outdoor scenes.
Method
XCube models sparse voxel hierarchies with level-conditioned latent diffusion and sparse structure VAEs in a coarse-to-fine process.
Results
XCube achieves state-of-the-art unconditional and conditional generation and produces complex 1024^3 shapes with millions of voxels in under 30 seconds.
Takeaways & Limitations
The model supports high-resolution object and scene generation, arbitrary multi-scale attributes, user-guided editing, and conditional generation from text or scans.
Abstract
from arXiv · showhide
We present XCube (abbreviated as $\mathcal{X}^3$), a novel generative model for high-resolution sparse 3D voxel grids with arbitrary attributes. Our model can generate millions of voxels with a finest effective resolution of up to $1024^3$ in a feed-forward fashion without time-consuming test-time optimization. To achieve this, we employ a hierarchical voxel latent diffusion model which generates progressively higher resolution grids in a coarse-to-fine manner using a custom framework built on the highly efficient VDB data structure. Apart from generating high-resolution objects, we demonstrate the effectiveness of XCube on large outdoor scenes at scales of 100m$\times$100m with a voxel size as small as 10cm. We observe clear qualitative and quantitative improvements over past approaches. In addition to unconditional generation, we show that our model can be used to solve a variety of tasks such as user-guided editing, scene completion from a single scan, and text-to-3D. The source code and more results can be found at https://research.nvidia.com/labs/toronto-ai/xcube/.
1. Introduction
XCube targets scalable 3D generation by combining sparse voxel hierarchies with hierarchical latent diffusion. It generates high-resolution objects and large outdoor scenes while supporting multiple attributes and conditional tasks.
- 1024^3 resolution is achieved through a sparse formulation that concentrates dense geometry near object surfaces.
- XCube generates high-resolution 3D objects and scenes with arbitrary attributes, including signed distances, normals, and semantics.
- The model uses latent diffusion over a sparse voxel hierarchy, generating levels progressively from coarse to fine.
- XCube supports multi-scale user-guided editing by modifying coarse levels and regenerating finer levels.
- The method achieves state-of-the-art unconditional and conditional generation on object datasets and high-quality generation on large outdoor scenes.
- It generates complex shapes with millions of voxels at 1024^3 resolution in under 30 seconds using a VDB-based sparse 3D framework.
2. Related Work
Prior 3D generative models use diverse representations and either distill 2D priors or learn directly from 3D data. XCube focuses on sparse voxel hierarchies to address high-resolution and large-scene generation.
- Generative Probabilistic Models: Latent diffusion performs diffusion in the latent space of a simpler generative model and allows different decoders to produce multiple attributes.
- 3D Generative Models: 3D generative methods either distill 2D image priors through inverse rendering or directly learn 3D priors from large datasets.
- Large-Scale Scene Generation: Procedural scene methods decouple components but restrict the possible scenes and require specialized training data.
- 3D Representation for Generative Tasks: Voxel grids are flexible, expressive for chunky and thin structures, and support fast querying and processing.
- 3D Generative Models: XCube combines hierarchical latent diffusion with sparse voxel representations to model large-scale 3D scenes without time-consuming test-time optimization.
3. Method
XCube represents scenes as coarse-to-fine sparse voxel hierarchies and models their latent distributions with level-conditioned diffusion. Sparse VAEs, progressive decoding, and sequential sampling support scalable high-resolution generation.
- Sparse Voxel Hierarchy: The representation contains L nested voxel grids with per-voxel attributes, where finer levels contain smaller voxels and more detail.
- Sparse Structure VAE: Sparse structure VAEs encode each grid and its attributes into compact continuous latents for downstream diffusion modeling.
- Sparse Structure VAE: VAE decoding progressively subdivides voxels into octants, prunes excessive voxels, and doubles resolution through upsampling layers.
- Hierarchical Voxel Latent Diffusion: Single-level latent diffusion lacks sufficient resolution for large outdoor scenes, motivating a hierarchical factorization conditioned on coarser levels.
- Hierarchical Voxel Latent Diffusion: The factorization assumes each level depends only on its coarser predecessor, enabling computation sharing and editing or resampling at different levels.
- Training and Sampling: Sampling begins with the coarsest latent and sequentially decodes grids and attributes while conditioning each finer diffusion model on the preceding level.
- Implementation Details: A refinement network mitigates error accumulation because higher-resolution grids cannot easily correct artifacts from earlier levels.
4. Experiments
XCube is evaluated across object- and scene-level generation, conditional tasks, scalability, and ablations. The experiments report strong text-to-3D preference, large-scale scene quality, and evidence that hierarchical design choices preserve detail and efficiency.
- Experiments: XCube is evaluated on unconditional ShapeNet generation, category- and text-conditional Objaverse generation, large-scale Karton City and Waymo scenes, and ablation studies.The evaluation uses 1-NNA for ShapeNet geometric quality and user studies for text-to-3D and scene realism.
- Object-level 3D Generation on ShapeNet: XCube generates millions of voxels, approximately 500× more than point-based methods limited to 2048 points, while supporting complex geometric structures and arbitrary attributes.The comparison notes that triplane effectiveness diminishes for intricate Car geometry, whereas XCube uses a sparse voxel hierarchy and outputs sparse voxel grids convertible to meshes.
- Object-level 3D Generation on Objaverse: 79.2% of 900 pairwise comparisons favored XCube over Shap·E for text matching and geometric fidelity.The study used 30 prompts and 30 users per prompt, comparing XCube with Shap·E without texture.
- Object-level 3D Generation on Objaverse: XCube generates more diverse 3D objects with higher geometric fidelity and finer details than Shap·E, while geometry and texture generation take about 1 minute per object.The reported runtime is approximately 30 seconds for geometry and 30 seconds for texture synthesis.
- Large-scale Scene-level 3D Generation: Single-scan conditioning produces complete scenes with plausible geometry and semantics even though the input LiDAR scan contains no semantics.The generated outputs include normals and semantic labels.
- Ablation Study: Removing progressive pruning lowers reconstruction grid IoU from 92.88% to 89.68% and increases GPU memory usage by 3×.The ablation replaces progressive pruning with a single pruning step for the 16^3→128^3 VAE on ShapeNet Chairs.
- Ablation Study: Hierarchical models outperform single-level models, while two- and three-level configurations achieve comparable performance; 16^3 is sufficient for the initial hierarchy level.The authors use two levels for fast unconditional sampling and three levels for easier user editing.
5. Discussion
XCube is presented as a hierarchical sparse-voxel generative model for high-resolution 3D objects and scenes, demonstrated across object- and scene-level generation. Its text-to-3D capability remains limited by the relative scarcity of 3D data compared with image datasets.
- XCube models large-scale 3D scenes as hierarchies of sparse voxel grids and learns their joint latent distribution with hierarchical voxel latent diffusion.The method targets high-resolution scenes with fine details at both object and scene scales.
- The model is effective for both object-level and scene-level generation, including high-resolution 3D scenes with fine details.
- Complex prompts remain difficult for XCube’s text-to-3D model because current 3D datasets are not yet comparable to image datasets.The authors identify image-conditioning and downstream-task extensions as future directions.
Appendices
The supplementary material supplies additional method, implementation, loss, metric, and qualitative-result details supporting the main paper’s experiments.
- Appendix A: Appendix A describes the sparse 3D deep learning framework and compares it with state-of-the-art implementations.
- Appendix B: Appendix B provides implementation details and precise definitions of the loss function and evaluation metrics.
- Appendix C: Appendix C presents additional qualitative results across the datasets used for training and evaluation.
A. Sparse 3D Learning Framework
XCube uses a customized sparse 3D learning framework built on NanoVDB to represent and process sparse voxel grids efficiently. Benchmark results indicate advantages in speed and memory efficiency, particularly at large resolutions.
- NanoVDB provides a GPU-friendly VDB representation for sparse feature grids and supports operations such as convolution and pooling.The VDB tree uses hierarchical layers for compact sparse-grid storage.
- The custom framework is benchmarked against TorchSparse, a state-of-the-art sparse deep-learning framework.The comparison is reported in Table 3.
- The custom framework is fast and memory-efficient, especially for large input grid resolutions.The VDB-based representation supports compact storage and more efficient nearest-neighbor lookup and processing than the hash-table counterpart.
B. Implementation Details
The implementation supports arbitrary voxel attributes, multiple conditioning mechanisms, and detailed surface reconstruction. Additional procedures address sparse outdoor scans, scale control, open surfaces, and texture generation.
- Voxel attributes: Each voxel can output surface normals, semantic labels, and neural kernel features for detailed subvoxel-level surface extraction.The neural kernel features can also be replaced with implicit features for alternative surface representations.
- Training losses: Training combines normal, semantic, and surface losses, comparing predicted TSDF values with ground-truth TSDF values sampled across 3D space.The reported weights are λ1 = 1, λ2 = 15, and λ3 = 1.
- Surface reconstruction: Neural kernels are interpolated at arbitrary positions and used in a linear solve to compute continuous TSDF values through a Bezier-kernel surface-fitting formulation.For open surfaces, neural kernels can instead be replaced with implicit features processed by a local MLP.
- Conditioning: Text prompts are fused with latent features through cross-attention, category conditions through AdaGN, and previous-level attributes may be concatenated before latent diffusion.User-control cases omit previous-level attribute concatenation to allow flexible voxel addition or deletion.
- Conditioning: Micro-conditioning injects voxel-count information into the diffusion backbone to mitigate missing voxels in Waymo LiDAR scans and control generated scene scale.The associated figure reports increasing voxel counts and more diverse sampled-scene contents as conditioning increases.
- Texture generation: TEXTure generates textures for generated shapes by applying depth-conditioned stable-diffusion models across multiple views, with later steps conditioned on earlier ones for consistency.The procedure targets texture maps rather than the primary 3D geometry generation.
B.4. Network Architecture
The VAE encodes sparse voxel grids and attributes into latent tensors for diffusion, while attribute-specific decoder heads reconstruct voxel attributes. The architecture also includes a sparse 3D diffusion backbone and supports diverse texture synthesis from shared geometry.
- VAE: The VAE positionally encodes each voxel, concatenates its attributes, and compresses the result through convolution and pooling layers into a latent tensor.At the coarsest hierarchy level, the bottleneck is dense; otherwise, it remains sparse.
- VAE: Attribute-specific MLP heads decode the latent representation to predict the associated attributes within each voxel.The representation can support attributes such as geometry-related quantities and semantics through separate decoding heads.
- Texture synthesis: Diverse textures can be synthesized from the same generated geometry using TEXTrue [52].
- Diffusion UNet: A sparse 3D variant of the backbone from [13] is used for voxel latent diffusion.
B.5. Training Details
The models are trained with Adam, exponential moving averages, PyTorch Lightning, and dataset-dependent NVIDIA GPU configurations.
- Hardware and framework: Training uses PyTorch Lightning, with 8× NVIDIA Tesla V100s for ShapeNet and 8× NVIDIA Tesla A100s for other datasets.
B.6. Metric Definition
The 1-NNA metric evaluates whether generated and reference point clouds can be distinguished by nearest-neighbor distances. Scores near 50% indicate that the generated set is close in distribution to the reference set.
- Definition: The 1-NNA metric is computed from generated and reference point-cloud sets using nearest-neighbor membership indicators.For each point cloud, the nearest neighbor is selected under a distance metric from the combined sets, excluding the point cloud itself.
- Interpretation: A 1-NNA score around 50% is considered best because it indicates that generated and reference sets are difficult to classify by nearest neighbors.
- Evaluation protocol: The evaluation samples 2048 surface points per shape and computes 1-NNA using Chamfer and Earth Mover’s distances.
C. More results
Additional qualitative results cover text-to-3D, ShapeNet objects, Waymo, and Karton City. The authors also analyze shape novelty and provide dataset-specific VAE and diffusion hyperparameter references.
- Qualitative results: The supplementary results include text-to-3D outputs on Objaverse and additional ShapeNet, Waymo, and Karton City generations.The ShapeNet results span retrieved-shape novelty analysis and car, airplane, and chair categories.
- Shape novelty: Shape novelty is assessed by retrieving the three most similar training shapes to each generated shape using Chamfer distance.
- Qualitative results: Additional qualitative figures report results for ShapeNet Chair, Waymo, and Karton City.
- Hyperparameters: Karton City uses the same VAE and voxel latent diffusion hyperparameters as Waymo.