Source-linked AI summary
Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention
Shuang Wu, Youtian Lin, Feihu Zhang, Yifei Zeng, Yikang Yang, Yajie Bao, Jiachen Qian, Siyu Zhu, Xun Cao, Philip Torr, Yao Yao
TL;DR
High-resolution SDF-based 3D generation is computationally and memory intensive. Direct3D-S2 uses sparse volumes, Spatial Sparse Attention, and a unified sparse VAE to improve scalability, with reported speedups and 1024^3 training using eight GPUs.
Problem
High-resolution 3D shape generation with volumetric SDF representations faces substantial computational and memory challenges, while heterogeneous 3D VAE representations compromise efficiency.
Method
Direct3D-S2 uses sparse volumetric representations, Spatial Sparse Attention with learnable compression and selection, sparse conditioning, and a symmetric sparse SDF VAE.
Results
Direct3D-S2 generates high-quality gigascale 3D shapes, achieves 3.9× forward and 9.6× backward speedups over FlashAttention-2 at 128k tokens, and trains at 1024^3 resolution using 8 GPUs.
Takeaways & Limitations
The framework makes high-resolution 3D generation more computationally efficient and enables training at 1024^3 resolution with relatively few GPUs.
Takeaways & Limitations
SSA’s forward-pass acceleration is smaller than its backward-pass acceleration because top-k sorting adds computational overhead.
Abstract
from arXiv · showhide
Generating high-resolution 3D shapes using volumetric representations such as Signed Distance Functions (SDFs) presents substantial computational and memory challenges. We introduce Direct3D-S2, a scalable 3D generation framework based on sparse volumes that achieves superior output quality with dramatically reduced training costs. Our key innovation is the Spatial Sparse Attention (SSA) mechanism, which greatly enhances the efficiency of Diffusion Transformer (DiT) computations on sparse volumetric data. SSA allows the model to effectively process large token sets within sparse volumes, substantially reducing computational overhead and achieving a 3.9x speedup in the forward pass and a 9.6x speedup in the backward pass. Our framework also includes a variational autoencoder (VAE) that maintains a consistent sparse volumetric format across input, latent, and output stages. Compared to previous methods with heterogeneous representations in 3D VAE, this unified design significantly improves training efficiency and stability. Our model is trained on public available datasets, and experiments demonstrate that Direct3D-S2 not only surpasses state-of-the-art methods in generation quality and efficiency, but also enables training at 1024 resolution using only 8 GPUs, a task typically requiring at least 32 GPUs for volumetric representations at 256 resolution, thus making gigascale 3D generation both practical and accessible. Project page: https://www.neural4d.com/research/direct3d-s2.
1. Introduction
Direct3D-S2 addresses high-resolution 3D generation with a unified sparse-volume framework. Its SSA mechanism improves DiT efficiency, while experiments report detailed generation using substantially fewer GPUs.
- Direct3D-S2 uses sparse volumetric representations to address the computational challenge of high-resolution 3D shape generation.
- SSA selectively attends to spatially important tokens through learnable compression and selection modules.
- SSA adapts sparse attention to irregular 3D data by preserving spatial coherence and redesigning its core modules.
- The SS-VAE maintains a consistent sparse volumetric format across input, latent, and output stages, avoiding cross-modality translation.
- 8 GPUs suffice for training at 1024^3 resolution, whereas prior methods typically require at least 32 GPUs at 256^3 resolution.
2. Related work
Prior 3D generation methods use multi-view reconstruction, implicit vecsets, or sparse voxel latents. These approaches face consistency, reconstruction, representation, or attention-efficiency challenges at scale.
- 2.1. Multi-view Generation and 3D Reconstruction: Multi-view methods generate images from 2D priors and reconstruct 3D shapes, but may suffer from multi-view inconsistency and reconstruction errors.
- 2.1. Multi-view Generation and 3D Reconstruction: Rendering-based supervision avoids direct 3D supervision but adds substantial training complexity and computational overhead.
- 2.2. 3D Latent Diffusion: Implicit vecset methods represent shapes with latent vecsets and reconstruct meshes through neural SDFs or occupancy fields.
- 2.3. Efficient Large Tokens Generation: NSA reduces attention cost through adaptive token compression, while SSA extends these ideas to preserve spatial coherence in sparse 3D token sets.
3. Sparse SDF VAE
The SS-VAE processes sparse SDF volumes instead of dense volumes and uses a symmetric encoder-decoder to produce and reconstruct sparse latent representations.
- 3D VAEs face heterogeneous representations across meshes, point clouds, and implicit fields, leading to asymmetric architectures and compromised efficiency.
- The SS-VAE encodes an SDF volume V into latent representation z and reconstructs it as a decoded SDF volume.
- Dense R^3 SDF processing is computationally prohibitive, so the method focuses on valid sparse voxels whose absolute SDF values fall below threshold τ.
- The symmetric encoder combines sparse 3D convolutions, mean pooling, and transformer processing for variable-length sparse voxel tokens.
SS-DiT
The SS-DiT generates image-conditioned 3D shapes from sparse latent tokens. Its sparse representation and training procedures support scalable reconstruction and efficient high-resolution generation.
- SS-DiT: SS-DiT trains an image-conditioned transformer on serialized sparse latent tokens produced by the SS-VAE.
- SS-DiT: The decoder progressively upsamples latent representations with attention layers and sparse 3D CNN blocks to reconstruct SDF volumes.
- SS-DiT: Decoded sparse voxels include both input and additional valid voxels, with SDF supervision applied across their spatial positions.
- SS-DiT: Training uses randomly sampled resolutions from 256^3, 384^3, 512^3, and 1024^3 before SS-VAE encoding.
4. Spatial Sparse Attention and DiT
Spatial Sparse Attention scales attention for high-resolution sparse 3D volumes by combining global compression, selective fine-grained block retrieval, and local windows. The framework integrates these modules with gated aggregation and sparse conditioning for efficient image-conditioned generation.
- Spatial Sparse Attention: SSA constructs spatially coherent blocks and combines sparse 3D compression, spatial blockwise selection, and sparse 3D window modules.Compression captures block-level global information, selection retains fine-grained token features, and windows provide localized interactions.
- Motivation: Over 100k tokens at 1024^3 resolution make conventional attention prohibitively inefficient, motivating spatially coherent block partitioning and blockwise selection.The method addresses spatial misalignment and unstable convergence caused by treating sparse 3D latent tokens as a fixed-index 1D sequence.
- Sparse 3D Compression: The compression module uses intra-block positional encoding, sparse 3D convolution, and mean pooling to reduce tokens while capturing block-level global information.The resulting block-level representations provide coarse context for subsequent attention computation.
- Spatial Blockwise Selection: Spatial blockwise selection ranks compression blocks by attention scores, selects the top-k blocks, and applies attention to their concatenated tokens.Grouped-Query Attention accumulates scores across shared query heads to improve computational efficiency.
- Sparse 3D Window: A sparse 3D window module dynamically aggregates active tokens within each non-overlapping window and performs localized self-attention over that subset.This auxiliary module explicitly incorporates local feature interactions.
- Sparse Conditioning Mechanism: Sparse conditioning extracts foreground image tokens before cross-attention, reducing conditioning tokens and mitigating background-related computational overhead.The tokens are produced from DINO-v2 features using foreground extraction, positional encoding, and a linear layer.
5. Experiments
Experiments evaluate Direct3D-S2 on curated public 3D data, comparing image-to-3D quality and training configurations with prior approaches. The reported results show strong image alignment and user preference, while the framework trains at 1024^3 resolution using limited hardware.
- 5.1. Datasets: Approximately 452k 3D assets were curated from public datasets through rigorous filtering for training.The datasets include Objaverse, Objaverse-XL, and ShapeNet.
- 5.1. Datasets: The evaluation measures generated-mesh alignment with input images using ULIP-2, Uni3D, and OpenShape.The benchmark uses highly detailed images sourced from Neural4D, Meshy, and CivitAI.
- 5.3. Quantitative and Qualitative Comparisons: Direct3D-S2 outperforms other approaches across all three image-alignment metrics and better preserves fine structures such as railings and tree branches.The comparison reports higher alignment and finer geometric details than competing methods affected by resolution limitations.
- 5.3. Quantitative and Qualitative Comparisons: 40 participants statistically preferred Direct3D-S2 over other methods for both image consistency and overall geometric quality.The study evaluated 75 unfiltered generated meshes using scores from 1 to 5.
6. Comparison of VAE
Direct3D-S2 combines sparse SDF reconstruction with efficient attention to improve high-resolution 3D generation. Experiments report better reconstruction and mesh quality, faster attention, and practical scaling to 1024^3 resolution.
- VAE reconstruction: SS-VAE achieves superior reconstruction accuracy at 512^3 and markedly improves complex-geometry reconstruction at 1024^3.The validation set contains meshes with complex geometric structures from Objaverse.
- Resolution scaling: Increasing image-to-3D resolution from 256^3 to 512^3 progressively improves mesh quality and high-frequency geometric detail.At lower resolutions, meshes show limited detail and image misalignment; 512^3 produces enhanced high-frequency geometry.
- SSA ablations: Adding sparse 3D compression and spatial blockwise selection improves mesh quality by combining global context with attention to important regions.Using only sparse 3D windows causes surface irregularities, while the complete configuration improves quality.
- Attention efficiency: At 128k tokens, SSA is 3.9× faster than FlashAttention-2 in forward execution and 9.6× faster in backward execution.SSA has comparable speed at low token counts, with a growing advantage as token counts increase.
- SSA ablations: SSA produces smoother, more organized surfaces than NSA, whose one-dimensional block partitioning causes positional ambiguity and training instability.The comparison is conducted at 512^3 resolution against full attention, NSA, and SSA configurations.
- Sparse conditioning: Sparse conditioning improves alignment with input images by excluding non-foreground conditioning tokens.This effect is reported in ablation experiments at 512^3 resolution.
7. Conclusion
Direct3D-S2 combines Spatial Sparse Attention with a symmetric sparse SDF VAE to accelerate high-resolution 3D generation while improving training stability and efficiency. Experiments report stronger generation quality than existing image-to-3D methods with training on only 8 GPUs.
- Spatial Sparse Attention significantly accelerates DiT training and inference for high-resolution 3D generation.
- The fully end-to-end symmetric sparse SDF VAE further improves training stability and efficiency.
- 8 GPUs are sufficient for training Direct3D-S2, while maintaining high generation quality.
8. Limitations
SSA provides substantial speed improvements over FlashAttention-2, but forward acceleration is limited by top-k sorting overhead.
- Top-k sorting operations introduce computational overhead that reduces SSA’s forward-pass acceleration relative to its backward-pass acceleration.