Source-linked AI summary
FLUX3D: High-Fidelity 3D Gaussian Generation with Diffusion-Aligned Sparse Representation
Haorui Ji, Weizhe Liu, Hongdong Li, Hengkai Guo
TL;DR
FLUX3D addresses two bottlenecks in sparse-voxel image-to-3DGS generation: reconstructive information loss from discriminative 2D features and weak alignment between dense 2D tokens and sparse 3D latents. It introduces DA-SLAT with decoder-only reconstruction plus SMDiT and MARoPE for sparse-structure-aware diffusion, and reports high-fidelity, image-consistent assets that outperform SOTA methods across key metrics.
Problem
Sparse-voxel methods struggle to preserve fine-grained appearance because discriminative 2D features suppress reconstructive cues and standard diffusion transformers poorly align dense 2D tokens with sparse 3D latents.
Method
FLUX3D uses FLUX diffusion features in DA-SLAT with a decoder-only architecture, and combines SMDiT with MARoPE for sparse-structure-aware diffusion.
Results
FLUX3D generates high-fidelity, image-consistent 3D assets and outperforms SOTA methods across key evaluation metrics.
Takeaways & Limitations
The combined representation and diffusion designs produce 3D assets with sharp structural details and intricate textures.
Takeaways & Limitations
Semantic subjects such as text and logos remain less capable than 2D methods, and input feature volumes rely heavily on multiview renderings.
Abstract
from arXiv · showhide
Sparse voxel representation has emerged as a scalable foundation for image-to-3D Gaussian Splatting (3DGS) generation, yet current methods struggle to preserve high-frequency visual details of input images due to two structural bottlenecks. First, they adopt discriminative 2D features optimized for semantic abstraction to construct sparse voxel latents, which suppress reconstructive cues and induce a representation bottleneck. Second, in the generation stage, standard diffusion transformers lack effective mechanisms to align dense 2D image tokens with sparse 3D voxel latents, resulting in a cross-modal correspondence bottleneck. To address these issues, we propose FLUX3D, a scalable image-to-3DGS framework that boosts both representation learning and cross-modal alignment during generation. We first revisit 2D feature selection for sparse-voxel-based 3D representation learning, propose Diffusion-Aligned Structured Latents (DA-SLAT) and couple it with a decoder-only architecture to improve 3DGS reconstruction fidelity. We also design a sparse-structure-aware diffusion framework, which integrates the Sparse-structure Multimodal Diffusion Transformer (SMDiT) and Modal-Aware Rotary Positional Embedding (MARoPE) to achieve geometry-agnostic 2D-3D alignment. Extensive benchmark experiments demonstrate that FLUX3D yields substantial improvements in appearance fidelity and significantly outperforms all state-of-the-art (SOTA) methods in generating high-quality 3DGS assets.
1 Introduction
FLUX3D targets appearance degradation in sparse-voxel image-to-3DGS generation by addressing representation and cross-modal alignment bottlenecks. It introduces diffusion-aligned structured latents, decoder-only reconstruction, and sparse-structure-aware diffusion, with benchmark improvements over SOTA methods.
- Sparse voxel representations reduce computational and memory overhead while supporting global and local 3D structure modeling.
- Discriminative 2D features suppress reconstructive cues, while inadequate dense-2D-to-sparse-3D alignment impairs appearance fidelity and visual consistency.
- FLUX3D introduces DA-SLAT with a decoder-only architecture, using pretrained FLUX diffusion features to directly map structured latents to 3DGS outputs.
- SMDiT and MARoPE form a sparse-structure-aware diffusion framework that enhances alignment between sparse 3D voxel latents and dense 2D image tokens.
- Benchmarks show high-fidelity, image-consistent 3D assets and outperformance of SOTA methods across key evaluation metrics.
2 Related Works
Related work spans 2D-guided, deterministic native-3D, and stochastic native-3D generation. Sparse voxel methods improve efficiency and geometric awareness, but fine-grained texture preservation remains challenging.
- 2D Generation from 2D Guidance: 2D-guided methods use diffusion priors for image-to-3D generation but require expensive optimization and lack explicit 3D geometric awareness.
- Deterministic 3D Reconstruction Models: Deterministic reconstruction models provide feed-forward 3D prediction but cannot model uncertainty, often producing blurry textures, distorted geometry, or inconsistent details.
- Stochastic 3D Generation Models: Stochastic 3D generation addresses ambiguity and includes vecset-based and voxel-based latent representation families.
- Voxel-based Methods: Sparse voxel methods distribute latent features across explicit spatial structures, reducing computational overhead while retaining geometric awareness.
- Denoising Diffusion and Flow Matching: Diffusion and flow matching achieve strong generative performance across image, video, dense prediction, and 3D applications.
- Denoising Diffusion and Flow Matching: FLUX3D applies rectified flow to 3D generation and adapts its architecture for sparse voxel data to improve cross-modal consistency.
3 Preliminaries
Sparse voxel representations encode active voxel coordinates with associated features, enabling efficient high-resolution modeling and downstream decoding. Sparse-voxel generation commonly separates layout generation from feature generation, while FLUX3D decodes features into 3DGS parameters.
- Sparse Voxel-based Representation: A sparse voxel asset is represented by feature-coordinate pairs, with each active voxel storing a spatial index and associated feature.
- Sparse Voxel-based Representation: Because active voxels are much fewer than the full grid, sparse representation enables efficient high-resolution modeling while preserving spatial locality.
- Sparse Voxel-based Representation: Sparse feature volumes can be decoded into downstream 3D representations through task-specific heads.
- Sparse-voxel-based 3D Generation: Sparse-voxel generation typically uses two stages: generating active-voxel spatial arrangement, then deriving features that encode shape and appearance.
- 3D Gaussian Splatting: 3DGS represents assets with parameterized Gaussian primitives, and FLUX3D decodes sparse voxel features into these parameters as the final representation.
- Representation Learning Pipeline: FLUX3D extends TRELLIS-SLAT by aggregating multiview FLUX features and using a decoder-only architecture to map structured latents directly to 3DGS output.
4 Method
FLUX3D addresses representation and cross-modal alignment bottlenecks in sparse-voxel image-to-3DGS generation through diffusion-aligned latents, decoder-only reconstruction, SMDiT, and MARoPE.
- Representation Learning: FLUX3D adapts Trellis-SLAT by voxelizing 3D assets and aggregating multiview image features at active voxel locations.The original pipeline uses DINOv2 features with a sparse transformer encoder-decoder for structured latent learning and 3DGS reconstruction.
- Representation Learning: DA-SLAT replaces DINOv2 with FLUX diffusion features to retain appearance information better suited to faithful 3D reconstruction.DINOv2 features prioritize semantic abstraction, whereas FLUX features are optimized for image reconstruction and synthesis and preserve richer high-frequency appearance cues.
- Representation Learning: A decoder-only architecture directly maps FLUX structured latents to 3DGS outputs, avoiding additional latent compression and its associated information loss.Pretrained FLUX features are used directly as 3D structured latents without re-encoding into a separate compact representation.
- Sparse-Structure-Aware Diffusion: SMDiT aligns sparse 3D latents with dense 2D image tokens using modality-specific double-stream blocks followed by single-stream multimodal interaction blocks.The condition image is tokenized with a FLUX encoder, stored in a sparse voxel structure, and processed jointly with latent tokens.
- Sparse-Structure-Aware Diffusion: MARoPE places 2D image patches on a virtual plane outside the latent volume while retaining raw 3D voxel coordinates for geometry-agnostic alignment.The virtual-plane construction provides a shared relative coordinate frame so attention favors nearby voxels without explicit geometric calibration.
- Training: The representation module is trained with perceptual L1 reconstruction and geometry-aware Gaussian volume and opacity regularization, while diffusion uses conditional flow matching.Training also samples stochastic FLUX VAE outputs using µ + σ · N(0, I) to improve robustness to continuous diffusion outputs.
5 Experiments
FLUX3D is evaluated through reconstruction, image-conditioned generation, qualitative comparisons, and component ablations. Results show improved appearance fidelity from diffusion-aligned features, decoder-only reconstruction, and the combined DA-SLAT, SMDiT, and MARoPE design.
- Experimental scope: The experiments cover reconstruction and generation benchmarks, qualitative comparisons, feature-selection studies, architecture comparisons, and component ablations.Evaluation uses Toys4k and compares FLUX3D with established 3DGS generation methods.
- Reconstruction experiments: FLUX3D outperforms all baselines across evaluated reconstruction metrics, validating the use of 2D diffusion features for representation learning.Reconstruction is assessed with SSIM, PSNR, and LPIPS against ground-truth renderings.
- Generation experiments: Qualitative comparisons show better preservation of color accuracy and appearance details across characters, buildings, and artifacts, especially from novel viewpoints.LGM shows shape and appearance distortions, while DiffusionGS and TRELLIS exhibit inconsistent or misaligned textures from novel views.
- Input feature selection: Diffusion-aligned SDXL and FLUX features improve reconstruction performance and fine-grained details such as text and logo rendering.The feature-selection study evaluates DINOv2, DINOv3, RGB pixels, SDXL, and FLUX under an encoder-decoder architecture.
- Architecture comparison: The decoder-only architecture outperforms the encoder-decoder architecture in both reconstruction and generation when paired with diffusion-aligned structured latents.The comparison uses results from Tables 1 and 3 and visualizations in Figure 7.
- Ablation studies: DA-SLAT improves all generation metrics, while SMDiT and MARoPE provide complementary gains for cross-modal alignment and artifact reduction.The full configuration achieves the best results across all evaluated benchmarks; DA-SLAT and SMDiT are reported as more influential than MARoPE.
6 Conclusion
FLUX3D combines diffusion features with sparse voxel representation learning and sparse-structure-aware diffusion to improve image-to-3DGS appearance fidelity. The resulting assets contain sharp structural details and intricate textures.
- Conclusion: FLUX3D integrates 2D diffusion features, SMDiT, and MARoPE to improve reconstruction and cross-modal alignment for sparse-voxel image-to-3DGS generation.The framework targets detailed appearance modeling in generated 3D assets.
- Conclusion: The combined components produce 3D assets with sharp structural details and intricate textures.This conclusion concerns the framework’s supported appearance-modeling outcome.
1 Additional Implementation Details
The implementation uses curated 3D assets, specified rendering settings, and a fixed training and inference configuration. The data curation process addresses missing-texture artifacts that can receive misleadingly high aesthetic scores.
- Data curation: The curation process evaluates average aesthetic scores across ten spherical-view renders, but missing-texture assets can still score relatively highly.The paper categorizes assets by texture and display condition during curation.
- Optimization: Training uses AdamW at 1 × 10^-4 on 8 NVIDIA A100 GPUs, with SMDiT and decoder training for 200K and 100K steps.The implementation also specifies batch size, loss weights, and 50 inference sampling steps.
2 Additional Quantitative Results
Additional experiments evaluate FLUX3D’s geometry reconstruction, cross-format generation, robustness to noisy layouts, component contributions, and multi-view consistency. Results consistently favor FLUX3D, with especially strong gains in appearance fidelity and model-design ablations.
- Geometry Reconstruction: FLUX3D outperforms baselines across all evaluated geometry-quality metrics, although its gains are smaller than those for appearance fidelity.Geometry is measured with Chamfer Distance, while surface details use normal-map PSNR and LPIPS.
- Cross-Type Comparison: Cross-type evaluation compares 3DGS methods with NeRF-based and textured-mesh methods using rendering, CLIP, Fréchet, and Kernel metrics.The evaluation uses SSIM, PSNR, LPIPS, CLIP Score, Fréchet Distance, and Kernel Distance on generated versus ground-truth assets.
- Layout Robustness: Under noisy predicted layouts, FLUX3D’s lead over TRELLIS widens: FDi advantage increases from 23% to 33%, and FDd from 14% to 21%.The result suggests that FLUX3D’s gains are attributed to the FLUX prior and model design rather than sparse voxel layout alone.
- Ablation: The model design contributes a 16.6% FDi gain, compared with 7.3% from the FLUX prior, in cumulative ablations against TRELLIS.The design combines the decoder-only variant with SMDiT and MARoPE.
- Multi-View Consistency: Across 24 views over the full 360° azimuth, FLUX3D achieves higher mean PSNR and 18% lower standard deviation than TRELLIS.This evaluates both reconstruction accuracy and cross-view consistency on Toys4K.
3 Additional Qualitative Results
The supplied passages point to additional qualitative evidence and identify a multi-view consistency evaluation, but provide no detailed qualitative comparison beyond the referenced demo video.
- Qualitative Results: The authors direct readers to a demo video for additional qualitative results and comparisons with state-of-the-art methods.
- Multi-View Consistency: Table 7 reports per-view PSNR across 24 views over the full 360° azimuth for multi-view consistency.
4 Limitations and Future works
FLUX3D retains limitations in semantically meaningful appearance modeling and in constructing input feature volumes from multiview renderings.
- Appearance Modeling: Appearance modeling for semantically meaningful subjects such as text and logos still trails 2D methods.The authors suggest more comprehensive, high-quality data as a possible way to improve this capability.
- Representation Learning: Input feature-volume construction relies heavily on multiview renderings, motivating exploration of alternative strategies and image or video modalities.