PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation

Chenguo Lin, Panwang Pan, Bangbang Yang, Zeming Li, Yadong Mu

arXiv:2501.16764v1cs.CV

TL;DR

Text- or image-conditioned 3D generation is limited by scarce high-quality 3D supervision and inconsistent multi-view generation. DIFFSPLAT fine-tunes image diffusion models to generate 3D Gaussian splats using reconstructed splat grids and rendering-aware training, and reports strong text- and image-conditioned generation with controllable 3D applications. Its main scope boundaries are unresolved high-quality mesh conversion and reliance on rendered multi-view datasets.

  • Problem

    Text- or single-image-conditioned 3D generation requires inferring unseen object parts while facing limited high-quality 3D datasets and inconsistent multi-view generation.

  • Method

    DIFFSPLAT fine-tunes pretrained image diffusion models to generate Gaussian-splat properties from splat latents, combining diffusion loss with a 3D rendering loss.

  • Results

    DIFFSPLAT reports strong prompt alignment and visual quality for text-conditioned generation, accurate image-aligned 3D content with geometric fidelity, and controllable high-quality 3D assets.

  • Takeaways & Limitations

    Compatibility with image diffusion models allows image-generation techniques and conditioning strategies to be adapted for 3D applications.

  • Takeaways & Limitations

    High-quality conversion from DIFFSPLAT's 3DGS representation to mesh remains unsolved, and the current work uses only rendered multi-view datasets.

Abstract

from arXiv · show

Recent advancements in 3D content generation from text or a single image struggle with limited high-quality 3D datasets and inconsistency from 2D multi-view generation. We introduce DiffSplat, a novel 3D generative framework that natively generates 3D Gaussian splats by taming large-scale text-to-image diffusion models. It differs from previous 3D generative models by effectively utilizing web-scale 2D priors while maintaining 3D consistency in a unified model. To bootstrap the training, a lightweight reconstruction model is proposed to instantly produce multi-view Gaussian splat grids for scalable dataset curation. In conjunction with the regular diffusion loss on these grids, a 3D rendering loss is introduced to facilitate 3D coherence across arbitrary views. The compatibility with image diffusion models enables seamless adaptions of numerous techniques for image generation to the 3D realm. Extensive experiments reveal the superiority of DiffSplat in text- and image-conditioned generation tasks and downstream applications. Thorough ablation studies validate the efficacy of each critical design choice and provide insights into the underlying mechanism.

1 INTRODUCTION

DIFFSPLAT addresses limitations of prior 3D generation approaches by directly generating 3D Gaussian splats with pretrained image-diffusion priors. Its structured splat representation and lightweight reconstruction model support scalable dataset curation while targeting multi-view consistency.

  • Motivation: Single-image or text-conditioned 3D generation is ill-posed because unseen object parts must be inferred, challenging fidelity and generalizability.The problem has applications including game design, digital arts, human avatars, and virtual reality.
  • Motivation: Prior native and rendering-based methods face limited 3D supervision, while reconstruction-based methods can produce inconsistent multi-view images.These limitations motivate using large-scale 2D generative priors without abandoning 3D coherence.
  • Approach: DIFFSPLAT directly generates 3D Gaussian splats while leveraging pretrained image-diffusion models to maintain multi-view consistency.Gaussian Splatting is selected for its balance of rendering efficiency and quality.
  • Approach: A lightweight model regresses structured splat 2D grids from multi-view images in less than 0.1 seconds, enabling scalable 3D dataset curation.Each grid stores Gaussian-splat properties that encode object texture and structure.
  • Approach: The framework fine-tunes image diffusion models and adapts their latent representations to generate Gaussian-splat properties directly.Reconstructed splat grids are treated as images in a special style, allowing pretrained 2D models to operate on them.
  • Implications: DIFFSPLAT is designed to transfer pretrained image-generation techniques and associated advances into 3D generation while preserving 3D consistency.The authors report extensive experiments and ablations to evaluate performance and design choices.

2 RELATED WORK

Prior 3D generative methods fall into native, rendering-based, and reconstruction-based categories, each using different supervision and encountering distinct scalability or consistency constraints.

  • Native 3D Generative Models: Native 3D generative models train denoising networks on explicit or implicit 3D representations, but dataset access, preprocessing cost, and limited use of pretrained 2D models constrain scale and training efficiency.Explicit representations are readily available but often have poor visual quality, whereas implicit representations can offer more faithful appearances.
  • Rendering-based 3D Generative Models: Rendering-based generative models use differentiable rendering to train with 2D supervision instead of ground-truth clean 3D data.They denoise images rendered from corrupted implicit 3D representations.
  • Reconstruction-based 3D Generative Models: Reconstruction-based methods reconstruct 3D fields from single images using multi-view supervision and can leverage frozen image-diffusion models to generate multiview images.Their deterministic reconstruction setup differs from denoising-based generative models.

3 METHOD

DIFFSPLAT structures 3D objects as multi-view Gaussian splat grids, adapts image diffusion models to splat latents, and adds rendering supervision to improve 3D consistency.

  • 3.1 DATA CURATION: STRUCTURED SPLAT RECONSTRUCTION: DIFFSPLAT uses structured multi-view Gaussian splat grids as an efficient 3D representation for scalable data curation.A lightweight network predicts per-pixel splats from posed images, while the method overview describes these representations as supporting pseudo-dataset curation.
  • 3.1 DATA CURATION: STRUCTURED SPLAT RECONSTRUCTION: Coordinate and normal maps supplement posed RGB images during reconstruction but are unnecessary during generation.These inputs provide auxiliary geometric guidance for reconstructing Gaussian splat grids.
  • 3.2 SPLAT LATENTS: The method fine-tunes image VAEs to encode Gaussian splat properties into image-diffusion-compatible splat latents.The VAE is trained with reconstruction and rendering losses, and the resulting latents undergo diffusion and denoising.
  • 3.3.1 MODEL ARCHITECTURE: View-concat and spatial-concat organize multi-view splat latents differently, with conditioning mechanisms adapted for image-conditioned generation.View-concat treats views like video frames, whereas spatial-concat arranges them in a spatial grid; image and splat latents require explicit separation.
  • 3.3.2 TRAINING OBJECTIVES: DIFFSPLAT combines regular diffusion loss with a rendering loss that decodes denoised splat latents and supervises renders from random viewpoints.The rendering term addresses the lack of guaranteed 3D consistency when diffusion operates on 2D splat grids.

4 EXPERIMENTS

The experiments train all models on G-Objaverse and evaluate text-conditioned generation with 300 prompts spanning several object-and-scene configurations.

  • 4 EXPERIMENTS: All models are trained on G-Objaverse, which contains images from 38 views of approximately 265K 3D objects.Captions are provided by Cap3D, and 300 T3Bench prompts evaluate text-conditioned generation across single and multiple-object settings.

4.2 TEXT-CONDITIONED GENERATION

DIFFSPLAT achieves the strongest prompt alignment and visual quality among evaluated text-conditioned 3D generation methods, particularly for complex prompts. Native 3D methods struggle with prompt matching, while reconstruction-based methods suffer from multi-view inconsistency.

  • DIFFSPLAT achieves the best prompt alignment and visual quality among cutting-edge text-conditioned 3D generation methods, especially for complex prompts.
  • Native 3D methods struggle to match text prompts because they train from scratch with limited text-3D pairs.
  • Reconstruction-based methods suffer from multi-view diffusion inconsistency, particularly when objects have surroundings or interact with other objects.

4.3 IMAGE-CONDITIONED GENERATION

The image-conditioned evaluation compares DIFFSPLAT with native 3D and reconstruction-based methods on GSO and in-the-wild images. DIFFSPLAT produces image-aligned 3D content while maintaining strong geometric fidelity.

  • The comparison includes two native 3D models and six reconstruction-based methods, including Gaussian Splatting- and FlexiCube-based approaches.
  • DIFFSPLAT delivers accurate 3D content aligned with input images while maintaining strong geometric fidelity compared with other state-of-the-art methods.
  • Single image-conditioned performance is assessed quantitatively on the GSO dataset and qualitatively on in-the-wild images.

4.4 APPLICATION: CONTROLLABLE GENERATION

DIFFSPLAT adapts image-generation conditioning techniques to controllable 3D generation. It responds to normal, depth, and Canny-edge controls while preserving text conditions and supports text-guided reconstruction from ambiguous single-view images.

  • DIFFSPLAT adapts ControlNet to generate diverse, high-quality 3D assets from normal, depth, and Canny-edge controls alongside text prompts.
  • The generated assets accurately respond to different control inputs while faithfully reflecting the text conditions.
  • Flexible conditioning also enables text-guided reconstruction from single-view ambiguous images.

4.5 ABLATION AND ANALYSIS

The ablations examine reconstruction inputs, auto-encoding, multi-view organization, training objectives, and diffusion backbones. They show that geometric guidance, rendering supervision, and compatibility with image diffusion models support efficient and coherent 3D generation.

  • Splat latent reconstruction: A lightweight reconstruction model using coordinate and normal maps provides high-quality Gaussian splat grids as pseudo-ground truth.Coordinate maps give more effective geometric guidance than normal maps.
  • Splat latent reconstruction: Rendering loss improves Gaussian splat auto-encoding because it supervises the VAE with real datasets rather than only reconstructed outputs.Freezing the original image VAE or encoder performs poorly because Gaussian splat properties differ from natural images.
  • DIFFSPLAT 3D generation: View-concat performs better for single image-conditioned generation, while both multi-view arrangements produce similar text-conditioned results.View-concat is preferred for flexibility with varying viewpoints and conditioning.
  • DIFFSPLAT 3D generation: The 3D rendering loss further improves aesthetic quality and geometric structure, while perceptual and mask losses contribute textured details and fewer translucent artifacts.Regular diffusion loss alone can perform well when splat latents are high quality.
  • DIFFSPLAT 3D generation: ControlNet adapts to DIFFSPLAT for controllable text-to-3D generation using normal maps, depth maps, and Canny edges.The generated 3D assets respond to control inputs while reflecting text conditions.
  • DIFFSPLAT 3D generation: DIFFSPLAT consistently benefits from stronger base text-to-image diffusion models in both text- and image-conditioned tasks.Splat latents can be visualized as RGB or grayscale images, supporting the use of image diffusion models for Gaussian splat generation.

5 CONCLUSION

DIFFSPLAT directly generates 3D Gaussian splat properties by fine-tuning image diffusion models with diffusion and 3D rendering losses. The framework leverages web-scale 2D priors and multi-view supervision while leaving high-quality mesh conversion unresolved.

  • Conclusion: DIFFSPLAT fine-tunes large image diffusion models to generate 3D Gaussian splat properties with diffusion and 3D rendering losses.The framework is designed to maintain 3D coherence while using web-scale 2D priors.
  • Limitations and Future Work: The conversion of DIFFSPLAT’s 3DGS representation to a high-quality mesh remains an unsolved problem.The paper identifies improved viewpoint coverage, supervision resolution, materials, and broader data sources as future directions.

A IMPLEMENTATION DETAILS

The implementation uses a lightweight reconstruction network, standard diffusion-model solvers, and limited training resources. DIFFSPLAT is reported to require substantially less training compute than several prior 3D generative models.

  • Training: The Gaussian splat reconstruction model is a 12-layer, 8-head Transformer with 42M parameters, using four input views and eight rendering views.Its parameter size is reported as 9.9%–23% of previous methods.
  • Inference: Diffusion-based models use DPM-Solver++ with 20 inference steps, while SD3 uses its flow-matching Euler ODE solver with 28 steps.The solver choices follow the corresponding model configurations.
  • Cost: DIFFSPLAT takes about 3 days on 8 A100 GPUs to produce decent results with fp16 mixed precision.The paper contrasts this cost with substantially larger GPU-time budgets for DMV3D, CLAY, and 3DTopia-XL.

B MORE VISUALIZATION RESULTS

The additional visualizations cover multimodal control, text-conditioned generation, and image-conditioned generation. They illustrate DIFFSPLAT outputs and its use of text and image conditions for 3D content creation.

  • Multimodal conditions: DIFFSPLAT uses both text and image conditions for single-view reconstruction with text understanding.The multimodal condition is presented as controllable generation.
  • Text-conditioned generation: Additional figures provide more text-conditioned DIFFSPLAT results.Figures 9–11 extend the text-conditioned visualizations.
  • Image-conditioned generation: Additional figures provide more image-conditioned DIFFSPLAT results.Figures 12–14 extend the image-conditioned visualizations.
Loading 2501.16764v1…