Source-linked AI summary
GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting
Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong, Jinlin He, Zhiwei Lin, Yongtao Wang, Deqing Sun, Ming-Hsuan Yang
TL;DR
Complex scene creation remains labor-intensive, and layout-guided generation faces challenges in obtaining reasonable priors and avoiding visual artifacts from layout constraints. GALA3D combines LLM-derived layouts, layout-guided Gaussian representations, and compositional optimization; the paper reports outperforming existing methods while supporting controllable scene editing.
Problem
Complex scene creation is labor-intensive, and layout-guided methods face challenges obtaining reasonable priors without manual design and limiting artifacts from layout constraints.
Source: §1 · p. 2, §3.1 · p. 4Method
GALA3D uses LLM-generated coarse layouts to guide Gaussian representations with adaptive geometry control and compositional optimization for multi-object scenes.
Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5Results
Takeaways & Limitations
The paper presents GALA3D as an efficient, user-friendly framework for interactive, controllable scene generation and editing.
Source: §5 · p. 9The authors note potential labor-market risks from automated 3D modeling and the capacity to generate deceptive or malicious 3D content.
Source: Impact Statement · p. 10
Abstract
from arXiv · showhide
We present GALA3D, generative 3D GAussians with LAyout-guided control, for effective compositional text-to-3D generation. We first utilize large language models (LLMs) to generate the initial layout and introduce a layout-guided 3D Gaussian representation for 3D content generation with adaptive geometric constraints. We then propose an instance-scene compositional optimization mechanism with conditioned diffusion to collaboratively generate realistic 3D scenes with consistent geometry, texture, scale, and accurate interactions among multiple objects while simultaneously adjusting the coarse layout priors extracted from the LLMs to align with the generated scene. Experiments show that GALA3D is a user-friendly, end-to-end framework for state-of-the-art scene-level 3D content generation and controllable editing while ensuring the high fidelity of object-level entities within the scene. The source codes and models will be available at gala3d.github.io.
1. Introduction
Complex 3D scene creation remains labor-intensive and difficult to customize, while existing text-to-3D methods struggle with multiple objects and interactions. GALA3D uses LLM-derived layouts, layout-guided Gaussians, and compositional optimization, and reports high-fidelity scene generation and controllable editing.
1. Introduction
- Existing text-to-3D models struggle with complex multi-object scenes, producing distorted geometry, 3D inconsistency, multi-face objects, and view-dependent content drift.The paper attributes these problems to optimizing a shared 3D representation without guidance on object interactions and spatial positions.Source: §1 · p. 2
- Hand-designed layouts can constrain geometry and capture object interactions, but require time-consuming manual work and may still yield blurred textures and geometric distortions.The cited problem concerns layouts used with implicit NeRF representations.Source: §1 · p. 2
- GALA3D uses LLMs to turn text into coarse layouts and introduces a layout-guided Gaussian representation with adaptive geometry control for complex scene generation.Its compositional optimization combines diffusion priors, while iterative layout position and scale updates address misalignment with generated scenes.Source: §1 · p. 2
- GALA3D reports strong compositional scene-generation results while preserving high-fidelity object-level entities and supporting controllable editing.The paper describes the framework as user-friendly and end-to-end.Source: §1 · p. 2
- The framework targets scene-level generation and conversational controllable editing while seeking high-fidelity, coherent scenes with precise interactions among multiple objects.The paper presents these capabilities as contributions of its generative 3D Gaussian Splatting approach.Source: §1 · p. 2
2. Related Work
Related work develops text-to-3D methods based on diffusion priors, NeRFs, Gaussian Splatting, and compositional layouts, but reports limitations in complex multi-object scenes. GALA3D combines LLM-interpreted layouts with layout refinement and layout-guided Gaussians to address layout quality and representation constraints.
2. Related Work
- Text-to-3D methods using NeRFs extend diffusion-based 3D generation, but implicit NeRF representations remain insufficient for complex scenes with intricate multi-object interactions.DreamFusion introduced SDS [Poole et al., 2022]; later work explored coarse-to-fine optimization, geometry-appearance disentanglement [Chen et al., 2023a], and other diffusion strategies [Wang et al., 2023a; Wang et al., 2023b].Source: §2 · pp. 2–3
- Compositional NeRF methods use customizable layouts as object proxies, but manual layout design is time-consuming and LLM-interpreted layouts can misalign with desired scenes.Prior LLM-layout methods connect interpreted layouts to NeRF-based generation; GALA3D instead refines those layouts and uses layout-guided Gaussians.Source: §2 · p. 3
- Diffusion-based 3D Gaussian Splatting methods include point-cloud initialization, two-stage geometry and texture optimization, and structured noise for multi-view consistency.The cited methods include Yi et al. (2023), Liang et al. (2023), Chen et al. (2023b), Tang et al. (2023), and Li et al. (2023).Source: §2 · p. 3
- Object-centric 3DGS methods optimize a single Gaussian set and lack constraints on Gaussian distribution and shape, limiting complex composite scenes and risking geometry distortions and view inconsistency.GALA3D introduces layout priors and adaptive geometry control for compositional multi-object scene construction.Source: §2 · p. 3
- Prior LLM-assisted 3D work includes restricted indoor scenarios, while zero-shot complex scene structuring and end-to-end generation remain unexplored in those methods.Because LLM layouts may be unreliable, GALA3D refines them to better align with generated scenes in 3D space.Source: §2 · pp. 3–4
3. Method
GALA3D turns LLM-interpreted layouts into a layout-guided Gaussian scene representation, then jointly optimizes Gaussian geometry, object interactions, and layout alignment using diffusion priors and layout refinement.
3. Method
- GALA3D combines layout-guided Gaussian construction, adaptive geometry control, diffusion-prior optimization, layout refinement, and an aggregated loss to optimize the scene pipeline.The method begins with coarse layouts interpreted by LLMs and jointly optimizes its components.Source: §3 · p. 4
3.1. Layout-guided Gaussian Representation
- LLMs extract object instances and coarse layout priors; each layout's instance Gaussians form the scene-level representation.Layout-guided Gaussians are transformed into a common scene coordinate system for global rendering.Source: §3.1 · p. 4
3.2. Adaptive Geometry Control for Gaussians
- Adaptive Geometry Control constrains Gaussian distribution and shape, sampling Gaussians near layout surfaces and regularizing their geometry.The method targets more refined geometric structures and detailed textures.Source: §3.2 · p. 5, Figure 3 · p. 5
3.3. Compositional Optimization with Diffusion Priors
- Multi-view diffusion optimizes each instance's Gaussians, while ControlNet-conditioned scene diffusion aligns multiple objects and their interactions with layout priors.A layout loss constrains instance scale, position, and geometry relative to the priors.Source: §3.3 · pp. 5–6
3.4. Layout Refinement
- Layout Refinement adaptively adjusts coarse LLM-generated layouts to make them more consistent with scene constraints.Layout parameters are continuously updated during optimization using rendered scene images and conditioned diffusion priors.Source: §3.4 · p. 6
3.5. Total Loss
- The total loss combines layout refinement, conditioned-diffusion global scene optimization, and Gaussian shape regularization.The terms supervise coarse layout priors, the whole scene, and Gaussian shape control, respectively.Source: §3.5 · p. 6
4. Experimental Results
GALA3D is evaluated on single- and multi-object text-to-3D generation, qualitative scene quality, human preference, interactive editing, and component ablations. The reported comparisons favor GALA3D, while ablations identify contributions from geometry control, layout refinement, compositional optimization, and loss functions.
4.1. Quantitative Comparison
- GALA3D excels over all competitors on prompts with varying object counts, including complex scenes with multiple interacting objects.Table 1 evaluates text prompts and text-plus-layout prompts, with an average over 22 scenes containing 1–10 objects; CLIP Score is used because zero-shot text-to-3D lacks ground truth.Source: §4.1 · pp. 6–7, Table 1 · p. 7
- GALA3D outperforms Latent-NeRF across evaluated metrics, boosts MVDream at object and scene levels, and compares favorably with layout-based and 3DGS approaches.Under the same specified layout, Set-the-Scene exhibits blurriness and artifacts, while GALA3D shows stronger scene consistency, spatial geometry, and quality, especially with multiple instances; Adaptive Geometry Control is associated with high-resolution geometry and texture.Source: §4.1 · p. 7
4.2. Qualitative Comparison
- Qualitative comparisons cover single-object, interactive multi-object, and composite scenes, with GALA3D maintaining specified instances, interactions, spatial positions, and a unified style.The reported contrasts include artifacts, distortions, multi-view inconsistencies, multi-face issues, and rough geometry in competing approaches.Source: §4.2 · p. 7, Figure 5 · p. 8
4.3. User Study
- In a user study, GALA3D scores higher than state-of-the-art approaches across scene quality, geometric fidelity, text alignment, and scene consistency.The study used eight text descriptions and 125 participants, who rated four assessment dimensions on a 1–10 scale; 39.2% were art-design or 3D-modeling professionals.Source: §4.3 · p. 8, Table 2 · p. 8
4.4. Conversational Interactive Editing
- Conversational editing translates text instructions into layout operations, then updates edited local regions while keeping other scene regions stable.Supported edits include adding or removing objects, changing positions or rotations, style transfer, and object interactions.Source: §4.4 · pp. 8–9, Figure 7 · p. 9
4.5. Ablation Studies
- Replacing Adaptive Geometry Control with raw 3DGS density control produces artifacts and blur, while continuous Gaussian-shape and distribution updates improve geometry and texture.Without layout refinement, scenes are poorly aligned; refinement adjusts layouts during denoising to better align instance interactions with real-world constraints.Source: §4.5 · p. 9
- Removing Global Scene Optimization leaves impoverished textures, weak scene coherence, and over-constrained boundaries; both layout and scene losses improve texture detail and text–3D alignment.The ablation retains instance-level SDS supervision and places objects according to adjusted layouts.Source: §4.5 · p. 9
5. Conclusion
GALA3D is a layout-guided 3D Gaussian framework for generating high-fidelity, consistent multi-object scenes. Experiments report stronger text-to-3D results than existing methods and support interactive, controllable scene editing.
5. Conclusion
- GALA3D generates high-fidelity, 3D-consistent scenes with multiple objects and interactions, and supports interactive, controllable scene editing.The conclusion describes an efficient, user-friendly scene-generation and editing framework.Source: §5 · p. 9
Impact Statement
The paper identifies possible labor-market risks from automating 3D modeling and the capacity to generate deceptive or malicious 3D content.
Impact Statement
- Automated 3D modeling may pose labor-market risks, and generated content may be deceptive or malicious, warranting caution in application.Source: Impact Statement · p. 10