PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs

Gege Gao, Weiyang Liu, Anpei Chen, Andreas Geiger, Bernhard Schölkopf

arXiv:2312.00093v2cs.CVcs.GRcs.LG

TL;DR

Complex multi-object text prompts challenge holistic text-to-3D methods’ ability to ground attributes and relationships. GraphDreamer decomposes scene graphs into object- and relation-specific descriptions and optimizes SDF-based scenes. Its full model achieves the highest CLIP score among the compared methods, though generation quality and object decomposition remain limited.

  • Problem

    Holistic text-to-3D methods can struggle to represent complex multi-object prompts, with attribute confusion and guidance collapse limiting accurate grounding.

    Source: §1 · pp. 1–2

  • Method

    GraphDreamer decomposes scene graphs into global, node-wise, and edge-wise text descriptions and optimizes SDF-based objects using corresponding text descriptions.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    GraphDreamer’s full model achieves the highest CLIP score among compared methods, and the authors report high-fidelity complex scenes with disentangled objects.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    GraphDreamer generates compositional 3D scenes from scene graphs or text-derived graphs without requiring 3D bounding boxes.

    Source: §1 · p. 2

    Single-object generation quality remains limited by SDS, and the Janus problem may also appear.

    Source: D. Failure Cases and Limitations · p. 18

Abstract

from arXiv · show

As pretrained text-to-image diffusion models become increasingly powerful, recent efforts have been made to distill knowledge from these text-to-image pretrained models for optimizing a text-guided 3D model. Most of the existing methods generate a holistic 3D model from a plain text input. This can be problematic when the text describes a complex scene with multiple objects, because the vectorized text embeddings are inherently unable to capture a complex description with multiple entities and relationships. Holistic 3D modeling of the entire scene further prevents accurate grounding of text entities and concepts. To address this limitation, we propose GraphDreamer, a novel framework to generate compositional 3D scenes from scene graphs, where objects are represented as nodes and their interactions as edges. By exploiting node and edge information in scene graphs, our method makes better use of the pretrained text-to-image diffusion model and is able to fully disentangle different objects without image-level supervision. To facilitate modeling of object-wise relationships, we use signed distance fields as representation and impose a constraint to avoid inter-penetration of objects. To avoid manual scene graph creation, we design a text prompt for ChatGPT to generate scene graphs based on text inputs. We conduct both qualitative and quantitative experiments to validate the effectiveness of GraphDreamer in generating high-fidelity compositional 3D scenes with disentangled object entities.

1. Introduction

Text-to-3D methods can confuse attributes or ignore parts of complex prompts, limiting accurate object grounding. GraphDreamer instead uses scene graphs to generate compositional scenes without requiring 3D bounding boxes.

1. Introduction

  • Complex prompts can intensify attribute confusion and guidance collapse, preventing accurate grounding and limiting text-to-3D methods’ practical utility.Attribute confusion arises from text-image embeddings; guidance collapse occurs when prompts are partly ignored or misinterpreted.Source: §1 · pp. 1–2
  • Non-overlapping 3D bounding boxes constrain scene diversity and can fail to represent irregular, occluding interactions such as an astronaut riding a horse.GraphDreamer uses scene graphs, which are spatially more relaxed, and prompts ChatGPT to generate them from unstructured text.Source: §1 · p. 2
  • GraphDreamer decomposes scene graphs into node- and edge-level text descriptions, using object feature fields and a shared SDF network to represent objects.It renders the scene by integrating objects according to the smallest SDF value at each sampled point, and can render objects separately as well as globally.Source: §1 · p. 2
  • Selective backpropagation sends text-dependent distillation gradients to corresponding objects, while global text simultaneously optimizes scene semantics.The contribution is described as enabling compositional generation from scene graphs or unstructured text without 3D bounding boxes, with object-level ambiguity avoided.Source: §1 · p. 2

2. Related Work

Related work covers diffusion-based text-to-image generation, lifting 2D guidance into 3D, and compositional representations. GraphDreamer uses SDFs for multi-object geometry and differs from prior hybrid methods that target narrower object sets.

Related Work

  • Text-to-3D methods commonly optimize a 3D model using randomly rendered 2D views guided by pretrained text-to-image models, including through score distillation sampling.Prior methods often represent scenes with NeRFs or combine NeRFs with mesh refinement.Source: §2 · p. 2
  • SDFs provide inside/outside information for geometry constraints against unexpected object intersections and support customization of object locations and scales.The paper uses SDFs rather than NeRF densities to model compositional scenes in which objects may be coupled in varied ways.Source: §2 · p. 2
  • Prior hybrid representations include avatar and head-generation methods, while GraphDreamer is designed for multiple objects rather than a parametric head or two-object setting.The cited head-generation method is limited to human heads, and the cited disentanglement approach applies to two objects such as face and hair.Source: §2 · p. 3

3. Preliminaries

The preliminaries describe score distillation sampling for optimizing 3D models from diffusion guidance and SDF-based volume rendering for producing pixel colors. The SDF representation is converted to opacity using NeuS’s weighting function.

Preliminaries

  • Score distillation sampling uses a pretrained text-to-image diffusion model’s noise prediction to guide updates to a 3D model’s parameters.The input is a noisy rendered image and a conditional text embedding; the predicted noise is used to guide gradients toward real image distributions.Source: §3 · p. 3
  • SDF volume rendering estimates a camera-ray pixel color by integrating samples between the near and far intervals.The rendering description defines the ray from camera origin and direction, and identifies color weights, cumulative transmittance, and opacity values for the samples.Source: §3 · p. 3
  • Unlike NeRF density prediction, implicit-surface methods map positions to signed distance values and derive density or opacity through a deterministic transformation.The paper follows NeuS, whose unbiased weighting function makes the ray color dominated by the SDF zero-level-set intersection; its opacity transform uses a trainable sigmoid steepness and sampled SDF values.Source: §3 · p. 3

4. Method

GraphDreamer converts a global prompt into a scene graph and decomposes it into global, node-wise, and edge-wise descriptions to guide SDF-based scene optimization. Its method models objects separately, renders related objects and the full scene for their respective prompts, and adds geometric constraints.

4. Method

  • A global prompt is converted into a scene graph whose global, node-wise, and edge-wise descriptions guide SDF-based scene optimization.The graph represents objects and their attributes as nodes and pairwise relationships as edges.Source: §4 · p. 3, §4.1 · p. 3, Figure 2 · p. 4

4.1. Leveraging Scene Graphs for Text Grounding

  • Scene-graph nodes encode objects and attributes, while edges encode pairwise relations, yielding prompts that guide object-level and relationship-level scene generation.The graph provides flexibility in attaching attributes to objects and accuracy in defining relationships.Source: §4.1 · pp. 3–4

4.2. Disentangled Modeling of Objects

  • Object-specific positional encoders form separate fields, and identity-aware rendering isolates each object to match its own prompt.The method uses SDF values to assign positions to object identities before rendering objects individually.Source: §4.2 · pp. 4–5

4.3. Pairwise Modeling of Object Relationships

  • Related objects are rendered jointly for edge prompts, while all object fields are combined for global scene guidance.Training applies an object loss and a connected-edge loss in sequence, then applies the scene loss after each traversal of the objects.Source: §4.3 · p. 5

4.4. Training Objectives

  • Training adds penetration and Eikonal geometry constraints to the SDS guidance for physically plausible completion of unobserved positions.The penetration constraint limits the predicted number of objects containing a sampled point to at most one.Source: §4.4 · p. 5

5. Experiments and Results

GraphDreamer is compared with prior methods and evaluated through graph-removal and object-decomposition analyses. The experiments report the full model’s highest CLIP score among the compared methods and evidence of object-wise separation.

5. Experiments and Results

  • Experiments train all methods for 20K steps and evaluate prompt-image similarity with CLIP Score.GraphDreamer uses two 10K-step stages, moving from 64×64 renders with DeepFloyd-IF guidance to 256px renders with Stable Diffusion guidance.Source: §5 · p. 7

5.1. Comparison to Other Methods

  • GraphDreamer (full model) achieves the highest CLIP score among the compared methods on 30 multi-object scenes.Qualitative examples also show more accurate modeling of individual attributes and inter-object relationships than the baselines.Source: §5.1 · p. 7

5.2. Ablation Study

  • Removing scene-graph guidance lowers mean CLIP score by more than 3 standard deviations and brings back missing objects and attribute confusion.The ablation collapses conditioning to the global prompt and omits object/edge rendering during training.Source: §5.2 · pp. 7–8

5.3. Decomposition Analysis

  • Object images score significantly higher against their own prompts than against other objects’ prompts, with the reported error bands showing no overlap.The analysis covers 64 objects rendered from four orthogonal views each.Source: Table 2 · p. 7, §5.3 · p. 8

6. Concluding Remarks

GraphDreamer generates compositional 3D scenes from scene graphs or text, using GPT4-V to turn text into a scene graph. Experiments show it can prevent attribute confusion and guidance collapse while generating disentangled objects.

6. Concluding Remarks

  • GraphDreamer generates compositional 3D scenes from scene graphs or text, using GPT4-V to generate the graph from text.It decomposes the graph into global, node-wise, and edge-wise descriptions, then optimizes SDF-based objects with corresponding SDS losses.Source: §6 · p. 8

A. Experimental Settings

The appendix describes implementation choices for the penetration constraint and training losses, alongside hyperparameter settings for guidance.

A.1. Additional Implementation Details

  • The penetration constraint prevents other objects’ SDF values from becoming negative when a sampled point lies inside or on an object, and has no effect outside all objects.Its implementation uses sampled positions on the current ray and a differentiable identity vector.Source: §A.1 · p. 12

A.2. Hyperparameter Settings

  • Training loss coefficients are β1 = 1 for LSDS, β2 = 100 for Lpenet, and β3 = 10 for Leknl.Classifier-free guidance weight is scheduled roughly according to object count, with more objects treated as harder for mode seeking.Source: §A.2 · p. 12

B.1. Ablation: Penetration Constraint

The penetration-constraint ablation shows that removing the constraint increases object inter-penetration and degrades object-level CLIP similarities. In the user study, raters more often selected GraphDreamer’s results as consistent with the prompts.

B.1. Ablation: Penetration Constraint

  • Without Lpenet, objects are more likely to inter-penetrate, making clean segmentation boundaries harder to identify.The ablation is presented in Figure 6.Source: §B.1 · p. 13, Figure 6 · p. 13
  • Without Lpenet, mean CLIP similarities with each object’s own prompt and the global scene prompt drop, while similarity with other object prompts increases slightly.The ablation results are summarized in Table 4.Source: §B.1 · p. 13, Table 4 · p. 13

B.2. User Study

  • 62.26% of raters selected GraphDreamer’s results as more consistent with the prompts than the two baselines.The study covered 30 multi-object prompts and included 31 raters.Source: §B.2 · pp. 13, 15

B.3. Extended Comparison to Other Text-to-3D Methods

The extended comparison adds DreamFusion and Fantasia3D, along with CLIP R1-Precision, to the evaluation of multi-object scenes. Qualitative results show guidance-collapse problems in both methods and failures for Fantasia3D in some cases.

B.3. Extended Comparison to Other Text-to-3D Methods

  • Qualitative comparisons show guidance-collapse problems in DreamFusion and Fantasia3D, while Fantasia3D fails in some cases.Fantasia3D’s performance may vary with SDF initialization; these results use the default sphere initialization.Source: §B.3 · p. 15
  • ProlificDreamer produced no content in the multi-object cases after 20K first-stage training steps with CFG-w set to 100 or 200.Its results are therefore not included.Source: §B.3 · p. 15
  • The extended evaluation reports CLIP-B/32 score and R1-Precision across 30 multi-object scenes, averaging precision over four orthogonal views per object.DreamFusion and Fantasia3D are added as baselines.Source: Table 6 · p. 15, §B.3 · p. 15

C. Inverse Semantics: A Paradigm of Semantic Reconstruction with GPT4V-guided GraphDreamer

Inverse semantics reconstructs a scene graph from an input image and then generates a compositional 3D scene from that graph. The GPT4V-guided implementation can also use estimated object positions to initialize the scene more precisely.

C. Inverse Semantics: A Paradigm of Semantic Reconstruction with GPT4V-guided GraphDreamer

D. Failure Cases and Limitations

GraphDreamer’s failure cases include Janus artifacts, incomplete object separation, and missing semantic parts. The authors associate some problems with approximate SDS gradients, challenges in rendering detail, and simultaneous optimization of multiple SDS losses.

D. Failure Cases and Limitations

  • GraphDreamer can produce Janus artifacts, fail to separate closely interacting objects, and omit object parts.Examples include multiple heads, a violin affected by the astronaut’s hand, and a missing hero leg.Source: D. Failure Cases and Limitations · p. 18, Figure 11 · p. 18
  • The authors believe some failures stem from approximate SDS gradients and diffusion models’ challenges with detailed parts.They also suspect simultaneous optimization of multiple SDS losses may affect disentanglement; enhanced part supervision and improved prompts could help.Source: D. Failure Cases and Limitations · p. 18, Figure 11 · p. 18

Related papers

Loading 2312.00093v2…