PaperlayerView exact revision on arXiv

Source-linked AI summary

InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior

Chenguo Lin, Yadong Mu

arXiv:2402.04717v1cs.CV

TL;DR

Natural-language scene instructions are difficult to control with methods that model object relations implicitly or require elaborate relation graphs. INSTRUCTSCENE combines a semantic graph prior and layout decoder, and the authors report improved controllability and fidelity over prior state-of-the-art methods. Its learned prior also supports diverse instruction-driven applications without fine-tuning.

  • Problem

    Natural-language instructions describe abstract object relationships, while implicit relation modeling limits controllability and elaborate relation graphs are burdensome for users to specify.

    Source: §1 · p. 1

  • Method

    INSTRUCTSCENE learns semantic graphs for holistic scenes from instructions, then uses a layout decoder, with multimodal semantic features representing object appearance and geometry.

    Source: §1 · pp. 1–2

  • Results

    INSTRUCTSCENE surpasses previous state-of-the-art methods by a large margin in controllability and fidelity, with ablations verifying essential components.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The learned semantic graph prior supports diverse instruction-driven generative applications without fine-tuning.

    Source: §1 · p. 1, §6 · p. 9

    The learned prior may inherit problematic object arrangements and misclassifications in 3D-FRONT, and the current scene dataset contains only hundreds of scenes.

    Source: D LIMITATIONS AND FUTURE WORK · p. 19

Abstract

from arXiv · show

Comprehending natural language instructions is a charming property for 3D indoor scene synthesis systems. Existing methods directly model object joint distributions and express object relations implicitly within a scene, thereby hindering the controllability of generation. We introduce InstructScene, a novel generative framework that integrates a semantic graph prior and a layout decoder to improve controllability and fidelity for 3D scene synthesis. The proposed semantic graph prior jointly learns scene appearances and layout distributions, exhibiting versatility across various downstream tasks in a zero-shot manner. To facilitate the benchmarking for text-driven 3D scene synthesis, we curate a high-quality dataset of scene-instruction pairs with large language and multimodal models. Extensive experimental results reveal that the proposed method surpasses existing state-of-the-art approaches by a large margin. Thorough ablation studies confirm the efficacy of crucial design components. Project page: https://chenguolin.github.io/projects/InstructScene.

1 INTRODUCTION

INSTRUCTSCENE addresses the difficulty of following natural-language instructions by combining a semantic graph prior with a layout decoder. It targets controllability and fidelity while supporting zero-shot instruction-driven tasks.

1 INTRODUCTION

  • The target system is expected to understand natural-language instructions, compose aesthetically and thematically coherent objects, and place them according to function and regular arrangements.Source: §1 · p. 1
  • Prior methods either model object relationships implicitly, use relation graphs that are complicated for users to specify, or represent objects with features that limit appearance detail.These approaches can constrain customization and contribute to style inconsistency.Source: §1 · p. 1
  • Two conditional diffusion models handle the graph prior and layout decoder separately, while quantized multimodal semantic features incorporate object geometry and appearance.The two-stage scheme separates discrete from continuous scene attributes to reduce network optimization burden.Source: §1 · p. 2
  • INSTRUCTSCENE surpasses previous state-of-the-art methods by a large margin in controllability and fidelity, and ablations verify its essential components.The authors also curate a high-quality dataset of paired scenes and instructions using large language and multimodal models.Source: §1 · p. 2
  • INSTRUCTSCENE combines a semantic graph prior with a layout decoder to improve controllability and fidelity in instruction-driven 3D scene synthesis.The graph represents objects as nodes with semantic features and spatial relationships as edges; the decoder predicts seven degrees of freedom for each object.Source: §1 · pp. 1–2, Figure 1 · p. 2

2 RELATED WORK

Prior scene-synthesis work uses graph structures to guide layouts, while language-driven methods often rely on implicit object relations. The paper distinguishes its approach by generating holistic semantic graphs from expressive instructions.

Graph-driven 3D Scene Synthesis

  • Graph-guided synthesis has used hierarchies, parse trees, and scene graphs, but relation-graph methods can require conditions more intricate than natural-language instructions.Some prior approaches use conditional VAEs and graph convolutional networks to generate layouts; their graph conditions provide control but demand elaborate specification.Source: §2 · p. 2

Language-driven 3D Scene Synthesis

  • Language-driven systems use self-attention and cross-attention to model object relations and condition synthesis on text, but prompts often omit layout descriptions or use simple object categories.Implicit relation modeling also hinders controllability.Source: §2 · p. 2

Generative Models for Graphs

  • Prior text-driven undirected graph generators lack semantics, whereas this work generates holistic semantic graphs from expressive instructions.Earlier graph-generation work includes autoregressive models, VAEs, GANs, and diffusion models.Source: §2 · p. 3

3 PRELIMINARY: DIFFUSION MODELS

Diffusion models progressively corrupt data and train a neural network to reverse that process. The reverse process can be conditioned, and the model is optimized using a variational upper bound.

3 PRELIMINARY: DIFFUSION MODELS

  • The forward process progressively corrupts a data point into increasingly noisy latent variables.The supplied formulation expresses the sequence of transitions as q(x1:T | x0).Source: §3 · p. 3
  • A neural network learns the reverse process by iteratively denoising latent variables, with an optional condition c to guide generation.The reverse transitions are represented by pψ(xt−1|xt, c).Source: §3 · p. 3
  • Training minimizes a variational upper bound on the negative log likelihood, with the KL-divergence term defined for reverse-step distributions and LT constant during training.The constant LT term can be ignored during optimization.Source: §3 · p. 3

4 METHOD

InstructScene models instruction-conditioned scenes through semantic graphs that encode object and relation information, then decodes those graphs into precise layouts. Its method quantizes object features and uses separate discrete and continuous diffusion processes for graphs and layouts.

4.1 PROBLEM STATEMENT

  • The framework learns instruction-conditioned semantic graphs for holistic scenes and uses them to generate precise layout configurations.The semantic graph prior models high-level object and relation distributions; the layout decoder produces scene layouts from those graphs.Source: §4.1 · p. 3

4.2 SEMANTIC GRAPH PRIOR

  • Spatial relations encode distances and relative orientations, while precise layout configurations are deferred to the layout decoder.Examples include “left,” “closely in front of,” “above,” and “too far away (none).”Source: §4.2 · p. 3

4.2.1 FEATURE QUANTIZATION

  • The fVQ-VAE quantizes multimodal-aligned object features into codebook indices to reduce the cost of modeling continuous features.The features represent visual appearance and geometric shape; the method identifies high-dimensional features as difficult to model directly.Source: §4.2.1 · p. 4

4.2.2 DISCRETE SEMANTIC GRAPH DIFFUSION

  • The graph prior uses discrete diffusion over categorical object, feature, and relation attributes, with independent masking and denoising.Mask states distinguish corrupted from clean attributes, and information in uncorrupted variables can help recover perturbed ones.Source: §4.2.2 · pp. 4–5

4.3 3D LAYOUT DECODER

  • The layout decoder learns pθ(L|G) with Gaussian diffusion, iteratively predicting noise in corrupted layouts conditioned on semantic graphs.The layout encodes object location, size, and rotation.Source: §4.3 · p. 5

4.4 MODEL ARCHITECTURE

  • The prior and decoder share a graph Transformer, while random permutation augmentation encourages exchangeability despite positional encodings.Graph attention and node-edge fusion process graph inputs; instructions condition the prior through cross-attention.Source: §4.4 · p. 6, Figure 3 · p. 6

5 EXPERIMENTS

Experiments evaluate instruction-paired scene synthesis and zero-shot applications. InstructScene improves controllability and fidelity over the baselines, while ablations examine diffusion settings and semantic-graph-prior design.

5.1 SCENE-INSTRUCTION PAIR DATASET

  • The scene-instruction dataset pairs 3D-FRONT scenes with instructions derived from spatial-relation triplets and filtered object captions.BLIP generates object captions, which ChatGPT filters using ground-truth object categories.Source: §5.1 · p. 6

5.3 INSTRUCTION-DRIVEN SCENE SYNTHESIS

  • About 15%∼25% iRecall improvement over current state-of-the-art across room types, while maintaining high fidelity and benefiting more complex living and dining scenes.The comparison averages five runs; living and dining rooms have about 20 objects on average, versus 8 in bedrooms.Source: Table 1 · p. 7, §5.3 · p. 7

5.2 EXPERIMENTAL SETTINGS

  • ATISS and DiffuScene baselines were augmented with quantized features; evaluation uses iRecall for triplets and rendered-image metrics for scene quality.iRecall measures the proportion of required subject–relation–object triplets present in synthesized scenes.Source: §5.2 · p. 7

5.4 ZERO-SHOT APPLICATIONS

  • Across four zero-shot tasks, InstructScene consistently outperforms both baselines on controllability and fidelity.The tasks are stylization, re-arrangement, completion, and unconditional generation; partial scene attributes serve as intermediate discrete denoising states.Source: §5.4 · pp. 7–8, Table 2 · p. 8

5.5.2 LEARNING SEMANTIC GRAPH PRIOR

  • Independent masking benefits controllability and fidelity, while permutation-invariant modeling lowers performance despite random object shuffling during training.The ablations compare transition-matrix strategies for the semantic graph prior.Source: Table 3 · p. 9, §5.5.2 · p. 9

5.5.1 CONFIGURATIONS FOR DIFFUSION MODELS

  • Reducing reverse-process steps avoids noticeable performance decline, while moderate classifier-free guidance improves controllability and large scales can hurt performance.The tested InstructScene timesteps include 100 for the prior and 10 for the decoder; classifier-free guidance training removes instructions on 20% of samples.Source: §5.5.1 · pp. 8–9, Figure 4 · p. 9

6 CONCLUSION

InstructScene combines a semantic graph prior with a layout decoder to improve controllability and fidelity in natural-language-driven 3D indoor scene synthesis. The prior also supports diverse downstream applications without fine-tuning.

6 CONCLUSION

  • InstructScene integrates a semantic graph prior and layout decoder for controllable, faithful 3D indoor scene synthesis through natural-language instructions.The conclusion describes the system as a user-friendly interface and identifies interior design, metaverse experiences, embodied-agent simulations, and VR/AR as possible practical scenarios.Source: §6 · p. 9
  • The semantic graph prior supports diverse downstream applications without fine-tuning.Source: §6 · p. 9
  • The authors identify limitations and future work in Appendix D.Source: §6 · p. 9

ETHICS STATEMENT

InstructScene uses several large pretrained models and may inherit imperfections from their web-collected training data. The authors plan to specify permissible applications through licensing to mitigate potential adverse societal impacts.

ETHICS STATEMENT

  • OpenShape, CLIP, BLIP, and ChatGPT provide object semantics, text features, object captions, and caption refinement, respectively.Source: ETHICS STATEMENT · p. 10
  • Because most incorporated models use web-collected data without rigorous content filtering, InstructScene may inherit those imperfections.Source: ETHICS STATEMENT · p. 10
  • Indoor-scene-only training is described as less likely to propagate negative consequences than synthesis and editing methods for faces or natural images; licensing will specify permissible uses.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility materials cover dataset processing, implementation, baseline reproductions, model hyperparameters, and evaluation metrics. The authors also provide the instruction-scene dataset and training and evaluation code on the project page.

REPRODUCIBILITY STATEMENT

A DATASET PREPARATION

The section describes how the authors prepare instruction-scene pairs from 3D-FRONT, define object relations, refine captions, and configure baselines and models. It also reports the dataset’s room counts and implementation settings.

A DATASET PREPARATION

  • The dataset pairs 3D-FRONT scenes with instructions generated from extracted spatial relations and object descriptions captioned by BLIP and filtered with ground-truth categories using ChatGPT.It contains 4,041 bedrooms, 813 living rooms, and 900 dining rooms, using the ATISS data split; processing scripts and the processed dataset are available on the project page.Source: A DATASET PREPARATION · p. 16

A.1 RELATION DEFINATIONS

  • Eleven spatial relations are defined using object orientation, ground distance, and whether a subject center falls inside an object’s ground bounding box.The definitions use X and Y for the ground plane and Z for the vertical axis.Source: §A.1 · p. 16

A.2 CAPTION REFINEMENT BY CHATGPT

  • ChatGPT (gpt-3.5-turbo) refines BLIP captions with temperature 0.2 and top p 0.1 to encourage more deterministic, focused outputs.Source: §A.2 · p. 16

B.1 BASELINE DETAILS

  • ATISS and DiffuScene are adapted for instruction-driven synthesis and quantized semantic features, with shared experimental settings across methods.ATISS is autoregressive and DiffuScene uses Gaussian diffusion; the comparison holds network architectures, training hyperparameters, object retrieval, and rendering schemes constant.Source: §B.1 · pp. 16–17

B.2 MODEL DETAILS

  • The models use 5-layer, 8-head Transformers and train for 500,000 iterations; OpenShape features are quantized with four ordered indices and CLIP extracts instruction features.The reported setup uses 512 attention dimensions, batch size 128, and learning rate 1e-4; the OpenShape codebook has size 64 and dimension 512.Source: §B.2 · pp. 17–18

B.3 EVALUATION DETAILS

Across zero-shot applications, INSTRUCTSCENE generates realistic scenes that follow instructions, while the reported comparisons identify weaknesses in baseline generations. Results also examine semantic-feature recovery, diversity, feature ablation, and runtime.

B.3 EVALUATION DETAILS

  • For evaluation, scenes are rendered from a top-down view at 256×256 resolution, and lower FID, FIDCLIP, and KID scores indicate greater distribution similarity while SCA closer to 50% indicates better generation.The rendering setup fixes textures to avoid evaluation bias; images are rendered with Blender and CYCLES.Source: §B.3 · p. 18

C.1 INSTRUCTION-DRIVEN SCENE SYNTHESIS

  • Qualitative bedroom, living-room, and dining-room results show stronger instruction adherence and generative quality than previous state-of-the-art approaches.The authors also report quantitative evaluations in Table 1, but the supplied passage gives no metric values.Source: §C.1 · p. 18

C.2 ZERO-SHOT APPLICATIONS

  • Across stylization, re-arrangement, completion, and unconditional synthesis, INSTRUCTSCENE generates realistic scenes that adhere to instructions, unlike baselines that often struggle to follow them.ATISS tends to generate redundant objects, while DiffuScene often produces semantically incoherent object styles or pairings.Source: §C.2 · p. 18

C.3 FEATURE RECOVERY

  • Semantic-feature recovery experiments mask either one object's feature or all but one, and the results are described as showing stylistic information and object co-occurrences captured from training data.Generated scenes tend toward consistent styles and thematic harmony, including coordinated chairs and nightstands and cohesive color palettes.Source: §C.3 · p. 19

C.4 DIVERSITY

  • The paper illustrates diverse scenes generated both from a single prompt and from the same semantic graph.Source: §C.4 · p. 19

C.5 INSTRUCTSCENE WITHOUT SEMANTIC FEATURES

  • Omitting semantic features significantly reduces appearance controllability and style consistency, as the degraded model focuses only on layout attributes.Without semantic features, generated object occurrences and combinations lack awareness of style and appearance.Source: §C.5 · p. 19

C.6 RUNTIM COMPARISON

  • At T = 20 + 5, generating 128 living rooms takes 3 seconds without a noticeable performance decline, compared with 12 seconds at the default T = 100 + 10.At default settings, ATISS takes 3 seconds and DiffuScene 22 seconds for the same batch.Source: §C.6 · p. 19

D LIMITATIONS AND FUTURE WORK

The paper identifies inherited dataset errors and limited scene-data scale as constraints, and confines its current synthesis focus to indoor scenes while outlining possible extensions.

D LIMITATIONS AND FUTURE WORK

  • The learned prior may inherit problematic object arrangements and misclassifications that remain in 3D-FRONT after filtering, while the current scene dataset contains only hundreds of scenes.The authors propose expanding the dataset or using large-scale, well-annotated 3D-object datasets to establish a new scene-synthesis benchmark.Source: D LIMITATIONS AND FUTURE WORK · p. 19
  • The work focuses on indoor synthesis, although the semantic graph prior may also model more intricate outdoor scenes.A fully generative pipeline could replace object retrieval with 3D object generative models conditioned on graph-provided categories and semantic features.Source: D LIMITATIONS AND FUTURE WORK · p. 19
  • The authors identify LLM integration into the instruction-driven pipeline as a promising direction for further improving controllability.Source: D LIMITATIONS AND FUTURE WORK · p. 19

E DISCUSSION ON DATASET

The authors discuss how their curated instructions extend across language patterns and vague spatial terms, while identifying difficult instructions and possible dataset-curation improvements.

E DISCUSSION ON DATASET

  • Although dataset instructions derive from predefined rules, experiments vary sentence patterns and vague location words such as “next to,” with generated objects placed in all possible spatial relations.For the example “Put a chair next to a double bed,” these relations include left, right, front, and behind.Source: E DISCUSSION ON DATASET · p. 20
  • INSTRUCTSCENE struggles with instructions requiring four or five triplets, repeated-object references, and abstract concepts absent from the curated instructions.The authors attribute these limits to the CLIP text encoder's tendency to capture global sentence semantics, given its contrastive training with image features.Source: E DISCUSSION ON DATASET · p. 20
  • The authors suggest using LLMs to refine dataset sentences or crowdsourcing to make instruction curation semi-supervised.They present the dataset and its creation pipeline as a starting point for high-quality instruction datasets.Source: E DISCUSSION ON DATASET · p. 20
Loading 2402.04717v1…