PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Language-guided Interactive 3D Generation: LLMs as Layout Interpreter with Generative Feedback

Yiqi Lin, Hao Wu, Ruichen Wang, Haonan Lu, Xiaodong Lin, Hui Xiong, Lin Wang

arXiv:2305.15808v1cs.CV

TL;DR

Interactive language-guided 3D generation remains challenging because prior language-guided generation focused on 2D and did not address arranging and editing multiple 3D objects. LI3D uses LLMs to translate language into semantic bounding-box layouts for off-the-shelf generative models, with LLaVA assessing rendered results and providing feedback for layout updates. Evaluations report multi-round 2D and 3D generation and editing, including spatial-relation reasoning and language-guided editing of individual object parts.

  • Problem

    Interactive language-guided 3D generation remains challenging because prior language-guided generation focused on 2D and did not address arranging and editing multiple 3D objects.

    Source: §1 · p. 1

  • Method

    LI3D uses LLMs to translate language into semantic bounding-box layouts for off-the-shelf generative models, with LLaVA assessing rendered results and providing feedback for layout updates.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Evaluations report multi-round 2D and 3D generation and editing, including spatial-relation reasoning and language-guided editing of individual object parts.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results indicate that LI3D can support language-guided generation and editing across 2D and 3D tasks and multi-round interactions.

    Source: §1 · p. 2

    The synthesized 3D models may not yet match the realism of state-of-the-art 2D image synthesis.

    Source: §5 · p. 9

Abstract

from arXiv · show

Generating and editing a 3D scene guided by natural language poses a challenge, primarily due to the complexity of specifying the positional relations and volumetric changes within the 3D space. Recent advancements in Large Language Models (LLMs) have demonstrated impressive reasoning, conversational, and zero-shot generation abilities across various domains. Surprisingly, these models also show great potential in realizing and interpreting the 3D space. In light of this, we propose a novel language-guided interactive 3D generation system, dubbed LI3D, that integrates LLMs as a 3D layout interpreter into the off-the-shelf layout-to-3D generative models, allowing users to flexibly and interactively generate visual content. Specifically, we design a versatile layout structure base on the bounding boxes and semantics to prompt the LLMs to model the spatial generation and reasoning from language. Our system also incorporates LLaVA, a large language and vision assistant, to provide generative feedback from the visual aspect for improving the visual quality of generated content. We validate the effectiveness of LI3D, primarily in 3D generation and editing through multi-round interactions, which can be flexibly extended to 2D generation and editing. Various experiments demonstrate the potential benefits of incorporating LLMs in generative AI for applications, e.g., metaverse. Moreover, we benchmark the layout reasoning performance of LLMs with neural visual artist tasks, revealing their emergent ability in the spatial layout domain.

1 Introduction

LI3D addresses the difficulty of interactive natural-language 3D generation by using LLMs to translate language into layouts for generative models, with visual feedback to refine them. Evaluations report multi-round 2D and 3D generation and editing, alongside layout reasoning and object-part capabilities.

1 Introduction

  • Open-vocabulary language interaction remains challenging: prior language-guided image-generation work focuses on 2D, while multi-object 3D arrangement and editing add distinct challenges.The passage also notes limits in prior models' capacity to comprehend and reason over open-vocabulary language.Source: §1 · p. 1
  • LLMs show reasoning, conversational, and zero-shot generation abilities, and prior work suggests they can interpret 3D space through code [3].This motivates using LLMs as a link between natural language and 3D generation without relying on 3D datasets.Source: §1 · pp. 1–2
  • LI3D uses LLMs to turn natural-language instructions into bounding-box and semantic layouts that condition existing generative models, with LLaVA feedback to refine layouts.The system is designed for generating scenes and editing objects over multiple rounds.Source: §1 · p. 2
  • LI3D shows multi-round language-guided generation and editing across 3D and 2D, with reported spatial reasoning and fine-grained object-part editing capabilities.The i-CLEVR evaluation assesses layout generation from sequential instructions; visual results cover 3D scenes and 2D image generation.Source: §1 · p. 2

2 Proposed Method

LI3D uses semantic bounding-box layouts as an interface between language models and off-the-shelf generators, supporting iterative scene creation and editing. LLaVA assesses rendered results and provides descriptive feedback that can make layouts more detailed.

2 Proposed Method

  • LI3D represents scenes as coarse boxes with semantic descriptions, giving LLMs an intermediate layout to interpret language and condition off-the-shelf generators.The layout is intended to help LLMs maintain conversational context and reason about language in a standardized way.Source: §2 · p. 2, §2.1 · p. 3

2.1 Language-guided Interactive 3D Generation Framework

  • Across interaction rounds, LI3D detects layout changes and applies predefined rules to update generated results for consistency.New objects can be initialized while existing ones remain unchanged; semantic descriptions identify boxes, and location edits modify their transformations.Source: §2.1 · p. 3
  • The framework also supports image generation by adapting its layout-based editing process with Stable Diffusion and Segment Anything.The figure describes language being mapped to a layout for CompoNeRF, with image-domain extension through adaptations.Source: Figure 1 · p. 3, §2.1 · p. 4

2.2 Generative Feedback Module

  • LLaVA evaluates whether rendered views satisfy the layout description and supplies more detailed feedback for the LLM to update an unsatisfactory layout.The feedback module can be integrated at each LI3D step.Source: §2.2 · p. 4
  • Box-based layouts support element-level decomposition and editing, which the authors describe as preserving context more accurately across rounds than code formats or plain language.The discussion presents layouts as an intermediate representation for connecting user intent to off-the-shelf generative models.Source: §2.2 · p. 4

3 Empirical Evaluation and Analysis

The evaluation examines LI3D’s language-to-layout robustness, interactive 3D generation, generative feedback, and extension to interactive 2D editing.

3.1 Experiment Setup

  • The implementation uses GPT-3.5-turbo for layout generation, LLaVA-13B for feedback, and CompoNeRF for rendering and rule-based scene updates.The initial scene is trained for 8,000 iterations; new local NeRFs receive 3,000 iterations before joint fine-tuning for 6,000.Source: §3.1 · p. 4

3.2 Robustness of Layout Generation from Language

  • The benchmark samples 50 i-CLEVR instruction sequences, each containing up to five steps that add objects and specify their placement relative to existing objects.The evaluation compares generated layouts with ground-truth arrangements.Source: §3 · p. 4, §3.2 · p. 5
  • LI3D recognizes objects from multi-round language input with high accuracy and shows strong reasoning about 3D spatial relationships in the i-CLEVR evaluation.The evaluation measures object recall and relational similarity, counting shared left-right and front-back relations.Source: §3.2 · p. 5

3.3 Qualitative Results of Interactive 3D Generation

  • LI3D supports multi-round scene edits, including adding, removing, adjusting, and duplicating objects, while some layouts require user correction.The reported bookshelf example has an unreasonable layout.Source: §3.3 · p. 6
  • LI3D generates individual objects from language-described parts and supports adding, removing, and adjusting those parts, but may place wheels outside a vehicle.The reported failure occurs in some views despite simple object structure.Source: §3.3 · p. 6
  • In the feedback ablation, LLaVA’s more detailed descriptions produce richer scene layouts and more satisfactory generated content.The comparison presents cases before and after adding generative feedback.Source: §3.3 · p. 6
  • LI3D can produce imbalanced object sizes, inconsistent layouts across trials or long interactions, and low-resolution scenes; CompoNeRF can be unstable on irregular objects and is not real-time.The feedback module also fails to correct some cases because of generation quality.Source: §3.3 · p. 7

3.4 Extension on Interactive Image Generation

  • The 2D extension uses Stable Diffusion for layout-conditioned generation and in-painting, and SAM to extract objects for add, remove, and move edits.The reported images use different random seeds across interactions.Source: §3.4 · p. 8

4 Related Work

Prior interactive visual-generation methods support multi-round editing but have difficulty with non-predefined, complex multi-object scenes. LI3D instead uses language-interpreted 3D layouts to connect user instructions with generative models.

Interactive Visual Generation and Editing

  • Earlier interactive editing methods such as TiGAN and Talk-to-Edit [15] support successive edits but face challenges with non-predefined, complex multi-object 3D scenes.LI3D addresses this with a task-specific intermediate representation for cooperation between LLMs and generative models.Source: §4 · pp. 8–9

Text-guided 3D Generative Models

  • Text-to-3D methods use pretrained text-to-image models and score distillation sampling, yet complex multi-object scenes still face view inconsistency, inaccurate content, and compositional challenges.DreamFusion [28], Magic3D [20], and Latent-NeRF [23] illustrate this line of work.Source: §4 · p. 9

Layout Conditional Visual Synthesis

  • Layout-conditioned synthesis uses labeled boxes or semantic maps, while LI3D interprets natural language into layouts without requiring users to specify predefined objects and locations.The cited 3D approaches condition on text and generated 3D layouts; LI3D supports multi-round generation and editing from language.Source: §4 · p. 9

5 Conclusion

LI3D uses LLMs to interpret language into layouts for interactive 2D and 3D generation, with LLaVA feedback from rendered views to update layouts. The paper reports effectiveness across these tasks and notes potential risks including inherited biases and creative-worker displacement.

5 Conclusion

  • LI3D integrates LLMs with existing generators for interactive 2D and 3D content, using LLaVA feedback from rendered views to update layouts.The authors report effectiveness and identify LLMs as a potential visual-commonsense reasoning engine.Source: §5 · p. 9
  • Using Stable Diffusion in LI3D may inherit that model’s biases or limitations, and the synthesized 3D models may not yet match state-of-the-art 2D realism.The paper also notes possible misuse involving convincing 3D disinformation.Source: §5 · p. 9
  • The authors note that automation from generative models such as LI3D may displace creative workers, while also potentially supporting growth and accessibility in the creative industry.Source: §5 · p. 10

A Implementation Details

This appendix section signals that it contains further details about the 2D and 3D generative models and their integration with LI3D.

A Implementation Details

  • The appendix section provides additional model and implementation details for integrating LI3D with 2D and 3D generative models.Source: A Implementation Details · p. 13

A.1 Layout Conditional 3D Generative Models

LI3D uses CompoNeRF with LLM-predicted layouts built from object boxes and local prompts. The implementation removes CompoNeRF’s global MLP for faster convergence and normalizes layout boxes before generation.

A.1 Layout Conditional 3D Generative Models

  • LI3D uses CompoNeRF [21], whose multi-object 3D generation accepts configurable layouts, while LLMs predict those layouts directly from user language.The layout contains object entities represented by 3D boxes and local text prompts.Source: §A.1 · p. 13
  • Each object’s local NeRF maps points in its local frame to volumetric density and emitted color, producing four pseudo-color channels and density for rendering.The pseudo-color is decoded by Stable Diffusion to obtain the final rendering.Source: §A.1 · p. 13
  • CompoNeRF training uses DreamFusion’s [28] Score Distillation Sampling to transfer the diffusion model’s text-to-image prior to NeRF parameters.The original CompoNeRF also proposes a global MLP to improve consistency across local NeRFs.Source: §A.1 · p. 13
  • LI3D removes CompoNeRF’s global MLP for faster convergence and lower computation when replacing objects or parts, and normalizes box boundaries to [-0.8, 0.8].The paper attributes the normalization to possible instability in LLM-generated layouts.Source: §A.1 · p. 13

A.2 Layout Conditional 2D Generative Models

The 2D extension uses Stable Diffusion for layout-conditioned image generation and editing, with SAM masks guiding inpainting after layout changes.

  • Backward guidance controls Stable Diffusion generation from the layout in the default setting.Source: §A.2 · p. 13
  • SAM segments objects detected as changed in the updated layout using each box’s center point, and the second-highest-scoring mask is used for extraction.Source: §A.2 · p. 13
  • The extracted object mask is dilated by 10 pixels before Stable Diffusion inpaints the unchanged background based on the updated layout.Source: §A.2 · p. 13

B Prompt Design for 3D Generation

The appendix presents tailored prompts and multi-round examples for 3D scene generation and single-object composition editing. It pairs layout and rendering visualizations with case dialogs across interaction rounds.

B.1 Scene Generation

  • The scene-generation context prompt targets generative-model needs, including expected box sizes and aligned elements.Source: §B.1 · p. 14
  • Table 4 provides the context prompt for 3D scene generation.Source: Table 4 · p. 14, §B.1 · p. 14
  • Table 5 presents a case dialog for multi-round interactive 3D scene generation.Source: §B.1 · p. 14, Table 5 · p. 15
  • Figure 7 shows layouts generated for the first, second, and third scene-generation prompts.Source: Figure 7 · p. 16
  • Figure 8 shows multiple rendering views and individual objects from the scenes generated across three prompts.Source: Figure 8 · p. 16

B.2 Single Object Generation

C Prompt Design for Generative Feedback

The feedback prompt uses LLaVA to assess a generated scene and provide feedback that revises the context for the LLM’s next layout. The castle example illustrates a mismatch and a more detailed updated layout.

D.1 Prompt Design for Benchmarking Layout Generation

The benchmark evaluates LLM layout generation using sequential i-CLEVR instructions and ground-truth arrangements. It assesses similarity between layout centers and ground-truth images.

  • The evaluation samples 50 sequential instructions from i-CLEVR, whose examples contain five image-instruction pairs describing object attributes and relative placement.Source: §D.1 · p. 24
  • All instructions are supplied simultaneously in a prompt, with up to five sequential instructions, to focus on layout generation from complex input.Source: §D.1 · pp. 24, 26
  • Numerical evaluation compares the front view of the 3D layout with ground-truth images using object center points; Figure 12 visualizes the ground-truth image for the multi-round instructions.Source: §D.1 · p. 24, Figure 12 · p. 26
Loading 2305.15808v1…