PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-GPT: Procedural 3D Modeling with Large Language Models

Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin, Stephen Gould

arXiv:2310.12945v2cs.CVcs.GRcs.LG

TL;DR

Procedural 3D generation can automate content creation, but requires knowledge of rules, algorithms, and parameters. 3D-GPT uses collaborating LLM agents to enrich instructions and translate them into procedural modeling actions. The authors report high-quality results, adaptability to expansive scenes, and 3D consistency, while identifying limits in curve control, shading, and dependence on procedural algorithms.

  • Problem

    Procedural generation can automate 3D content creation, but demands understanding of rules, algorithms, and parameters, while LLMs may struggle to determine which elements to modify and how.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    3D-GPT coordinates task-dispatch, conceptualization, and modeling agents to enrich scene descriptions and infer procedural parameters for instruction-driven 3D modeling.

    Source: §1 · p. 2

  • Results

    The authors report high-quality results, adaptability to expansive scenes, 3D consistency, material modeling and editing, and real ray tracing.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The framework reduces manual definition of procedural parameters and supports text-guided 3D generation with Blender manipulation capabilities.

    Source: §1 · p. 2

    The framework lacks advanced precise curve control and intricate shading, and its effectiveness depends on procedural algorithms, limiting some categories such as hair and fur.

    Source: §5 · p. 9

Abstract

from arXiv · show

In the pursuit of efficient automated content creation, procedural generation, leveraging modifiable parameters and rule-based systems, emerges as a promising approach. Nonetheless, it could be a demanding endeavor, given its intricate nature necessitating a deep understanding of rules, algorithms, and parameters. To reduce workload, we introduce 3D-GPT, a framework utilizing large language models~(LLMs) for instruction-driven 3D modeling. 3D-GPT positions LLMs as proficient problem solvers, dissecting the procedural 3D modeling tasks into accessible segments and appointing the apt agent for each task. 3D-GPT integrates three core agents: the task dispatch agent, the conceptualization agent, and the modeling agent. They collaboratively achieve two objectives. First, it enhances concise initial scene descriptions, evolving them into detailed forms while dynamically adapting the text based on subsequent instructions. Second, it integrates procedural generation, extracting parameter values from enriched text to effortlessly interface with 3D software for asset creation. Our empirical investigations confirm that 3D-GPT not only interprets and executes instructions, delivering reliable results but also collaborates effectively with human designers. Furthermore, it seamlessly integrates with Blender, unlocking expanded manipulation possibilities. Our work highlights the potential of LLMs in 3D modeling, offering a basic framework for future advancements in scene generation and animation.

1 INTRODUCTION

The introduction frames procedural modeling as a way to automate 3D content creation, but notes that its rules and parameters make it demanding. It presents 3D-GPT as an LLM-based framework for developing scene descriptions and translating them into procedural 3D content.

1 INTRODUCTION

  • Procedural generation can automate content creation, but requires expertise in generation rules, algorithms, and parameters, while client-vision alignment adds communication demands.The authors describe the traditional modeling workflow as time-intensive, from shaping basic forms through detailing, texturing, and rendering.Source: §1 · p. 1
  • LLMs' ability to characterize object attributes, parse concise text, understand code functions, and interact with users motivates their use in procedural 3D modeling.The stated objective is to control 3D creation software according to client requirements.Source: §1 · p. 1
  • 3D-GPT divides modeling into manageable tasks handled by conceptualization, modeling, and task-dispatch agents, with dispatch coordinating initial and subsequent instructions.The conceptualization and modeling agents manipulate 3D generation functions.Source: §1 · p. 2
  • The agents enrich scene descriptions in response to later instructions and use procedural-generation functions to infer parameter values from the enriched text.This approach uses adjustable parameters and rule-based systems to interface with 3D software rather than manually crafting every element.Source: §1 · p. 2
  • The framework is presented as controllable text-guided generation that reduces manual parameter definition and supports Blender operations including material adjustment, animation, mesh editing, and physical simulation.The authors also describe it as training-free and as generating Python code to control 3D software.Source: §1 · p. 2

2 RELATED WORK

The related work contrasts conventional text-to-3D systems that produce neural representations with 3D-GPT's use of LLM-generated code to control Blender. It also situates the work among LLM research on reasoning, planning, coordination, and code generation.

2.1 TEXT-TO-3D GENERATION

  • Prior text-to-3D approaches commonly optimize shapes using differentiable rendering and CLIP or diffusion guidance, while other work targets speed, geometry fidelity, or subject-specific assets.Examples include DreamFields and CLIP-Mesh using CLIP guidance, Dreamfusion optimizing NeRF with diffusion guidance, and Magic3D addressing speed and visual quality.Source: §2.1 · p. 2
  • Unlike those approaches, 3D-GPT generates Python code to control Blender modeling from instructions rather than producing conventional neural representations as its final output.Source: §2.1 · pp. 2–3

2.2 LARGE LANGUAGE MODELS

  • The paper draws on LLM work in reasoning, planning, multi-agent coordination, and specialized code generation, and applies LLMs to control procedural 3D generation.The cited examples include modular prompting without finetuning and Codex for Python code generation.Source: §2.2 · p. 3

3 3D-GPT

3D-GPT models a scene as a response to an initial natural-language description and any later edits, selecting procedural functions and inferring their parameters. It prepares function-specific resources and coordinates dispatch, conceptualization, and modeling agents to generate Blender code and render 3D content.

3.1 TASK FORMULATION

  • The task is to generate a 3D scene from an initial natural-language description and modify it through subsequent instructions.The paper represents these instructions as a sequence L = [Li].Source: §3.1 · p. 3
  • Because LLMs may struggle to decide which scene elements to modify and how, 3D-GPT uses procedural functions with adjustable parameters and rule-based systems.For each instruction, the framework selects a subset of functions, infers their parameters, and generates a Python script to manipulate Blender.Source: §3.1 · p. 3

3.2 MODELING TOOL PREPARATION

  • The framework uses Infinigen and supplies each function with documentation, readable code, required inference information, and examples.For a flower function, the required information can include flower color, petal appearance, and center appearance.Source: §3.2 · pp. 3–4
  • The function-specific resources are intended to help LLMs use planning, reasoning, and tool-use abilities for language-driven 3D generation.Source: §3.2 · p. 4

3.3 MULTI-AGENTS FOR 3D REASONING, PLANING AND TOOL USING

  • The three-agent workflow assigns function selection to task dispatch, description enrichment to conceptualization, and parameter inference and Python-code generation to modeling.The generated calls use Blender's API to create content, which can then be rendered.Source: §3.3 · p. 4, Figure 1 · p. 5
  • Dispatch identifies functions relevant to each instruction, reducing the need for the other agents to analyze the full function set and potentially avoiding undesired modifications.For a winter-setting instruction, the paper gives adding a snow layer and updating trees as example functions.Source: §3.3 · p. 4
  • Conceptualization enriches descriptions with function-specific appearance details because sparse instructions can leave modeling parameters unspecified and direct inference may yield simplistic, repetitive values.The paper gives tree branch length, tree size, and leaf type as details absent from a brief meadow description.Source: §3.3 · p. 5
  • The modeling agent uses function code, documentation, and examples to infer parameters and generate typed Python calls; Blender uses those calls for mesh production and rendering.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

The experiments assess instruction-following across large scenes, fine-grained object control, sequential editing, and individual modeling functions, then examine the contributions of the task dispatch and conceptualization agents.

4.1 3D MODELING

  • 3D-GPT generates diverse large scenes that generally align with descriptions, with Blender-rendered meshes supporting 3D consistency.The study evaluates 100 scene descriptions generated from repeated ChatGPT prompts.Source: §4.1 · p. 6
  • Random flower prompts test fine-grained control of curves, key appearance features, and color across real-world flower types.The reported results describe faithful capture of the flowers' distinct appearances.Source: §4.1 · p. 7, Figure 3 · p. 7
  • Across sequential editing steps, 3D-GPT comprehends follow-up instructions and modifies scenes while retaining prior modifications as context.The method is reported to require no additional networks for controllable editing, unlike Zhang et al. (2023b).Source: §4.1 · p. 7, Figure 4 · p. 7
  • For Nishita-sky modeling, 3D-GPT infers parameters from text by relating them to real-world sky and weather conditions.The sky-texture function does not directly map color information to sky appearance.Source: §4.1 · pp. 7–8, Figure 5 · p. 8

4.2 ABLATION STUDY

  • Without the Task Dispatch Agent, CLIP score falls from 29.16 to 22.79 in a test of 100 initial descriptions each followed by one instruction.The reported effect primarily concerns subsequence instructions, since all functions are used for the initial instruction.Source: §4.2 · p. 8, Table 1 · p. 8
  • Without the Conceptualization Agent, text alignment and parameter diversity decrease, while failure rate increases; with it, flower models better match intended appearances.The study measures CLIP score using 100 initial scene descriptions.Source: §4.2 · p. 9, Figure 6 · p. 9

5 DISCUSSION AND CONCLUSION

3D-GPT is a training-free framework for instruction-driven 3D modeling that integrates procedural generation, with experiments indicating LLMs’ potential for reasoning, planning, and tool use in this setting.

5 DISCUSSION AND CONCLUSION

  • 3D-GPT combines three collaborative agents to produce a 3D modeling file and reports high-quality results, expansive-scene adaptability, 3D consistency, material editing, and ray-traced visualization.The authors frame the empirical results as evidence of LLM potential for reasoning, planning, and tool use in procedural 3D modeling.Source: §5 · p. 9
  • The framework’s limitations include restricted curve control and shading design, reliance on available procedural algorithms, and challenges with audio or image instructions.The authors note that algorithm dependence may limit categories such as hair and fur, while multimodal processing can lose information.Source: §5 · p. 9
  • Proposed directions include fine-tuning LLMs for geometry and shading, autonomous discovery of generation rules, and improved multimodal instruction processing.These directions target finer modeling control, reduced human involvement in discovering rules, and broader user inputs.Source: §5 · p. 9

6 APPENDIX

The appendix presents additional 3D results, prompt examples, ablation evaluation details, and an example of parameter inference by the modeling agent.

6.1 ADDITIONAL RESULT

  • The appendix directs readers to an online page to view additional 3D results.The text describes the results as high quality.Source: §6.1 · p. 13

6.2 PROMPT EXAMPLE

  • The Adding Trees prompt example presents document, code, information, and usage-example components; the appendix also labels a sky-modeling-function prompt example.Figure 7 is the Adding Trees prompt example, and Figure 8 is the sky modeling function prompt example.Source: §6.2 · p. 13, Figure 7 · p. 13, Figure 8 · p. 14

6.3 ABLATION STUDY DETAILS

  • The ablations assess the Conceptualization and Task Dispatch Agents using CLIP scores, failure rates, and parameter diversity.CLIP evaluation uses 50 rotating-camera views; the conceptualization test uses an initial instruction, while task dispatch uses an initial and a subsequent instruction.Source: §6.3 · p. 14
  • Failure rates track parser, datatype, or function-call parameter errors across five functions per scene, while diversity is measured over 500 parameters using Shannon Diversity Index.Float parameters are divided into 100 bins.Source: §6.3 · p. 14

6.4 AGENT RESPONSES AND VISUAL RESULT

  • The single-function control example reports accurate inference of algorithm parameters, including parameters not directly connected to visual appearance.Figure 9 pairs a visual result with a modeling-agent response example.Source: Figure 9 · p. 15
Loading 2310.12945v2…