PaperlayerView exact revision on arXiv

Source-linked AI summary

L3GO: Language Agents with Chain-of-3D-Thoughts for Generating Unconventional Objects

Yutaro Yamada, Khyathi Chandu, Yuchen Lin, Jack Hessel, Ilker Yildirim, Yejin Choi

arXiv:2402.09052v1cs.AI

TL;DR

Text-to-image and text-to-3D models struggle to follow prompts specifying precise spatial configurations, particularly for unconventional objects. L3GO builds meshes through structured, iterative reasoning in Blender, and is reported to outperform other agents and generative models in its evaluations. Its agent communicates through text rather than visual input.

  • Problem

    Text-to-image and text-to-3D models can produce outputs that do not follow prompts requiring precise spatial configurations, such as a chair with five legs.

    Source: §1 · p. 1

  • Method

    L3GO uses a language agent to construct objects part by part in Blender, using environmental feedback to refine the mesh.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    L3GO outperforms GPT-4, ReAct-B, and Reflexion-B in ShapeNet evaluations and text-to-image and text-to-3D models in human preference on UFO.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The findings suggest structured prompting could generate unconventional data that may further improve diffusion models.

    Source: §7 · p. 8

    L3GO is text-based and does not use visual information; spatial orientations are set as assumptions in prompts.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Diffusion-based image generation models such as DALL-E 3 and Stable Diffusion-XL demonstrate remarkable capabilities in generating images with realistic and unique compositions. Yet, these models are not robust in precisely reasoning about physical and spatial configurations of objects, especially when instructed with unconventional, thereby out-of-distribution descriptions, such as "a chair with five legs". In this paper, we propose a language agent with chain-of-3D-thoughts (L3GO), an inference-time approach that can reason about part-based 3D mesh generation of unconventional objects that current data-driven diffusion models struggle with. More concretely, we use large language models as agents to compose a desired object via trial-and-error within the 3D simulation environment. To facilitate our investigation, we develop a new benchmark, Unconventionally Feasible Objects (UFO), as well as SimpleBlenv, a wrapper environment built on top of Blender where language agents can build and compose atomic building blocks via API calls. Human and automatic GPT-4V evaluations show that our approach surpasses the standard GPT-4 and other language agents (e.g., ReAct and Reflexion) for 3D mesh generation on ShapeNet. Moreover, when tested on our UFO benchmark, our approach outperforms other state-of-the-art text-to-2D image and text-to-3D models based on human evaluation.

1. Introduction

Text-to-image and text-to-3D models struggle to follow prompts requiring precise spatial configurations. L3GO uses a structured, iterative Blender workflow and outperforms other agents and generative models in the reported evaluations.

1. Introduction

  • Diffusion models can miss precise spatial requirements such as a five-legged chair, even after repeated human-feedback attempts to correct the output.The introduction notes that DALL-E 3 still fails after five feedback iterations.Source: §1 · p. 1, Figure 1 · p. 1
  • GPT-4 can create simple objects such as lamps, but its success is limited for objects with more than four parts or complex objects such as airplanes.The examples include assembling a table's legs and constructing airplanes.Source: §1 · p. 2, Figure 2 · p. 2
  • SimpleBlenv, built on Blender, provides a setting for evaluating LLM agents' text-to-3D mesh generation.The paper reports creating and releasing the environment.Source: §1 · p. 2
  • L3GO outperforms GPT-4, ReAct-B, and Reflexion-B on ShapeNet evaluations, and text-to-image and text-to-3D models on UFO human preference.The ShapeNet comparisons use both human and automatic evaluation.Source: §1 · p. 2
  • L3GO constructs objects part by part, specifying and critiquing parts and placements before rendering the mesh into an image with ControlNet.Its process critiques spatial placement and completion before the rendering stage.Source: §1 · p. 2

2. Related Work

Related work covers language-model spatial reasoning, LLM agents that interact with external environments, and text-to-3D generation. L3GO applies these lines of work to constructing 3D meshes from language instructions.

Spatial Understanding of Language Models

  • Prior studies examine spatial reasoning in simulated 2D environments, directional representations, and language-based spatial question answering.The cited work is used to support the claim that language models exhibit spatial reasoning capabilities.Source: §2 · p. 2

Large Language Models as Agents

  • LLM agents combine reasoning and interaction with external APIs; iterative approaches have been applied to language and embodied tasks including games, web navigation, and robot navigation (Schick et al., 2023; Yao et al., 2022b; Wang et al., 2023a; Lin et al., 2023a; Yao et al., 2022a; Huang et al., 2022).The paper positions its use of LLMs for 3D meshes within this developing field.Source: §2 · p. 2

Text to 3D models

  • Text-to-3D work adapts pretrained text-to-image diffusion models, while LLM-based approaches offer another perspective on out-of-distribution mesh creation (Poole et al., 2022; Lin et al., 2023b; Wang et al., 2023b).The cited text-to-3D studies suggest similar challenges to those found in text-to-image generation.Source: §2 · p. 2

3. L3GO framework

L3GO decomposes mesh construction into sequential part placement and correction, using Blender feedback to help language agents refine objects. SimpleBlenv supplies the action and feedback interface, while L3GO's generators and critics plan and check construction.

3. L3GO framework

  • L3GO replaces one-shot object construction, which can compound spatial inaccuracies, with stepwise part placement and iterative feedback in a 3D environment.The framework collects feedback from Blender and uses it to correct construction.Source: §3 · p. 3

3.1. SimpleBlenv

  • SimpleBlenv limits actions to five primitive shapes, with wrappers exposing parameters such as scale, location, and radius.These controls allow agents to create varied objects despite the restricted set of shape APIs.Source: §3.1 · p. 3
  • SimpleBlenv wraps Blender with action commands and feedback, including part bounding boxes, intersections, and unnecessary gaps.Its state representation tracks each part's size along the x, y, and z axes and its global location.Source: §3.1 · p. 3

3.2. L3GO: LLM-based 3D Generation of Objects

  • L3GO identifies and critiques part specifications, selects spatial relationships and coordinates, then runs Blender actions and checks continuity, overlap, and completion.A coordinate calculator uses three code-generated coordinate estimates and selects the majority result, with ties broken arbitrarily.Source: §3.2 · pp. 3–4
  • After mesh construction, the completion critic decides whether to continue building or render the object for ControlNet; the agent operates through text and uses spatial assumptions in its prompts.The prompts establish spatial orientations such as front and back in advance.Source: §3.2 · pp. 4–5

4. Experiments on ShapeNet

The ShapeNet experiments compare LLM-based mesh generation approaches using GPT-4V and human judgments. Structured agents outperform one-shot GPT-4, and L3GO leads the structured methods.

4.1. Baselines

  • Baselines adapt ReAct and Reflexion to Blender feedback, alongside one-shot GPT-4 code generation.Source: §4.1 · p. 5

4.2. Automatic Evaluation via GPT-4V

  • GPT-4V evaluates meshes from ten views per object across four methods and 13 ShapeNet categories; human assessments use the same views.Source: §4.2 · p. 5, §4.1 · p. 5
  • Structured agents outperform one-shot GPT-4, with L3GO best; Figure 5 highlights its advantage on airplanes and rifles.Source: §4.2 · p. 5, Table 1 · p. 5, Figure 5 · p. 6
  • Human judgments also favor L3GO over GPT-4, except in a few categories including lamps, benches, and loudspeakers.Source: §4.2 · p. 6
  • GPT-4V aligns most closely with human judgments among the evaluated models, although agreement among humans is imperfect.Source: §4.2 · p. 6, Table 2 · p. 6

5. Experiments on UFO: Constructing Unconventionally Feasible Objects

The UFO experiments test whether L3GO can construct objects from unusual prompts, and human evaluations favor L3GO over the compared language agents and text-to-image and text-to-3D models.

Benchmark

  • UFO contains 50 difficult prompts requiring precise spatial understanding and featuring unusual combinations, such as a chair with one armrest.The benchmark is designed to probe prompts less likely to occur during text-only pre-training.Source: §5 · p. 6
  • Its nine object categories pair everyday objects with varied attributes, helping distinguish prompt-following ability from object-creation ability.Examples include a chair with five legs and a mug with two handles.Source: §5 · p. 6

Evaluation

  • For each prompt, human evaluators compare two sets of ten outputs; 3D outputs are rendered from ten camera angles as rotating GIFs.Each experiment recruits ten evaluators and includes attention checks.Source: §5 · p. 7

Results

  • L3GO receives higher human preference than ReAct-B, Reflexion-B, DALL-E-3, SDXL, and Shap-E on UFO.Figure 7 reports the human-preference comparison.Source: Figure 7 · p. 7, §5 · p. 7

Background and texture

  • In both background-and-texture scenarios, L3GO outperforms text-to-image models in human preference.The evaluation varies prompt styles and SDXL guidance scales; Figure 9 also uses gray-shaded backgrounds and DALL-E-3 quality=hd.Source: §5 · p. 7

Evaluation caveat

  • GPT-4V refused to answer in over 20% of its initial UFO evaluation cases.The authors note this may relate to characteristics of the generated images.Source: §5 · p. 7

6. Ablation studies

The ablations compare L3GO’s spatial critic and coordinate calculator and test a smaller open-source language model. Results show lower average accuracy with Mixtral than with GPT-4 and identify category-specific losses when coordinate calculation is removed.

Ablations

  • 0.515 accuracy without the spatial critic and 0.585 without program-based coordinate calculation, compared with L3GO’s 0.6 average accuracy.These are mean GPT-4V evaluation scores across ShapeNet’s 13 categories.Source: §6 · p. 7
  • Without coordinate calculation, cabinet and car accuracy falls to 0, while L3GO scores 0.5 and 0.4 in those categories, respectively.The overall average for the no-coordinate-calculator variant is similar to L3GO’s.Source: §6 · p. 7

UFO examples

  • Figure 8 examples show that LLM-based approaches ReAct-B, Reflexion-B, and L3GO successfully create the desired objects, while DALL-E 3, Stable Diffusion XL, and Shap-E still struggle to follow prompts perfectly.Source: Figure 8 · p. 8

Background and texture

  • Figure 9 compares L3GO with DALL-E-3 and SDXL using gray-shaded backgrounds, SDXL guidance scales 15 and 30, and DALL-E-3 quality=hd.The caption identifies these settings but does not state a result.Source: Figure 9 · p. 8

Open-source LLMs

  • 0.138 average accuracy for ReAct-B(Mixtral-8x7B), versus 0.423 for ReAct-B(GPT-4) on ShapeNet-13.The passage reports that Mixtral-8x7B is known to match or surpass Llama-2 70B and GPT-3.5 on most benchmark tests.Source: §6 · p. 8

Conclusion

  • The authors conclude that constructing mesh objects demands precise 3D spatial understanding and reasoning at GPT-4 level.The Mixtral comparison is the evidence supplied for this conclusion.Source: §6 · p. 8

7. Conclusion

L3GO is presented as a language agent for building 3D objects, with reported advantages over other models on ShapeNet and UFO. The conclusion also identifies possible applications and ethical concerns, while the GPT-4V evaluation was limited by refusals.

7. Conclusion

  • L3GO outperformed GPT-4, ReAct, and Reflexion on 13 large ShapeNet object categories, and the authors introduced UFO to test unconventional object characteristics.The conclusion notes that structured prompting could supply unconventional data for diffusion models and that studying language models’ spatial representations may offer insight into their 3D modeling abilities.Source: §7 · p. 8
  • The authors describe L3GO as having potential to support design, engineering, arts, and education, while emphasizing responsible use of image-generation technology.They identify misleading or harmful content as an ethical concern and call for continued research into safeguards.Source: §7 · pp. 8–9

A. Algorithm

B. GPT-4V as an evaluator for UFO

  • GPT-4V declined to evaluate more than 20% of UFO prompts in each of two trials, so the authors used human evaluators for the main experiments.The test compared two sets of 10 images and asked GPT-4V to choose a row or express no preference.Source: B. GPT-4V as an evaluator for UFO · p. 12

C. Details for human evaluation

The human evaluation recruited English-fluent participants through Prolific and obtained consent before the study began. The authors note that the sample may be biased toward English-speaking people with computer and internet access.

C. Details for human evaluation

D. Additional details for experiments

The appendix points to additional SimpleBlenv implementation materials, the UFO prompt list, and examples of generated outputs. It also states that the code was uploaded.

D. Additional details for experiments

D.1. Additional generated examples

E.1. Creative Combinations of Object Parts

L3GO is shown composing unusual combinations of object parts, with the authors reporting appropriate placement of components such as wings and umbrellas. They also describe it as an early-stage language interface for complex 3D software.

E.1. Creative Combinations of Object Parts

  • L3GO successfully interpreted and constructed unusual designs, including placing wings on a chair’s sides and an umbrella on top.The examples use basic shapes for wings and umbrellas.Source: §E.1 · p. 13

E.2. Language interface for complex softwares

  • The authors propose L3GO as a language interface that can generate an initial object draft from a prompt, but say it needs further development for practical use.Source: §E.2 · p. 14

F. Limitations

L3GO’s mesh quality remains below diffusion-based text-to-3D methods and human standards, while generation can take minutes, especially for complex objects.

F. Limitations

  • L3GO-generated meshes lag behind diffusion-based text-to-3D methods and remain far from human standards.Source: F. Limitations · p. 15
  • Simple objects take roughly 3 to 5 minutes to create, while complex ones take 10 to 20 minutes depending on retries.Source: F. Limitations · p. 15
  • Additional ShapeNet examples include creative chair variants such as a chair with wings or an attached umbrella.Source: Figure 16 · p. 16, Figure 15 · p. 16
  • An LLM agent can help Blender newcomers by creating a draft object.Source: Figure 17 · p. 17
Loading 2402.09052v1…