PaperlayerView exact revision on arXiv

Source-linked AI summary

BrickWright: Agentic Creation of Physically Feasible Brick Builds

Ava Pun, Kevin Dai, Sally Chen, Maxwell Jones, Ruixuan Liu, Kangle Deng, Changliu Liu, Maneesh Agrawala, Deva Ramanan, Jun-Yan Zhu

arXiv:2610.31651v1cs.CV

TL;DR

Agents that can model 3D objects still struggle to create physical assemblies without collisions, floating parts, or disconnections. BrickWright guides pre-trained agents to create and revise brick-build code using BrickLayer’s assembly API and physical-validation tools. BrickWright-equipped lightweight agents achieved 100% validity on 512 samples tested, while improving prompt adherence at a similar cost to agents without BrickWright.

  • Problem

    Agents that can model 3D objects still struggle to create physical assemblies without collisions, floating parts, or disconnections.

    Source: §1 · pp. 1–2

  • Method

    BrickWright guides pre-trained agents to create and revise brick-build code using BrickLayer’s assembly API and physical-validation tools.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    BrickWright-equipped lightweight agents achieved 100% validity on 512 samples tested, while improving prompt adherence at a similar cost to agents without BrickWright.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The results support the value of construction APIs and physical validation for improving LLM-agent performance on building tasks.

    Source: §5 · p. 7

    The feasibility definition excludes valid assembly sequences and stability analysis, which are left to future work.

    Source: §5 · p. 7

Abstract

from arXiv · show

Frontier coding agents are becoming increasingly adept at 3D modeling. Yet, they still face challenges when building objects meant to be assembled in the physical world, often creating invalid assemblies with floating, colliding, or disconnected parts. We argue that providing agents with tools for assembling builds and verifying their physical feasibility is sufficient for them to produce physically valid assemblies that still adhere to input prompts. Focusing on the domain of LEGO brick builds, we present BrickWright, a framework for generating physically feasible brick builds from text or image prompts using pre-trained agents, where a build is physically feasible if it uses only real parts, has no colliding or floating parts, and forms a connected structure if possible. BrickWright guides agents to create builds by writing code with BrickLayer, a Python toolkit with two main components: an ergonomic brick assembly API that halves inference cost compared to working in raw LDraw, and a validator that provides feedback on physical feasibility. Whereas baseline agents achieve physical validity rates ranging from only 9% (GPT-6 Luna) to 69% (GPT-6 Astra), BrickWright-guided agents achieve 100% validity on a sample of 512 prompts, while improving prompt adherence for lightweight agents and maintaining it for stronger agents. Additionally, they outperform specialized models on the text-to-brick task. Code is available on our project page: https://avalovelace1.github.io/brickwright/

1. Introduction

BrickWright targets a gap between agents’ 3D modeling abilities and the physical constraints of multi-part assemblies. It pairs a brick-assembly toolkit with validation, and reports improved physical validity while maintaining or improving prompt adherence.

1. Introduction

  • Multi-part assemblies must use preset parts, avoid collisions and floating parts, and typically form a connected structure.The paper focuses on LEGO builds, a complex system with thousands of parts and dozens of connection types.Source: §1 · p. 2
  • The authors argue that agents need an ergonomic assembly API and a validator that gives feedback on physical feasibility.They report that even agents that write their own validation scripts often produce builds with physical violations.Source: §1 · p. 2
  • BrickLayer provides assembly and validation tools, while BrickWright guides agents to create validated builds from text or image prompts.The task is to generate code using an API grounded in brick production rules and export a build in standard LDraw format.Source: §1 · p. 2
  • 100% validity across agents on 512 prompts, including Luna’s increase from a 9% baseline, is reported for BrickWright-guided builds.The introduction also reports improved prompt adherence for lightweight models and higher validity and fidelity than specialized brick-generation models.Source: §1 · p. 2

2. Related work

Prior brick-generation work includes optimization, retrieval, learned generators, and agentic modeling, but these approaches face limits in vocabulary, generalization, or physical validation. BrickWright instead uses fixed part connectors and an executable verifier to let pre-trained agents iteratively repair builds.

2. Related work

  • Earlier optimization and retrieval approaches are constrained by small brick vocabularies, explicit geometric guidance, or fixed templates.Methods include voxel-to-brick optimization, sequential assembly [17] [49], and retrieval from prebuilt figurines.Source: §2 · p. 2
  • Learned brick generators span graph, voxel, policy, diffusion, and autoregressive methods, with some limited in category or orientation and others prone to disconnection or collisions.BrickGPT [37] uses eight rectangular types across 21 categories; LEGO-Maker supports axis-aligned builds in two categories; LegoACE can produce disconnected components, while BrickNet [19] accumulates collisions over longer sequences.Source: §2 · p. 2
  • Agentic 3D modeling uses iterative code execution and visual inspection, but often resolves collisions and floating objects through optimization or visual judgment.BrickWright’s explicit connector rules enable deterministic attachment checks.Source: §2 · p. 3
  • Unlike prior code-based modeling systems that generate geometry or infer relationships, BrickWright assembles fixed parts whose connectors determine attachment and motion.The related-work discussion contrasts this approach with articulated asset modeling using custom SDKs [59].Source: §2 · p. 3
  • Physics-aware generation has used post-generation refinement, simulator feedback, constrained diffusion, or force-based and simulation-based stability assessment.BrickWright instead uses an executable verifier for collisions, floating parts, and connectivity, and the authors identify force-based stability analysis as a possible extension.Source: §2 · p. 3

3. BrickWright

BrickWright defines physical feasibility for catalog-based brick builds and guides an agent through code creation, testing, and revision. BrickLayer supplies the assembly and validation interface; the agent can inspect rendered views and finalize once checks pass.

3. BrickWright

  • A build is physically feasible when it uses catalog parts, has no collisions or floating parts, and has as few connected components as possible.Stability and executable assembly sequences are outside this definition; multiple components may still be appropriate for some prompts.Source: §3 · p. 4
  • BrickWright guides a single agent to write and revise Python code, accepting the build only when predefined and agent-written tests pass.The process follows a plan-execute-validate loop informed by test results.Source: §3 · p. 4

3.1. BrickLayer

  • BrickLayer provides a unified representation and functions for discovering parts, assembling builds, and validating them, with conversion between BrickLayer and LDraw.Its assembly model includes elements and nested builds, typed connectors, and placements that can be made relative to named components.Source: §3.1 · p. 4
  • BrickLayer includes high-level constructors for common parts and sub-builds, plus functions that fill cuboids or wedges validly and connected when possible.These convenience functions were added after repeated patterns appeared in agent-generated code.Source: §3.1 · p. 4
  • BrickLayer checks collisions, floating elements, and disconnected components; its floater test traces connector links or supporting surface contacts to grounded elements.Surface contact counts as supporting only when it has an upward component; lateral contact alone is insufficient.Source: §3.1 · pp. 4–5

3.2. Building with BrickLayer

  • The agent uses validate for fixed collision and floating checks alongside its tests, render for visual inspection, and finalize when finished.The agent may also use explore to investigate the API and the written build; the overview’s arrows indicate information flow, not a fixed tool-call sequence.Source: §3.2 · p. 5

4. Experiments

BrickWright is evaluated against specialized brick-generation models and agentic baselines, with measures of physical feasibility, prompt adherence, and generation cost. The experiments compare its baseline performance, ablate the assembly API and validator, and demonstrate extensions to other build settings.

4. Experiments

  • The evaluation uses 512 text prompts and compares BrickWright with specially trained brick models and pre-trained agentic methods on feasibility and prompt adherence.It also measures brick counts, time, and cost, and tests lightweight, balanced, and frontier agents.Source: §4 · pp. 5–6
  • Human review changes collision-free rates by up to 1.4 percentage points and component counts by up to 0.13, and Table 1 reports post-review numbers.The review checks potential false positives such as collisions resolved by shifts of at most 0.2 mm and components containing parts without connector annotations.Source: §4 · p. 6

4.1. Baseline comparison

  • GPT-6 Luna rises from 9% baseline validity to 100% with BrickWright, while lightweight agents improve prompt adherence and stronger agents maintain it at broadly comparable token costs.BrickWright agents also outperform specially trained text-to-brick models in overall feasibility and prompt adherence; most use 2–3 validate calls, while Luna uses 4–5.Source: §4.1 · p. 6

4.2. Ablation study

  • Without the assembly API, agents spend around twice the token cost and three times as many validate calls; without the validator, all agents score below 79% validity.The API’s grid-snapping and connector-mating functions help agents, especially lightweight ones, produce valid builds; Luna’s average part count drops from 27.4 to 7.2 without the API.Source: §4.2 · p. 7

4.3. Extensions

  • BrickWright supports image-conditioned builds, indoor and outdoor scenes, builds exceeding 29,000 parts, articulated objects, editing, and constraints on available bricks.These capabilities arise from pre-trained agent capabilities without toolkit updates.Source: §4.3 · p. 7, Figure 5 · p. 7

5. Discussion

BrickWright closes a gap between aesthetically pleasing prompted builds and physically feasible builds. Lightweight agents reach 100% validity on 512 tested samples while improving prompt adherence at similar cost; the paper leaves stability and assembly-sequence checks to future work.

5. Discussion

  • BrickWright-equipped agents reach 100% validity on 512 tested samples while improving prompt adherence for lightweight agents and maintaining it for stronger agents.The paper reports similar costs to agents without BrickWright.Source: §5 · p. 7
  • The authors identify well-written construction APIs and physical validation as important for improving LLM-agent performance on building tasks.Source: §5 · p. 7
  • The feasibility definition excludes valid assembly sequences and stability analysis, and agentic methods are generally slower and more costly than specially trained models.Distilling powerful agents’ capabilities into smaller, faster models remains future work.Source: §5 · p. 7

A. Additional results

The additional results describe how quickly agents reach validation and characterize failed validation calls. The appendix also points to qualitative comparisons in Figure 7.

A. Additional results

  • With the assembly API, 81% of GPT-6.1 Sol builds pass validation within two calls, compared with 37% without the API.Table 3 reports valid@k—the frequency of a passing validation call within the first k calls—and mean token usage.Source: A. Additional results · p. 12
  • Physical checks account for 63–85% of failed validation calls, followed by runtime errors at 11–28%.For GPT agents, collisions are the main physical failure; Haiku’s main failure mode is connectivity.Source: A. Additional results · p. 12
  • As agent capability increases, physical failures shift toward collisions and away from floaters and connectivity.Source: A. Additional results · p. 12

B. System prompt and BrickLayer reference

The system-prompt and reference appendix documents the prompts used for BrickWright and baseline agents, along with selected BrickLayer API classes.

B. System prompt and BrickLayer reference

C. Example prompts

The examples pair text and image prompts with LEGO builds spanning recognizable objects, scenes, articulated designs, and large-scale settings. Further examples illustrate baseline comparisons, an ablation, image-conditioned generation, and iterative editing.

C. Example prompts

  • The paper provides full prompts for all examples shown in its figures.Source: C. Example prompts · p. 12
  • Examples span text and image prompts for articulated creatures, large-scale scenes, and varied recognizable builds.The teaser includes a poseable crab and a harbour scene specified at 10,000+ parts; image-conditioned examples include an Arctic observatory and a mountainside city specified at 15,000+ parts.Source: C. Example prompts · pp. 12–13
  • Baseline comparisons use prompts for vehicles, creatures, flowers, and indoor or outdoor scenes, including an ATV, helicopter, fox, and backyard.Additional prompts cover a sports car, tugboat, locomotive, snake, squirrel, dragon, station, and garden.Source: C. Example prompts · pp. 12–13
  • The ablation examples prompt a compact grand piano and a dilapidated shelter or bunker.The piano prompt specifies a hinged lid and multiple brick types; the shelter prompt includes a bed, rifle, sack, and money.Source: C. Example prompts · p. 13
  • The editing example starts with a timber-framed cottage, then changes its depth and reshapes its roof while preserving specified features.The first edit doubles depth while preserving width, height, and style; the second changes the roof to a dark-blue gambrel while keeping the stretched walls and chimney.Source: C. Example prompts · p. 13

D. Brick build generation with vanilla agents

This section examines brick-build generation with vanilla and BrickWright-guided agents, alongside BrickLayer’s geometry and connector API for describing valid mating connections. It also emphasizes physical checks and notes that physical validity alone does not establish visual fidelity.

D. Brick build generation with vanilla agents

  • 471 of 512 builds (92%) used a generator, and 424 (83%) used a validator, according to an AST-based analysis of baseline GPT-6 Astra programs.Programs were classified as generators when they wrote to model.ldr, and as validators when they read it and asserted or reported errors using derived values.Source: D. Brick build generation with vanilla… · p. 13
  • BrickWright instructions require collision-free, supported builds that are connected unless separate pieces are requested, while preserving the requested identity, proportions, features, and detail.The harness independently checks collisions and floating parts; connectivity and request-specific requirements are checked through run_tests().Source: D. Brick build generation with vanilla… · p. 15
  • For articulated builds, optional motion checks sample annotated joint ranges for collisions, floating parts, and connector disengagement; static checks omit motion, and sampled poses do not cover every intermediate position.Unannotated connections remain rigid, which may produce false positives when those connections would also move in reality.Source: D. Brick build generation with vanilla… · p. 15
  • The suggested workflow plans features and support, checks catalog parts, builds incrementally and repairs validation failures, inspects renders when available, and runs checks on deterministic source.The instructions caution that physical validity alone does not establish visual fidelity.Source: D. Brick build generation with vanilla… · p. 15
  • BrickLayer filters and orders connector queries by type, facing, target, and count; valid connections depend on connector compatibility and relative frame alignment.Connector positions are expressed in the queried geometry’s frame, while placements use world coordinates.Source: D. Brick build generation with vanilla… · pp. 19–20

Related papers

Loading 2610.31651v1…