BrickBench: Evaluating Agentic Brick Design

1. Introduction

• Prior evaluations leave full-set physical complexity and design quality unresolved; design quality has no definitive pass/fail test.BrickNet evaluates objects of at most 100 parts, unlike sets with hundreds or thousands of interconnected pieces and inventory constraints.Source: §1 · p. 2
• BrickBench evaluates agentic text-conditioned LEGO-set design across 300 prompts in Model, Set, and fixed-inventory Alt-Build settings.BrickAgent provides construction tools and interpretable validation, while pairwise judgments validated against human preferences measure design quality.Source: §1 · p. 2
• With BrickAgent, leading agents produce valid assemblies for nearly every prompt, while the best satisfies approximately 95% of semantic requirements; human raters identify human designs in 323 of 360 comparisons.Removing BrickAgent sharply reduces physical validity, showing that reliable construction depends on grounded tools and feedback.Source: §1 · p. 2
• The paper contributes a benchmark, a construction-and-validation environment, and an evaluation of eleven frontier agents and two data-driven baselines with human studies.Metrics cover physical validity, semantic alignment, and design quality.Source: §1 · p. 2

Agentic Benchmarks

• Agentic benchmarks execute agent-written programs and score outcomes, spanning software tests, graphics, CAD, modular assembly, and simulated block placement.BrickBench executes programs into real LEGO-part assemblies and validates connections, collisions, and simulated stability.Source: §2 · p. 3

LEGO and Evaluation

• Prior LEGO generation includes specialized models for grid placement, connector-based structures, and image-conditioned facades, alongside tasks for inspecting or reasoning about assemblies.BrickGPT places eight brick types on a 20×20×20 grid, while BrickNet predicts connections and generates structures through typed connectors.Source: §2 · p. 3
• BrickBench has general coding agents generate set-scale assemblies from text prompts, then scores physical validity, prompt alignment, and design.Source: §2 · p. 3

3.1. LDraw

• LDraw is a community-maintained digital LEGO modeling standard whose part instances encode type, color, and 6D pose, but it does not determine whether an assembly is buildable.Source: §3.1 · p. 3

3.2. Connectivity and Physical Validity

• BrickNet annotates LDraw parts with typed connectors; compatible connector pairs aligned within tolerance define connections, and assemblies are tested for collisions and stability.The connector system includes stud, hinge, axle, ball, and fixed types; stability simulation treats connected components as rigid bodies under gravity.Source: §3.2 · pp. 3–4

3.3. Text-Conditioned Assembly

• Text-conditioned assembly requires selecting library part types and assigning each instance a color, position, and 3D rotation to depict a prompt.Prior BrickNet training used online human-designed assemblies, and its validation set contains 512 held-out captions for objects of at most 100 parts.Source: §3.3 · p. 4
• Without task-specific training, GPT-5.6 Luna exceeds BrickNet and BrickGPT on BrickNet validation, scoring within 0.02 of held-out assemblies on each metric.The evaluation uses captions for single objects of at most 100 parts and PE, SigLIP 2, and VQAScore; the result motivates a new benchmark.Source: Figure 2 · p. 4, §3.3 · p. 5

4. BrickBench

• BrickBench asks agents to create physically buildable LEGO assemblies from text prompts and evaluates validity, semantic alignment, and design quality.Source: §4 · p. 5

4.1. Settings

• Model permits at most 400 parts, Set requires 400–4000 parts, and Alt-Build restricts designs to a fixed retail-set inventory.The settings test primarily single-component assemblies, full-size-set construction, and design with a scarce fixed inventory, respectively.Source: §4.1 · p. 5

4.2. Prompts

• Each setting has 100 prompts across ten topic categories, manually validated after LLM-assisted construction; average prompt lengths range from 31 to 57 words.Alt-Build prompts average 31 words, Model 45, and Set 57.Source: §4.2 · p. 5

4.3. BrickAgent

• BrickAgent combines the LDraw library, BrickNet connectors, programmatic construction and viewing tools, and a validator that reports the parts involved in failures.Without its tools, frontier coding agents cannot reliably construct valid builds.Source: §4.3 · p. 6

4.4. Metrics

• Validity requires meeting setting-specific part requirements while avoiding collisions and remaining stable; VQA measures prompt-question satisfaction, and ELO aggregates pairwise alignment and design preferences.VQA uses eight rendered views and yes/no questions; ELO fits Bradley–Terry models to pairwise judge preferences.Source: §4.4 · p. 6

5. Evaluations

• Eleven frontier agents and two data-driven baselines were evaluated; Opus 5.5 and GPT-6.1 Sol received automated scores only after the human studies.The baselines apply only to the Model setting.Source: §5 · pp. 6, 8

5.1. Results

• Five agents deliver valid assemblies for every prompt, none falls below 0.80 overall Valid, and Design ELO spans 545 points from GLM to Astra.Figure 3 summarizes metrics averaged across the three settings; Table 1 reports disaggregated results.Source: §5.1 · p. 8, Figure 3 · p. 8
• The reference agents span a factor of 30 in cost without rank following price; Muse costs nine times DeepSeek but rates 138 points lower.Both trained baselines rank below every agent.Source: §5.1 · p. 8

5.3. Validation of the Design Judge

• Human and VLM rankings agree on 34 of 36 agent pairs (Kendall τ=0.89), supporting Design ELO as an effective proxy for human judgment.The study covered the nine reference agents; Opus 5.5 and GPT-6.1 Sol were evaluated afterward.Source: Figure 5 · p. 10, §5.3 · p. 11

5.1. Results

• Set is hardest for validity (Muse 0.68, Qwen 0.79, DeepSeek 0.87), while Alt-Build costs least and has the highest validity.Alignment falls for every agent in Set, and many agents build near the setting’s 400-part floor.Source: §5.1 · p. 10

5.2. Environment Ablation

• Without tools, Astra is valid on 40% of prompts and Luna on less than 1%, while Luna’s VQA rises 0.02 and ELO rises 62 points.The ablation replaces BrickAgent tools with the LDraw library and a primer on its file format.Source: §5.2 · p. 10, Table 2 · p. 11

5.4. Comparison with Human Designs

• Raters chose human designs in 323 of 360 comparisons; agent builds often meet prompts but lack humans’ part usage, proportions, and surface detail.The comparisons used part-count-matched designs from Model and Set, but the pairs were not prompt-matched.Source: Figure 6 · p. 11, §5.4 · p. 12

6. Discussion and Limitations

• LEGO-familiar participants distinguished agent-designed from human-designed assemblies nine out of ten times, indicating a remaining design-quality gap.The authors interpret this as evidence that agents meet requirements but fall short of designed-set standards.Source: §6 · p. 12
• General coding agents exceed task-specific models on their benchmark and BrickBench, while cost remains an open axis.BrickNet-14B produces an assembly for six hundredths of a cent, while Astra does so in $4.06.Source: §6 · p. 12
• The physical-validity simulator is a proxy: it does not test connection strength, so balanced assemblies may pass despite potentially sagging or coming apart.The authors describe this evaluation as necessary but not sufficient until a general force model is available.Source: §6 · p. 12

7. Conclusion

• Nine frontier agents build valid assemblies for most prompts and satisfy the majority of requests, but the conclusion highlights an open gap to human design.Displacing task-specific data-driven models comes at an increase of 900–6,800 times the cost.Source: §7 · p. 12
• BrickBench contains 300 prompts across three settings with different constraints and metrics for validity, alignment, and design confirmed by human raters.The conclusion also describes BrickAgent as the reference environment in which agents build.Source: §7 · p. 12

A. Prompts and Question Graphs

• Set graphs average 24 questions per prompt, as their prompts often describe scenes with multiple objects.Figure A1 shows the share of each setting’s questions by kind and subtype.Source: A. Prompts and Question Graphs · p. 15, Figure A1 · p. 15
• In a Set graph, roots are the named entities, while attribute and relation questions depend on questions establishing their referents.A missing entity therefore fails every subsequent question about it.Source: A. Prompts and Question Graphs · p. 15

B. Judge Prompts

• The alignment judge compares prompt coverage regardless of build quality, while the design judge compares design quality without seeing the prompt.Figure A3 reproduces the three prompts as the judge receives them.Source: B. Judge Prompts · p. 15
• The VQA judge answers each question-graph item yes or no from the rendered views alone.Figure A3’s VQA prompt lists every graph question and requires a yes-or-no answer for each.Source: B. Judge Prompts · p. 15
• In the sample question graph, an arrow means the second question is satisfied only if the judge answers yes to the first.Figure A2 depicts this dependency for a set prompt.Source: Figure A2 · p. 16

• Table A1's per-setting statistics show 20–40 average disconnected components for Qwen, DeepSeek, Muse, and GLM in Set, versus under five for Astra, Opus, Gemini, and Sol.Source: C. Additional Results · p. 16, Table A1 · p. 17
• Figures A4–A5 plot Tables 1 and A1 by setting, while A6 shows additional agent assemblies across the three settings.Source: C. Additional Results · p. 16, Figure A5 · p. 20, Figure A6 · p. 21
• Figure A7 disaggregates VQA scores by prompt category and attribute type.Source: C. Additional Results · p. 16, Figure A7 · p. 22
• Most agents peak near Set's 400-part floor, whereas Astra uses considerably more parts on average in both Model and Set.Source: C. Additional Results · p. 16, Figure A8 · p. 22
• Judge prompt construction differs by task: pairwise questions use tiled four-view images, the design question omits the prompt, and VQA uses eight separate views.Source: Figure A3 · p. 18
• Figure A4 evaluates BrickNet-14B and BrickGPT only in Model, and BrickNet-14B's reported Valid metric excludes stability.Source: Figure A4 · p. 19