PaperlayerView exact revision on arXiv

Source-linked AI summary

LMBuild: Evaluating LLM Agents for Generating Buildable and Functional Structures

Jiateng Liu, Rushi Wang, Cheng Qian, Xuejun Zhang, Sun Li, Jiayu Liu, Yifan Shen, Xu Cao, Jiarui Yao, Bingxuan Li, Ruhi Sarikaya, Heng Ji

arXiv:2610.04292v1cs.AI

TL;DR

Existing 3D-generation evaluations largely emphasize geometry and visual criteria, while systematic evaluation of buildability and function remains lacking. LMBuild provides an interactive benchmark and four-dimension evaluation framework. Across systems, functional affordance and physical operability remain more challenging than structural and visual criteria.

  • Problem

    Existing 3D-generation evaluations largely emphasize geometric quality, visual consistency, and semantic alignment, while systematic evaluation of buildable, functional structures under physical constraints is lacking.

    Source: §1 · p. 2

  • Method

    LMBuild combines an interactive tool-based construction environment, a curated benchmark augmented with real-world knowledge, and evaluation across four dimensions of structure quality and realization.

    Source: §1 · p. 2

  • Results

    Frontier closed-source models show near-perfect soundness and strong design quality, while all model families struggle with functional-part completeness, kinematics, and simulation-based operability.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LMBuild can diagnose open-source agents’ part-creation and scene-state-tracking challenges and provide a foundation for measuring progress toward buildable, functional structure generation.

    Source: §1 · p. 2

    LMBuild scores do not establish readiness for safety-critical deployment; evaluation omits several physical factors and requires domain-specific validation and human oversight.

    Source: ETHICS STATEMENT · p. 11, §A.6 · p. 63

Abstract

from arXiv · show

LLM-based agents are increasingly capable of generating complex 3D structures, with the potential to reshape how objects are designed and realized in the physical world. Yet, producing elegant geometry is fundamentally different from producing objects that can be built and perform their intended functions. Existing evaluations largely focus on geometric quality while overlooking physical realizability. We introduce LMBuild, a benchmark for evaluating LLM agents on generating buildable and functional structures. LMBuild represents generated objects as assembled structures comprising part decompositions, joints, materials, and sequences. To support reproducible evaluation, we provide a unified framework consisting of: (1) an interactive environment in which agents can use tools to retrieve, create, and place components to construct objects; (2) a curated benchmark that repurposes established CAD datasets and augments them with knowledge from Wikipedia; and (3) a evaluation framework covering structural soundness, functional affordance, design quality, and physical realization. Evaluations across 30 systems reveal several intriguing findings: (a) Soundness and alignment are no longer the primary bottlenecks for frontier closed-source models, while functional affordance and physical operability remain substantially more challenging; (b) stronger models more effectively create new components, whereas weaker models tend to rely on retrieval; and (c) providing functional specifications substantially improves part completeness, kinematics, and physical operability. These results show that generating real-world structures requires deeper reasoning about functional affordances, mechanics, and designing and creating novel components. We expect LMBuild to provide a foundation for measuring progress and incentivizing research toward agents that generate buildable and functional structures.

1 INTRODUCTION

LMBuild addresses the gap between visually plausible 3D generation and structures that can be built and function under physical constraints. It evaluates agents with a unified interactive benchmark.

1 INTRODUCTION

  • The paper identifies a lack of systematic evaluation of whether LLM agents can generate structures that are buildable, functional, and aligned with real-world physical constraints.Existing evaluations largely emphasize geometric quality, visual consistency, and semantic alignment.Source: §1 · p. 2
  • LMBuild combines an interactive construction environment, a curated benchmark augmented with real-world knowledge, and a four-dimension evaluation hierarchy.The hierarchy covers structural soundness, functional affordance, design quality, and physical realization.Source: §1 · p. 2
  • Frontier closed-source models show near-perfect soundness and strong design quality, but all model families struggle with functional parts, kinematics, and simulation-based operability.These results distinguish structural and visual performance from functional affordance and physical operability.Source: §1 · p. 2
  • Stronger models create new components more effectively, whereas weaker models rely heavily on retrieval; functional specifications improve part completeness, kinematics, and physical operability.The reported prompt improvements concern these functional measures.Source: §1 · p. 2
  • LMBuild diagnoses open-source agents’ part-creation gap and persistent scene-state-tracking challenges during structure construction.The authors position the benchmark as a foundation for measuring progress toward agents that generate buildable and functional structures.Source: §1 · p. 2

LMBUILD

LMBuild combines an interactive construction environment, a curated set of real-world-grounded structures, and a hierarchical evaluation framework. Its protocols vary how agents construct and refine structures, while the benchmark spans diverse construction regimes.

LMBUILD

  • LMBuild’s interactive environment equips agents with tools to inspect, select or create, place, and adjust components, then specify joints, materials, and assembly sequences.The workflow is illustrated with a wheelchair construction example.Source: §2.1 · p. 3, Figure 1 · p. 3

2.1 INTERACTIVE GENERATION OF BUILDABLE AND FUNCTIONAL STRUCTURES

  • In the baseline, agents receive a target label and reference image and construct in one round using retrieval or new-part creation; constrained settings isolate each mechanism.Domain-specific systems retain their native input and output protocols.Source: §2.1 · p. 3
  • Revision rounds return renderings and rule-based diagnostics, while fine-grained prompts add salient attributes and functional-part requirements.The evaluated revision budgets are K ∈ 1, 2, 3 rounds.Source: §2.1 · pp. 3–4

2.2 REPURPOSING EXISTING DATASETS FOR LMBUILD

  • LMBuild repurposes complementary geometry sources into 2,549 full-benchmark objects and a 200-object core set, selected for coverage across construction regimes and real-world-verifiable functions.Selection spans structural and kinematic families, varied part counts, and product-level CAD; experiments use the core set.Source: §2.2 · pp. 4–5

2.3 HIERARCHICAL EVALUATION FRAMEWORK FOR LMBUILD

  • The evaluation hierarchy covers structural soundness, functional affordance, design quality, and physical realization, with twelve sub-metrics normalized to a 0–100 scale.The dimensions assess stability, functional parts and kinematics, decomposition and appearance, and fabrication and operation.Source: §2.3 · p. 6

3 EXPERIMENTS

Across 30 systems, frontier closed-source models perform strongly on structural soundness and design, while functional affordance and physical operability remain challenging. The experiments also show that part creation benefits stronger models, explicit functional specifications improve results, and open models exhibit recurring creation and scene-tracking errors.

3.1 SETUP

  • The evaluation spans 30 systems, including proprietary APIs, open-weight models, and domain-specific generators.The systems cover diverse model families, generations, scales, and architectures.Source: §3.1 · p. 6

3.2 MEASURING MODEL CAPABILITIES FOR FUNCTIONAL STRUCTURE GENERATION

  • Frontier closed-source models perform strongly on soundness and design, but functional-part completeness, kinematics, and operability remain the largest capability gaps.Most open-source models score below 25 on joints and kinematics; the best simulation-based operability scores reach only the mid-30s.Source: Table 2 · p. 7, §3.2 · pp. 7–8

3.3 STRONG MODELS CREATE, WHILE WEAKER MODELS RETRIEVE

  • Retrieval helps weaker models by narrowing the design space, while frontier models benefit from creating structure-specific parts.Qwen3-VL-32B scores 61.7 under retrieval and 34.1 under creation-only across Soundness and Design metrics.Source: §3.3 · p. 8, Figure 3 · p. 8

3.4 IMPROVING CAPABILITIES WITH ITERATIVE REVISION AND PROMPTING

  • Three rounds of iterative revision yield only about 2% overall gains, mainly offering more chances to retry rather than addressing underlying reasoning limitations.Agents can already inspect and interact with the environment under the baseline protocol.Source: §3.4 · p. 8
  • Explicit functional requirements improve functional-part completeness by 10.3 points and physical operability by 12.0 points, with additional gains in kinematic correctness.The reported gains have p < 0.001; the figure summarizes average changes across models.Source: Figure 4 · p. 9, §3.4 · p. 9

3.5 LMBUILD IDENTIFIES FAILURE PATTERNS OF EVALUATED LLMS

  • Closed-source models succeed on 95.7% of part-creation attempts, versus 51.7% for open-source models, with Qwen3.5-27B the close exception at 89.4%.Recurring failures include violating CAD-kernel conventions and producing invalid program structures.Source: §3.5 · p. 9
  • Open-source models lose track of scene state more often, referring to unplaced parts at 7.2 errors per 100 calls versus 0.3 for closed-source models.They also redundantly add existing entities at 3.2 errors per 100 calls, versus 0.5 for closed-source models.Source: §3.5 · p. 9

4 RELATED WORK

Prior 3D-generation research connects language models to procedural or parametric modeling, part-aware generation, and physical assembly. LMBuild complements these efforts by evaluating assembled structures across geometry, function, and physical realization.

3D Generation with Language Models

  • Language-model 3D-generation methods produce scripts, scene blueprints, or CAD sequences, but generally target editable or visually correct geometry rather than physical assembly and function.The cited work includes Hu et al. (2024), Sun et al. (2025), and Tang et al. (2025b) for scripts or blueprints, and Khan et al. (2024) and Li et al. (2025) for CAD sequences.Source: §4 · p. 10

Part-Aware Generation and Physical Assembly

  • Related work represents objects as parts, predicts joints and articulation, and studies stable construction, assembly programs, placement, or robotic execution.These approaches motivate evaluating part structure, physical behavior, and whether completed objects realize intended functions.Source: §4 · p. 10

Evaluation of 3D Generation

  • Prior 3D evaluations emphasize visual quality, text alignment, and geometric consistency, while LMBuild jointly assesses soundness, affordance, design, and physical realization.LMBuild uses geometric checks, grounded component assessments, simulation-based motion tests, and VLM judgments.Source: §4 · p. 10

5 CONCLUSION

LMBuild provides a benchmark for evaluating structured 3D assemblies across soundness, functional affordance, design quality, and physical realization. Results identify continuing functional and operability challenges, while real-world readiness requires further validation.

5 CONCLUSION

  • LMBuild evaluates structured 3D assemblies across structural soundness, functional affordance, design quality, and physical realization.The authors present it as a reproducible foundation for studying these capabilities.Source: §5 · p. 10
  • Systems perform better on structural and visual criteria than on functional-part completeness, kinematic correctness, and simulation-based operability.The results highlight challenges in translating object specifications into components, joints, assembly plans, and task-specific parts.Source: §5 · p. 10
  • LMBuild performance alone is insufficient evidence of readiness for safety-critical engineering or other physical-world applications without domain-specific validation and human oversight.The authors note that benchmarks capture only a subset of deployment requirements.Source: ETHICS STATEMENT · p. 11

A.1 AGENT HARNESS FOR LMBUILD

LMBuild uses a common interactive assembly environment for general-purpose agents, adapting how observations and actions move across model interfaces. Its environment tiers vary part access, and revision rounds preserve the evolving assembly state.

Overview

  • A model-specific harness gives general-purpose agents common task information and environment semantics while accommodating different APIs and action formats.Closed-source agents use shell commands, while locally served open-source models use an OpenAI-compatible chat interface with text-serialized tool calls.Source: §A.1 · p. 17

Environment tiers

  • Tier A allows catalog retrieval and placement, Tier B adds part creation, and Tier C requires creating every component without a catalog.Episodes allow up to 160 tool calls, with separate limits for checking, rendering, placed parts, and part creation.Source: §A.1 · p. 17

Multi-round interaction

  • When another revision round remains, the environment returns diagnostics and multiple views, and revisions modify the persistent assembly rather than restarting it.A final submit ends the episode.Source: §A.1 · p. 17

A.1.1 SHARED LMBUILD AGENT PROMPT

The shared LMBuild prompt provides a common system prompt for general-purpose LLM/VLM agents, with tier-specific instructions and tool documentation appended.

A.1.1 SHARED LMBUILD AGENT PROMPT

  • A shared system prompt is used for all general-purpose LLM/VLM agents.Source: §A.1.1 · p. 17
  • The {PARTS SOURCE} placeholder is instantiated according to the environment tier.Source: §A.1.1 · p. 17
  • LMBuild appends tier-specific part-creation instructions and generated tool documentation to the prompt.Source: §A.1.1 · p. 17

Shared environment system prompt

The system prompt asks agents to construct a real-scale object from parts and declare how it is assembled and functions, then review and revise within the available rounds.

Shared environment system prompt

  • The target is a real, buildable, working product that holds together, stands, matches the request, moves as expected, uses sensible materials, and can be built in the stated order.Source: Shared environment system prompt · p. 18
  • Parts are positioned by their bounding-box centers; attach can align face centers, and an unrotated part rests on the ground when its center is at half its height.Source: Shared environment system prompt · p. 18
  • Parts must connect through touching surfaces or joints near both surfaces; extensive interpenetration causes collisions, and unconnected parts float.Source: Shared environment system prompt · p. 18
  • Each part needs a plain-language role and a listed material, while joints declare their type, parent, child, world-coordinate axis and origin, and applicable motion limits.Source: Shared environment system prompt · p. 18
  • The assembly sequence must include every part once in the order a person would build the object, with optional insertion directions.Source: Shared environment system prompt · p. 18
  • After submission, agents may inspect checks and renders and revise during remaining review rounds; checks and renders consume budget and are intended for verification.Source: Shared environment system prompt · p. 18
  • The task prompt supplies a target category and may include an image, attributes, functional constraints, and the current environment tier; objects should use real-world size and the stated coordinate frame.Source: Shared environment system prompt · p. 18

Example user prompt: Tier B with reference image

The example Tier B prompt asks an agent to build an office chair from a reference image, using catalog parts and creating components when needed.

Example user prompt: Tier B with reference image

A.1.2 CLOSED-SOURCE AGENT HARNESS

The closed-source harness evaluates Claude models in isolated episode directories and routes their environment-tool use through shell commands.

A.1.2 CLOSED-SOURCE AGENT HARNESS

  • Claude Haiku 4.5, Sonnet 5, Opus 5, and Fable 5.1 are evaluated through the Claude command-line interface.Source: §A.1.2 · p. 19
  • Each episode runs in an isolated working directory, and the model receives the session interface and images returned by that session.Source: §A.1.2 · p. 19
  • The agent requests the LMBuild prompt, invokes tools through shell commands, and receives JSON results; image-producing tools return PNG paths for inspection.Source: §A.1.2 · p. 19

Claude shell wrapper

Claude is instructed to build the object through the session command and its tools, then continue submitting rounds until the episode is complete. The final response reports the part and joint counts.

Claude shell wrapper

  • Use the session command to read the task and tool instructions, then build the object through tool calls.The prompt provides the session state file and specifies the command for reading the task output.Source: Claude shell wrapper · p. 19
  • Continue working after each submit until a result says the episode is done.Each submit ends a round and returns a review; errors should be read and corrected.Source: Claude shell wrapper · p. 19
  • Report the number of parts and joints in the final design.Source: Claude shell wrapper · p. 19

GPT shell wrapper

The GPT shell wrapper directs agents to use the session tools, inspect permitted images, and revise the same design through successive submissions. It limits direct file access and requires final part and joint counts.

GPT shell wrapper

  • Do not edit files or inspect reference assemblies, evaluator data, other results, or repository code.Only the condition image, catalog sheets, and session-named renders may be read.Source: GPT shell wrapper · p. 19
  • Use the session command and tool calls to build the object, opening the image files named by the prompt.The prompt directs the agent to read the full task output before beginning.Source: GPT shell wrapper · p. 19
  • Revise the same design after each successful submit until episode_done is true, then report final part and joint counts.If the saved session is partly complete, get_scene and continue from that state rather than starting over.Source: GPT shell wrapper · p. 19
  • Claude and GPT receive the same LMBuild environment prompt and tool interface, with provider-specific execution and isolation instructions differing.GPT episodes permit up to three execution attempts.Source: GPT shell wrapper · p. 19

A.1.3 OPEN-SOURCE LLM/VLM AGENT HARNESS

The harness serves open-source models through vLLM and an OpenAI-compatible endpoint, using an in-process agent loop. It assembles prompts from tier-specific instructions and supplies task inputs according to model capability and tier.

A.1.3 OPEN-SOURCE LLM/VLM AGENT HARNESS

  • Open-source LLMs and VLMs use vLLM through an OpenAI-compatible chat endpoint and interact through an in-process agent loop.Source: §A.1.3 · pp. 19–20
  • The system message combines the tier-specific system prompt, tool documentation, and tool-call protocol.Source: §A.1.3 · p. 20
  • The user message includes task text, a condition image when vision is supported, and a catalog sheet for Tier A/B.Condition images are resized to a maximum side of 1024 pixels and catalog sheets to 1600 pixels.Source: §A.1.3 · p. 20

Open-source tool-call protocol

Locally served agents use a text-based tool-call format with limited fallback parsing and repair. Parsing errors and missing actions prompt corrective responses, while repeated non-action responses end an episode.

Open-source tool-call protocol

  • Tool calls are written in standalone <tool_call> blocks with strict JSON arguments, and multiple blocks can run in sequence.The example invokes place_part with a part ID, instance ID, and position.Source: Open-source tool-call protocol · p. 20
  • The parser supports explicit tool-call blocks and selected fallback formats, but repairs only simple numeric arithmetic and truncated JSON.On parsing failure, the model receives the concrete error and is instructed to resend a strict-JSON call.Source: Open-source tool-call protocol · p. 20
  • After three consecutive responses without an executable tool call, the episode terminates; locally served models use temperature 0.6 and seed 0.Models producing no action are reminded to continue through tools or submit once the object is complete.Source: Open-source tool-call protocol · p. 20

A.1.4 DOMAIN-SPECIFIC GENERATION SYSTEMS

Domain-specific generators use their native inference pipelines rather than being forced into LMBuild’s iterative agent interface. Their evaluation is limited to supported benchmark conditions and outputs.

A.1.4 DOMAIN-SPECIFIC GENERATION SYSTEMS

  • Domain-specific systems generate once using their native pipelines, and Tier B matches Tier A when those pipelines lack a create-part action.They receive only the subset of each benchmark condition their original inference pipeline supports.Source: §A.1.4 · p. 20

A.2 ENVIRONMENT DETAILS FOR LMBUILD

LMBuild is a stateful construction environment whose episode state can be serialized, checkpointed, and replayed. It supports component construction and assembly while providing physical feedback, with evaluation interpreted according to each system’s native output capabilities.

A.2 ENVIRONMENT DETAILS FOR LMBUILD

  • Episode state—including poses, created-part programs, joints, assembly sequence, counters, and call trace—is stored in a JSON-serializable dictionary for checkpointing and exact replay.LMBuild is a stateful, pure-Python environment.Source: §A.2 · p. 20
  • Custom parts use declarative CSG programs built as DAGs of up to 96 nodes, with primitives and Boolean operations compiled into watertight solids.Parameters are range-checked, curved primitives are tessellated, and disconnected parts are flagged.Source: §A.2 · pp. 20–21
  • Physical feedback uses a geometry kernel shared with the evaluator, voxelizing each placed part on a 4-mm grid; materials are restricted to 23 submaterials in 9 classes.Renders are produced by a headless Blender worker from six fixed viewpoints; episodes allow up to three cumulative rounds and impose tool, check, and render budgets.Source: §A.2 · p. 21
  • General-purpose agents are evaluated end-to-end, while domain-specific systems may produce only native-output subsets; unsupported metrics are N/A, so cross-family comparisons belong on shared metrics.The full end-to-end LMBuild evaluation applies to LLM agents; domain-specific systems provide context on pipeline portions they support.Source: §A.2 · p. 21, Table 3 · p. 21
  • Agents receive generic construction rules and diagnostics, but not target-specific evaluation sheets or final metric outcomes; functional-prompt trials expose additional task requirements.Those trials measure translating explicit specifications, while the baseline also tests inference from a category label and reference image.Source: §A.2 · p. 22
  • LMBuild tools support retrieving and inspecting parts, creating custom parts, placing them, assigning materials, declaring joints, and specifying assembly sequences.Placement returns feedback on bounding boxes, contacts, collisions, and ground penetration.Source: §A.2 · pp. 22–23

A.3 DATA SELECTION AND DATA EXAMPLES IN LMBUILD

These parts document LMBuild’s benchmark data selection, annotation, examples, and evaluation conventions. They define measures spanning structural soundness, functional parts and motion, design quality, and physical realization and operability.

Candidate Admission and Structural Validity

  • Only assets with at least four identifiable components, renderable geometry, and resolvable joint annotations enter the candidate catalog, leaving approximately 74.7K structurally valid objects.The filters exclude corrupted meshes, monolithic objects, incomplete assemblies, and assets with joint annotations referring to missing or external components.Source: §A.3 · p. 24

Source-aware and Diversity-aware Sampling

  • Source-specific part-count ranges and medoid-seeded farthest-point sampling select structurally diverse instances within each source-category pair.Structural descriptors count semantic part roles and moving-part/joint-type combinations; subsequent selections maximize ℓ1 distance from retained examples. LEGO metadata prioritizes movable connectors, while Fusion 360 categories are assigned from rendered views using a VLM.Source: §A.3 · p. 24

Semantic Concept Grounding and Visual Verification

  • Categories are grounded to Wikidata and WordNet, then VLM-checked against two rendered views to remove semantically mismatched or incomplete instances.A language model selects only among retrieved entity candidates, and categories without a suitable Wikidata entity are removed.Source: §A.3 · p. 24

Kinematic Coverage and Core-set Construction

  • LMBuild-Core covers 50 tasks across six kinematic families, while LMBuild-Full adds 2,349 non-overlapping instances across 152 categories.Core objects range from four to 417 parts; the families span static load-bearing structures, hinged and sliding mechanisms, rolling or rotating systems, serial chains, and many joints attached to a common body.Source: §A.3 · pp. 24–25

Retrieval-based Real-world Knowledge Grounding and Evidence-constrained Claim Extraction and Verification

  • Annotations draw on Wikidata statements, pinned Wikipedia revisions, WordNet meronyms, and ConceptNet only as corroboration, with conservative inheritance from taxonomic ancestors.Qwen3.5-27B proposes claims with source quotations; Gemma-4-31B independently checks support and category typicality, permitting one constrained re-grounding step. Retained claims remain traceable to source statements or revisions and quotations.Source: §A.3 · p. 25

Concrete Data Examples in LMBuild

  • Desk, toilet, fan, and oven examples show specifications spanning component functions, materials, functional subsystems, and kinematic constraints.Examples include drawer slides moving relative to a desk frame, toilet flush components swinging, a fan head oscillating, and oven doors and spits rotating.Source: §A.3 · pp. 27, 28, 30, 32

A.3 DATA SELECTION AND DATA EXAMPLES IN LMBUILD

  • The examples’ factual claims are checked against pinned Wikipedia and Wikidata sources; the verifier dropped 11 claims for the car example and 5 for the other example.Source: §A.3 · pp. 32, 34
  • LMBuild’s concrete examples include car and excavator designs specified through parts, functional systems, attributes, and joint motions.The appendix also visualizes declared joint motion in generated structures using three poses, outlined moving parts, and an indicated joint axis.Source: §A.3 · pp. 33, 35

A.4 DETAILED EVALUATION METRICS FOR LMBUILD

  • The evaluation framework covers structural soundness, functional affordance, design quality, and physical realization, with shared conventions intended to support reproducible comparison across heterogeneous structures.It uses task-specific references and curated factual claims; its fixed heuristic parameters are practical comparative proxies, not theoretically optimal values or definitive models of real-world behavior.Source: §A.4 · pp. 35, 40

Structural Soundness – S.1 Connectivity

  • Connectivity combines largest-component volume, part-level fragmentation, and physically supported connections to assess whether generated parts form an integrated assembly.A connection counts only when supported by physical evidence such as a plausible joint, interlock, or brick–stud connection; a valid single-part structure receives a score of one.Source: Structural Soundness – S.1 Connectivity · p. 41

Structural Soundness – S.2 Collision

  • Collision scoring penalizes only overlaps exceeding both absolute and relative thresholds, allowing small artifacts and plausible joint geometry without treating all contact as invalid.The absolute threshold filters negligible intersections, while a more permissive relative threshold applies to parts with a valid declared joint.Source: Structural Soundness – S.2 Collision · p. 42

Structural Soundness – S.3 Reliability

  • Reliability estimates tipping stability from support margin and center-of-mass height, using critical tilt angle and lateral-force proxies normalized against the reference structure.The two proxies capture angular perturbation and lateral loading; normalization reflects that suitable stability requirements vary by object category.Source: Structural Soundness – S.3 Reliability · pp. 42–43
  • Reliability averages clipped, reference-normalized tipping and lateral-force proxies, while functional component targets come from Wikipedia claims and class-level instance statistics.The stability measure addresses tipping reliability, not sliding, deformation, impact, fracture, joint strength, or fatigue. Components are relevant if present in the reference or at least 40% of the class population; plural counts use a conservative lower-quartile expectation capped at eight.Source: Structural Soundness – S.3 Reliability · p. 43

Functional Affordance – A.1 Geometry

  • Geometry compares normalized generated and reference surfaces, and adds role-aware component correspondence when at least half of parts have informative semantic names.The whole-object comparison samples 20,000 points and retains the best match across four 90-degree yaw orientations; created and retrieved parts are treated identically.Source: Functional Affordance – A.1 Geometry · pp. 43–44

Functional Affordance – A.2 Parts

  • Parts scoring checks canonical-component coverage and multiplicity while allowing alternative decompositions, grouping fragments and giving only partial credit to floating or inferred hidden structure.Semantic roles or substantial spatial correspondence assign components; fixed-jointed or overlapping parts can form one instance. Connected instances score higher than floating ones, and anonymous interior modules can support partial credit for non-visible components.Source: Functional Affordance – A.2 Parts · pp. 44–45

Functional Affordance – A.3 Kinematics

  • Kinematics scores expected articulation by motion type, axis, placement, range, moving-part extent, and collision-free feasibility, aggregating target recall and joint precision.Targets combine reference joints and curated claims, with population-derived properties where evidence permits and deterministic fallbacks otherwise; harmonic aggregation penalizes omitted expected motion and unsupported joints.Source: Functional Affordance – A.3 Kinematics · pp. 45–47
  • System-level correlations with human judgments were 0.77 for decomposition, 0.83 for aesthetics, and 0.94 for alignment.Four annotators rated 90 objects from six systems; metrics were calibrated on eight objects and evaluated on a disjoint held-out set. Item-level correlations were more moderate but positive across all three dimensions.Source: Functional Affordance – A.3 Kinematics · p. 48

Design Quality – D.1 Decomposition

  • Decomposition quality combines calibrated perceptual judgments of boundaries, granularity, and separability with conservative signals for granularity mismatch, fragmentation, and unsupported joint placement.It does not treat a single reference segmentation as uniquely correct; fragmentation is flagged only when interfacing parts share a semantic role and material without a declared joint.Source: Design Quality – D.1 Decomposition · pp. 48–50

Design Quality – D.2 Aesthetics

  • Aesthetics scores reference-free proportion, coherence, and refinement from neutral-gray clay renders, without reference images or appearance cues.It uses perceptual judgment alone, calibrated to human ratings; color, texture, semantic labels, and material assignments are excluded.Source: Design Quality – D.2 Aesthetics · p. 50

Design Quality – D.3 Structural Alignment

  • Structural Alignment measures target-specific component inventory, placement, scale, and detail, blending perceptual assessment with complementary geometric correspondence signals.Its computed signals include volume-weighted part IoU and, when available, agreement with the reference’s symmetry; perceptual and computed scores receive 3:1 weighting.Source: Design Quality – D.3 Structural… · pp. 51–52

Physical Realization – R.1 Assembly Sequence

  • Assembly Sequence evaluates whether declared placements admit collision-aware insertion and maintain grounded, stable intermediate subassemblies throughout construction.The score penalizes omitted parts by normalizing against all generated parts, and combines the executable prefix with intermediate-state supportability.Source: Physical Realization – R.1 Assembly… · pp. 53–54

Physical Realization – R.2 Material

  • Material scores compatibility with acceptable classes supported by text, population statistics, or reference overlap, using functional plausibility only when those sources offer none.Acceptable classes are combined by union to accommodate legitimate variation; parts without an evaluation basis are excluded, while undeclared library materials score zero when a basis exists.Source: Physical Realization – R.2 Material · pp. 54–56

Physical Realization – R.3 Operability

  • Operability tests whether task capabilities integrate required components and relationships, articulated behavior, and applicable human-use conditions into a plausibly functional structure.The fixed task-level capability sheet covers functional subsystems, motions, human-use geometry, and additional required parts; motion feasibility is tested in MuJoCo under shared simulation assumptions.Source: Physical Realization – R.3 Operability · pp. 56–57
  • Operability tests settle artifacts for 0.6 s, zeroing motion credit after ≥20° tilt or ≥0.3 object-diagonal translation, then scores each capability by its weakest required link.This compositional measure checks whether required components, relationships, mechanisms, and human-use geometry work together under fixed comparative simulation criteria.Source: Physical Realization – R.3 Operability · pp. 57–58

A.5 INTERPRETATION OF EVALUATION METRICS AND RESULTS

  • GPT-6 Astra scores 88.5/50.0/74.0/53.5 on Soundness/Affordance/Design/Realization; Qwen3.5-27B scores 82.9/28.3/52.7/57.2, while closed-source APIs lead overall.The strongest open model is competitive on Soundness and Realization but substantially lower on Affordance and Design.Source: §A.5 · pp. 59–60
  • Metric-specific advantages can reflect enforced output representations, simpler assemblies, or visual optimization, so local Soundness, Realization, and aesthetic scores do not establish complete functional design capability.Brick and mesh-generation procedures can enforce connectivity or geometric compatibility, while smaller assemblies ease local criteria but can omit required components and mechanisms.Source: §A.5 · pp. 60–62
  • Cube3D+CubePart benefits from an input part schema, Cube3D excels at aesthetic quality, PartPacker at visual fidelity, and PhysX-Anything at material plausibility, without these strengths ensuring functionality.Cube3D+CubePart’s supplied schema corresponds to task-required components, yet its parts rarely produce meaningful motion; PhysX-Anything’s generated objects likewise show almost no successful simulated operation.Source: §A.5 · pp. 61–62
  • 37.9 is the mean A.3 score and 18.3 the mean R.3 score across six closed-source APIs; 57% of failed motion targets lack the moving component and 17% lack a moving joint.All systems submit a design for every task, and tool-call failures are uncommon; functional decomposition is the more consequential weakness.Source: §A.5 · p. 62
  • 54.5%–68.5% of designs have at least half of required components missing or misplaced, while 48%–68% of assembly sequences are not fully executable.Product-level CAD and Fusion 360 objects have more missing or misplaced components than Artiverse furniture and appliances; frontier models often omit internal parts or mechanisms despite recognizable exterior geometry.Source: §A.5 · pp. 62–63

A.6 LIMITATIONS AND FUTURE WORK

  • LMBuild’s fixed geometric, semantic, kinematic, and simulation tests measure comparative plausibility, not fabrication readiness or safety, and omit several real-world physical factors.The evaluation uses simplified physical assumptions, a coverage-oriented 50-object Core split, limited human calibration, and one final generation trajectory per model-task pair; future work includes broader evaluations, repeated runs, physical validation, and reusable physical reasoning skills.Source: §A.6 · pp. 63–64
Loading 2610.04292v1…