PaperlayerView exact revision on arXiv

Source-linked AI summary

AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents

Jiahao Zhang, Yeying Fan, Moitreya Chatterjee, Suhas Lohit, Bernhard Egger, Tim K. Marks, Anoop Cherian, Stephen Gould

arXiv:2609.40353v1cs.CVcs.RO

TL;DR

3D assembly asks whether pretrained general-purpose agents can translate visual and spatial understanding into precise arrangements of parts. AssemblyWorld and its cross-domain benchmark evaluate agents through rendered observations and pose-editing tools. Results show strong assembly capability but a gap between part accuracy and complete reconstruction.

  • Problem

    3D assembly requires translating understanding of parts and their relationships into precise spatial arrangements, raising whether pretrained general-purpose agents can do so through visual interaction without assembly-specific fine-tuning.

    Source: §1 · p. 2

  • Method

    AssemblyWorld lets agents inspect rendered views and manipulate supplied rigid parts, while AssemblyWorldBench evaluates them across 100 tasks and 80 objects in three assembly domains.

    Source: §1 · p. 2

  • Results

    The strongest system achieves 80.9% part accuracy but 59.4% complete-assembly success, while evaluated open-source systems lag stronger closed-source peers in execution reliability and assembly accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across the evaluated domains, the same pretrained agent can assemble furniture, industrial objects, and fractured objects, but broad capability does not ensure geometric precision.

    Source: §5 · p. 9

    Evaluation covers free-space geometry and does not establish collision-free or physically stable assembly; sampled tasks and limited robustness and refinement studies constrain the findings.

    Source: §5 · p. 9

Abstract

from arXiv · show

The task of 3D assembly requires translating an understanding of parts and their relationships into precise spatial arrangements. Can pretrained general-purpose agents assemble objects through visual interaction without additional assembly-specific fine-tuning? To investigate this question, we introduce AssemblyWorld, an interactive 3D environment in which agents inspect rendered views and manipulate supplied rigid parts, guided by images or assembly manuals when available. Agents perceive part geometry through 2D views rather than direct access to mesh vertices or faces, while their resulting assemblies are evaluated geometrically. Building on this environment, we construct AssemblyWorldBench, comprising 100 assembly tasks across 80 objects spanning furniture, industrial assembly, and fracture reassembly. Evaluating eight agent systems reveals substantial differences in their capabilities. The strongest system achieves 80.9% part accuracy but 59.4% complete-assembly success. The evaluated open-source systems lag substantially behind their stronger closed-source peers in both execution reliability and assembly accuracy. Analyses of visual references, interaction trajectories, and failures show how agents revise assemblies while leaving residual positioning errors. AssemblyWorld provides a common setting for both assessing the capabilities of interactive assembly agents and characterizing the gap between approximate structure recovery and precise reconstruction.

1 INTRODUCTION

3D assembly tests whether an agent can turn visual and spatial reasoning into precise part arrangements, across domains with different reference information. AssemblyWorld and AssemblyWorldBench provide an interactive setting and cross-domain benchmark for assessing general-purpose agents, whose results reveal a gap between partial accuracy and complete reconstruction.

1 INTRODUCTION

  • Correct assembly requires aligned connections, correct part orientation, and precise positioning; having all parts in roughly the right place may still leave the object incomplete.The available clues range from illustrated manuals to part geometry alone.Source: §1 · p. 2
  • AssemblyWorld lets agents inspect rendered 2D views, choose viewpoints, and translate or rotate supplied rigid parts, with images or manuals guiding assembly when available.Geometric evaluation measures the resulting configurations, and recorded states support analysis of how an assembly develops.Source: §1 · p. 2
  • AssemblyWorldBench contains 100 tasks across 80 objects, enabling comparisons across furniture, industrial assembly, and fracture reassembly.The benchmark combines standardized initial configurations and a common execution budget across eight agent systems.Source: §1 · p. 2
  • The strongest system achieves 80.9% part accuracy but only 59.4% complete-assembly success, while evaluated open-source systems lag stronger closed-source peers in execution reliability and assembly accuracy.Visual references improve performance for stronger systems, and interaction trajectories show revisions that can still leave positional errors unresolved.Source: §1 · p. 2

2 RELATED WORK

Prior work studies learned part poses, instruction-guided assembly, fracture correspondence, and agents that act in executable environments. This paper examines general-purpose agents assembling fixed parts through rendered observations and pose-editing tools.

Learning-Based and Instruction-Guided Assembly

  • Learning-based assembly methods use geometric and structural cues, generative pose modeling, or visual and instructional guidance; fracture reassembly relies more on geometric correspondence.The paper contrasts these approaches with general-purpose agents assembling through rendered observations and pose-editing tools without assembly-specific fine-tuning.Source: §2 · p. 3

3D Assembly Datasets and Benchmarks

  • Existing datasets cover complementary domains: annotated furniture parts, furniture manuals, industrial instructions, and simulated or scanned fractures.AssemblyWorldBench unifies subsets in a common interactive environment with standardized initialization and geometric evaluation.Source: §2 · p. 3

Interactive Agents in 3D Environments

  • Tool-using agents and 3D content-creation systems connect reasoning to executable actions or rendered feedback, while AssemblyWorld focuses on interactive assembly of fixed parts.Its evaluation follows the execution-based paradigm of OSWorld (Xie et al., 2024).Source: §2 · p. 3

3 ASSEMBLYWORLD

AssemblyWorld frames assembly as arranging fixed rigid parts into a coherent object, using optional reference information where available. Agents inspect rendered views and edit poses through tools, while AssemblyWorldBench standardizes tasks and reference conditions across three assembly domains.

3.1 TASK FORMULATION

  • The task supplies rigid-part meshes in a separated initial configuration and optional references, and requires the agent to determine final poses while keeping geometry and relative scale fixed.References may include a completed-object image or an assembly manual.Source: §3.1 · p. 3

3.2 INTERACTIVE 3D ENVIRONMENT

  • Agents inspect rendered 2D views, adjust camera viewpoints, query part identifiers and pose records, and use translation, rotation, or group operations to edit parts.Mesh vertices and faces are hidden, so agents infer part geometry from rendered views; recorded operations and states preserve the interaction sequence and final configuration.Source: §3.2 · pp. 3–4

3.3 BENCHMARK CONSTRUCTION

  • AssemblyWorldBench combines furniture, industrial assemblies, and fractured objects, retaining dataset-specific reference conditions while evaluating final geometry rather than illustrated operation sequences.PartNet objects are tested with and without an assembled-object image; Fantastic Breaks has no reference, and the benchmark totals 100 tasks across 80 objects, including two-part fracture objects.Source: §3.3 · p. 4

4 EXPERIMENTS

Experiments evaluate general-purpose agents on a shared cross-domain benchmark and compare Astra with specialized assembly methods. Results show strong but uneven performance, with interactive revisions and residual errors shaping outcomes.

4.1 EXPERIMENTAL SETUP

  • Eight agents were evaluated on 100 tasks with shared tools, initial configurations, instructions, and a 60-minute limit; geometric scoring reports SCD, PA, and all-parts SR after alignment and matching.Astra was also evaluated on larger dataset subsets for comparison with specialized methods.Source: §4.1 · pp. 4–5

4.2 RESULTS ON ASSEMBLYWORLDBENCH

  • 59.4% overall SR made Astra the top system, ahead of Fable at 50.0% and Opus at 44.4%, despite domain-level variation.Astra averaged 5.8 minutes and 94 calls per evaluation, versus Sol’s 8.7 minutes and 208 calls.Source: §4.2 · p. 5, Table 1 · p. 5

4.3 COMPARISON WITH SPECIALIZED ASSEMBLY METHODS

  • With image guidance, Astra exceeded compared methods in PA and SR across all three PartNet categories, reaching 85.35% PA and 63.60% SR on tables.Removing the reference image reduced PA in every category; image-conditioned Storage reached 63.87% PA but 20.27% SR.Source: §4.3 · p. 5, Table 2 · p. 6
  • On IKEA-Manual, Astra achieved category-averaged PA/SR of 93.27%/75.44% on Chair/Table and full-dataset SCD of 0.9 with 100% SRa.Astra used original IKEA pages without assembly-specific fine-tuning or a dedicated pose predictor.Source: §4.3 · p. 6, Table 3 · p. 6
  • 78.32% PA and 55.91% SR across 279 AssemblyBench objects contrasted with 91.67% PA on 150 Fantastic Breaks objects, where specialized methods reported stronger fragment placement.Astra used stepwise rendered instructions for industrial assembly and no target-object visual reference for fracture reassembly.Source: §4.3 · p. 7, Table 4 · p. 7, Table 5 · p. 7

4.4 ANALYSIS OF INTERACTIVE ASSEMBLY

  • Inspection concentrated early and checking late as edits shrank; reference images raised SR by 25, 20, and 15 points for Astra, Fable, and Opus on 20 PartNet objects.Across larger PartNet evaluations, Astra’s SR rose by 14.6 points on tables and 7.4 points on storage.Source: Figure 2 · p. 6, §4.4 · p. 7, Figure 4 · p. 8
  • Failures include execution errors, detached components, and residual seating errors; small-sample, single-run task variations show aggregate SR stability can mask object-level switches.Translation-dominated errors account for 67.4% of Astra’s and 75.1% of Fable’s incorrect parts.Source: §4.4 · pp. 8–9, Figure 5 · p. 9
  • GARF refinement raised PA from 83.67% to 87.00%, below agent-only 91.67%; Astra’s visual correction of GARF reached 93.00% PA and reduced CD from 8.82 × 10−3 to 4.35 × 10−3.GARF was not trained to refine agent outputs; its refinement reduced rotation error but increased translation error.Source: §4.4 · pp. 8–9, Table 6 · p. 9

5 CONCLUSION

AssemblyWorld and AssemblyWorldBench evaluate general-purpose agents assembling objects across domains, with Astra achieving the highest overall success among eight systems. The results show that broad assembly capability does not guarantee precise, complete reconstruction.

5 CONCLUSION

  • Across eight systems, Astra achieves the highest overall success, while stronger closed-source systems substantially outperform evaluated open-source systems.The conclusion also identifies a gap between part accuracy and complete-assembly success.Source: §5 · p. 9
  • The same pretrained agent can assemble furniture, industrial objects, and fractured objects without additional assembly-specific fine-tuning.AssemblyWorldBench uses a common interface and geometric criteria across domains.Source: §5 · p. 9
  • The findings motivate combining general-purpose cross-domain reasoning with specialized geometric pose estimation and refinement, while jointly designing their interaction because refinement results depend on composition order.The evaluation does not establish collision-free or physically stable assembly; sampled tasks, limited robustness and refinement studies, and unknown pretraining exposure further constrain scope.Source: §5 · p. 9

A.1 DATASETS AND TASK CONSTRUCTION

The benchmark samples objects and reference conditions from several source datasets, then evaluates agents under shared initial configurations and interaction instructions. Its task construction and execution protocols distinguish the common benchmark from larger source-level evaluations.

A.1 DATASETS AND TASK CONSTRUCTION

  • The common benchmark covers 100 tasks across 80 objects, including paired image and no-reference conditions for 20 PartNet objects and 20 objects per source.These counts describe the evaluated subsets, not the full datasets; the larger source-level evaluations are separate.Source: Table A-1 · p. 13, §A.1 · p. 13
  • Objects are sampled with part-count stratification and category coverage, except Fantastic Breaks, which contains two-part objects and is sampled by category; these subsets support system comparisons, not full-dataset performance estimates.The larger AssemblyBench evaluation retains 279 of 280 objects after screening, with reported means conditional on screening and one successful retry.Source: §A.1 · p. 13
  • Each task begins with rigid parts in a separated configuration; initialization uses vertex-PCA frames, shared scaling, independent yaw, and randomized XY placement, with seed zero.Initial placements are baked into mesh vertices, so body transforms start as identity; body origins need not coincide with visual part centers.Source: §A.1 · p. 13
  • Target poses and evaluator data are unavailable to agents, while references vary by condition: none, a final manual page, all IKEA-Manual pages, or step diagrams.Part identifiers are anonymized, and reference images are supplied separately from the parts.Source: §A.1 · p. 13

A.2 AGENTS, TOOLS, AND INSTRUCTIONS

  • All systems use the same environment tools and task text, but deployed model–harness combinations differ in model, CLI, effort setting, and tool and image handling.Pose-edit operations return execution status, so agents must make a separate observation to inspect the effect.Source: §A.2 · p. 14
  • The common benchmark allows 60 minutes and scores final states after normal termination, errors, or timeouts; source-level runs do not all share that limit.Resource means use incomplete records for some systems and may reflect outcome-dependent coverage; elapsed time includes intervals between calls.Source: §A.2 · pp. 14–15

A.3 GEOMETRIC EVALUATION

The evaluation measures whole-assembly shape distance and matched-part correctness after global alignment, then aggregates source means for the benchmark. Paired resampling and corrected tests quantify uncertainty, with intervals describing object sampling rather than repeated-run reliability.

A.3 GEOMETRIC EVALUATION

  • SCD is squared bidirectional Chamfer distance between assembled point sets after one rigid registration, selected from 24 initializations and refined with symmetric ICP.Evaluation samples 1,000 points per part and normalizes the largest part’s PCA bounding-box diagonal to one.Source: §A.3 · p. 15
  • PA is the fraction of matched parts with squared bidirectional Chamfer distance at most 0.01, while SR requires every part of an object to meet that threshold.Geometrically equivalent parts are grouped using pairwise shape matches and assigned with Hungarian matching; no part is independently repositioned to improve its scored pose.Source: §A.3 · p. 15
  • Benchmark quality uses equal source weights, with PartNet’s image and no-reference means each receiving twice the weight of the other source means.All benchmark tasks have scored final states; a missing outcome would receive zero PA and SR, while SCD requires finite geometry.Source: §A.3 · p. 15
  • Published source-level comparisons retain the original methods’ evaluation implementations, so their anchor alignment and sampled inputs are not matched reproductions; the four-way refinement experiment uses matched inputs.The alternative SRa metric measures whole-shape proximity rather than requiring every part to be correct.Source: §A.3 · p. 15

B.1 UNCERTAINTY AND REFERENCE EFFECTS

  • Bootstrap intervals resample objects within each source while preserving paired PartNet conditions and benchmark weights, and therefore describe object-sampling variation within recorded runs, not repeated-run reliability.The benchmark quality table includes all 100 tasks per system, including partial assemblies after errors or timeouts.Source: §B.1 · p. 16, Table A-3 · p. 16
  • System comparisons use paired randomization tests with Holm correction across 56 tests; Astra’s descriptive advantage over Fable is not significant under that correction for either metric.Intervals remain pointwise even though the p-value correction covers the family of tests.Source: §B.1 · p. 16, Table A-4 · p. 16

B.2 GEOMETRY, COMPLEXITY, AND METRIC SENSITIVITY

The appendix examines part-size and part-count patterns, SCD concentration, and sensitivity to correctness thresholds. These comparisons are descriptive where geometry, category, and complexity vary together.

B.2 GEOMETRY, COMPLEXITY, AND METRIC SENSITIVITY

  • AssemblyBench has a higher median largest-to-smallest part-size ratio than the furniture sets, but PartNet Storage’s 90th percentile of 17.68 exceeds AssemblyBench’s 15.73.The ratio compares PCA bounding-box diagonals within each object.Source: §B.2 · p. 17
  • For PartNet, Table A-7 reports PA and SR by 2–5, 6–10, and 11–20 part bands; table-category results decline across bands, but Chair/no-reference is not monotonic.Storage’s lowest band contains one object, and the comparisons cannot attribute patterns to part count alone because geometry, category, and complexity vary together.Source: §B.2 · p. 17
  • For Storage/image, eight objects contribute 99.32% of summed SCD, with a median of 0.53 versus a mean of 184.80; six retain every initial part pose.Separated parts can remain far apart after whole-object registration, producing large squared distances; normal execution termination does not rule out geometric failure.Source: §B.2 · p. 17, Table A-8 · p. 18
  • Figure A-1 varies the correctness threshold while keeping registration, correspondence, and input-shape equivalence threshold fixed, isolating cutoff sensitivity from alignment or grouping changes.Table A-9 supplies separate IKEA Chair and Table results underlying the equal-category summary.Source: §B.2 · p. 17, Figure A-1 · p. 18, Table A-9 · p. 18

C.1 INTERACTION DYNAMICS

The section defines how interaction activity and manipulation magnitude are summarized, then examines quality over time, geometric changes, and completion reports.

C.1 INTERACTION DYNAMICS

  • Activity shares are averaged across active evaluation bins with source weights, while grouping and episode-control calls remain in the denominator.Ten equal-width bins span each evaluation; bins without calls are omitted and weights renormalized.Source: §C.1 · p. 19, Table A-10 · p. 19
  • Manipulation magnitude uses per-evaluation medians followed by weighted medians across evaluations, with translation normalized by the largest-part diagonal and rotation measured in degrees.Smaller late-stage motions occur in both successful and failed runs, so they do not alone establish successful refinement.Source: §C.1 · p. 19, Table A-10 · p. 19
  • Retrospective time-budget scores use the last completed operation, and final states persist after termination rather than being interpolated between checkpoints.Five episodes had edits after 60 minutes of end-to-end time, but PA and SR agree at 60 minutes and termination for all systems.Source: §C.1 · p. 19, Figure A-2 · p. 19
  • Among 239 sampled-checkpoint PA decreases, 96 changed an assignment, 140 persisted with alignment fixed, and 137 persisted with alignment and matching fixed.Intermediate accuracy changes are interpreted alongside the actual manipulation.Source: §C.1 · p. 20
  • Astra succeeds in 43 of 74 declared completions, compared with Fable in 39 of 88; unsuccessful claims have median incorrect-part fractions of 33.3% and 42.9%, respectively.Missing reports are excluded, and declarations are compared with geometric success rather than probabilistic confidence calibration.Source: §C.1 · p. 20

C.2 EXECUTION AND GEOMETRIC FAILURES

The section distinguishes execution failures from geometric errors, illustrates trajectory and visual comparisons, and tests how agent poses interact with geometric refinement.

C.2 EXECUTION AND GEOMETRIC FAILURES

  • Astra and Fable terminate normally in all 100 tasks, while Qwen has 65 timeouts and 14 execution failures and DeepSeek has 17 and 34, respectively.All 49 normally terminated DeepSeek runs fail geometrically; these are failures of evaluated model–harness combinations, not necessarily model reasoning alone.Source: §C.2 · p. 20
  • The failure cases distinguish logical grouping, numerical pose agreement, visible alignment, and residual positioning error in otherwise assembled objects.The classifier applies ordered categories after global alignment and equivalent-part matching, using centering to separate translation from orientation error.Source: §C.2 · p. 20
  • Translation-dominated errors account for 67.4% of Astra and 75.1% of Fable under default thresholds, but vary substantially when cutoffs change.Across tested cutoffs, the fractions range from 48.3–78.0% for Astra and 50.5–86.1% for Fable; the majority classification is not threshold-invariant.Source: §C.2 · p. 20

C.3 QUALITATIVE COMPARISONS AND MANUAL USE

  • Figures A-3 and A-4 show eight systems on four common objects with shared view directions and part colors, but each rendering is fitted to its own geometry.Figure A-5 compares shape CD across the same 150 objects; points below the diagonal improve under the corresponding composition order.Source: §C.3 · p. 20, Figure A-3 · p. 21, Figure A-4 · p. 21
  • In the IKEA bench example, the agent adds the stretcher after the seat and both end frames, unlike the manual’s order, while industrial placements are revised around blade inspection.The examples show instructional relationship correspondence without reproducing every illustrated step; placement order alone does not establish failure to use a manual.Source: §C.3 · p. 22

D COMBINING AGENTS WITH GEOMETRIC MODELS

  • Agent-initialized GARF replaces random initialization with agent poses but retains 20 refinement steps, aligning the agent anchor while supplying no non-anchor target poses.GARF-to-Agent instead initializes the original meshes from predicted poses without preceding interaction history; all 150 runs terminate normally.Source: D COMBINING AGENTS WITH GEOMETRIC MODELS · p. 22
  • Agent initialization improves PA over standard GARF on 10 objects and leaves it unchanged on 140, while GARF-to-Agent improves PA on 33 and degrades it on five.GARF-to-Agent improves shape CD on 125 objects, worsens it on 23, and leaves it unchanged on two; these single-run two-fragment results do not establish seed-averaged superiority or transfer to multi-part industrial assemblies.Source: D COMBINING AGENTS WITH GEOMETRIC MODELS · p. 22

E ROBUSTNESS EXPERIMENTS

The robustness experiments vary layouts and part inventories on fixed benchmark samples, then inspect agent actions, geometry, and reports under those changes.

E ROBUSTNESS EXPERIMENTS

  • The study runs 95 new conditions once with medium-effort Astra, using alternative layouts, one removed part, or an added part while holding geometry and references fixed.Five fixed samples per benchmark block are selected without performance-based selection, and the prompt remains unchanged while requesting use of all supplied parts.Source: E ROBUSTNESS EXPERIMENTS · pp. 22–23
  • The missing-part measure scores retained parts against original-scale targets, so retained-part completeness is not complete-object success.The task-variation table reports PA/SR except for missing parts, where it reports retained-part PA/completeness; each condition is run once.Source: E ROBUSTNESS EXPERIMENTS · p. 23, Table A-12 · p. 23
  • Reports and trajectories distinguish recognizing an absent or extra part from attempting to use it or leaving it unincorporated.Under distractors, some reports identify uninstalled parts that were never moved; Fantastic Breaks runs include attempts to incorporate or set aside the extra fragment.Source: E ROBUSTNESS EXPERIMENTS · pp. 23–24
  • The unchanged all-parts instruction influences distractor handling, and the small paired sample with one run per condition supports exploratory observations rather than repeated-run reliability.Without a reference, intended inventory may also be underdetermined; the study does not establish robustness to larger numbers of missing or extra parts.Source: E ROBUSTNESS EXPERIMENTS · p. 24
Loading 2609.40353v1…