Towards Physically Grounded Articulated 3D Assets: A Multi-Agent Pipeline and Benchmark

1 INTRODUCTION

• Existing approaches often handle appearance, shape, articulation, and physical properties separately, making it difficult to satisfy them together and limiting asset usability.The cited properties include materials, mass, stability, and collision-free motion.Source: §1 · p. 2
• Single-agent construction and verification can produce self-evaluation bias, while checks without direct simulation may not establish that assets work in physics environments.Procedural systems offer controllable physical properties but limit object variety and are difficult to use for image-based reconstruction.Source: §1 · p. 2
• ARTISAN separates blueprint design, asset coding, and review across three agents, and adds physical-simulator feedback to validate collisions and stability.The agents exchange the blueprint, 3D asset, and feedback; their independence is intended to reduce self-evaluation bias.Source: §1 · p. 2
• The benchmark jointly assesses spatial fidelity, physical attributes, downstream task performance, and overall quality using multiple metrics for each aspect.Physical attributes include mass, material, stability under gravity, and collision during motion.Source: §1 · p. 2
• ARTISAN is reported to be faithful to input references, physically valid, and useful in downstream tasks, outperforming recent strong models on most metrics and in human evaluation.The paper highlights improvements in collision and stability, and reports support for simulated robotic manipulation.Source: §1 · p. 2

Creating Functional 3D Assets

• Earlier per-instance optimization was limited to specific instances; later methods generate articulated objects from noise, text, images, video, or 3D models.Recent work also predicts physical properties alongside joint motion or uses simulation feedback to refine support, clearances, and mobility.Source: §2 · p. 3

Procedural and Agentic 3D Asset Creation

• Procedural systems create editable variations within authored rules, while agent-based pipelines extend coverage; ARTISAN adds multiple agents and physical-simulator feedback.The passage contrasts ARTISAN with Articraft’s single-agent construction approach.Source: §2 · p. 3

Datasets and Benchmarks for Physically Grounded 3D Assets

• Prior benchmarks separately assess geometry, semantic fidelity, program quality, materials, or motion; ARTISAN’s benchmark evaluates spatial fidelity, physical plausibility, and downstream task utility.Source: §2 · p. 3

3 METHOD: A MULTI-AGENTIC PIPELINE WITH PHYSICAL VALIDATION

• The system maps a natural-language prompt and optional images to an articulated asset with rigid-part geometry, mass, material, kinematic structure, and joint specifications.The target asset should reproduce its inputs, avoid collisions, remain stable under gravity, and be graspable and manipulable.Source: §3 · p. 3

3.1 DESIGN AGENT

• The Design Agent creates and checks a textual blueprint; its functional descriptions specify coupled motions and interaction tasks that guide verification and downstream task annotations.It also generates a design board with orthographic, exploded, kinematic, and cutaway views.Source: §3.1 · pp. 3–4

3.2 CODING AGENT

• The Coding Agent iteratively edits and executes programs using design inputs, prior assets, physics reports, and review feedback; reusable connectors reduce floating connections and unintended interpenetration.It can annotate tasks with target parts, grasp or press regions, joints to actuate, and required joint travel for downstream use without manual labelling.Source: §3.2 · pp. 4–5

3.3 REVIEW AGENT

• The Review Agent checks repairs against the blueprint and references, flags functional regressions, and returns scored criteria and structured issues to guide coding revisions.Critical structural or functional failures cannot be offset by a high overall score; the Review Agent does not alter the asset program itself.Source: §3.3 · p. 5

3.4 PHYSICAL VALIDATION

• MuJoCo validation records colliding part pairs, penetration depths, and tilt under gravity; collision checks sample joint motion, while stability checks measure tilt during gravity simulation.The report guides coding and review; the described stability test runs for up to 5 seconds and flags tilt beyond a threshold.Source: §3.4 · p. 5

4 BENCHMARK

• The benchmark combines curated data and evaluation protocols to assess physically grounded articulated 3D assets across multiple properties.The stated scope includes spatial fidelity, physical plausibility, and downstream task utility.Source: §4 · p. 6

4.1 EVALUATION PROTOCOL AND METRICS

• Spatial fidelity compares generated and reference assets in rest and articulated states, using aligned geometry and rendered appearance metrics.The protocol samples points, aligns meshes with ICP, matches parts by centers, and tests both state pairings to reduce erroneous penalties.Source: §4.1 · p. 6
• Physical evaluation measures materials, mass, collisions during motion, and stability under gravity, while robotic-task evaluation reports success and asset qualification.Stability uses 10-second simulations of 50 articulated objects in rest and fully articulated states; an asset must remain below the stated tilt, position, and rotation thresholds.Source: §4.1 · pp. 6–7

4.2 DATASETS

• The benchmark includes ArtVIP and LightWheel reference assets, ArtVIP material and collision evaluations, 50 ABO-500 objects for mass, and 50 curated objects for stability.Downstream evaluation covers tasks from RoboTwin 2.0 and RoboCasa365.Source: §4.2 · p. 7

4.3 BASELINES

• Baselines span task-specific learned articulation models, specialized agentic asset-generation methods, and general-purpose coding agents.The comparisons include Instruct-Particulate, PhysX-Anything, URDF-Anything+, Articraft, Articulate-Anything, Codex, Claude, and Antigravity.Source: §4.3 · p. 7

5.1 ANALYSIS OF BENCHMARK RESULTS

• 8 of 11 spatial-fidelity metrics favor ARTISAN; collision occurrences are 1.7% on single-joint and 13.0% on multi-joint evaluations, with stability at 98% in rest and 96% in articulated states.Source: §5.1 · p. 8
• Against Instruct-Particulate, ARTISAN consistently achieves superior results in most metrics and shows better appearance and articulated structure in two reference-guided examples.Source: §5.1 · p. 8
• Without physical validation, appearance, geometry, and kinematics remain comparable, but collision behavior degrades; Figure 5 shows ARTISAN alone avoids intersections and stays stable.The ablated assets cannot execute their intended articulated motion within the simulator.Source: §5.1 · pp. 8–9, Figure 5 · p. 9
• 5.4% in pCD, 4.9% in pIoU, and 11.9% in collision rates are the average improvements of ARTISAN (GPT-5.5 w/o phys.) over Articraft (GPT-5.5).Source: §5.1 · p. 9

5.2 ASSESSING ROBOTIC TASKS AND HUMAN JUDGEMENTS

• π0.5 trained on ARTISAN demonstrations achieves higher success rates than Articraft-trained models on both benchmarks, and adding ARTISAN assets raises success on the same test set with unseen instances.Source: §5.2 · p. 9, Table 3 · p. 9
• ARTISAN assets achieve pass rates 37 percentage points higher on average than Articraft assets across two qualification rounds on two benchmarks.Source: §5.2 · p. 9, Table 3 · p. 9
• 43.3% Top-1 and 85.1% Top-3 votes go to ARTISAN, versus runner-up Codex at 21.1% and 68.7%, respectively.Votes followed object manipulation in the MuJoCo simulation environment.Source: Figure 6 · p. 10, §5.2 · p. 10

• ARTISAN combines multiple agents and physical simulation feedback for physically grounded articulated 3D asset generation.Source: §6 · p. 10
• The benchmark evaluates appearance, geometry, kinematics, materials, collision, mass, stability, and robotic task performance.Source: §6 · p. 10
• The authors identify efficient agent harnesses, richer simulator feedback, and broader physical simulation as directions for future work.Source: §6 · p. 10

• Generative AI tools assisted with manuscript editing.Source: AI USE STATEMENT · p. 11
• The editing assistance covered grammar, sentence structure, clarity, and conciseness.Source: AI USE STATEMENT · p. 11
• The authors reviewed and revised the AI-assisted text and assume responsibility for the final manuscript and its scientific claims and reported results.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

• Human evaluation combines rendered-asset inspection with interaction in a physics simulator.Participants assess visual quality, articulation, and physical behavior through simulated interactions.Source: ETHICS STATEMENT · p. 11
• The supplementary offline demo contains no real participant data.Source: ETHICS STATEMENT · p. 11
• Real-world robotic applications require additional validation appropriate to the deployment setting.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

• Appendices A and B provide implementation details, dataset construction procedures, and evaluation protocols, including the human-study setup.Source: REPRODUCIBILITY STATEMENT · p. 11, APPENDIX CONTENTS · p. 18

APPENDIX CONTENTS

• Appendices C–F document complete results, component ablations, generation budgets and computational costs, robotic experiment settings, and representative agent traces.Source: REPRODUCIBILITY STATEMENT · p. 11, APPENDIX CONTENTS · p. 18

REPRODUCIBILITY STATEMENT

• The supplementary material includes an offline demonstration of the human-evaluation interface for inspecting rendered candidates and interacting with simulated assets.Source: REPRODUCIBILITY STATEMENT · p. 11, APPENDIX CONTENTS · p. 18

A.1 SDK EXTENSIONS

• The SDK exposes materials, part masses, and joint damping and friction, while anchor-based ATTACH, CENTER, and SNAP operations derive poses from geometric relationships.Mass is estimated from blueprint dimensions, materials, and construction rather than density times geometric volume.Source: §A.1 · p. 19

A.2 AGENT TOOLS

• The Coding Agent edits and executes asset programs, while the Review Agent writes tests and inspects structure, geometry, images, and physics evidence.Source: §A.2 · pp. 19–20, Table 4 · p. 20

A.3 PHYSICAL SIMULATION

• MuJoCo checks sampled joint configurations for collisions and tests free-standing assets under gravity at rest and with each joint opened in turn.Persistent single-joint collisions are sent for repair; final tilt above 3° is flagged for stability review.Source: §A.3 · p. 20

B.1 SPATIAL FIDELITY PROTOCOL

• Spatial correspondence uses surface sampling, pose alignment, normalization, and Hungarian matching of part centroids in the closed state, reused in the open state.Source: §B.1 · pp. 20–21

B.2 MATERIAL TRANSFER PROTOCOL

• Material transfer matches parts and components by position, shape, motion, and geometric attributes before rendering transferred materials on reference geometry for PSNR comparison.Numeric parameters are copied directly; texture inputs are reduced to surface-area-weighted means.Source: §B.2 · p. 21

B.3 EVALUATION DATASETS

• LightWheel contributes 50 kitchen-appliance objects, while ArtVIP contributes 50 stratified objects spanning broader categories and varied joint counts.Source: §B.3 · p. 21, Figure 7 · p. 22
• The stability set contains 50 prompts across destabilization mechanisms; the mass set contains 50 articulated ABO objects with ground-truth masses from 0.086–205 kg.Source: §B.3 · pp. 21, 23, Figure 8 · p. 23, Figure 9 · p. 23
• Downstream evaluation covers three RoboTwin 2.0 tasks and eight RoboCasa365 tasks, with generated assets and demonstrations collected under task-specific train-test splits.Source: §B.3 · p. 23

B.4 HUMAN EVALUATION

• The human study asks participants to assess visual quality, articulation, and physical interaction by inspecting renders and manipulating candidates in MuJoCo.Source: §B.4 · pp. 23–24
• Thirty non-expert students evaluated randomized subsets of objects, yielding 275 valid responses; rankings compare ARTISAN with five representative baselines.Source: §B.4 · pp. 23–24

C.1 FULL METRIC TABLES

• ARTISAN leads on 8 of 11 ArtVIP metrics and 4 of 6 LightWheel geometry metrics, while Instruct-Particulate leads on 3 of 5 LightWheel appearance metrics and open-state part IoU.ARTISAN has the lowest object Chamfer distance in both LightWheel states.Source: Table 7 · p. 26, Table 8 · p. 26, §C.1 · p. 26
• ARTISAN (GPT-6-Astra) has lower collision rates than every baseline at all sweep levels on both datasets and stability rates of 98% at rest and 96% open.It also achieves the lowest mass APE, although Agy has lower ADE and higher MnRE.Source: §C.1 · p. 26, Table 9 · p. 27
• Moving from GPT-5.5 to GPT-6-Astra improves every reported spatial metric for Articraft, Codex, and ARTISAN, but the stronger baselines still have substantially higher collision rates than ARTISAN.The authors conclude that stronger models improve reconstruction quality but do not remove the need for explicit physical validation.Source: §C.1 · p. 27

C.2 ABLATION

• Removing physical validation raises single-joint collisions from 1.9% to 35.7% and multi-joint collisions from 16.4% to 73.2%; removing Design or Review also worsens structural or mass measures.Without the Design Agent, part IoU falls from 0.2993 to 0.2788 and mass APE rises from 0.566 to 0.710; without Review, mass APE rises from 0.566 to 0.736.Source: §C.2 · p. 28

C.3 ADDITIONAL QUALITATIVE EXAMPLES

• Qualitative comparisons show closer reference appearance, part decomposition, and open-state motion for ARTISAN, while baselines often lose color, merge or fragment parts, or deviate from the ground truth.Additional visualizations show ARTISAN stable under gravity and collision-free, while baselines often fail stability checks or show interpenetration.Source: §C.3 · p. 28

D API AND TIME COST

• ARTISAN incurs higher API costs but produces assets with fewer interpenetrations and greater stability, making them more usable in physical simulation.Figure 14 separates API-cost shares across the Design, Coding, and Review Agents from generation-time shares that include the simulator.Source: D API AND TIME COST · p. 32, Figure 14 · p. 32, Table 12 · p. 32

E ROBOTIC EXPERIMENTS

• Robotic evaluation checks physical validity and task feasibility, compatibility with demonstration-generation pipelines, and policy generalization, comparing native, generated, and combined training assets.Assets must pass physical-stability and task-feasibility checks, then support demonstration collection; only qualified assets are used for training.Source: E ROBOTIC EXPERIMENTS · pp. 32–33

E.1 ROBOTWIN 2.0

• Adding ARTISAN assets improves average held-out native-object success over native-only training, while incorrect collision meshes exclude 5 of 9 generated laptops and 3 of 6 microwaves from demonstration collection.The excluded assets cause grasp failures despite being loadable in simulation.Source: §E.1 · pp. 33–34

E.2 ROBOCASA365

• Among 80 assets per group, Round 2 retains 61 native, 35 ARTISAN, and 6 Articraft assets after Round 1 counts of 78, 76, and 31, respectively.Thus, passing physical-stability and task-feasibility checks does not guarantee successful demonstration generation.Source: §E.2 · p. 34
• Articraft achieves zero successful demonstrations in six of eight task groups, with reported failures including instability, incorrect collision geometry, and mesh interpenetration.The paper identifies physical and collision failures as barriers to reliable demonstration collection.Source: §E.2 · pp. 34–35
• Physically feasible assets can still fail trajectory transfer when their mechanism differs from the interaction in the reference demonstration, as with a hinged kettle button versus a sliding one.The hinged button is plausible, but MimicGen's transferred manipulation trajectory does not actuate it successfully.Source: §E.2 · p. 35, Figure 15 · p. 36

F.1 STAND MIXER: CONTROL ACCESSIBILITY

• The stand-mixer trace shows that collision and gravity checks can pass while a selector remains inaccessible; review identifies a 4.499 mm clearance and the Coding Agent relocates the control while preserving masses and joint ranges.The revision addresses access without requiring bowl removal.Source: §F.1 · p. 37

F.2 SHOE CABINET: DRAWER CONNECTIONS

• For the shoe cabinet, longer runners and revised front rails restore drawer engagement without shortening travel.The design requires runners to remain engaged at full extension.Source: §F.2 · p. 38, Figure 17 · p. 39