PaperlayerView exact revision on arXiv

Source-linked AI summary

USDCraft: Geometrically Grounded Programmatic Modeling of Articulated 3D Assets for Simulation

Chuanrui Zhang, Zaijia Yang, Duomin Wang, Lu Shi, Daquan Zhou, Ruihua Zhang, Ziwei Wang

arXiv:2610.11322v1cs.RO

TL;DR

Reconstructing specific articulated objects remains difficult when meshes are incomplete or corrupted and existing methods generalize poorly to out-of-distribution objects. USDCraft has a pretrained LLM write and revise simulation-ready asset programs using measured geometry and visual feedback. It leads evaluated baselines on USDCraft-bench, performs competitively on Lightwheel, and is validated for real-to-sim-to-real manipulation.

  • Problem

    Mesh-based articulation methods face deployment challenges with incomplete or corrupted meshes and objects outside their training distribution, despite the need for faithful assets in real-to-sim manipulation.

    Source: §1 · pp. 1–2

  • Method

    USDCraft has a pretrained LLM write, compile, and revise editable articulated-asset programs using metric mesh analysis, iterative geometric rechecking, and visual feedback.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    USDCraft leads evaluated baselines on USDCraft-bench, achieves competitive geometry and strong part and joint recovery relative to published Lightwheel results, and is validated in real-to-sim-to-real manipulation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results show that geometry-grounded program reconstruction can produce assets whose simulation-trained policies transfer to real objects with little loss.

    Source: §5 · p. 8

    Fine lattices remain difficult to reconstruct faithfully, and deformable objects are supported only in annotated states and rest geometry; flexible-body dynamics have not been validated in Isaac Sim.

    Source: §5 · p. 9

Abstract

from arXiv · show

Geometrically faithful and functional articulated 3D assets are essential for real-to-sim robot manipulation, where policies trained in simulation must transfer to physical objects. Recent mesh-based methods learn to infer articulation from annotated 3D assets, but deployment remains challenging when real-world objects fall outside the training distribution or their meshes are incomplete or corrupted. To address these limitations, we formulate articulated asset reconstruction as programmatic modeling grounded in partial geometric evidence and introduce USDCraft, a framework in which a pretrained LLM writes and revises executable programs for simulation-ready articulated assets without task-specific training. We propose source geometry analysis, which converts the source mesh into a metric textual description that distinguishes observed surface from unknown space, and iterative geometric rechecking, which re-encodes each candidate in the same representation so that discrepancies point to program edits while unobserved regions remain open to completion. Visual feedback and physical authoring guidance complete the modeling process, which produces articulated USD assets with explicit physical properties that load into Isaac Sim without manual adjustment. Experiments demonstrate leading articulation recovery on two benchmarks and validate USDCraft's effectiveness for real-to-sim-to-real robot manipulation.

1. Introduction

USDCraft addresses the difficulty of reconstructing specific articulated objects from incomplete or corrupted meshes, where existing approaches struggle with missing geometry and out-of-distribution objects. It grounds editable asset programs in measured geometry and reports strong benchmark and manipulation results.

1. Introduction

  • Mesh-based methods infer articulation by partitioning static meshes and estimating the joint axes and connectivity governing motion.Incomplete surfaces and fused components can undermine part segmentation.Source: §1 · p. 1
  • Existing approaches struggle to recover missing geometry or functional openings and to generalize to objects outside their task-specific training data.Programmatic generation can cover diverse categories without a complete mesh, but without geometric measurements its dimensions and part placement may not match the target.Source: §1 · p. 2
  • A pretrained LLM writes and revises editable asset programs against partial geometric evidence, using source measurements and visual feedback to guide changes.Compiled assets carry physical properties for direct use in Isaac Sim.Source: §1 · p. 2
  • USDCraft leads evaluated baselines on USDCraft-bench, performs competitively on Lightwheel, and is validated in real-to-sim-to-real manipulation.The work also introduces USDCraft-10k, a library of generated articulated assets.Source: §1 · p. 2
  • The paper contributes source geometry analysis and iterative geometric rechecking, alongside empirical reconstruction validation and real-to-sim-to-real manipulation demonstrations.These methods make source geometry readable to LLMs and return candidate discrepancies in the same representation.Source: §1 · p. 2

2. Related Work

Prior work includes mesh-based part and articulation inference, asset generation, and agentic program construction. USDCraft distinguishes its reconstruction approach by combining broad object knowledge with measured geometric feedback and iterative correction.

2. Related Work

  • Mesh representation and articulation research spans part datasets, learned segmentation, static-mesh motion inference, 3D-language methods, and visual prompting.Particulate jointly estimates parts, connectivity, and motion constraints; Instruct-Particulate adds kinematic instructions and heterogeneous supervision [5] [6].Source: §2 · p. 2
  • PhysX-Anything learns to generate geometry, articulation, and physical attributes from a single image, while URDF-Anything+ jointly generates part geometry and joints; URDFormer and Articulate-Anything predict kinematic structure and assemble objects from library meshes [2] [19] [20] [21].Source: §2 · p. 2
  • Agentic modeling uses program authoring or structured assembly feedback, but validating an assembly does not establish agreement with measured target geometry.USDCraft combines source geometry analysis, iterative geometric rechecking, and visual feedback to ground reconstruction in the source mesh [1] [8].Source: §2 · p. 3

3. Method

USDCraft uses an LLM-guided executable asset program, grounded in partial mesh evidence and visual feedback, to reconstruct or generate articulated assets. Its method makes observed geometry measurable, candidate discrepancies actionable, and unobserved regions available for justified completion.

3. Method

  • An off-the-shelf LLM iteratively refines an executable asset program against the source using tools, guidance, and geometric and visual feedback.Source geometry analysis represents observed surfaces separately from unknown space; rechecking compares each candidate in the same representation.Source: §3 · p. 3

3.1. Problem setup

  • Reconstruction uses a source mesh, reference image, and optional text request to produce an articulated program whose geometry and motion match the available evidence.The mesh guides dimensions and placement; the image guides object identity, visible parts, and likely motion, including components absent from an incomplete scan.Source: §3.1 · p. 3

3.2. Source geometry analysis

  • Source geometry analysis serializes sampled mesh surfaces as metric axial slices and summaries, marking all cells outside the observed surface as unknown.The agent can query local bounds, point distributions, and surface orientations for selected regions when needed.Source: §3.2 · p. 3

3.3. Programmatic asset representation

  • Explicit program parameters let the agent make local edits to parts, joints, and physical properties, or add parts and connections absent from the source.The toolkit combines parametric solid modeling and SDF modeling with analytic primitives.Source: §3.3 · p. 3

3.4. Iterative geometric rechecking

  • Rechecking compares the candidate and source on a shared grid, making discrepancies traceable to the program; missed observed surface is direct evidence of error.Candidate surface in unknown source regions may instead be a justified completion, so it is assessed using the image and object function rather than uniformly penalized.Source: §3.4 · p. 4
  • First-surface maps and region-to-part depth comparisons distinguish placement shifts from shape differences, guiding different program edits.The median offset measures front–back shift, while the 90th-percentile residual after removing that offset measures shape difference.Source: §3.4 · pp. 4–5

3.5. Visual feedback

  • Visual feedback renders candidates from selected viewpoints and joint configurations to reveal missing components, implausible assemblies, appearance errors, and motion-dependent clearance problems.For generation without a source mesh, visual feedback is the main check on the candidate.Source: §3.5 · p. 5

4. Experiments

The experiments evaluate reconstruction and articulation recovery across two benchmarks, test the modeling harness and real-to-sim-to-real transfer, and assess generation and component contributions.

4.1. Evaluation setup

  • The evaluation covers 60 agent-generated USDCraft-bench assets and a 40-object extension, alongside 243 human-modeled Lightwheel objects, with static meshes supplied without part labels or articulation metadata.Metrics assess part recovery, geometric overlap and distance, and joint-axis and location errors.Source: §4.1 · p. 5, §4 · p. 5

4.2. Reconstruction results of USDCraft

  • On USDCraft-bench, both USDCraft configurations outperform all four baselines on every reconstruction and articulation metric.The comparison includes Table 1; the reported Astra results average three runs, with variation far smaller than the margins.Source: §4.2 · p. 5, Table 1 · p. 6
  • On Lightwheel, USDCraft–Astra leads on part matching, static and articulated gIoU, part Chamfer, and joint location error, while mesh-segmentation methods retain lower whole-object Chamfer.Articraft–Astra has lower angular error but lower movable-part recall; shared category-level guidance further improves results without retraining.Source: §4.2 · pp. 5–6, Table 2 · p. 6

4.3. Effectiveness of the USDCraft harness

  • USDCraft improves all nine reconstruction metrics over Mini Workflow, and USDCraft–Sol outperforms Mini Workflow–Astra on every metric, with gains transferring across model families.Analysis and rechecking add about one to two minutes per asset for GPT backbones.Source: §4.3 · pp. 6–7, Table 3 · p. 6

4.4. Real-to-sim-to-real manipulation

  • USDCraft loses at most 10 success-rate points from simulation to real on any task, compared with Articraft’s 60–80-point losses; incomplete Particulate assets support only lever pressing.The reported comparison covers drawer opening, toaster-lever pressing, and toaster switching.Source: §4.4 · pp. 7–8, Table 4 · p. 8

4.5. Text- and image-conditioned generation

  • For text- and image-conditioned generation, USDCraft scores highest in every evaluated dimension while taking the least time, and its toolkit also produces the 10,000-asset USDCraft-10k library.The library spans more than 500 everyday categories and uses 70% text-conditioned and 30% image-conditioned requests.Source: §4.5 · p. 8

4.6. Component ablations

  • Ablations identify the authoring harness as the largest single gain, source geometry analysis as the next largest, and rechecking and visual feedback as addressing different error types.An independent observation agent lowers every metric; the authors suggest its image-only inventory can mislead the modeling agent.Source: §4.6 · p. 8

5. Conclusion

USDCraft grounds programmatic asset modeling in measured source geometry, improving geometry and articulation over segmentation-based and image-only programmatic methods. Fine lattices and deformable-object support remain limitations.

  • Grounding improves geometry and articulation over segmentation-based and image-only programmatic methods, matters more than backbone choice, and yields policies that transfer with little loss.Source: §5 · p. 8
  • USDCraft uses a pretrained LLM to revise asset programs against measured geometry, with geometric rechecking and visual feedback.Source geometry analysis makes the mesh readable to the agent; visual feedback catches missing parts and implausible motion.Source: §5 · p. 8
  • Fine lattices remain hard to reconstruct faithfully, while deformable objects are supported only in annotated states and rest geometry.Flexible-body dynamics have not been validated in Isaac Sim.Source: §5 · pp. 8–9

Guide to the supplementary material

The supplement collects implementation details, benchmark and evaluation protocols, additional results, and prompts across appendices A–G.

  • The supplement provides implementation details, benchmark and evaluation protocols, additional results, and prompts.Source: Guide to the supplementary material · p. 12
  • Appendices A–C cover the authoring and source-analysis pipeline, benchmark inputs and categories, and reconstruction and generation evaluation.They include harness execution, annotation, part matching, score aggregation, and generation scoring.Source: Guide to the supplementary material · p. 12
  • Appendices D–G provide additional comparisons, manipulation protocols, limitations, and modeling-agent prompts and examples.Topics include consistency, benchmark sensitivity, real-world task sequences, fine-lattice failures, and deformable-object support.Source: Guide to the supplementary material · p. 12

A. Implementation Details

The implementation combines an interactive modeling harness, a programmatic authoring toolkit, and metric source-geometry evidence to create and revise simulation assets.

A.1. Harness execution and tool access

  • A single modeling agent maintains the asset program, chooses construction and inspection actions, and interprets refreshed geometry and visual feedback after recompilation.The host compiles each edit and verifies that the exported asset is complete and can be opened.Source: §A.1 · p. 13

A.2. Authoring toolkit and compilation

  • The toolkit composes SDF shapes, materials, textures, and decals, then exports USD and URDF assets with geometry, articulation, and physical properties.Compilation derives collision geometry and packages referenced meshes and textures.Source: §A.2 · p. 13

A.3. Source preparation and evidence interpretation

  • For inputs lacking known physical scale, an image-based scale estimate is fixed during modeling; only Table 7 ablations use an advisory part-and-motion inventory.Evaluation in the reference frame retains errors in the estimated scale.Source: §A.3 · p. 13

A.4. Source geometry analysis details

  • Source geometry analysis encodes ray-intersection support, vertices, and triangle centroids as metric XY slices along Z, with local summaries for geometric details.The agent can inspect the fuller encoding or request local measurements beyond its compact digest.Source: §A.4 · p. 13

B. Benchmarks and Annotation

Benchmark inputs are standardized into static meshes, while reference annotations and category inventories define the evaluation coverage.

B.1. Input preparation and reference annotation

  • Inputs are reduced to a single static mesh by removing hierarchy, joints, animation, and metadata while preserving observed geometry and appearance.Transforms are baked, and UVs and vertex and face indices are rebuilt consistently across sources.Source: §B.1 · p. 13
  • Generated references retain their articulation labels; challenging meshes receive P3-SAM initial segmentations followed by manual functional-part merging and joint annotation, following SIMART [7].Source-exclusion comparisons test whether reconstruction advantage depends on USDCraft-generated references or agent-generated references generally.Source: §B.1 · p. 13

B.2. Object-category coverage

  • The benchmarks cover 55 distinct categories: 48 in USDCraft-bench and 14 in Lightwheel, with seven shared categories.Source: §B.2 · p. 14, Table 8 · p. 14

C.1. Normalization and geometric evaluation

The appendix specifies object-normalized geometry and joint evaluation, then checks robustness across failed outputs, repeated runs, and qualitative galleries.

C.1. Normalization and geometric evaluation

  • Unit-longest-side normalization removes global translation and scale while retaining relative part and joint geometry; reference-frame evaluation leaves USDCraft best on all nine metrics.Image-only Articraft drops sharply without metric scale, while Mini Workflow also drops noticeably.Source: §C.1 · p. 14

C.2. Part matching, aggregation, and failed outputs

  • Movable-part matches require the same joint type plus an overlap or centroid-and-size criterion, with axis-bearing matches subject to angular and axis-line thresholds.F1 is averaged per object; missing or mistyped joints receive specified penalties for AE and LE.Source: §C.2 · p. 15
  • 86.524% F1 for USDCraft–Astra versus 80.824% for USDCraft–Sol on the same 72 successful cases checks selection effects after ArtLLM’s 28 failures.The 72 cases comprise 45 agent-generated and 27 mixed-source objects.Source: §C.2 · p. 15

C.3. Generation scoring

  • Each generated asset is judged in a fresh context on condition adherence, part and operation requirements, geometry, joints, appearance, and motion evidence.The judge assigns evidence-based integer scores from 1 to 5.Source: §C.3 · p. 15

D.1. Run-to-run consistency

  • Across three repeated runs, USDCraft–Astra varies less than Mini Workflow–Astra on every geometric metric except OC, and its margin exceeds run variation on every metric.Table 9 reports means and standard deviations across the three runs.Source: §D.1 · p. 15, Table 9 · p. 15

D.2. Additional reconstruction comparisons

  • Figure 5 compares six selected objects across GT, Particulate, SIMART, Mini-Astra, and USDCraft–Astra in rest and articulated poses.The displayed Astra results are the first of three runs.Source: §D.2 · p. 16, Figure 5 · p. 16

D.3. Reconstruction galleries

D.4. Generation gallery

  • Figure 8 presents ten text-conditioned and ten image-conditioned USDCraft–Astra outputs, including complex mechanisms and household objects.Text panels identify object names, while image panels show the supplied inputs.Source: §D.4 · p. 19, Figure 8 · p. 19

D.5. Sensitivity to benchmark sources

The appendix tests benchmark-source sensitivity, evaluates predictions in the reference frame, and reports modeling-agent token use and estimated cost.

D.5. Sensitivity to benchmark sources

  • After excluding either USDCraft-generated references or all agent-generated references, USDCraft–Astra and USDCraft–Sol give the best result on each metric in both settings.Table 10 reaggregates recorded predictions under the two source exclusions.Source: §D.5 · p. 20, Table 10 · p. 20

D.6. Evaluation in the reference frame

  • Table 11 reports USDCraft-bench results with predictions evaluated in the reference frame, retaining absolute size and position errors.The table identifies best and second-best results; ArtLLM values include successful outputs only.Source: §D.6 · p. 20, Table 11 · p. 20

D.7. Token usage and cost

  • USDCraft uses more input tokens than Mini Workflow with the same backbone at a similar number of turns because geometric evidence is returned as text.Table 12 reports mean token usage in thousands and estimated cost in USD.Source: §D.7 · p. 20

E. Real-to-Sim-to-Real Manipulation

The manipulation study reconstructs three scanned objects and trains policies in simulation for direct real-robot deployment; the appendix also identifies difficult geometry and unvalidated flexible dynamics.

E. Real-to-Sim-to-Real Manipulation

  • USDCraft–Astra reconstructs scanned drawer, toaster, and box assets, including support objects, from LiDAR and RGB scans acquired in about two minutes per object.Figure 9 shows the reference images, scans, and reconstructions at a shared camera and scale within each row.Source: E. Real-to-Sim-to-Real Manipulation · pp. 20–21, Figure 9 · p. 21
  • The toaster is supported by the drawer for lever pressing and by the box for switching, so preserving support height sets the toaster controls’ positions relative to the robot.The corresponding assets are stacked in simulation to reproduce these arrangements.Source: E. Real-to-Sim-to-Real Manipulation · pp. 21–22
  • For supported settings, cuMotion collects 200 demonstrations in simulation, and a Diffusion Policy trained for 80 epochs is deployed directly on the real robot without real-world training or fine-tuning.The policy uses RGB observations and outputs absolute end-effector poses.Source: E. Real-to-Sim-to-Real Manipulation · pp. 22–23
  • Each policy is tested in 20 simulation and 20 real-world trials with randomized starts; success is defined by task-specific drawer, lever, or switch motion thresholds.Dangerous-contact rollouts count as failures; this rule applies to the executed Mini Workflow–Astra drawer rollouts reported as 0% real-world success.Source: E. Real-to-Sim-to-Real Manipulation · p. 23
  • Particulate recovers the toaster-lever joint but cannot support drawer opening or toaster switching; its bodies are fixed in simulation because they cannot stand stably.The reported 50% simulation success for lever pressing uses this fixed-base condition.Source: E. Real-to-Sim-to-Real Manipulation · p. 23

F. Difficult Geometry and Unsupported Dynamics

  • Fine lattice structures remain difficult: the shopping cart’s overall form is recovered, but local wire spacing and connections differ from the source.Closely spaced members require accurate local geometry and consistent connectivity.Source: F. Difficult Geometry and Unsupported… · p. 23, Figure 11 · p. 23
  • Generated flexible-member assets capture rest geometry, but do not establish bending, stretching, or contact response; the umbrella’s canopy tension and rigid–flexible coupling remain unvalidated in Isaac Sim.The examples include extension and microwave cords and an umbrella canopy with rigid rib and stretcher joints.Source: F. Difficult Geometry and Unsupported… · p. 23, Figure 12 · p. 24

G. Prompts and Authoring Guidance

The appendix documents USDCraft’s reconstruction and evaluation prompts, category-level articulation guidance, and generation requests, while illustrating source-geometry encoding with a scanned toaster. The example uses fine-grid measurements to locate and size sampled slot boundaries and outer-shell traces while leaving unknown cells unspecified.

G.1. USDCraft reconstruction instructions

  • The reconstruction instructions treat the image and mesh as complementary evidence, preserve the metric frame, and distinguish measured dimensions from hypotheses and unknown surfaces.They direct the agent to build an executable candidate, use focused inspection and geometric rechecking to repair supported discrepancies, and author articulation and physical properties explicitly.Source: §G.1 · p. 25

G.2. Mini Workflow reconstruction prompt

  • The Mini Workflow prompt requires independently editable procedural geometry from image and mesh evidence, with scale decisions documented and source-mesh geometry excluded from the final asset.It also specifies floating standalone PhysX and Newton entries from one definition, explicit physical properties, per-joint resistance and gravity-hold checks, and validation that does not treat static checks as proof of runtime success.Source: §G.2 · pp. 26–27

G.3. Generation-evaluation rubric

  • Review the supplied task, manifest, geometry, joint records, and reference images; assess each required part, operation, and joint, then score each dimension independently from 1–5 with a dimension-specific reason.Do not infer the generation method, inspect other assets, repair the output, or rank methods. Use not_applicable only for articulation when the task requires no movable operation; a missing required joint is a failure.Source: §G.3 · p. 27
  • The rubric scores condition adherence, structural quality, articulation functionality, completeness, and appearance, while distinguishing insufficient evidence from confirmed defects.Motion samples do not prove physics simulation, stability, load bearing, collision-free dynamics, or simulator success.Source: §G.3 · p. 27

G.4. Lightwheel articulation conventions (+G)

  • The +G guidance gives every case the same descriptions of 14 categories, leaving category selection and applicable mechanisms to the modeling agent using the image and mesh.Shared rules call for evidence-supported moving parts, mechanically appropriate joints, preservation of measured rest shape, and inspection of joint axes and posed renders.Source: §G.4 · p. 28

G.5. Generation gallery prompts

  • Generation-gallery requests pair object-specific prompts with shared instructions for realistic scale, plausible materials, motion clearances, articulated moving parts, and a complete standalone assembly.Image-conditioned examples ask for a complete, functionally usable, simulator-ready asset of the pictured subject.Source: §G.5 · pp. 28–29

G.6. Source geometry analysis example

  • The toaster example’s axial encoding reports metric bounds and sampled XY slices, marking observed source surface separately from unknown geometry.The excerpt shows three representative slices and a fine-resolution boundary summary; other slices and metadata are omitted.Source: §G.6 · p. 29
  • The upper slice exposes sampled boundaries around the toaster slots.Source: §G.6 · p. 30
  • The lower slices primarily trace the toaster’s outer shell.Source: §G.6 · p. 30
  • Unknown cells do not establish empty space or solid interiors.Source: §G.6 · p. 30
  • Fine-grid summaries retain local bounds and boundary contours.Source: §G.6 · p. 30
  • The summaries define mz and xy as half-open fine-grid index ranges, f as the sampled-cell fraction within the XY bounding box, and p as a simplified boundary contour.Source: §G.6 · p. 30
  • These measurements let the modeling agent locate and size geometry beyond the coarse character grid.Source: §G.6 · p. 30
Loading 2610.11322v1…