Source-linked AI summary
Articraft: An Agentic System for Scalable Articulated 3D Asset Generation
Matt Zhou, Ruining Li, Xiaoyang Lyu, Zhaomou Song, Zhening Huang, Chuanxia Zheng, Christian Rupprecht, Andrea Vedaldi, Shangzhe Wu
TL;DR
Articulated-object learning is constrained by small, narrow, and uneven datasets. Articraft has an LLM write asset-building programs through a domain-specific SDK and iterative harness, producing over 10K assets across 245 categories and improving downstream articulated-object modeling.
Problem
Existing articulated-asset datasets are small, narrow in coverage, and uneven in quality, with practical categories underrepresented.
Method
Articraft uses an agent harness and LLM-friendly SDK to generate articulated 3D assets as programs through iterative editing, execution, validation, and repair.
Results
Articraft produces higher-quality assets than prior articulated-asset generators and general-purpose coding agents, and creates Articraft-10K with over 10K assets spanning 245 categories.
Takeaways & Limitations
Articraft-10K improves training of articulated-object models and supports applications in robotics simulation and virtual reality.
Abstract
from arXiv · showhide
A bottleneck in learning to understand articulated 3D objects is the lack of large and diverse datasets. In this paper, we propose to leverage large language models (LLMs) to close this gap and generate articulated assets at scale. We reduce the problem of generating an articulated 3D asset to that of writing a program that builds it. We then introduce a new agentic system, Articraft, that writes such programs automatically. We design a programmatic interface and harness to help the LLM do so effectively. The LLM writes code against a domain-specific SDK for defining parts, composing geometry, specifying joints, and writing tests to validate the resulting assets. The harness exposes a restricted workspace and interface to the LLM, validates the resulting assets, and returns structured feedback. In this way, the LLM is not distracted by details such as authoring a URDF file or managing a complex software environment. We show that this produces higher-quality assets than both state-of-the-art articulated-asset generators and general-purpose coding agents. Using Articraft, we build Articraft-10K, a curated dataset of over 10K articulated assets spanning 245 categories, and show its utility both for training models of articulated assets and in downstream applications such as robotics simulation and virtual reality.
1 Introduction
Articraft addresses limited and uneven articulated-object data by having an LLM generate asset-building programs through a task-specific SDK and iterative harness. The resulting system supports scalable generation, yields Articraft-10K, and improves downstream articulated-object modeling.
- Motivation: Existing articulated-asset datasets are small, narrow, and uneven, leaving important categories underrepresented and causing learning-based methods to overfit.The paper identifies poor generalization to new categories as a consequence of the available data distribution.
- Approach: Articraft generates articulated 3D objects as code, transferring the recursive, compositional structure of object design into an agentic programming workflow.The workflow includes decomposing objects, specifying connections and motion, instantiating geometry, validating behavior, and revising failures.
- Design goals: Articraft is designed to be automatic, lightweight, and expressive, avoiding heavy external graphics software and image-based feedback while covering complex articulated mechanisms.These design goals target low per-asset inference cost and broad category coverage.
- Approach: The system combines an agent harness with an LLM-friendly SDK so off-the-shelf LLMs can create articulated assets without retraining.The SDK supports low- and high-level geometric abstractions, while the harness provides a minimal edit–execute–feedback loop.
- Results: Articraft outperforms prior articulated-asset generators and general-purpose coding agents, while producing Articraft-10K with over 10K objects spanning 245 categories.The dataset is intended for large-scale generation and is accompanied by code representations, reasoning traces, and the agent environment.
- Applications: Articraft-10K substantially boosts Particulate’s performance when used to augment training data, and the system is demonstrated in robotics simulation and virtual reality.The paper also reports public release plans for the dataset and associated generation artifacts.
2 Related Work
Prior work spans direct articulated-asset generation, programmatic CAD design, language-agent interfaces, and articulated simulation datasets. Articraft relates these directions through task-specific programming interfaces and iterative feedback.
- Articulated asset generation: Prior articulated-asset generators use ad-hoc representations with diffusion models or token-based representations with autoregressive transformers.Other approaches map images to URDF proxy representations or reconstruct URDF objects sequentially by part.
- Programmatic 3D design: Programmatic 3D design methods target CAD systems through SDKs, reusable primitives, encoded CAD commands, or Python code against the CadQuery API.These methods differ in whether programs are generated by specialized code generators or autoregressive transformers.
- Language agents: Language-agent research interleaves reasoning, actions, tool use, and feedback, with SWE-agent emphasizing task-specific software interfaces and structured feedback.Articraft follows this interface-design principle for articulated 3D generation.
- Datasets: Existing articulated and simulation datasets include synthetic objects, real-object scans, affordance annotations, scenes, and simulated environments with varying articulated-category coverage.The cited datasets differ in whether they focus on individual objects, complete scenes, or robotics environments.
- Articraft interface: Articraft’s overview combines a restricted workspace, iterative program editing, harness execution, and validation feedback for successive revisions.The figure describes the operational pattern that connects Articraft to prior agentic programming systems.
3 Articraft
Articraft generates articulated 3D assets by having an off-the-shelf LLM iteratively write and refine a program through a task-specific SDK and harness. The system restricts the workspace, validates assets, and provides structured feedback while supporting image-conditioned generation.
- Articraft converts a natural-language object description, optionally paired with an image, into a Python program that outputs a URDF containing meshes, semantic parts, and articulated joints.
- The agent follows a reasoning-action-observation loop in which the LLM edits, compiles, or probes the current program and asset, with each state and feedback appended to history.
- Generation terminates when the LLM stops issuing edits and validation criteria are met, after which the final asset version is accepted.
- A task-specific programmatic interface and harness improve the general-purpose LLM backend while avoiding expensive visual feedback by providing direct geometry-authoring and checking tools.
- The interface supports articulated-object authoring through a single model.py program, geometric primitives, semantic parts, multiple joint types, motion limits, and object-specific validation.
- The harness limits the agent to a restricted workspace and action space, then returns structured failure, warning, and note signals while recording an auditable execution trace.
- For image-conditioned generation, the reference image remains the primary source for geometry, proportions, articulation, and materials throughout iterative repair.
4 Articraft-10K
Articraft-10K is a curated dataset of over 10K articulated 3D models across 245 categories, distributed with code and agent traces. Generated assets are filtered using manual ratings of realism, articulation, and physical plausibility.
- Articraft-10K contains over 10K articulated 3D models spanning 245 object categories mapped into 15 super-categories.
- Each dataset asset includes a URDF file, its corresponding model.py program, and the full agent trace of reasoning, feedback, and tool use.
- The dataset construction process selected categories through manual exploration, guideline development, LLM-proposed candidates, and manual review and filtering.
- Generated objects were manually scored from 1 to 5 for geometric realism, expected articulated motions, and adherence to basic physical constraints.
5 Evaluating the Articraft Agent
The Articraft agent is evaluated against articulated-asset generators and coding agents through a user study, foundation-model ablations, and image-conditioned and scene-reconstruction applications. Results emphasize the contribution of Articraft’s domain-specific interface and harness.
- The evaluation compares Articraft with Articulate-Anything, PhysX-Anything, URDF-Anything+, and Codex, including variants powered by GPT-5.4 and GPT-5.5.
- The user study evaluates six methods on prompts from 46 PartNet-Mobility categories, using 125 participants and 5000 submitted comparisons.
- GPT-5.5 alone ranks second to last, whereas Articraft using the same underlying LLM performs substantially better, highlighting the effect of the domain-specific interface and harness.
- Across LLM ablations, GPT-5.5 produces more visual detail than Gemini 3.1 Pro and Claude Opus 4.7, although all recover the requested kinematic structure.
- Articraft also supports image-conditioned articulated-object generation and articulated room-level reconstruction from indoor scans through integration with LiteReality.
- Generated assets can be imported into physics simulation and manipulated with robot arms and VR headsets.
6 Evaluating the Articraft-10K Dataset
Articraft-10K is evaluated as training data and as a basis for interactive applications. Augmenting articulation-model training with the dataset improves performance, while its assets support robotic simulation and natural VR interactions.
- Adding Articraft-10K to PartNet-Mobility and GRScenes training data boosts Particulate’s performance on articulated-object estimation.
- The gains from Articraft-10K are particularly pronounced for categories outside the original training distribution.
- In NVIDIA Isaac Sim, Articraft-10K assets support robot-arm interactions such as using a Franka arm to pull a drawer open through inverse kinematics.
- In VR, collision detection and custom scripts controlling URDF-defined joints produce natural and realistic object interactions.
7 Conclusions
Articraft combines a task-specific SDK with a restricted execution harness to generate articulated assets through validated edit–execute–repair cycles. It produces Articraft-10K, a dataset of over 10K assets with structured programs and annotations for downstream training, simulation, and VR interaction.
- Articraft pairs a domain-specific SDK with a restricted execution harness for scalable articulated 3D asset generation.The system uses validation and geometric feedback without relying on rendered visual inspection.
- Articraft-10K contains over 10K high-quality articulated assets with source programs, semantic part structures, joint specifications, and generation traces.
- The resulting dataset supports feed-forward model training, simulation, and VR interaction.
A.1 Articraft-10K Dataset
Articraft-10K documents broad dataset composition across object categories, generation effort, part structure, and curation retention. Its statistics include category distributions, generation cost and turns, object links, mesh properties, and backend-specific retention.
- Dataset statistics: Articraft-10K covers 15 super-categories, with category-level statistics relating average generation cost to average agent turns.The two quantities show a positive correlation.
- Dataset statistics: The dataset statistics include distributions of generation cost, agent turns, and number of links, alongside mesh vertices, triangles, and edges.
- Curation: Curation retention is defined as the fraction of generated assets receiving a final manual rating of at least 4.Retention statistics are broken down by generation backend.
A.3 Compute Requirements for Articraft Generation
Articraft generation uses external LLM APIs for inference and a local CPU-only pipeline for program execution, geometry construction, export, testing, and quality control. No GPU is required for generation, dataset materialization, or the compile/QC loop.
- Local pipeline: The local Articraft pipeline runs model.py, CAD and mesh construction, URDF export, authored tests, and quality-control checks on CPU workers.The CPU workers are heterogeneous, and no GPU is required for these local steps.
- Execution split: External LLM API backends provide the expensive inference step, while the harness performs local compilation and validation.
- Execution split: The same lightweight CPU-only local pipeline supports retained objects, filtered objects, and exploratory runs.The system logs provider, model, turns, API cost, generated programs, compile outputs, and curator ratings.
A.4 API Cost and Token Usage
Articraft combines lightweight API-based generation with programmatic asset construction, articulated structure, and image-conditioned reconstruction. The reported pipeline supports diverse assets and downstream simulation while documenting generation cost, dataset statistics, and practical fallback behavior.
- API usage: $12.39K total API cost covered 10,880 cost-logged generation attempts, averaging $1.14 per generated attempt.Retained Articraft-10K objects accounted for approximately $11.33K, or $1.13 per retained object on average.
- API usage: 85.7% of prompt tokens in the main-backend cost logs were served from cache.Prompt caching was identified as important for scalability.
- Generated assets: Articraft-10K examples span small handheld objects, household and workshop objects, larger structures, and multiple articulation types.The examples include explicit semantic parts and articulated structure rather than only static geometry.
- SDK expressivity: The SDK allows prompts to control both mechanism and construction idiom, illustrated by a block-based gatehouse with controlled drawbridge motion.
- Image-conditioned generation: Image-conditioned generation combines reference-grounded geometry with retrieved and optimized PBR materials to produce articulated assets with valid joints.The two stages are shown to produce faithful geometry, valid joints, and complete PBR materials.
- Scene reconstruction: For irregular or unconventionally designed objects, the scene pipeline routes non-articulated cases to LiteReality retrieval instead of Articraft generation.
B.4 LLM and Reasoning-Effort Ablation Details
The ablation compares LLMs and reasoning effort under a fixed prompt, using the same Articraft harness and SDK. It is intended to illustrate qualitative differences in geometry and surface detail rather than provide a definitive model comparison.
- The ablation holds the prompt, Articraft harness, and SDK constant across runs.
- Model comparison: OpenAI GPT-5.5, Google Gemini 3.1 Pro, and Anthropic Claude Opus 4.7 are compared at high reasoning effort.
- Reasoning-effort comparison: GPT-5.5 is evaluated at low, medium, and high reasoning effort.
- The comparison is designed to show qualitative differences in geometry and surface detail, not to establish a definitive model ranking.
- Controlled prompt: The controlled prompt specifies a folding quadcopter with hinged arms, continuously spinning rotors, landing skids, and a tilting camera.
B.5 Failure Cases and Validation Trade-offs
Articraft’s lightweight validation emphasizes high-value structural checks while limiting exhaustive pose testing to control generation cost. The resulting trade-off leaves some global visual, mechanism-specific, and category-specific failures outside the default validation envelope.
- Validation trade-offs: Validation prioritizes detecting floating parts and unintended overlaps, while targeted tests are encouraged instead of exhaustive pose sampling.The SDK can check many articulated poses, but exhaustive sampling increases runtime.
- Validation trade-offs: Some failure modes remain outside the default validation envelope despite the design’s efficient generation.Examples are presented in Fig. 18.
- Global shape quality: A malformed screwcap-bottle shell can pass local structural checks because connectivity, overlap constraints, and authored local tests do not fully assess global visual plausibility.Similar visually unsatisfactory cases occur for skateboard and revolving-door objects despite avoiding floating parts and unintended overlaps.
- Mechanism-specific failures: Trigger-spray-bottle mechanisms can be difficult to express compactly, causing awkward motion and trigger–bottle overlap.The passage suggests richer mechanism-specific abstractions or additional pose checks for such categories.
- Category-specific failures: Complex categories may sporadically omit interior structure or fail to hollow shapes even when exterior geometry and articulation appear plausible.These errors reflect a trade-off between cheap validation and stronger semantic or functional checks.
- Agent interface: The agent operates in a restricted workspace centered on one editable model.py file, read-only SDK guidance, and tool-mediated compilation, testing, and inspection.The workspace and prompt constrain authoring while leaving system execution and infrastructure management to Articraft.