Source-linked AI summary
ArtLLM: Generating Articulated Assets via 3D LLM
Penghao Wang, Siyuan Xie, Hongyu Yan, Xianghui Yang, Jingwei Huang, Chunchao Guo, Jiayuan Gu
TL;DR
Articulated-object generation is limited by slow optimization and fixed-library retrieval, despite its importance for interactive environments and robot simulation. ArtLLM predicts part layouts and kinematic structures autoregressively, conditions part generation on that blueprint, and applies physics-based joint-limit correction. It achieves superior articulation prediction and generates novel geometries that generalize to real-world objects, supporting digital-twin construction and scalable robot learning.
Problem
Existing articulated-object methods are limited by slow per-object optimization, simple joint structures, and fixed part libraries that restrict geometric novelty.
Method
ArtLLM autoregressively predicts part layouts and kinematic relationships, uses multi-task training, conditions part-aware geometry synthesis on the resulting blueprint, and refines joint limits physically.
Results
ArtLLM achieves superior part placement, joint accuracy, and kinematic relationship modeling, while generating novel geometries and generalizing well to real-world images.
Takeaways & Limitations
The framework supports simulation-ready articulated assets and digital twins, highlighting potential for scalable robot learning.
Takeaways & Limitations
The curated training data has limited category diversity, causing weaker generalization to complex categories such as vehicles or robots; physical properties are not jointly modeled.
Abstract
from arXiv · showhide
Creating interactive digital environments for gaming, robotics, and simulation relies on articulated 3D objects whose functionality emerges from their part geometry and kinematic structure. However, existing approaches remain fundamentally limited: optimization-based reconstruction methods require slow, per-object joint fitting and typically handle only simple, single-joint objects, while retrieval-based methods assemble parts from a fixed library, leading to repetitive geometry and poor generalization. To address these challenges, we introduce ArtLLM, a novel framework for generating high-quality articulated assets directly from complete 3D meshes. At its core is a 3D multimodal large language model trained on a large-scale articulation dataset curated from both existing articulation datasets and procedurally generated objects. Unlike prior work, ArtLLM autoregressively predicts a variable number of parts and joints, inferring their kinematic structure in a unified manner from the object's point cloud. This articulation-aware layout then conditions a 3D generative model to synthesize high-fidelity part geometries. Experiments on the PartNet-Mobility dataset show that ArtLLM significantly outperforms state-of-the-art methods in both part layout accuracy and joint prediction, while generalizing robustly to real-world objects. Finally, we demonstrate its utility in constructing digital twins, highlighting its potential for scalable robot learning.
1. Introduction
ArtLLM addresses limitations in articulated-object generation by jointly predicting part layouts and kinematic structures, then synthesizing novel geometries and refining joint limits for physically plausible assets.
- Articulated assets support interactive digital worlds, robot training, and simulation through their part geometry and kinematic structures.
- Optimization-based methods are slow, often produce low-fidelity geometry, and typically handle only simple single-joint objects.
- Retrieval-based methods provide fast inference but depend on fixed part databases, limiting geometric novelty and generalization.
- ArtLLM autoregressively predicts tokenized part layouts and kinematic relationships from point clouds, using structured language modeling for articulation understanding.
- The predicted blueprint conditions a part-aware generative model that synthesizes high-fidelity, coherent geometries without relying on fixed databases.
- Physics-based joint-limit correction prevents self-collisions and improves asset stability and realism in physical simulations.
4. Experiments
The experiments compare ArtLLM with recent articulated-object generation methods using standardized evaluation protocols, ablations, physical-limit correction, and real2sim demonstrations. Results show stronger articulated-structure recovery, collision-free motion after refinement, and successful simulation replay of three real tasks.
- Comparisons: The evaluation compares ArtLLM with URDFormer, Singapo, and Articulate-Anything under a consistent protocol, including retrieval-library adjustments and standardized resources.Evaluation uses PartNet-Mobility categories and matching procedures based on aligned objects, Hungarian part matching, and derived joint correspondences.
- Comparisons: ArtLLM closely matches input geometry and recovers more coherent articulations than baselines that mispredict parts, joint types, axes, scales, or positions.Qualitative comparisons report errors from fixed structural assumptions and retrieval-based methods, while ArtLLM produces accurate shapes and articulation structures.
- Comparisons: ArtLLM achieves a clear advantage in part layout prediction, joint accuracy, and hierarchical structure modeling, while also providing a faster inference pipeline.Table 2 averages metrics across seven PartNet-Mobility categories; retrained Singapo improves but remains behind ArtLLM.
- Ablation: Removing multitask learning, augmentation, or multi-stage training degrades most part and joint metrics, while direct continuous prediction weakens coordinate and direction inference.The ablations attribute these effects to varied task difficulty, improved spatial perception, pretrained encoder initialization, and the difficulty of autoregressive continuous prediction.
- Ablation: Physics-based limit refinement removes self-collisions and improves the stability and realism of generated assets for simulation training.The correction adjusts predicted joint ranges when they cause collisions, producing smooth, physically plausible motion.
- Real2sim Evaluation: In real2sim evaluation, generated assets successfully reproduce laptop-closing, box-closing, and bucket-handle-moving tasks in simulation.Real Franka Panda trajectories were recorded, reconstructed as URDF assets, and replayed in SAPIEN.
5. Limitation
The method’s category coverage remains limited despite training on a curated articulation dataset, and it does not jointly model physical properties. These boundaries affect generalization to complex object categories and physics-aware prediction.
- Category Coverage: Limited category diversity lets the model generalize well to common household items but makes complex categories such as vehicles or robots difficult.The authors suggest open-vocabulary approaches to expand category coverage and broader object modeling.
- Physical Properties: The framework does not jointly model physical properties, leaving physics-aware prediction for future work.The authors propose training on large-scale datasets with annotated physical attributes.
6. Conclusion
ArtLLM rapidly generates articulated objects from modalities including images and text by autoregressively modeling variable part layouts and articulation parameters. Its geometry synthesis and physical-limit correction support detailed, collision-mitigated assets for simulation and digital-twin applications.
- Conclusion: ArtLLM autoregressively represents part layouts and articulation parameters, accommodating objects with varying numbers of joints and topologies.The framework uses a data discretization design to improve numerical stability in next-token prediction.
- Conclusion: Integration with high-quality part-generation modules produces geometrically detailed and diverse parts without the low-novelty limitation of retrieval-based methods.The framework accepts modalities such as a single image or text description.
- Conclusion: Physical constraint-based limit correction mitigates mesh collisions and produces physically grounded assets for robotic simulation and related downstream tasks.The conclusion connects the pipeline to narrowing the real-to-simulation gap, accelerating digital twins, and supporting scalable robot learning.
8. Implementation Details
Implementation uses staged supervised training for ArtLLM, point-cloud augmentation, and an existing part-generation model. The resulting articulated geometry is combined with predicted articulations and exported for simulation.
- Training: ArtLLM is trained with cross-entropy supervised fine-tuning, a 3:2:5 multitask mixing ratio, cosine scheduling, and a maximum learning rate of 1e-5.The warmup ratio is 0.03.
- Data Augmentation: Training augments point clouds, layouts, and articulations with random scaling and y-axis rotations applied with probability 0.75.Scale factors range from 0.8 to 1.05, and rotation angles are selected from 90, 180, and 270 degrees.
- Training: A two-stage training schedule first initializes the point encoder and projector, then trains all three tasks on mixed data.Both stages use eight H20 GPUs; the first runs for 50 epochs and the second for 30 epochs.
- Asset Export: The part-generation model uses XPart, combines generated link geometry with predicted articulations, and exports articulated assets in URDF format.URDF export enables integration into simulators for further analysis and simulation.
9. ArtLLM Template
ArtLLM uses concise structured templates to predict part layouts and four joint articulation types through three task formulations.
- Training tasks: ArtLLM distinguishes part layout, kinematic, and end-to-end prediction tasks.The tasks respectively predict part boxes, joints conditioned on boxes, or both from the point cloud.
- Output format: The output template includes part bounding boxes followed by Revolute, Continuous, Screw, and Prismatic joints.Each joint representation encodes parent and child boxes plus axis and motion parameters where applicable.
- Joint parameters: Revolute joints encode rotation limits, while prismatic joints encode translation limits.Both formats also specify parent and child box identifiers and an axis direction; revolute joints additionally specify axis position.
10. Detail of Training Dataset
The curated training dataset is organized around articulated objects with varied link counts and serialized point-cloud, layout, and articulation data.
- Dataset statistics: The dataset filters objects with more than 20 joints, limiting retained objects to at most 21 links.Although most objects have few parts, the collection includes sufficient examples with larger part counts.
- Dataset statistics: The varied link-count distribution is intended to support handling objects with different structural complexities.The dataset includes both predominantly small-part objects and samples with larger part counts.
- Data ordering: Layout bounding boxes are sorted by minimum coordinates in z–y–x order, while joints are sorted by ascending child-link IDs.Deterministic ordering converts unordered annotations into sequences for autoregressive training.
- Training representation: Training examples are provided in ShareGPT format with raw continuous values and discretized values used during training.Special tokens represent bounding-box coordinates, axis directions, axis origins, and limit ranges.
11. Detail of Experiments
Experiments evaluate ArtLLM’s part and articulation predictions with aligned object-level metrics and category-wise comparisons against prior methods.
- Metrics: Predicted parts and joints are scale- and coordinate-aligned, matched to ground truth with the Hungarian algorithm, then scored per component.Final results average first within categories and then across categories.
- Metrics: Evaluation uses mIoU for part layout and joint type accuracy, axis error, pivot error, and range IoU for articulation.These metrics separately assess geometric overlap, joint semantics, axis direction, joint origin, and motion limits.
- Metrics: Joint range IoU accounts for direction reversal by taking the maximum overlap between original and reversed predicted directions.This prevents equivalent direction conventions from lowering the range score.
- Comparison: ArtLLM shows clear superiority across most metrics and categories in the quantitative comparison.Articulate Anything performs better on limit range, while SINGAPO occasionally performs better on joint axis prediction.
- Qualitative results: Additional results use Hunyuan3D 3.0-generated input geometry and show canonical and sampled articulated states in paired images.The figure presents further PartNet-Mobility test-set outputs and failure-case analysis.
- Comparison: Articulate Anything’s limit-range advantage coexists with weaker part-layout and joint-origin performance.SINGAPO is also significantly weaker on the remaining metrics, particularly joint origin.
12. More Results
ArtLLM generates realistic articulated assets across many PartNet-Mobility categories from input point clouds generated by Hunyuan3D 3.0.
- Qualitative results: ArtLLM generates realistic articulation assets with accurate geometry across many PartNet-Mobility categories.The model is evaluated using input point clouds generated by Hunyuan3D 3.0.
13. Failure Cases Analysis
Failure cases show that ArtLLM can produce accurate layouts while downstream geometry generation may miss internal structures or create intersecting parts. These failures reduce the realism of reconstructed articulation assets.
- ArtLLM produces accurate layouts, but generated geometries can miss internal concave structures, reducing realism.
- Highly overlapping parts may generate intersecting geometries in reconstructed assets.
- Fine structures hidden inside an object may be omitted when generating the full object from a single image.The omitted internal structure causes the resulting articulation asset to fail to reproduce that detail.
- Future work includes fine-tuning XPart or training a 3D generative model to reconstruct occluded internal structures.