PaperlayerView exact revision on arXiv

Source-linked AI summary

Kinematify: Open-Vocabulary Synthesis of High-DoF Articulated Objects

Jiawei Wang, Dingyou Wang, Jiaming Hu, Qixuan Zhang, Jingyi Yu, Lan Xu

arXiv:2511.01294v4cs.ROcs.CV

TL;DR

High-DoF articulated descriptions are difficult to create, and existing approaches rely on motion sequences or struggle with complex kinematic structures. Kinematify synthesizes articulated objects from RGB images or text by combining MCTS topology inference with geometry-driven joint estimation. Across everyday objects and robot platforms, it improves joint estimation accuracy and kinematic tree fidelity over prior work.

  • Problem

    Creating articulated models, particularly for high-DoF objects, is challenging, while existing methods typically rely on motion sequences or strong assumptions from hand-curated datasets.

    Source: §I · p. 1

  • Method

    Kinematify generates articulated objects from RGB images or text, using MCTS for kinematic-tree inference and geometry-driven DW-CAVL optimization for joint parameters.

    Source: §I · p. 2

  • Results

    Kinematify improves joint estimation accuracy and kinematic tree fidelity over prior work across everyday objects and multiple robot platforms.

    Source: §V · p. 7

  • Takeaways & Limitations

    The framework generates physics-aware articulated objects from arbitrary RGB images or text without motion data, training, or predefined articulation priors.

    Source: §I · p. 2

Abstract

from arXiv · show

A deep understanding of kinematic structures and movable components is essential for enabling robots to manipulate objects and model their own articulated forms. Such understanding is captured through articulated objects, which are essential for tasks such as physical simulation, motion planning, and policy learning. However, creating these models, particularly for objects with high degrees of freedom (DoF), remains a significant challenge. Existing methods typically rely on motion sequences or strong assumptions from hand-curated datasets, which hinders scalability. In this paper, we introduce Kinematify, an automated framework that synthesizes articulated objects directly from arbitrary RGB images or textual descriptions. Our method addresses two core challenges: (i) inferring kinematic topologies for high-DoF objects and (ii) estimating joint parameters from static geometry. To achieve this, we combine MCTS search for structural inference with geometry-driven optimization for joint reasoning, producing physically consistent and functionally valid descriptions. We evaluate Kinematify on diverse inputs from both synthetic and real-world environments, demonstrating improvements in registration and kinematic topology accuracy over prior work.

I. INTRODUCTION

Accurate articulated descriptions require substantial modeling and kinematics work, especially for high-DoF systems, while existing approaches face limits in capture requirements or structural complexity. Kinematify generates articulated objects from images or text and combines topology search with geometry-driven joint estimation.

I. INTRODUCTION

  • Creating articulated descriptions for high-DoF systems requires labor-intensive part-aware modeling, resolving kinematic dependencies, and estimating precise joint parameters.Such descriptions encode geometry, kinematic dependencies, and dynamic constraints for robotic tasks including manipulation, locomotion, and policy learning.Source: §I · p. 1
  • Kinematify generates articulated objects from RGB images or text, using a part-aware 3D foundation model, MCTS, for tree inference, and DW-CAVL for joint parameters.The resulting description is exported to URDF and can be converted to MJCF or USD.Source: §I · p. 2
  • Its MCTS-based objective encodes hierarchy and structural regularity to resolve ambiguous attachments in complex, high-DoF objects with multiple branches.The framework does not require motion data, training, or predefined articulation priors.Source: §I · p. 2
  • DW-CAVL estimates revolute and prismatic joint parameters from static geometry with an SDF-based, contact-aware objective under virtual motions.It preserves near-contact regions while penalizing collisions under virtual motion.Source: §I · p. 2

A. 3D Articulated Object Modelling

  • Motion-based methods can achieve high fidelity but depend on multi-view or temporal data and controlled capture, while program-synthesis methods often struggle with multibranch kinematics.AutoURDF uses motion data and is largely limited to serial-chain structures, unlike the multibranched linkages common in high-DoF objects.Source: A. 3D Articulated Object Modelling · p. 2, §I · p. 2

B. Robot Self-Modeling

Robot self-modeling typically relies on an embodied robot's motor and sensory signals or visual motion sequences. Kinematify instead generates kinematics-aware objects from RGB images or text, using part meshes and geometric contact to build its representations.

B. Robot Self-Modeling

  • Robot self-modeling is the online discovery of a robot’s body plan, commonly using motor actions and sensory feedback; AutoURDF instead uses time-series point clouds of a moving robot.AutoURDF demonstrated topology inference for serial-chain objects.Source: B. Robot Self-Modeling · p. 2

III. KINEMATIFY

  • Kinematify generates kinematics-aware articulated objects directly from RGB images or textual descriptions in a zero-shot context.The framework is described as generating objects without motion data, training, or predefined articulation priors.Source: §III · p. 2

A. Preliminaries

  • An assembly is a set of parts with meshes and geometric attributes, while a connection graph links parts in contact and a rooted kinematic tree orients those links and annotates joints.Joints can be fixed, revolute, or prismatic; movable joints store an axis and revolute joints may also store a pivot.Source: A. Preliminaries · pp. 2–3

B. Part-Aware 3D Representations

  • The method generates part-level meshes from the input image or text, trains a continuous SDF for each prospective parent part, and uses SDF distances to identify contacting part pairs.Pairs whose minimum bidirectional surface distance falls below a tolerance are connected by an undirected graph edge.Source: B. Part-Aware 3D Representations · p. 3

C. Kinematic Topology Inference

Kinematify turns a contact graph into a rooted kinematic tree, using a staged search and reward terms to resolve structural choices. Its topology inference accounts for symmetry and physical structure.

C. Kinematic Topology Inference

  • The base is a highest-degree graph node, and BFS supplies a warm-start orientation while avoiding cycles and attaching nearby disconnected components.A disconnected component is attached by a virtual edge only when its nearest neighbor is within dmax.Source: C. Kinematic Topology Inference · p. 3

1) State, actions, constraints:

  • Search states track a partial directed tree, visited nodes, and broken edges; expansion forbids links between parts in the same multi-member symmetry cluster.Adding an edge provisionally treats its joint as fixed and records edges that would otherwise form a cycle.Source: 1) State, actions, constraints · p. 3, 2) Transition · p. 3

2) Transition:

3) Reward:

  • The terminal reward combines structural regularity, centre-of-mass support, contact strength, symmetry, and parent-child volume hierarchy.These terms penalize depth and degree irregularity, favor gravitational support and stronger contacts, and prefer symmetric parts to share parents.Source: 3) Reward · pp. 3–4

4) Search:

Kinematify searches candidate tree structures with MCTS, using UCT selection and greedy rollouts to evaluate completed trees. Its objective is designed to resolve attachment ambiguities in high-DoF structures.

4) Search:

  • MCTS selects among feasible edge additions using UCT, balancing each child’s cumulative return and visit count against its parent’s visit count.The selection rule uses an exploration constant C > 0.Source: 4) Search · p. 4
  • Rollouts greedily complete the tree by choosing the available action with the highest immediate score, then back up the terminal return along the simulation path.Source: 4) Search · p. 4
  • The search objective helps resolve symmetric attachments and multibranch ambiguities at scale.The paper attributes its reasoning about high-DoF, multi-branched structures to the MCTS-based objective.Source: 4) Search · p. 4

D. Joint Reasoning

Joint reasoning combines visual joint-type decisions with geometry-derived candidate parameters. Candidate revolute joints are optimized and selected by score.

4) Joint Reasoning

  • A VLM assesses rendered whole-assembly and joint close-up views for each edge, with abstention; identified types are grouped by child cluster and resolved by majority.Outliers are corrected after the representative joint type is selected.Source: D. Joint Reasoning · p. 4
  • For each part pair, weighted near-contact samples yield a contact centroid and covariance; the smallest-variance direction estimates a hinge axis, while surface differences estimate a contact normal.Source: D. Joint Reasoning · p. 4
  • Revolute candidates use the contact centroid as an initial pivot and draw axes from geometric directions and random unit directions, with pivots sampled along each axis.The candidate set includes the PCA direction, contact normal, an orthogonal completion, and principal directions of the contact covariance.Source: D. Joint Reasoning · p. 4

1) Differentiable rigid motions:

Kinematify represents revolute and prismatic motions as differentiable rigid transforms of child samples. It also normalizes an unconstrained vector to parameterize the motion direction.

1) Differentiable rigid motions:

2) DW-CAVL objective:

The DW-CAVL objective combines contact consistency, collision penalties, and a pivot regularizer to evaluate candidate joints. The method ranks candidates before refining the top-K on full point sets.

2) DW-CAVL objective:

  • The objective penalizes separation near contact and penetration, and regularizes revolute pivots toward the weighted contact centroid.The terms use nonnegative weights λc and λcoll.Source: 2) DW-CAVL objective · p. 5
  • The figures show generated objects across joint configurations and a qualitative comparison of articulation recovery by Kinematify, Articulate Anymesh, and ArtGS.Figure 4 marks the joint direction with a red line.Source: Figure 4 · p. 5, Figure 3 · p. 5
  • Candidates (p, u) are ranked on subsampled points, then the top-K are refined on full points by minimizing J.The output score is 1/(1 + J).Source: 3) Candidate selection · p. 5

3) Candidate selection:

Kinematify infers joint parameters from static geometry and selects the candidate with the highest score. The accompanying table describes robot comparisons across three metrics.

3) Candidate selection:

  • Joint parameters are inferred from static geometry, and the candidate with the highest score is selected.The optimized results are shown for each joint in the input.Source: 3) Candidate selection · p. 5
  • The robot comparison table reports axis angle error, axis position error, and tree edit distance, with lower values better.The table is sorted by degrees of freedom.Source: Table I · p. 6
  • The table compares robots ordered by degrees of freedom.Its reported measures cover joint-axis orientation, pivot position, and kinematic-tree structure.Source: Table I · p. 6

IV. EXPERIMENTS

The experiments evaluate everyday articulated objects, robotic platforms, and an end-to-end pipeline starting from RGB images. They assess joint parameters and kinematic-tree quality using three metrics.

IV. EXPERIMENTS

  • Everyday-object comparisons use ground-truth segmented PartNet-Mobility meshes to isolate the impact of 3D segmentation and compare against Articulate AnyMesh and ArtGS.The kinematic graphs and joint parameters are erased before reconstruction.Source: §IV · p. 6
  • Six robot models spanning a range of DoF are evaluated, with AutoURDF added for kinematics reconstruction because the other baselines lack explicit tree structures.ArtGS and Articulate AnyMesh do not expose explicit kinematic tree structure.Source: §IV · p. 6
  • An end-to-end evaluation starts from RGB images and uses ground truth to quantify output discrepancies.Source: §IV · p. 6

A. Metrics

  • The metrics are axis-angle error, axis-position error, and tree edit distance.They measure joint-axis direction deviation, pivot-position distance, and the number of tree edits needed to match kinematic trees.Source: A. Metrics · p. 6

B. Quantitative results

Kinematify improves joint-axis and kinematic-tree reconstruction over prior methods, while end-to-end image inputs incur larger errors on Fetch and Panda than geometry-only inputs.

B. Quantitative results

  • On PartNet-Mobility, Kinematify has the lowest axis angle error among Articulate Anymesh and ArtGS; across six robots, it substantially reduces average Tree Edit Distance.Representative robot visualizations show Unitree Go2 (12 DoF) and Unitree H1 (19 DoF).Source: B. Quantitative results · p. 6, Figure 5 · p. 6

C. End-to-end evaluation

  • For single-RGB end-to-end evaluation, errors rise modestly on EO and more noticeably on Fetch and Panda versus geometry-only results; Table II reports absolute values.Existing baselines do not support image-to-articulated-object evaluation at comparable scope, so the paper reports absolute performance rather than head-to-head comparisons.Source: C. End-to-end evaluation · p. 6, Table II · p. 7

D. Ablation study

Ablations distinguish the contributions of MCTS to tree structure and DW-CAVL to joint parameters, while real-world demonstrations use recovered URDFs for planning and execution. The conclusion identifies segmentation and contact reliability as assumptions.

D. Ablation study

  • Replacing MCTS with BFS increases TED across robots, while removing the DW-CAVL anchor degrades joint parameters; the full model achieves the best balance.BFS lacks long-range regularization, and without the anchor term, optimized axes can drift from true pivots while reducing interpenetration.Source: D. Ablation study · p. 7

E. Real-world robot manipulation

  • Recovered Fetch and cabinet URDFs supported drawer-opening and cup-pouring tasks, with planned trajectories executed without collision in Isaac Sim and on hardware.The URDFs were loaded into a MoveIt planning scene; the drawer-opening demonstration transferred to the real robot.Source: E. Real-world robot manipulation · p. 7, Figure 6 · p. 7

V. CONCLUSION

  • Across everyday objects and robot platforms, Kinematify improves joint estimation accuracy and kinematic tree fidelity over prior work.The authors characterize it as a step toward open-vocabulary synthesis of high-DoF articulated structures.Source: §V · p. 7
  • Kinematify assumes accurate part segmentation and a reliable contact graph; spurious seams, missed contacts, or decorative geometry can mislead topology inference.The authors propose jointly refining segmentation and contact reliability during structure inference as future work.Source: §V · p. 7
Loading 2511.01294v4…