Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model

• Existing static-object libraries contain over 10 million objects, whereas open articulated-object libraries contain only around 2,300 objects because annotation is labor-intensive and expertise-demanding.Source: §1 · p. 2
• Articulate-Anything uses a vision-language actor-critic system to articulate objects from text, images, and videos through iterative feedback.Source: §1 · p. 2
• Grounded inputs let the system resolve ambiguous affordances, such as determining that a window should slide rather than tip from an in-the-wild video.Source: §1 · p. 2
• The system synthesizes high-level Python code that compiles into URDF files, while a vision-language critic evaluates rendered predictions against ground truth.Source: §1 · p. 2
• 75% success versus 8.7–12.2% for prior work demonstrates substantially improved articulation quality, accuracy, and generalizability.Source: §1 · p. 2

• Existing 3D repositories largely lack articulation, while PartNet-Mobility and GAPartNet provide manually annotated links and joints at modest scale.Source: §2 · p. 2
• Articulated-object modeling includes perception, reconstruction from images or point clouds, and generation using connectivity graphs or neural methods.Source: §2 · pp. 2–3
• The paper seeks an end-to-end pipeline from perception through reconstruction and generation using language or videos instead of specialized point clouds or graphs.Source: §2 · p. 3

• URDF represents objects as hierarchical trees of links and joints, with prismatic joints translating and revolute joints rotating parts.Source: §3 · p. 3
• The objective constructs a predicted URDF with the same links as the ground truth while minimizing differences between the two models.Source: §3 · p. 3
• The loss combines mesh appearance, link pose, and joint discrepancies, including joint type, axis, origin, and motion-limit terms.Source: §3 · p. 3
• Because jointly optimizing the objective is intractable, the system sequentially performs mesh retrieval, link placement, and joint prediction.Source: §3 · p. 3
• Vision-language agents approximate losses from visual inputs and iteratively refine code-generated solutions using critic feedback when ground-truth access is limited.Source: §3 · p. 3

4 ARTICULATED OBJECT GENERATION VIA ACTOR-CRITIC VLMS

• The system decomposes articulation into mesh retrieval, link placement, and joint prediction, with optional targeted affordance extraction.It produces high-level Python code compiled into URDFs rather than directly generating verbose URDF/XML.Source: §4 · p. 3, Figure 2 · p. 4

4.1 MESH RETRIEVAL

• Visual inputs are matched to library objects through CLIP-guided category narrowing, VLM selection, and divide-and-conquer retrieval.Video motion cues assist object identification, while candidate selection proceeds through recursive tournament comparisons.Source: §4.1 · p. 4, Figure 3 · p. 5
• Text inputs are expanded into part descriptions and dimensions, then matched to mesh descriptions with CLIP language embeddings and rescaled.A TaskSpecifier densifies the prompt before a LayoutPlanner describes the parts and their dimensions.Source: §4.1 · pp. 4–5

4.2 LINK PLACEMENT

• The link-placement actor writes spatial arrangement code, while a critic compares rendered models with images, rates realism, and triggers revisions below a threshold of 5.Placement also uses link-center alignment and collision checks to avoid intersections.Source: §4.2 · p. 5, Figure 4 · p. 6

4.3 JOINT PREDICTION

• The joint-prediction actor extends placement code with kinematic joints, while video-based criticism identifies failure cases and guides rewrites below a realism threshold of 5.The critic compares simulated joint motion with the input video and categorizes errors from joint type through joint limit.Source: §4.3 · p. 5, Figure 4 · p. 6

4.4 TARGETED AFFORDANCE EXTRACTION

• Targeted affordance extraction selects a specific child link for annotation from an input video and color-segmented simulated asset.This optional mode reduces the number of joints considered and can lower VLM input-token costs.Source: Figure 2 · p. 4, §4.4 · pp. 5–6

5 EXPERIMENTS

• The benchmark masks link and joint information from PartNet-Mobility URDFs and evaluates reconstructed link placement and joint prediction with success-rate metrics.Joint success requires joint type, axis, origin, and limit errors to remain within thresholds; link placement uses 50mm and 0.25-radian thresholds.Source: §5 · p. 6

5.1 HOW WELL DOES ARTICULATE-ANYTHING PERFORM VERSUS PRIOR WORK?

• ARTICULATE-ANYTHING significantly outperforms all baselines in joint prediction across the reported classes.Figure 5 reports all classes using few-shot prompting and includes 95% confidence intervals.Source: Figure 5 · p. 7, §5.1 · p. 7
• Video-based reconstruction performs best on in-the-wild objects, resolving articulation ambiguities that remain plausible but incorrect from static images or text.The method handles casually captured cluttered inputs, whereas baselines fail under minor misalignment or segmentation inaccuracies.Source: §5.1 · p. 7, §5.2 · p. 7, Figure 7 · p. 8

5.2 HOW DO DIFFERENT INPUT MODALITIES AFFECT ARTICULATION ACCURACY?

• Richer grounded modalities consistently improve articulation success rates, with videos used for joint prediction and first-frame images used for link placement.The ablation compares text, image, and video inputs on Faucet link placement and StorageFurniture joint prediction using an actor-only one-iteration system.Source: §5.2 · pp. 7, 9, Figure 6 · p. 7

5.3 CAN ARTICULATE-ANYTHING EVALUATE AND REFINE ITS OWN PREDICTIONS?

• Iterative critic refinement improves accuracy by around 5.8% for link placement and 2.4% for joint prediction.Critic evaluations correlate highly with human ground truth but tend to overestimate success.Source: §5.3 · p. 9, Figure 9 · p. 9
• Success rates improve with more in-context prompting examples, while performance remains high across different leading VLM choices.These findings come from an actor-only StorageFurniture ablation using input videos.Source: §5.4 · pp. 9–10, Figure 10 · p. 9, Figure 11 · p. 10

• 100% success rate was achieved in the real world by policies trained on both ARTICULATE-ANYTHING and human-annotated assets.Source: §6 · p. 10
• 40.58 minutes of human labor per task were saved on average using ARTICULATE-ANYTHING instead of manual annotation.Source: §6 · p. 10, Figure 12 · p. 10
• Four finer-grained manipulation tasks were trained with PPO in Robosuite simulation and executed on a real Franka arm.Source: §6 · p. 10, Figure 13 · p. 10

• ARTICULATE-ANYTHING articulates diverse objects from text, images, and videos through vision-language program synthesis and self-improving actor-critic feedback.Source: §7 · p. 10
• The current approach relies on mesh retrieval for high-quality 3D assets, while upstream mesh reconstruction is proposed for more customized assets and broader generality.Source: §7 · p. 10

• Joint prediction failures are visualized across a range ordered from joint type, the most egregious, to joint limit, the least.Source: Figure 14 · p. 14

• Link states represent position in R3 and orientation as a unit quaternion, enabling comparison between predicted and ground-truth link states.Source: §A.1.1 · p. 14
• Orientation error is the geodesic distance on the unit sphere, corresponding to the smallest rotation angle between predicted and ground-truth orientations.Source: §A.1.1 · p. 14
• The total link error averages the position and orientation error components.Source: §A.1.1 · p. 14

A.1.2 JOINT ERROR

• Joint predictions are evaluated in descending severity: type, axis, origin, and limit errors.Failures are assigned in this natural order, with joint type detected first and joint limit last.Source: §A.1.2 · p. 15
• Joint type error is a binary mismatch between the predicted and ground-truth joint types.Source: §A.1.2 · p. 15
• Joint axis error measures the angular difference between predicted and ground-truth axes, ranging from 0 to π radians.The axis represents rotation for revolute joints or translation for prismatic joints.Source: §A.1.2 · p. 15
• Joint origin error measures the shortest distance between revolute joint axes or the Euclidean distance between prismatic-joint origins.For revolute joints, the computation uses the cross product of the predicted and ground-truth axes.Source: §A.1.2 · p. 15
• Joint limit error combines motion-range and motion-direction differences, with direction error ranging from 0 for identical to 2 for opposite directions.Source: §A.1.2 · p. 15

A.2 CRITIC-GROUNDTRUTH AGREEMENT

• The visual critic is highly correlated with ground truth, but its largest disagreement is false-positive success judgments on difficult-to-notice articulation errors.This comparison is shown with confusion matrices for two articulation tasks.Source: §A.2 · p. 15, Figure 15 · p. 16

A.4.1 CASUAL INPUTS

• Casually captured videos in cluttered environments are handled without the extensive manual curation required by URDFormer and Real2Code baselines.The baselines required adjusted bounding boxes, curated segmentation masks, tuned hyperparameters, and other preprocessing.Source: §A.4.1 · p. 16, §A.4 · p. 16

A.4.3 EVALUATING URDFORMER

• ARTICULATE-ANYTHING outperforms prior methods on joint prediction errors, with lower-is-better Type, Axis, and Origin metrics.Table 1 reports average joint prediction errors, with Type as incorrect-joint-type fraction, Axis in radians, and Origin in meters.Source: Table 1 · p. 20, §A.4.3 · p. 20

A.6 ADDITIONAL STATISTICS

• ARTICULATE-ANYTHING has the smallest average origin and axis errors among compared methods.Category breakdowns show that small parts and exotic irregular movements are more difficult, while camera zoom is suggested as a possible optimization.Source: Figure 20 · p. 20, §A.6 · p. 20, Figure 21 · p. 21

A.5 COMPARABLE INPUT MODALITIES

• ARTICULATE-ANYTHING achieves higher accuracy than prior work under the same impoverished input modalities.The comparison attributes the advantage to foundation models’ common-sense and high-level planning capabilities.Source: §A.5 · p. 20, Figure 19 · p. 20

A.7 MESH RECONSTRUCTION

• For in-the-wild mesh reconstruction, mesh generation integrated with ARTICULATE-ANYTHING is reported as better than all baselines, while retrieval also surpasses prior work including Real2Code.The evaluation uses Chamfer distance and excludes the evaluated object from the retrieval candidate pool on PartNet-Mobility.Source: §A.7 · p. 23, Table 2 · p. 23, Table 3 · p. 23

A.8 GENERATING NEW ASSETS

• The current system relies on mesh retrieval from existing 3D datasets, while large-scale mesh generation is identified as a future direction for more customized assets.Source: §A.8 · p. 23, Figure 23 · p. 23
• The asset-generation extension selects a mesh-generation model, segments the moving part using video-conditioned VLM guidance and Grounded SAM, then lifts multiview masks into 3D.The resulting segmentation supports articulation of generated meshes with realistic movement in simulation.Source: §A.8 · pp. 23–24, Figure 24 · p. 24, Figure 25 · p. 24