PaperlayerView exact revision on arXiv

Source-linked AI summary

Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modeling

Xiaowen Qiu, Jincheng Yang, Yian Wang, Zhehuan Chen, Yufei Wang, Tsun-Hsuan Wang, Zhou Xian, Chuang Gan

arXiv:2502.02590v2cs.CVcs.RO

TL;DR

Existing methods struggle to produce diverse articulated objects in an open-vocabulary setting because they rely on limited training categories or omit motion parameters. Articulate Anymesh segments parts and estimates joints from geometric cues, outperforming previous approaches in joint-parameter estimation in both in-domain and out-of-domain settings.

  • Problem

    Existing methods rely on limited articulated-object training categories or produce objects without the joint parameters needed for a broad range of articulated objects.

    Source: §1 · p. 2

  • Method

    Articulate Anymesh segments movable parts semantically, prompts GPT-4o with geometric cues to estimate joints, and optionally completes geometry and texture for surface meshes.

    Source: §1 · pp. 2–3

  • Results

    Articulate Anymesh outperformed previous approaches in joint parameter estimation in both in-domain and out-of-domain settings, while matching state of the art on trained PartNet-Mobility categories and modeling broader objects.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The resulting assets support manipulation-policy learning in simulation and can be created from generated, handcrafted, or reconstructed meshes.

    Source: §1 · p. 3, §5 · p. 8

    Articulation parameters are not always accurate or physically grounded, owing to mesh structures, segmentation errors, GPT-4o biases, and uncommon cases such as front-pulled drawers.

    Source: §6 · p. 9

Abstract

from arXiv · show

3D articulated objects modeling has long been a challenging problem, since it requires to capture both accurate surface geometries and semantically meaningful and spatially precise structures, parts, and joints. Existing methods heavily depend on training data from a limited set of handcrafted articulated object categories (e.g., cabinets and drawers), which restricts their ability to model a wide range of articulated objects in an open-vocabulary context. To address these limitations, we propose Articulate Anymesh, an automated framework that is able to convert any rigid 3D mesh into its articulated counterpart in an open-vocabulary manner. Given a 3D mesh, our framework utilizes advanced Vision-Language Models and visual prompting techniques to extract semantic information, allowing for both the segmentation of object parts and the construction of functional joints. Our experiments show that Articulate Anymesh can generate large-scale, high-quality 3D articulated objects, including tools, toys, mechanical devices, and vehicles, significantly expanding the coverage of existing 3D articulated object datasets. Additionally, we show that these generated assets can facilitate the acquisition of new articulated object manipulation skills in simulation, which can then be transferred to a real robotic system. Our Github website is https://articulate-anymesh.github.io.

1 Introduction

Existing approaches struggle to create diverse articulated objects in open vocabulary because they depend on limited categories or lack joint parameters. Articulate Anymesh converts 3D meshes into articulated assets by combining semantic part segmentation with geometry-guided joint estimation.

1 Introduction

  • The pipeline converts generated or handcrafted 3D meshes into open-vocabulary articulated assets without relying on existing labeled articulated-object data.It addresses the need for semantic structures and articulation parameters, alongside geometry and appearance.Source: §1 · p. 2
  • Movable Part Segmentation adopts PartSlip++; Articulation Estimation prompts GPT-4o with connected-part geometry to infer joint parameters for unseen objects.The authors state that performance can improve as vision-language models advance.Source: §1 · pp. 2–3
  • An optional post-processing stage uses 3D generative models to complete missing geometry and generate texture for surface meshes.Surface meshes can yield occluded or incomplete parts, such as a microwave whose internal cavity is absent.Source: §1 · p. 3
  • Articulate Anymesh outperformed previous approaches in joint-parameter estimation in both in-domain and out-of-domain settings, matched state of the art on trained PartNet-Mobility categories, and modeled broader objects.The experiments also used manipulation-policy learning to demonstrate the usefulness of collected data.Source: §1 · p. 3
  • Manipulation-policy learning experiments demonstrate the usefulness of data collected with the pipeline.The pipeline is intended to create a wider range of articulated objects than existing datasets cover.Source: §1 · p. 3

2 Related Work

Prior articulated-object perception and generation methods rely on observations or training datasets, limiting open-vocabulary creation. Part-aware generators add structural detail but do not provide the joint parameters needed for articulated manipulation.

2.1 Articulated object modeling

  • Perception methods reconstruct or estimate articulation from point clouds, interaction, multiview images, or single images, while generation methods use graph diffusion; existing approaches remain data-dependent and do not automatically create open-vocabulary articulated objects.Perception methods may require extensive observations of an object or rely on datasets spanning only a small range of categories.Source: §2.1 · p. 3

2.2 Part aware 3D generation

  • Part-aware generators use approaches including VAEs, graphs, sequence models, and diffusion to represent structure and part geometry, but often depend on annotated datasets and omit joint parameters.Without articulation parameters, parts are not individually manipulatable in simulation, limiting downstream robot-learning applications.Source: §2.2 · pp. 3–4

3 Method

Articulate AnyMesh converts meshes into articulated objects through movable-part segmentation, articulation estimation, and optional post-processing. It uses open-vocabulary visual prompting and geometric cues to estimate parts and joints.

3.1 Overview

  • The pipeline converts meshes into articulated assets through movable-part segmentation, geometry-aware articulation estimation, and optional post-processing.Post-processing enhances geometry and texture, while articulation estimation leverages geometric cues and foundation-model prior knowledge.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Movable Part Segmentation via VLM assistant

  • A VLM identifies movable parts to label; PartSlip++ grounds those labels in rendered views and fuses multi-view detections into 3D segmentation.Source: §3.2 · p. 4

3.3 Articulation Estimation via Geometry-aware Visual Prompting

  • The method defines a connecting area from nearby points on adjacent parts, using it as geometric evidence for estimating joints.If the distance threshold finds no points, it is doubled until a connecting area is identified.Source: §3.3 · p. 5
  • For revolute joints, clustered connecting-area points are rendered and GPT-4o selects hinge points that define the rotation axis.For a partly hidden hinge with one visible endpoint, the axis uses the connecting-plane normal and that point; hidden structures also prompt sampling candidate points across the part surface.Source: §3.3 · p. 5
  • Prismatic joints are classified as inward/outward, using the connecting-plane normal, or surface-sliding, whose direction GPT-4 selects from annotated 2D arrows.The method addresses surface meshes' lack of ideal internal geometry and the frequent concealment of sliding mechanisms.Source: §3.3 · pp. 5–6

3.4 Geometry and Texture Post-Processing

  • Optional post-processing completes incomplete surface-mesh geometry and generates textures for the completed parts.The pipeline uses HoloPart for shape completion and Meshy for texture generation.Source: §3.4 · p. 6

4 Experiments

Experiments apply ARTICULATE ANYMESH to generated, hand-crafted, and reconstructed meshes, evaluate joint estimation, and test generated assets in simulation and real-world manipulation. The applications include object generation, dataset annotation, trajectory transfer, and policy learning with augmented training data.

4 Experiments

  • Implementation uses PartSlip++ segmentation with set-of-mark prompting for grounding, GPT-4o for part labels and joint configurations, HoloPart for shape completion, and Meshy for texture generation.Baselines lack a direct counterpart that converts the same input meshes into articulated outputs, so comparisons use joint-estimation methods trained on seen categories.Source: §4 · p. 6

4.1 Quantitative Experiments

  • The evaluation compares ARTICULATE ANYMESH with URDFormer and Real2Code on PartNet-Mobility joint estimation, measuring angular error and distance between joint lines.Table 1 covers joint angle and position error; the supplied passages do not report its numerical results.Source: §4 · p. 6, §4.1 · p. 6, Table 1 · p. 8

4.2 Applications

  • Combining Meshy surface meshes with ARTICULATE ANYMESH produces high-quality articulated objects across varied categories.Figure 3 shows raw surface meshes, completed geometry, and textured objects in two poses.Source: Figure 3 · p. 7, §4.2 · p. 8
  • The pipeline also annotates hand-crafted Objaverse meshes with part segmentation and joint structures, using their existing geometry and textures.Because these objects include inner structures, this application uses only segmentation and articulation estimation.Source: Figure 4 · p. 7, §4.2 · p. 8
  • Trajectories for pushing a drill trigger, opening a microwave door, and rotating a steering wheel transfer successfully from simulation to the real world, with minimal part-segmentation and joint-prediction error.The pipeline reconstructs real objects, converts meshes to URDF, plans trajectories in simulation, and replicates them in the real world.Source: Figure 5 · p. 7, §4.2 · p. 8
  • Augmenting DexArt training with ARTICULATE ANYMESH objects effectively enhances robot learning on the augmented test set.The evaluation covers opening a laptop lid and lifting a bucket; Table 2 compares fine-tuning on original versus augmented training sets.Source: §4.2 · p. 8, Table 2 · p. 8

5 Conclusion

ARTICULATE ANYMESH converts open-vocabulary 3D meshes into articulated counterparts by segmenting parts, estimating their articulation, and refining geometry and texture. It produces assets from generated, hand-crafted, and reconstructed meshes that can support simulation-based manipulation learning and real-world robot transfer.

5 Conclusion

  • ARTICULATE ANYMESH converts open-vocabulary 3D meshes into articulated counterparts through semantic part segmentation, articulation estimation, and geometry and texture refinement.The pipeline uses vision-language models, visual prompting, and common geometric cues to segment parts and estimate articulation structures.Source: §5 · p. 8
  • The pipeline creates high-quality textured articulated objects from generated, hand-crafted, and reconstructed meshes.Source: §5 · p. 8
  • The resulting assets can support articulated-object manipulation learning in simulation, with learned skills transferred to real-world robots.Source: §5 · p. 8

6 Limitations

The method can produce semantically correct parts even when articulation parameters are inaccurate or not physically grounded. The authors attribute this limitation to mesh structure, segmentation errors, and GPT-4o biases, and note that some uncommon cases are not covered.

6 Limitations

  • Semantically correct parts do not guarantee accurate, physically grounded articulation parameters.The authors cite incorrect physical structures in meshes, occasional segmentation errors, and GPT-4o biases as contributing challenges.Source: §6 · p. 9
  • Some uncommon cases fall outside coverage, including drawers designed to pull out up-front.Source: §6 · p. 9
  • Enhanced 3D reasoning in future large VLMs is a proposed route to greater pipeline robustness and reliability.The authors identify fully accurate and physically grounded articulation parameters as future work.Source: §6 · p. 9

A More quantitative results on joint parameter estimation

Additional comparisons assess joint-parameter estimation from single observations and ground-truth part meshes. Articulate AnyMesh outperforms the compared methods in the reported single-observation and held-out-category settings.

Compared methods

  • The comparison spans generative methods NAP and CAGE and discriminative methods ANCSH, OPD, and OPDmulti.NAP and CAGE synthesize articulated objects using diffusion models; the discriminative methods jointly predict part segmentation and articulation parameters from an observation.Source: A More quantitative results on joint… · p. 14

Single-observation comparison

Part-mesh comparison

  • With ground-truth part meshes, all three methods perform comparably in-domain, while Articulate AnyMesh substantially outperforms NAP and CAGE on held-out categories.Table 4 distinguishes in-domain categories included in CAGE’s training set from out-of-domain PartNet-Mobility categories not included in that training set.Source: Table 4 · p. 14, A More quantitative results on joint… · p. 14

VLM comparison

  • Table 5 compares joint-estimation angle and position error using different vision-language models.Source: Table 5 · p. 15

B Ablation study of VLM

The VLM ablation finds a small performance gap among GPT-4o, Claude 3.7 Sonnet, and Gemini 2.0 Flash, with all three outperforming URDFormer and Real2Code in both in-domain and out-of-domain settings.

VLM ablation

  • GPT-4o slightly outperforms Claude 3.7 Sonnet and Gemini 2.0 Flash, but all three significantly outperform URDFormer and Real2Code in both in-domain and out-of-domain settings.Source: B Ablation study of VLM · p. 15

C Implementation of policy learning experiment

The policy-learning experiment evaluates bucket lifting and laptop opening using DexArt’s original data and additional samples generated by Articulate AnyMesh. Policies are tested on both original and augmented test sets.

Tasks

  • The two DexArt tasks are lifting a bucket by its handle and opening a laptop lid to 90 degrees.The bucket handle is thickened in the original implementation to prevent severe penetration, and the experiment retains that setting.Source: C Implementation of policy learning… · p. 15

Training and evaluation

  • Policies are fine-tuned from DexArt’s no-pretrain checkpoint using either the original training samples alone or those samples combined with Articulate AnyMesh-generated Objaverse samples.The baseline uses the original dataset for the same number of fine-tuning steps.Source: C Implementation of policy learning… · p. 15
  • New policies were tested on both original and additional test samples; 100 evaluations on the test set included all test set objects with randomized position and rotation, and final success rate was averaged across evaluations.Figures 6, 7, and 8 visualize the datasets used.Source: C Implementation of policy learning… · p. 15

Dataset visualizations

Loading 2502.02590v2…