PaperlayerView exact revision on arXiv

Source-linked AI summary

DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects

Chen Bao, Helin Xu, Yuzhe Qin, Xiaolong Wang

arXiv:2305.05706v1cs.ROcs.CVcs.LG

TL;DR

DexArt addresses the challenge of generalizing dexterous manipulation across diverse, unseen articulated objects. It benchmarks reinforcement learning with 3D point-cloud representations and finds that object diversity, encoder size, and part-focused pre-training shape policy performance.

  • Problem

    Dexterous manipulation of diverse articulated objects remains challenging because hand and object joints have high degrees of freedom, and generalization to unseen objects is a major bottleneck for reinforcement learning.

    Source: §1 · pp. 1–2

  • Method

    DexArt benchmarks an Allegro Hand across tasks, training on diverse articulated objects and testing on unseen objects with RL policies that use PointNet-encoded 3D point clouds.

    Source: §1 · p. 2

  • Results

    More training objects improve unseen-object generalization, the simplest PointNet achieves the best sample efficiency and generalizability, and part-segmentation pre-training benefits manipulation, especially of small functional parts.

    Source: §1 · p. 2, §5.1 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    Part-focused visual representations can support manipulation of delicate functional parts, while policies trained with partial point clouds remain resilient to camera-pose variation.

    Source: §1 · p. 2, §5.1 · p. 7

Abstract

from arXiv · show

To enable general-purpose robots, we will require the robot to operate daily articulated objects as humans do. Current robot manipulation has heavily relied on using a parallel gripper, which restricts the robot to a limited set of objects. On the other hand, operating with a multi-finger robot hand will allow better approximation to human behavior and enable the robot to operate on diverse articulated objects. To this end, we propose a new benchmark called DexArt, which involves Dexterous manipulation with Articulated objects in a physical simulator. In our benchmark, we define multiple complex manipulation tasks, and the robot hand will need to manipulate diverse articulated objects within each task. Our main focus is to evaluate the generalizability of the learned policy on unseen articulated objects. This is very challenging given the high degrees of freedom of both hands and objects. We use Reinforcement Learning with 3D representation learning to achieve generalization. Through extensive studies, we provide new insights into how 3D representation learning affects decision making in RL with 3D point cloud inputs. More details can be found at https://www.chenbao.tech/dexart/.

1. Introduction

DexArt addresses the challenge of generalizing dexterous manipulation across unseen articulated objects, where both hand and object degrees of freedom complicate learning. It benchmarks multi-object tasks and examines how 3D visual representations affect policy learning and decision-making.

1. Introduction

  • Earlier benchmarks focused on single objects per task or used parallel grippers, while articulated-object manipulation adds high-DoF challenges and generalization to unseen objects.These limitations motivate a benchmark combining multi-finger dexterity with diverse articulated objects.Source: §1 · pp. 1–2
  • DexArt trains Allegro-hand policies on diverse articulated objects and evaluates generalization on a separate set of unseen objects using point-cloud observations and PointNet representations.The benchmark is intended to support study of generalizable dexterous skills and visual perception for decision-making.Source: §1 · p. 2
  • Training with more objects consistently improves success rates on unseen objects, despite making it harder for one RL policy to handle different objects simultaneously.Policies were trained with varying numbers of objects and tested on the same unseen set.Source: §1 · p. 2
  • The simplest PointNet achieves the best sample efficiency and success rate, while larger encoders can make RL optimization more challenging [41].The result holds whether or not the encoder is pretrained.Source: §1 · p. 2
  • Object-part segmentation pretraining significantly improves sample efficiency and success rate over approaches without pretraining and with other pretraining methods.The authors motivate part reasoning by the hand's interactions with different object parts.Source: §1 · p. 2
  • Policies trained with partial point clouds remain accurate under large camera-viewpoint variations, a property useful when aligning cameras between simulation and reality.The authors report this robustness as consistent with prior studies using complete point clouds [32].Source: §1 · p. 2
  • The benchmark is also proposed as a platform for studying how visual perception can be improved for better decision-making.Source: §1 · p. 2

2. Related Work

Prior work studies dexterous manipulation, articulated-object perception, and point-cloud representations, but the cited approaches leave open how 3D representation learning supports closed-loop policy decisions for articulated manipulation.

2. Related Work

  • Many dexterous manipulation methods use planning and trajectory optimization requiring tuned dynamics models, while learned methods often focus on single-body objects rather than articulated ones.The paper positions its benchmark as addressing generalizable manipulation of articulated objects with point-cloud observations.Source: §2 · p. 2
  • Articulated-object research includes perception of pose, joints, parts, and dynamics, alongside model-based control and planning.Source: §2 · p. 2
  • Some approaches learn actionable information from raw sensory input but use single-step actions with predefined open-loop controllers; DexArt instead studies sequential decisions with visual feedback in closed-loop control.Source: §2 · p. 3
  • Point-cloud representations have supported 3D vision and robotic tasks, while DexArt examines their role in high-DoF hand-and-object manipulation and decision-making.Source: §2 · p. 3

3. DexArt Benchmark

DexArt defines four articulated-object tasks for evaluating policy sample efficiency and generalizability, with seen and unseen objects in each task. The benchmark specifies a simulated Allegro/XArm setup and a staged reward design.

3. DexArt Benchmark

  • DexArt provides four tasks of varying difficulty for evaluating policy sample efficiency and generalizability across seen and unseen articulated objects.The tasks are Faucet, Bucket, Laptop, and Toilet.Source: §3 · p. 3

3.1. Task Description

  • The tasks test faucet rotation, bucket lifting, laptop opening, and toilet-lid opening, each with a task-specific success criterion.Success is measured by faucet handle angle, bucket lift height, laptop lid angle, or reaching a threshold toilet-lid angle.Source: §3.1 · p. 3

3.2. Environment Setup

  • The simulator uses a 6-DoF XArm6 with a 16-DoF Allegro hand and proprioceptive plus partial point-cloud observations.The 22-dimensional action combines target palm linear and angular velocity with target positions for the hand's 16 joints; both controllers use PD control.Source: §3.2 · pp. 3–4

3.3. Reward Design

  • Rewards are organized around reaching, constructing hand-object contact, and manipulating the articulated part toward its task-specific goal.The design aims for dense, natural, safe, and standardized rewards across tasks.Source: §3.3 · p. 4
  • The reach reward uses palm-object distance, while contact shaping encourages the palm and at least two fingers to touch the object.The contact check uses collision detection between links; distance alone may encourage undesirable motions such as opening a laptop with a clenched hand.Source: §3.3 · p. 4
  • Task progress is rewarded in the final stage, and an action-norm plus task-specific penalty discourages jerky or unstable motion.For the Faucet task, progress is measured by change in the handle joint angle.Source: §3.3 · p. 4

3.4. Asset Selection and Annotation

  • PartNet-Mobility models are selected and annotated per object, and training randomizes object positions while keeping goals achievable.Scale and initial position are annotated to ensure reasonable sizes and prevent initial intersections with the robot.Source: §3.4 · p. 4

4. Method

DexArt uses PPO with PointNet point-cloud features for dexterous manipulation, with perception pre-training to initialize the RL visual backbone. It also augments observed points with robot geometry and evaluates supervised and self-supervised pre-training methods.

4.1. Policy Overview

  • PPO combines PointNet point-cloud features with robot proprioception, sharing the extracted features between policy and value networks.The simple PointNet uses a one-hidden-layer local MLP and max pooling; the paper reports that increasing extractor size harms policy learning.Source: §4.1 · pp. 4–5

4.3. Pre-training Methods

  • Perception-only PointNet pre-training initializes the PPO visual backbone before RL training.The evaluated pre-training methods include supervised segmentation and classification, and self-supervised reconstruction and SimSiam.Source: §4.1 · p. 5, Figure 3 · p. 5, §4.3 · p. 6

4.1. Policy Overview

  • The policy receives observed point clouds plus imagined finger points computed from the robot model through forward kinematics.The imagined points supply robot geometry that may be occluded or underrepresented in low-resolution observations.Source: §4.1 · p. 5

4.3. Pre-training Methods

  • Reconstruction pre-training uses a PointNet encoder and PCN decoder on DAM, with Chamfer loss, and transfers the encoder to PPO.SimSiam trains a PointNet on two augmented views of a point cloud, predicting similarity on one branch while stopping gradients on the other.Source: §4.3 · p. 6

5. Experiment

Across four tasks, part-segmentation pre-training performs best, with particular gains for delicate tasks involving small functional parts. Ablations report benefits from more training objects, smaller PointNet encoders, and 3D representations; the PointNet policy also remains robust to viewpoint changes.

5.1. Main Results

  • Part-segmentation pre-training performs best on all four tasks, especially for delicate manipulation of small functional parts.The authors report that segmentation identifies small faucet handles, whereas reconstruction emphasizes global shape and misses small-part detail.Source: §5.1 · pp. 6–7

5.2. Ablation Study

  • Training with 100% rather than around 50% of seen objects maintains higher unseen-object success rates, although convergence by episodic return is slower.The comparison uses segmentation-pretrained policies; the figure caption specifies three random seeds for unseen-object success rates.Source: Figure 8 · p. 7, Figure 6 · p. 7
  • The smallest PointNet achieves the best success rate and episodic return, while training faster and generalizing better than larger encoders.The compared encoders have one, three, and five hidden layers.Source: Figure 7 · p. 7, §5.2 · p. 8
  • On Laptop, PointNet-based 3D representation learning outperforms the R3M-pretrained ResNet-18 on both seen and unseen objects.The comparison is between DAM-segmentation-pretrained PointNet and R3M, which uses Ego4D pre-training.Source: §5.2 · p. 8, Table 3 · p. 8

5.3. Robustness to Viewpoint Change

  • The PointNet policy remains robust with viewpoint shifts up to 60° azimuth and 20° polar angle, while ResNet-18 success drops sharply as viewpoints diverge.The viewpoint evaluation is conducted on the Laptop task.Source: §5.3 · p. 8, Figure 9 · p. 8

6. Conclusion

DexArt benchmarks dexterous manipulation of articulated objects and studies RL policy generalizability. Its experiments find that more training objects, simple PointNet encoders, part segmentation, and geometric representation learning support generalization, sample efficiency, or viewpoint robustness within the tested settings.

6. Conclusion

  • DexArt provides four tasks—Faucet, Bucket, Laptop, and Toilet—with seen and unseen objects to evaluate sample efficiency and policy generalizability.The conclusion reports that more training objects consistently improve unseen-object performance, while the simplest PointNet offers the best sample efficiency and generalizability.Source: §6 · p. 8
  • Part segmentation benefits manipulation of small functional parts, while all visual pre-training methods benefit tasks with larger functional parts.The conclusion also reports strong camera-viewpoint robustness from PointNet geometric representation learning.Source: §6 · p. 8

Related papers

Loading 2305.05706v1…