PaperlayerView exact revision on arXiv

Source-linked AI summary

Uni-PNP: Unified Prehensile and Non-prehensile Policy Learning for Generalizable Object Pose Reaching

Wenxuan Zhu, Yuze Wang, Jiangran Lyu, Jiayi Chen, Yixin Zheng, Mi Yan, Li Yi, Zhizheng Zhang, He Wang

arXiv:2610.21735v1cs.RO

TL;DR

Prior approaches commonly separate prehensile and non-prehensile skills, and their integration can depend on object, gripper, and environmental geometry. Uni-PNP learns both modes jointly using contact-motion prediction pretraining and a two-stage reinforcement-learning curriculum. It reports 76.39% success on training objects and grippers and 57.91% on unseen objects with the unseen Panda gripper.

  • Problem

    Prior work typically develops prehensile and non-prehensile capabilities separately, while methods are also sensitive to object, gripper, and environmental geometry.

    Source: §I · p. 2

  • Method

    Uni-PNP pretrains a geometry encoder to predict contact-induced object motion, then trains a unified policy with a two-stage reinforcement-learning curriculum from graspable to challenging objects.

    Source: §I · p. 2

  • Results

    76.39% pose-reaching success on training objects and grippers, and 57.91% on unseen objects with the unseen Panda gripper, outperforming pose-reaching baselines across all settings.

    Source: §I · p. 2

  • Takeaways & Limitations

    Uni-PNP exhibits coordinated repositioning, grasp acquisition, and table-assisted reorientation in simulation and real-world trials.

    Source: §V · p. 7

    The study focuses on tabletop interactions, two-finger grippers, and basic object pose reaching, leaving cluttered environments, multifinger hands, and semantic tasks outside its scope.

    Source: §V · p. 7

Abstract

from arXiv · show

Object manipulation often requires combining non-prehensile actions with grasping. However, existing approaches typically develop these capabilities separately and integrate them by sequencing distinct primitives through prescribed intermediate objectives, limiting the coordination of contact motions and grasping. Their integration also remains sensitive to object and gripper geometry and depends on particular environmental supports, complicating real-world deployment. We present Uni-PNP, a unified policy that jointly learns prehensile and non-prehensile manipulation for object pose reaching across diverse objects and grippers. To achieve this, we first pretrain a geometry encoder to predict contact-induced object motion using a contact-dynamics dataset constructed from simulated interactions between diverse objects and grippers. We then train the policy in a two-stage reinforcement learning curriculum from graspable objects to ungraspable peers, encouraging the discovery of non-prehensile interactions and their integration with grasping. Experiments show 76.39% success on hundreds of seen objects with grippers, and 57.91% on unseen peers, outperforming the strongest pose-reaching baseline by 20.52 and 17.79 percentage points, respectively. We also analyze emergent behaviors, including nonprehensile and prehensile repositioning, grasp acquisition that requires grasping after tipping, and table-assisted reorientation while grasping, and transfer to the real-world scenario. Project page available at lgswdn.github.io/Uni-PNP.

I. INTRODUCTION

Uni-PNP jointly learns prehensile and non-prehensile manipulation for object pose reaching across diverse objects and grippers. It combines contact-motion prediction pretraining with a two-stage learning curriculum and outperforms pose-reaching baselines across the reported settings.

Motivation

  • Prior work develops prehensile and non-prehensile capabilities separately, then prepares objects for grasping or plans sequences of distinct manipulation primitives.Treating the two modes as separate skills limits learning unified manipulation behaviors.Source: §I · p. 2
  • Prior methods are sensitive to environment, object, and gripper geometry, including dependence on suitable supports and gripper geometries seen during training.Geometry affects contact interactions and resulting object motions, not only grasp-pose selection.Source: §I · p. 2

Approach

  • Uni-PNP jointly learns prehensile and non-prehensile object-pose reaching across diverse objects and grippers using motion-prediction pretraining and a two-stage reinforcement-learning curriculum.The curriculum starts with smaller, readily graspable objects before challenging configurations encourage non-prehensile interactions combined with grasping.Source: §I · p. 2

Results

  • 76.39% pose-reaching success on training objects and grippers, and 57.91% on unseen objects with the unseen Panda gripper, outperforming pose-reaching baselines across all settings.The evaluation includes 1,218 challenging-set objects, 249 unseen objects, and grippers drawn from a training catalog of 200 geometries.Source: §I · p. 2

Approach

  • A contact-dynamics dataset and motion-prediction pretraining provide a compact geometry representation for the policy.The dataset records object motions resulting from sampled object–gripper contact configurations under gripper actions.Source: §I · p. 2

Evaluation

  • The paper reports generalization to unseen objects and grippers in simulation and real-world experiments.The benchmarks are complemented by analysis of emergent skills including tip-assisted grasping and table-assisted reorientation.Source: §I · p. 2

II. RELATED WORK

Prior work studies non-prehensile repositioning, grasp-oriented preparation, and explicit skill integration, while geometry-representation methods differ in whether they supervise contact-induced motion.

A. Learning-based Non-Prehensile Manipulation

  • Non-prehensile learning methods use geometric observations, contact-aware representations, or interaction learning for repositioning, but do not secure objects for stable lifting.Grasping and post-grasp repositioning therefore remain separate objectives in most existing systems.Source: A. Learning-based Non-Prehensile… · p. 2

B. Combining Prehensile and Non-Prehensile Manipulation

  • Grasp-preparation methods use pushes, affordances, or environmental supports, while skill-composition and contact-mode approaches coordinate manipulation through planned skills or contacts.Skill-library approaches are constrained by available primitives and transition conditions; Uni-PNP instead learns coordinated arm and gripper actions through a unified policy.Source: B. Combining Prehensile and… · p. 2

C. Geometry Representations for Contact-Rich Control

  • Geometry representation methods range from point-cloud features and self-supervised geometric prediction to contact and dynamics prediction, with differing supervision for induced motion.Contact prediction identifies where contact occurs, whereas DAPL incorporates action-conditioned scene-motion prediction and refines its encoder using policy-collected trajectories.Source: C. Geometry Representations for… · pp. 2–3

III. METHOD

Uni-PNP targets object pose reaching across two-finger grippers by conditioning an actor–critic policy on a geometry encoder pretrained to predict contact-induced motion. A curriculum progresses from ordinary grasping scenes to challenging configurations.

Method overview

  • Object pose reaching requires combining non-prehensile contacts for repositioning or reorientation with prehensile actions that secure the object for final motion.The stated scope is arbitrary stable initial poses, specified target poses, and different two-finger grippers.Source: §III · p. 3
  • A pretrained geometry encoder predicts object motion induced by gripper contact under varying physical conditions, and its representation conditions an actor–critic policy.The policy is optimized with a curriculum from ordinary grasping scenes to challenging object configurations.Source: §III · p. 3

Contact-dynamics dataset

  • Dataset generation samples object shapes, rotations, and surface contacts alongside gripper fingertip contacts and orientations, retaining collision-free configurations.The algorithm header identifies centered object and gripper shapes and a contact-dynamics dataset.Source: §III · p. 3

A. Representation Pretraining

Uni-PNP pretrains a geometry encoder on simulated object–gripper contacts to predict induced object motion, then uses the learned representation in policy learning.

A. Representation Pretraining

  • The pretraining data pairs object–gripper geometry and physical conditions with the object motion induced by gripper actions.Objects and generated grippers are placed in sampled contact configurations; configurations penetrating the object or ground are rejected.Source: A. Representation Pretraining · p. 3
  • The encoder represents object and gripper point clouds in an object-centered frame, forming 2M local patches from sampled centers and their neighbors.The passage identifies farthest-point sampling and K-nearest neighbors as the patch construction methods.Source: A. Representation Pretraining · p. 3
  • Cross-attention conditions geometry tokens on gripper position, gripper motion, and physical parameters to predict object pose change.The prediction includes translation and rotation, and the pretraining objective combines their losses.Source: A. Representation Pretraining · pp. 3–4
  • The encoder is optimized jointly with the dynamics prediction head, and its resulting representation is used by the policy.The overview states that the pretrained encoder is frozen during actor–critic policy learning.Source: A. Representation Pretraining · p. 4, Figure 2 · p. 4

B. PNP Policy Training via Reinforcement Learning

The policy is trained with reinforcement learning in a two-stage curriculum, moving from basic grasping and pose reaching to challenging ungraspable objects.

B. PNP Policy Training via Reinforcement Learning

  • PPO optimizes the actor–critic policy in 8,192 parallel environments, with ungraspable objects selected to challenge direct grasping.Objects must have generated grasp candidates but no collision-free candidate; StereoVLA further filters out objects graspable with the Panda gripper.Source: B. PNP Policy Training via… · p. 4
  • At each timestep, the policy observes object and gripper point clouds, a goal pose, physical parameters, and robot proprioception, then outputs joint-position and gripper-opening increments.The relative action has eight dimensions: seven joint-position increments and one gripper-opening increment.Source: B. PNP Policy Training via… · p. 4
  • The policy fuses geometry-encoder tokens with query features from the remaining observations, while its reward combines success, contact, and goal-directed object motion.The actor and critic use MLP heads; the reward design follows DAPL [16].Source: B. PNP Policy Training via… · p. 4
  • The first curriculum stage develops basic grasping and pose-reaching skills, while the second encourages combining nonprehensile interactions with grasping.Training proceeds from smaller-scale DGN objects to selected ungraspable objects.Source: B. PNP Policy Training via… · p. 5

IV. EXPERIMENTS

The experiments evaluate contact-dynamics pretraining and its generalization across objects and grippers.

IV. EXPERIMENTS

  • The experiments assess contact-dynamics pretraining and generalization across objects and grippers.The supplied passage states the evaluation aims but gives no outcomes.Source: §IV · p. 5

A. Simulation Experiments

Uni-PNP is evaluated on ungraspable objects with training grippers and on zero-shot transfer to the Panda gripper. It outperforms pose-reaching baselines and learns more quickly after the curriculum transition.

Setup and Baselines

  • The evaluation tests selected ungraspable objects and unseen objects from Objaverse, with ten episodes per object and zero-shot Panda gripper transfer.Success requires maintaining the object within 0.05 m and 0.1 rad of its target pose for 1 second.Source: A. Simulation Experiments · p. 5

Main Results

  • 76.39% success on ungraspable objects with training grippers is 20.52 percentage points above UniCORN (retrained); Uni-PNP reaches 66.57% success on Panda.Table 1 compares pose-reaching success across object and gripper settings; GOMP reports grasping, lifting, and holding success instead.Source: Table 1 · p. 5, A. Simulation Experiments · p. 5

Training Efficiency

  • Under the same two-stage curriculum, Uni-PNP improves more quickly after the 5,000-iteration stage transition and reaches higher training success.The authors attribute this learning efficiency to contact-dynamics pretraining.Source: A. Simulation Experiments · p. 5, Figure 4 · p. 5

B. Ablation Study

The ablations show that contact-dynamics pretraining strongly improves PointNet success, while curriculum performance depends on ordering interaction difficulty rather than only simplifying target poses.

Ablation of Architecture and Pretraining

  • Dynamics pretraining increases PointNet’s success rate from 10.89% to 76.39%, and the dynamics-pretrained compact PointNet exceeds UniCORN by 21.28 percentage points.UniCORN achieves similar success rates with contact and dynamics pretraining.Source: B. Ablation Study · p. 5, Table 2 · p. 6

Ablation Study

  • Fig. 5 presents four behavior combinations in rows: NP only, P only, NP+P, and P+NP.The figure organizes emergent prehensile and non-prehensile manipulation skills.Source: Figure 5 · p. 6

Ablation of Curriculum Learning

  • Training directly on ungraspable objects without a curriculum remains near zero success for 15,000 iterations, while the adopted curriculum orders interaction difficulty.The adopted curriculum first establishes prehensile grasping and repositioning on simpler objects, then introduces configurations requiring non-prehensile repositioning.Source: B. Ablation Study · pp. 5–6
  • Stable target poses reach 69.98% success but fail to support progress after switching to arbitrary targets.The stable-pose policy can close the gripper and rely on table contact without acquiring grasped pose control.Source: B. Ablation Study · p. 6

C. Skill

The policy exhibits four ways of combining or using prehensile and non-prehensile manipulation, including tipping before grasping and reorienting an object while maintaining a grasp. Successful episodes spend most manipulation time acquiring and controlling grasps.

Emergent Skills

  • Grasp acquisition can follow non-prehensile tipping: the policy tips an object against the table to expose an accessible region, then closes the gripper.For some objects, the other finger supports tipping and then slides downward to complete the grasp.Source: C. Skill · p. 6

Skill Analysis

  • Non-prehensile repositioning uses pushing and pivoting on the table, whereas prehensile repositioning moves a held object with arm and wrist motion.The analysis identifies these skills using contact, closure, and object-motion cues.Source: C. Skill · p. 6

Emergent Skills

  • Table-assisted reorientation changes object orientation within an established grasp by pressing the object against the table while maintaining the grip.The table contact produces a new object–gripper relative pose.Source: C. Skill · p. 6

Qualitative Rollouts

  • The policy also revises an acquired grasp by releasing the object onto the table and regrasping it when the original grasp constrains goal-directed rotation.One qualitative rollout shows the object flipping onto its opposite face before regrasping.Source: C. Skill · p. 6

Skill Analysis

  • Most manipulation time in successful episodes is spent acquiring and controlling grasps, while non-prehensile repositioning also appears across many episodes.Free motion and transitions account for the remaining episode time.Source: C. Skill · pp. 6–7, Table 3 · p. 7

D. Real-World Experiments

The real-world experiments transfer the simulation-trained policy to a Franka robot setup and qualitatively demonstrate object pose-reaching behavior. The reported evidence is a representative sequence rather than a quantitative success rate.

Real-World Setup

  • The real-world setup uses three Intel RealSense cameras, XMem segmentation tracking, and FoundationPose for 6D pose estimation.Policy commands run through CRISP low-level controllers with end-effector speed limits of 0.1 m/s linear and 1 rad/s angular.Source: D. Real-World Experiments · p. 7

Real-World Experiments

  • A real-world trial sequence shows object repositioning, grasp acquisition, and reorientation with the simulation-trained policy.The policy is transferred through teacher–student distillation; the setup uses a Franka Research 3 and Robotiq 2F85 gripper.Source: D. Real-World Experiments · p. 7
  • The authors report that teleoperation trials did not achieve similar behaviors in their trials.They cite the fine-grained coordination required between gripper and arm actions as motivation for learning coordinated manipulation in simulation.Source: D. Real-World Experiments · p. 7

V. CONCLUSIONS AND LIMITATIONS

Uni-PNP combines prehensile and non-prehensile manipulation for object pose reaching, using contact-dynamics pretraining and a two-stage reinforcement learning curriculum. Its scope is limited to tabletop interactions, two-finger grippers, and basic pose reaching without semantic information.

V. CONCLUSIONS AND LIMITATIONS

  • Uni-PNP combines prehensile and non-prehensile manipulation for general object pose reaching, with coordinated behaviors observed in simulation and real-world trials.Reported behaviors include repositioning, grasp acquisition, and table-assisted reorientation.Source: §V · p. 7
  • The approach introduces a contact-dynamics dataset and motion-prediction pretraining for a compact geometry representation, then uses two-stage reinforcement learning to transfer grasping and pose control to ungraspable objects.Source: §V · p. 7
  • The work is limited to tabletop interactions, two-finger grippers, and basic object pose reaching; cluttered scenes, multifinger hands, and semantic tasks remain outside its scope.The authors identify planning-based integration for long-horizon tasks as future work.Source: §V · p. 7

Related papers

Loading 2610.21735v1…