PaperlayerView exact revision on arXiv

Source-linked AI summary

Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation

Siddhant Haldar, Lerrel Pinto

arXiv:2502.20391v1cs.RO

TL;DR

Robotics lacks the large-scale internet data available to vision and language because collecting robot data requires physical interaction. Point Policy learns robot policies from offline human videos using key-point representations, achieving substantial in-domain and novel-object performance gains.

  • Problem

    Robot policy learning commonly relies on costly teleoperation datasets that require months or years of human effort and remain much smaller than computer-vision and language datasets.

  • Method

    Point Policy learns robot policies solely from offline human videos by using vision-derived hand and object key points to predict robot points and recover end-effector poses.

  • Results

    75% absolute improvement over prior methods across 8 real-world tasks in matched training settings, with 74% average success across novel object instances.

  • Takeaways & Limitations

    Point Policy provides a morphology-agnostic, offline-video framework that generalizes to new object instances and remains robust to background clutter.

  • Takeaways & Limitations

    Failures in hand-pose detection or point tracking under occlusion can substantially reduce performance, while point abstractions discard scene context needed for cluttered navigation.

Abstract

from arXiv · show

Building robotic agents capable of operating across diverse environments and object types remains a significant challenge, often requiring extensive data collection. This is particularly restrictive in robotics, where each data point must be physically executed in the real world. Consequently, there is a critical need for alternative data sources for robotics and frameworks that enable learning from such data. In this work, we present Point Policy, a new method for learning robot policies exclusively from offline human demonstration videos and without any teleoperation data. Point Policy leverages state-of-the-art vision models and policy architectures to translate human hand poses into robot poses while capturing object states through semantically meaningful key points. This approach yields a morphology-agnostic representation that facilitates effective policy learning. Our experiments on 8 real-world tasks demonstrate an overall 75% absolute improvement over prior works when evaluated in identical settings as training. Further, Point Policy exhibits a 74% gain across tasks for novel object instances and is robust to significant background clutter. Videos of the robot are best viewed at https://point-policy.github.io/.

I. INTRODUCTION

Point Policy addresses robotics data scarcity and the human–robot morphology gap by learning policies solely from offline human videos. It represents human and object states with key points, predicts future robot point tracks, and reports strong performance across real-world tasks.

  • Motivation: Robotics data collection requires physical interaction, making robot datasets far smaller and more labor-intensive than internet-scale CV and NLP datasets.Existing teleoperation-based collection can require months or years of human effort.
  • Motivation: Human-video policy learning is constrained by the morphology gap between human bodies and robot manipulators.Prior approaches commonly require downstream robot data or use human videos to derive rewards for reinforcement learning.
  • Contribution: Point Policy learns robot policies solely from offline human data without requiring robot interactions during training.The method ties human and robot behavior through key points in shared 3D space.
  • Results: 75% absolute improvement over prior state-of-the-art algorithms was reported across 8 real-world tasks in identical training and evaluation settings.Point Policy also reported 74% absolute improvement on novel object instances and robustness to background distractors.
  • Implications: Point-based representations target generalization beyond image-based policies, including diverse object types and scene components.Object-centric representations structure individual scene components rather than treating the entire scene as one image.
  • Method: The framework converts hand and object key points into transformer-predicted 3D future point tracks, then computes robot actions using rigid-body geometry constraints.The computed action is executed with end-effector position control at 6Hz.

C. Human-to-Robot Transfer for Policy Learning

Prior human-video approaches either pretrain representations or coarse policies and then use robot data, or derive rewards for reinforcement learning. Point Policy instead operates in an observation-based imitation-learning setting while aiming to remove the need for robot data.

  • Prior approaches: Human-video policy methods commonly learn visual representations or coarse policies first, then fine-tune them with limited robot data.This creates a remaining requirement for collecting robot demonstrations despite abundant human videos.
  • Point Policy: Point Policy learns generalizable policies from human videos without requiring robot data during training.Its representation uses key points rather than raw images as policy inputs.
  • Imitation learning: The imitation-learning setting uses observation-based expert trajectories containing observations and actions.The paper selects this setting because such trajectories are feasible to obtain from real-world demonstrators.
  • Behavior cloning: Behavior cloning formulates policy learning as maximum likelihood over expert demonstrations and can be framed as regression under a fixed-variance normal model.After training, the policy mimics actions associated with demonstrated observations.

C. Semantic Correspondence and Point Tracking

Point Policy transfers human hand motion to a robot through semantically corresponding and tracked key points. Two camera views provide 3D coordinates, while rigid transformations recover robot end-effector poses and gripper states.

  • Human-to-robot transfer: Human hand key points are extracted with a hand-pose detector, focusing on the index finger and thumb.These points provide the basis for transferring demonstrated human motion to the robot.
  • 3D reconstruction: Key points from two camera views are triangulated to obtain 3D world coordinates.The paper uses point triangulation because it is more accurate than camera sensor depth.
  • Human-to-robot transfer: The robot position is computed as the midpoint between the index-finger and thumb tips, while orientation is derived from the hand geometry.A rigid transform relates first-frame hand key points to the robot pose.
  • Gripper state: The gripper is considered closed when the index-thumb distance is less than 7cm and open otherwise.The gripper state is computed from the same hand key points used for pose transfer.
  • Human-to-robot transfer: Rigid transformations generate robot key points and represent the robot end-effector pose in the transferred coordinate frame.This provides a morphology-agnostic bridge between human hands and robot manipulators.

2) Environment state through point priors:

Object states are represented by semantically meaningful point priors initialized with minimal annotation and propagated automatically through human demonstrations. Semantic correspondence and point tracking provide object key points for policy learning across instances and scenes.

  • Point priors: A user annotates task-relevant object points on one first-frame demonstration, requiring only a few seconds of input.The annotations serve as priors for subsequent point generation.
  • Semantic correspondence: DIFT transfers annotated points from the reference frame to corresponding locations in other frames and demonstrations.This uses semantic correspondence to identify equivalent object points across scenes.
  • Point tracking: Co-Tracker automatically tracks the initialized object key points throughout each demonstration trajectory.The resulting process computes object key points for every frame with single-frame user input.
  • 3D representation: Multiple camera views are used only for point triangulation, while the policy operates on 3D key points grounded in the robot base frame.This separates multi-view geometric reconstruction from policy input representation.

B. Policy Learning

Point Policy represents observations as robot and object key points, uses a transformer to predict future robot-point tracks, and converts those predictions into robot actions. Semantic correspondence and point tracking support identifying and following meaningful points across scenes and trajectories.

  • Observation representation: Robot and object points grounded in the robot’s base frame replace raw images as policy inputs.A history of observations for each key point is flattened, encoded with an MLP, and provided to a BAKU transformer with a gripper token.
  • Point correspondence: Semantic correspondence identifies the same object points across different positions, new object instances, and background clutter.Figure 3 shows annotated object points alongside correspondence results for the put bottle on rack and sweep broom tasks.
  • Policy prediction: The BAKU transformer predicts future tracks for robot points and the robot gripper state using a deterministic action head.The policy uses a history of observations and action chunking with exponential temporal averaging for smoother predicted point tracks.
  • Action backtracking: Predicted robot points are mapped to robot pose through rigid-body geometry, using the wrist point for position and a fixed known reference orientation for orientation.The resulting action is executed with end-effector position control at 6 Hz.

V. EXPERIMENTS

Experiments evaluate Point Policy on eight real-world Franka tasks with varied object positions, object instances, and backgrounds. The protocol includes held-out objects and spatial variation to test generalization beyond training examples.

  • Evaluation goals: Five research questions assess policy learning, novel-object generalization, background distractors, robot-data co-training, and influential design choices.These questions organize the evaluation of Point Policy across performance, robustness, data sources, and architecture decisions.
  • Experimental setup: 190 human demonstrations cover 8 real-world tasks executed by a Franka Research 3 robot with RGB and RGB-D observations from two third-person views.The action space contains the robot end-effector pose and gripper state.
  • Experimental setup: Tasks vary object position, type, and background context, while evaluations introduce object instances unseen during training.Figure 4 distinguishes in-domain objects from unseen objects used in generalization experiments.
  • Task descriptions: The close-drawer and close-oven tasks vary positions during evaluation but use the same drawer or oven as training.These tasks test spatial variation without introducing new object instances.
  • Task descriptions: The put-bread-on-plate, fold-towel, sweep-broom, put-bottle-on-rack, and put-bowl-in-oven tasks include held-out plates, towels, brooms, bottles, or bowls.The task protocols vary positions and, where specified, introduce new object instances during evaluation.
  • Task descriptions: The make-bottle-upright task uses 30 demonstrations from two bottles and evaluates two new bottles.Bottle position also varies across evaluations.

C. Baselines

Point Policy is compared with image-based behavior cloning, depth-augmented behavior cloning, Motion Track Policy, and P3-PO. The baselines differ in their image or point inputs, dimensionality, and use of history.

  • Compared methods: Four baselines are evaluated: behavior cloning with RGB, behavior cloning with RGB-D, Motion Track Policy, and P3-PO.The comparison includes both image-based and point-based policy-learning approaches.
  • Image-based baselines: RGB behavior cloning uses the BAKU architecture to map human-hand RGB images to extracted robot actions.The depth variant adds depth images to the RGB input.
  • Point-based baselines: Motion Track Policy predicts future 2D robot-point tracks from scene images and robot key points, then triangulates them into 3D robot actions.The 3D points are converted to the robot’s absolute pose similarly to Point Policy.
  • Point-based baselines: P3-PO projects image points for robots and objects into 3D using camera depth, then feeds them to a transformer that predicts robot actions.Its 3D point representation is intended to support spatial generalization, novel-object adaptability, and clutter robustness.
  • Implementation comparison: Point Policy, MT-π, and P3PO use ten point observations as history, whereas the image-based baselines do not use history.Point-based methods operate on 640 × 480 images in the stated implementation setting.

E. How well does Point Policy work for policy learning?

Point Policy performs strongly on in-domain and novel objects, remains robust to background clutter, and is examined alongside teleoperated robot data and triangulated depth.

  • In-domain objects: 88% average success rate across all tasks, outperforming the strongest baseline MT-π by 75% in the in-domain evaluation.The evaluation uses objects seen during training across 8 real-world tasks.
  • Novel object instances: 74% average success rate across tasks, outperforming the strongest baseline by 73% on novel object instances.The novel-object evaluation covers a subset of tasks and tests instances unseen during training.
  • Background clutter: Point Policy exhibits comparable performance or only minimal degradation when background distractors are present.The clutter study covers three tasks and includes both in-domain and novel object instances.
  • Teleoperated robot data: Robot-only demonstrations perform poorly on complex-motion tasks when matched in number to human demonstrations, due to noisier VR teleoperation data.The co-training analysis considers tasks including sweeping a broom and making a bottle upright.
  • Qualitative rollouts: Real-world rollouts visualize Point Policy’s ability on in-domain objects across 8 tasks.The rollout figure complements the quantitative in-domain evaluation.

I. What design choices matter for human-to-robot learning?

The design analysis examines triangulated depth and object-point inputs as choices for human-to-robot policy learning, while the framework targets generalization beyond training conditions.

  • a) Depth Sensing: Triangulated depth improves 3D key-point quality compared with noisy sensor depth, which otherwise produces unreliable robot actions.The comparison evaluates P3PO and Point Policy with and without triangulated depth on four real-world tasks.
  • b) Significance of Object Points: Adding object points improves MT-π performance on select tasks, suggesting that explicit object information can benefit policy learning.MT-π otherwise uses robot key points and an input image for scene information.
  • b) Significance of Object Points: Point Policy outperforms MT-π with and without object points by 68% across all tasks.The result emphasizes predicting 3D key points rather than 2D key points in image space.
  • Generalization: Point Policy is designed to generalize to novel object instances and remain robust to background distractors.The reported framework also targets spatial variation and does not require real-world online interactions.
  • Limitations: Point-based abstractions improve generalization capabilities but sacrifice scene context needed for cluttered or obstacle-rich environments.The authors identify preserving sparse contextual cues as a direction for future work.

APPENDIX

The appendix describes correspondence, point tracking, and triangulation procedures used to represent motion and reconstruct 3D points from multiple camera views.

  • 2) Point Tracking: Point tracking follows reference points from the first video frame across subsequent frames despite illumination changes, occlusions, and camera motion.Point Policy uses Co-Tracker to track initialized points through robot trajectories for training generalizable policies.
  • 1) Point Triangulation: Point triangulation reconstructs a 3D point X from corresponding 2D projections observed by cameras with known projection matrices.The formulation seeks the 3D point that best explains the multi-view observations.
  • 1) Point Triangulation: The Direct Linear Transform combines two linear equations from each camera view into AX = 0 and solves it using SVD.The solution is the unit vector associated with the smallest singular value of A.
  • Implementation: The experiments train all models using a single NVIDIA RTX A4000 GPU, with hyperparameters and demonstration counts listed separately.The complete hyperparameter list appears in Table VII, while demonstration counts are summarized in Table VIII.

D. Implementation Details for MT-π

The implementation details cover the MT-π baseline, evaluation setup, task variations, depth comparisons, and tables reporting in-domain and novel-object performance.

  • D. Implementation Details for MT-π: MT-π is implemented with a Diffusion Transformer that predicts future 2D robot-point tracks from one image and robot motion tracks.The predicted tracks are converted to 3D using corresponding tracks from two camera views.
  • D. Implementation Details for MT-π: MT-π achieves 18/20 on the put bread on plate task using 60 demonstrations, including 30 teleoperated robot demonstrations.This evaluation uses the setting described in the MT-π paper to check implementation correctness.
  • Evaluation Setup: The experiments vary object positions, object instances, and backgrounds, with novel objects held out from training for generalization evaluation.Figures 7 and 8 illustrate spatial and object variations used across tasks.
  • Depth Evaluation: Noise in sensor depth produces noisy robot points and unreliable actions, motivating triangulated depth for the bottle-on-rack task.Figure 9 illustrates the discrepancy between actions obtained from sensor depth and triangulated depth.
  • Performance Tables: Tables IX and X compare MT-π with and without object points against Point Policy across in-domain and novel-object settings.The tables are titled in-domain policy performance and policy performance on novel object instances.
Loading 2502.20391v1…