PaperlayerView exact revision on arXiv

Source-linked AI summary

Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation

Siddhant Haldar, Lerrel Pinto

arXiv:2502.20391v1cs.RO

TL;DR

Robotics lacks the large-scale internet data available to vision and language because collecting robot data requires physical interaction. Point Policy learns robot policies from offline human videos using key-point representations, achieving substantial in-domain and novel-object performance gains.

  • Problem

    Robot policy learning commonly relies on costly teleoperation datasets that require months or years of human effort and remain much smaller than computer-vision and language datasets.

    Source: §I · pp. 1–2

  • Method

    Point Policy learns robot policies solely from offline human videos by using vision-derived hand and object key points to predict robot points and recover end-effector poses.

    Source: §I · p. 2

  • Results

    75% absolute improvement over prior methods across 8 real-world tasks in matched training settings, with 74% average success across novel object instances.

    Source: §I · p. 2, F. How well does Point Policy work for… · p. 7

  • Takeaways & Limitations

    Point Policy provides a morphology-agnostic, offline-video framework that generalizes to new object instances and remains robust to background clutter.

    Source: §I · p. 2, §VI · p. 9

    Failures in hand-pose detection or point tracking under occlusion can substantially reduce performance, while point abstractions discard scene context needed for cluttered navigation.

    Source: §VI · p. 9

Abstract

from arXiv · show

Building robotic agents capable of operating across diverse environments and object types remains a significant challenge, often requiring extensive data collection. This is particularly restrictive in robotics, where each data point must be physically executed in the real world. Consequently, there is a critical need for alternative data sources for robotics and frameworks that enable learning from such data. In this work, we present Point Policy, a new method for learning robot policies exclusively from offline human demonstration videos and without any teleoperation data. Point Policy leverages state-of-the-art vision models and policy architectures to translate human hand poses into robot poses while capturing object states through semantically meaningful key points. This approach yields a morphology-agnostic representation that facilitates effective policy learning. Our experiments on 8 real-world tasks demonstrate an overall 75% absolute improvement over prior works when evaluated in identical settings as training. Further, Point Policy exhibits a 74% gain across tasks for novel object instances and is robust to significant background clutter. Videos of the robot are best viewed at https://point-policy.github.io/.

I. INTRODUCTION

Point Policy addresses robotics data scarcity and the human–robot morphology gap by learning policies solely from offline human videos. It represents human and object states with key points, predicts future robot point tracks, and reports strong performance across real-world tasks.

Motivation

  • Robotics data collection requires physical interaction, making robot datasets far smaller and more labor-intensive than internet-scale CV and NLP datasets.Existing teleoperation-based collection can require months or years of human effort.Source: §I · pp. 1–2
  • Human-video policy learning is constrained by the morphology gap between human bodies and robot manipulators.Prior approaches commonly require downstream robot data or use human videos to derive rewards for reinforcement learning.Source: §I · p. 2, A. Imitation Learning · p. 2

Contribution

  • Point Policy learns robot policies solely from offline human data without requiring robot interactions during training.The method ties human and robot behavior through key points in shared 3D space.Source: §I · p. 2

Results

  • 75% absolute improvement over prior state-of-the-art algorithms was reported across 8 real-world tasks in identical training and evaluation settings.Point Policy also reported 74% absolute improvement on novel object instances and robustness to background distractors.Source: §I · p. 2

Implications

Method

  • The framework converts hand and object key points into transformer-predicted 3D future point tracks, then computes robot actions using rigid-body geometry constraints.The computed action is executed with end-effector position control at 6Hz.Source: §I · p. 2, Figure 2 · p. 3

C. Human-to-Robot Transfer for Policy Learning

Prior human-video approaches either pretrain representations or coarse policies and then use robot data, or derive rewards for reinforcement learning. Point Policy instead operates in an observation-based imitation-learning setting while aiming to remove the need for robot data.

Prior approaches

  • Human-video policy methods commonly learn visual representations or coarse policies first, then fine-tune them with limited robot data.This creates a remaining requirement for collecting robot demonstrations despite abundant human videos.Source: C. Human-to-Robot Transfer for Policy… · p. 3

Point Policy

Imitation learning

  • The imitation-learning setting uses observation-based expert trajectories containing observations and actions.The paper selects this setting because such trajectories are feasible to obtain from real-world demonstrators.Source: A. Imitation learning · p. 3

Behavior cloning

  • Behavior cloning formulates policy learning as maximum likelihood over expert demonstrations and can be framed as regression under a fixed-variance normal model.After training, the policy mimics actions associated with demonstrated observations.Source: B. Behavior Cloning · p. 3

C. Semantic Correspondence and Point Tracking

Point Policy transfers human hand motion to a robot through semantically corresponding and tracked key points. Two camera views provide 3D coordinates, while rigid transformations recover robot end-effector poses and gripper states.

Human-to-robot transfer

  • Human hand key points are extracted with a hand-pose detector, focusing on the index finger and thumb.These points provide the basis for transferring demonstrated human motion to the robot.Source: 1) Human-to-Robot Pose Transfer · p. 4

3D reconstruction

  • Key points from two camera views are triangulated to obtain 3D world coordinates.The paper uses point triangulation because it is more accurate than camera sensor depth.Source: 1) Human-to-Robot Pose Transfer · p. 4

Human-to-robot transfer

  • The robot position is computed as the midpoint between the index-finger and thumb tips, while orientation is derived from the hand geometry.A rigid transform relates first-frame hand key points to the robot pose.Source: 1) Human-to-Robot Pose Transfer · p. 4

Gripper state

  • The gripper is considered closed when the index-thumb distance is less than 7cm and open otherwise.The gripper state is computed from the same hand key points used for pose transfer.Source: 1) Human-to-Robot Pose Transfer · p. 4

Human-to-robot transfer

  • Rigid transformations generate robot key points and represent the robot end-effector pose in the transferred coordinate frame.This provides a morphology-agnostic bridge between human hands and robot manipulators.Source: 1) Human-to-Robot Pose Transfer · p. 4

2) Environment state through point priors:

Object states are represented by semantically meaningful point priors initialized with minimal annotation and propagated automatically through human demonstrations. Semantic correspondence and point tracking provide object key points for policy learning across instances and scenes.

Point priors

  • A user annotates task-relevant object points on one first-frame demonstration, requiring only a few seconds of input.The annotations serve as priors for subsequent point generation.Source: 2) Environment state through point… · p. 4

Semantic correspondence

  • DIFT transfers annotated points from the reference frame to corresponding locations in other frames and demonstrations.This uses semantic correspondence to identify equivalent object points across scenes.Source: 2) Environment state through point… · p. 4

Point tracking

  • Co-Tracker automatically tracks the initialized object key points throughout each demonstration trajectory.The resulting process computes object key points for every frame with single-frame user input.Source: 2) Environment state through point… · p. 4

3D representation

  • Multiple camera views are used only for point triangulation, while the policy operates on 3D key points grounded in the robot base frame.This separates multi-view geometric reconstruction from policy input representation.Source: 2) Environment state through point… · p. 4

B. Policy Learning

Point Policy represents observations as robot and object key points, uses a transformer to predict future robot-point tracks, and converts those predictions into robot actions. Semantic correspondence and point tracking support identifying and following meaningful points across scenes and trajectories.

Observation representation

  • Robot and object points grounded in the robot’s base frame replace raw images as policy inputs.A history of observations for each key point is flattened, encoded with an MLP, and provided to a BAKU transformer with a gripper token.Source: B. Policy Learning · p. 4

Point correspondence

  • Semantic correspondence identifies the same object points across different positions, new object instances, and background clutter.Figure 3 shows annotated object points alongside correspondence results for the put bottle on rack and sweep broom tasks.Source: Figure 3 · p. 5

Policy prediction

  • The BAKU transformer predicts future tracks for robot points and the robot gripper state using a deterministic action head.The policy uses a history of observations and action chunking with exponential temporal averaging for smoother predicted point tracks.Source: B. Policy Learning · p. 5

Action backtracking

  • Predicted robot points are mapped to robot pose through rigid-body geometry, using the wrist point for position and a fixed known reference orientation for orientation.The resulting action is executed with end-effector position control at 6 Hz.Source: C. Backtrack Robot Actions from… · p. 5

V. EXPERIMENTS

Experiments evaluate Point Policy on eight real-world Franka tasks with varied object positions, object instances, and backgrounds. The protocol includes held-out objects and spatial variation to test generalization beyond training examples.

Evaluation goals

  • Five research questions assess policy learning, novel-object generalization, background distractors, robot-data co-training, and influential design choices.These questions organize the evaluation of Point Policy across performance, robustness, data sources, and architecture decisions.Source: §V · p. 5

Experimental setup

  • 190 human demonstrations cover 8 real-world tasks executed by a Franka Research 3 robot with RGB and RGB-D observations from two third-person views.The action space contains the robot end-effector pose and gripper state.Source: A. Experimental Setup · p. 5
  • Tasks vary object position, type, and background context, while evaluations introduce object instances unseen during training.Figure 4 distinguishes in-domain objects from unseen objects used in generalization experiments.Source: B. Task Descriptions · p. 5, Figure 4 · p. 6

Task descriptions

  • The close-drawer and close-oven tasks vary positions during evaluation but use the same drawer or oven as training.These tasks test spatial variation without introducing new object instances.Source: B. Task Descriptions · p. 5
  • The put-bread-on-plate, fold-towel, sweep-broom, put-bottle-on-rack, and put-bowl-in-oven tasks include held-out plates, towels, brooms, bottles, or bowls.The task protocols vary positions and, where specified, introduce new object instances during evaluation.Source: B. Task Descriptions · pp. 5–6
  • The make-bottle-upright task uses 30 demonstrations from two bottles and evaluates two new bottles.Bottle position also varies across evaluations.Source: B. Task Descriptions · p. 6

C. Baselines

Point Policy is compared with image-based behavior cloning, depth-augmented behavior cloning, Motion Track Policy, and P3-PO. The baselines differ in their image or point inputs, dimensionality, and use of history.

Compared methods

  • Four baselines are evaluated: behavior cloning with RGB, behavior cloning with RGB-D, Motion Track Policy, and P3-PO.The comparison includes both image-based and point-based policy-learning approaches.Source: C. Baselines · p. 6

Image-based baselines

  • RGB behavior cloning uses the BAKU architecture to map human-hand RGB images to extracted robot actions.The depth variant adds depth images to the RGB input.Source: C. Baselines · p. 6

Point-based baselines

  • Motion Track Policy predicts future 2D robot-point tracks from scene images and robot key points, then triangulates them into 3D robot actions.The 3D points are converted to the robot’s absolute pose similarly to Point Policy.Source: C. Baselines · p. 6
  • P3-PO projects image points for robots and objects into 3D using camera depth, then feeds them to a transformer that predicts robot actions.Its 3D point representation is intended to support spatial generalization, novel-object adaptability, and clutter robustness.Source: C. Baselines · p. 6

Implementation comparison

  • Point Policy, MT-π, and P3PO use ten point observations as history, whereas the image-based baselines do not use history.Point-based methods operate on 640 × 480 images in the stated implementation setting.Source: D. Considerations for policy learning · p. 6

E. How well does Point Policy work for policy learning?

Point Policy performs strongly on in-domain and novel objects, remains robust to background clutter, and is examined alongside teleoperated robot data and triangulated depth.

In-domain objects

  • 88% average success rate across all tasks, outperforming the strongest baseline MT-π by 75% in the in-domain evaluation.The evaluation uses objects seen during training across 8 real-world tasks.Source: E. How well does Point Policy work for… · p. 6

Novel object instances

Background clutter

Teleoperated robot data

  • Robot-only demonstrations perform poorly on complex-motion tasks when matched in number to human demonstrations, due to noisier VR teleoperation data.The co-training analysis considers tasks including sweeping a broom and making a bottle upright.Source: H. Can Point Policy be improved with… · p. 7, Table IV · p. 7

Qualitative rollouts

  • Real-world rollouts visualize Point Policy’s ability on in-domain objects across 8 tasks.The rollout figure complements the quantitative in-domain evaluation.Source: Figure 5 · p. 8

I. What design choices matter for human-to-robot learning?

The design analysis examines triangulated depth and object-point inputs as choices for human-to-robot policy learning, while the framework targets generalization beyond training conditions.

a) Depth Sensing

  • Triangulated depth improves 3D key-point quality compared with noisy sensor depth, which otherwise produces unreliable robot actions.The comparison evaluates P3PO and Point Policy with and without triangulated depth on four real-world tasks.Source: §I · p. 8

b) Significance of Object Points

  • Adding object points improves MT-π performance on select tasks, suggesting that explicit object information can benefit policy learning.MT-π otherwise uses robot key points and an input image for scene information.Source: §I · p. 8
  • Point Policy outperforms MT-π with and without object points by 68% across all tasks.The result emphasizes predicting 3D key points rather than 2D key points in image space.Source: §I · p. 8

Generalization

  • Point Policy is designed to generalize to novel object instances and remain robust to background distractors.The reported framework also targets spatial variation and does not require real-world online interactions.Source: §VI · pp. 8–9, Figure 6 · p. 9

Limitations

  • Point-based abstractions improve generalization capabilities but sacrifice scene context needed for cluttered or obstacle-rich environments.The authors identify preserving sparse contextual cues as a direction for future work.Source: §VI · p. 9

APPENDIX

The appendix describes correspondence, point tracking, and triangulation procedures used to represent motion and reconstruct 3D points from multiple camera views.

2) Point Tracking

  • Point tracking follows reference points from the first video frame across subsequent frames despite illumination changes, occlusions, and camera motion.Point Policy uses Co-Tracker to track initialized points through robot trajectories for training generalizable policies.Source: 2) Point Tracking · p. 14

1) Point Triangulation

  • Point triangulation reconstructs a 3D point X from corresponding 2D projections observed by cameras with known projection matrices.The formulation seeks the 3D point that best explains the multi-view observations.Source: 1) Point Triangulation · p. 14
  • The Direct Linear Transform combines two linear equations from each camera view into AX = 0 and solves it using SVD.The solution is the unit vector associated with the smallest singular value of A.Source: 1) Point Triangulation · p. 14

Implementation

  • The experiments train all models using a single NVIDIA RTX A4000 GPU, with hyperparameters and demonstration counts listed separately.The complete hyperparameter list appears in Table VII, while demonstration counts are summarized in Table VIII.Source: C. Hyperparameters · p. 14

D. Implementation Details for MT-π

The implementation details cover the MT-π baseline, evaluation setup, task variations, depth comparisons, and tables reporting in-domain and novel-object performance.

D. Implementation Details for MT-π

  • MT-π is implemented with a Diffusion Transformer that predicts future 2D robot-point tracks from one image and robot motion tracks.The predicted tracks are converted to 3D using corresponding tracks from two camera views.Source: D. Implementation Details for MT-π · p. 14
  • MT-π achieves 18/20 on the put bread on plate task using 60 demonstrations, including 30 teleoperated robot demonstrations.This evaluation uses the setting described in the MT-π paper to check implementation correctness.Source: D. Implementation Details for MT-π · p. 14

Evaluation Setup

Depth Evaluation

Performance Tables

Related papers

Loading 2502.20391v1…