PaperlayerView exact revision on arXiv

Source-linked AI summary

From Pixel to Poses: Object-centric Tool Manipulation Learning from Human Demonstrations

Bangjun Wang, Longyan Wu, Yukun Wei, Shenghe Shao, Chaoyi Huang, Wenze Cui, Zetong Xu, Hanlin Wu, Long Chen, Yi Ma, Hongyang Li

arXiv:2609.35375v1cs.ROcs.AI

TL;DR

Robot-data scarcity and the paired-alignment and compute demands of human-video approaches complicate precise tool manipulation. P2P-T learns object-centric pose priors from human demonstrations and integrates them into a low-level policy. It reports a 73% average improvement over the strongest baseline on real-world tool-manipulation tasks, with evaluation limited to one robot embodiment.

  • Problem

    Robot-data scarcity and the expense of paired alignment constrain learning from human videos, while existing models struggle with precise complex tool manipulation.

    Source: §I · p. 1

  • Method

    P2P-T extracts 6D tool-pose trajectories from human demonstrations, pretrains an object-centric world model, and integrates its pose priors into a pose-aware low-level policy.

    Source: §I · p. 2, §V · p. 8

  • Results

    73% average improvement over the strongest baseline, π0.5 (0.33), with P2P-T achieving an average success rate of 0.57 on real-world tool manipulation tasks.

    Source: C. Result Analysis · p. 7

  • Takeaways & Limitations

    P2P-T demonstrates strong execution performance on complex tool tasks with minimal per-task fine-tuning, without paired human-robot data.

    Source: §I · p. 2

    Evaluation is limited to a single robot embodiment; multi-fingered dexterous hands and scaling under massive parameter regimes remain unexplored.

    Source: §V · p. 8

Abstract

from arXiv · show

Scaling up robotic manipulation is primarily bottlenecked by the scarcity of real-world robot data. While recent approaches leverage human video demonstrations to mitigate this shortage, they remain computationally expensive and still rely on paired human-robot data for domain alignment. Although current state-of-the-arts excel at long-horizon tasks, they struggle with the delicate and precise control required for complex tool manipulation. To overcome these limitations, we introduce P2P-T, from Pixel to Poses for Tool Manipulation, a data-efficient, object-centric framework that learns tool use directly from human demonstrations. P2P-T bridges the cognitive and physical execution gap through a two-stage approach. First, pretraining an object-centric world model to extract stable pose priors; second, integrating these priors into an efficient, pose-aware low-level policy. By utilizing a robust automated data processing pipeline powered by modern foundation models, P2P-T completely bypasses the need for human-robot aligned data. This reduces overall training overhead drastically. With minimal per-task fine-tuning, our framework achieves a 73% improvement over the previous state of the art in execution performance on complex, real-world tool manipulation tasks that currently remain out of reach for standard large-scale pretrained models.

I. INTRODUCTION

The scarcity and cost of real-world robot data constrain scaling, while existing human-video approaches still need paired alignment, heavy computation, or struggle with precise tool use. P2P-T combines object-centric pose-prior pretraining with a pose-aware execution policy and automated data processing, bypassing aligned data and targeting complex tool manipulation.

I. INTRODUCTION

  • Real-world robot data scarcity and the expense of paired teleoperation data remain major bottlenecks to scaling manipulation.Human-video demonstrations are a promising scalable source of supervision, but existing approaches retain practical limitations.Source: §I · p. 1
  • Existing human-video approaches still require paired alignment and costly training, while standard large-scale models struggle with precise complex tool manipulation.Some full-stage training pipelines demand computing resources on the scale of hundreds of GB200 GPUs; current models also tend toward simple pick-and-place or fall short on delicate control.Source: §I · pp. 1–2
  • Robots in household settings need to use human-designed tools because frequently changing hardware end-effectors for each task is impractical.Prior rule-based and reinforcement-learning approaches to complex tool use often struggle to generalize across real-world scenarios.Source: §I · p. 2
  • P2P-T pretrains an object-centric world model for stable pose priors, then integrates them into a low-level pose-aware execution policy.The two stages target high-level task understanding and low-level physical precision, respectively.Source: §I · p. 2
  • A foundation-model-powered pipeline streamlines data curation and produces clean, low-noise training data.The pipeline uses Grounded-SAM [1], SAM3D [2], and FoundationPose.Source: §I · p. 2
  • P2P-T bypasses paired human-robot data and intermediate representation-mapping stages, reducing training overhead while achieving strong execution performance with minimal per-task fine-tuning.The reported scope is complex tool manipulation tasks beyond standard large-scale pretrained models.Source: §I · p. 2

II. RELATED WORK

Prior work uses human videos, structured tool representations, and world models to address robot-data and tool-use challenges, but alignment and precise action transfer remain concerns. P2P-T learns object pose trajectories from human demonstrations and uses them to guide a low-level policy.

II. RELATED WORK

  • Human-video learning can reduce reliance on robot action labels, but approaches often still need retargeting, paired alignment, or robot fine-tuning across the human-robot morphology gap.Examples include LAPA, Dreamitate [15], EgoVLA [5], and EgoScale [4].Source: §II · p. 2
  • Tool manipulation requires affordance reasoning, contact-rich interaction, precise orientation, and often forceful or long-horizon execution.Prior structured approaches include functional keypoints in FUNCTO [8], simulation-trained RL in SimToolReal [9], and SE(3) pose trajectories in SPOT.Source: §II · p. 2
  • FUNCTO and SPOT were not included in same-protocol comparisons because neither is open-sourced; P2P-T instead learns pose priors from in-the-wild videos and injects them into a pose-aware action policy.This retains an object-centric focus while differing in learning source and policy integration.Source: §II · p. 3
  • P2P-T predicts explicit 6D tool poses rather than latent visual dynamics, using automated video-derived supervision to guide a downstream diffusion policy.The related-work discussion presents explicit poses as structured, physically meaningful predictions that are more directly transferable across embodiments.Source: §II · p. 3

III. METHODS

  • The method isolates tool motion from embodiment-specific hand motion, extracts 6D pose trajectories from human videos, pretrains a world model, and uses its priors to guide real-world execution.The stated premise is that the spatial trajectory for manipulating a specific tool remains consistent across human and robot hands.Source: §III · p. 3

1) Data Distribution of Stage 1:

Stage 1 uses a diverse, long-tailed video corpus and manually verified pose trajectories to supervise embodiment-independent spatial dynamics. Tool poses are represented as flattened homogeneous matrices, chosen to provide a continuous regression target.

1) Data Distribution of Stage 1:

  • 3,094 video clips across 17 tool categories form the Stage-1 corpus, with spoons (479) and knives (464) most frequent.The corpus has a long-tail distribution, including plates (16), glue (9), and soap (2).Source: 1) Data Distribution of Stage 1 · p. 3, Figure 2 · p. 3
  • Data intentionally concentrates on functional, dynamic tools to supervise robust 6D pose priors for intricate SE(3) maneuvers.The combined in-the-wild and TACO data are described as a diverse foundation for learning embodiment-independent spatial dynamics.Source: 1) Data Distribution of Stage 1 · p. 3
  • Human verification discarded 27% of clips with unreliable trajectories; discard rates vary with tool geometry and motion, reaching 37.875% for screwdrivers.The passage reports 19% for cups and says verification improves the retained trajectories’ supervision reliability.Source: 1) Data Distribution of Stage 1 · pp. 3–4
  • The preprocessing pipeline creates a 2D tool mask, reconstructs an instance-specific 3D mesh with SAM 3D [2], then tracks 6D poses with FoundationPose [3].FoundationPose uses the reconstructed mesh and raw video frames as inputs.Source: Figure 3 · p. 4

2) Pose Representation:

  • Each tool state is a 6DoF pose relative to the camera, with translation for centroid location and rotation for orientation, represented as a homogeneous transformation matrix.The matrix is flattened into a 16-dimensional vector for training.Source: 2) Pose Representation · p. 4
  • Flattened pose matrices provide an unambiguous, continuous regression target, avoiding quaternion sign ambiguity and Euler-angle wrapping or singularities.The vector is reshaped to a 4 × 4 matrix for decoding or evaluation.Source: 2) Pose Representation · p. 4

3) Pre-processing Pipeline:

P2P-T converts in-the-wild human videos into tool-pose supervision, then learns future object-centric pose latents as a reusable prior for downstream policy learning. The Stage 1 world model predicts pose and visual latents over a future horizon and is jointly supervised by decoded poses and auxiliary frame predictions.

3) Pre-processing Pipeline:

  • Foundation models extract 6D tool poses from in-the-wild video despite dynamic backgrounds, occlusions, and missing 3D annotations.Grounded-SAM isolates the tool, SAM 3D reconstructs its mesh, and FoundationPose tracks its poses; human validation follows automated processing.Source: 3) Pre-processing Pipeline · p. 4

B. Stage 1: Object-centric World Model Pretraining

  • Stage 1 learns reusable object-centric pose latents from human demonstrations without introducing a robot-specific action space.The latents span future steps t + 1 through t + H −1 and encode task-conditioned tool dynamics.Source: B. Stage 1: Object-centric World Model… · p. 4
  • The world model autoregressively predicts future pose and visual latents from current observations, language instructions, and tool pose.Separate heads decode predictions at each future step into tool poses and visual observations, supervising temporally coherent tool dynamics.Source: B. Stage 1: Object-centric World Model… · p. 4
  • Frozen DINOv2, SigLIP, and T5 encoders feed projected visual, language, and pose features into an autoregressive multimodal transformer.The transformer produces future pose and visual latents; separate heads decode poses and future observations.Source: B. Stage 1: Object-centric World Model… · pp. 4–5
  • The two-stage framework feeds future pose latents from the human-demonstration world model through a trainable adaptor to condition an RDT diffusion action expert.The objective jointly supervises decoded poses and auxiliary frame predictions across the prediction horizon.Source: Figure 4 · p. 5, B. Stage 1: Object-centric World Model… · p. 5
  • λ = 0.01 weights auxiliary visual prediction while prioritizing pose learning in the joint Stage 1 objective.Joint multi-step supervision yields structured pose latents encoding future tool trajectories for Stage 2 policy learning.Source: B. Stage 1: Object-centric World Model… · p. 5

C. Stage 2: Pose-aware Post-training

Stage 2 uses the frozen Stage 1 world model’s future pose latents as a task-space prior for a robot-specific action expert. The expert conditions on those latents alongside visual, language, and proprioceptive inputs to generate action chunks for receding-horizon execution.

C. Stage 2: Pose-aware Post-training

  • The pretrained RDT-based action expert conditions on future pose latents to use predicted tool trajectories as a structured task-space prior.A shared trainable MLP maps the frozen Stage 1 model’s pose latents into the RDT embedding space.Source: C. Stage 2: Pose-aware Post-training · p. 5
  • The frozen Stage 1 model produces pose latents for future steps from the current RGB observation, instruction, and available tool pose.The action expert receives projected pose tokens in temporal order, without requiring decoded pose matrices as policy inputs.Source: C. Stage 2: Pose-aware Post-training · p. 5
  • The conditional diffusion policy predicts action chunks using pose latents, robot proprioception, multi-view RGB history, and language instructions.The Stage 1 model and policy T5 and SigLIP encoders remain frozen while the action expert and pose adaptor are optimized.Source: C. Stage 2: Pose-aware Post-training · p. 5
  • The action expert learns to recover clean action chunks from noise, then uses DPM-Solver for receding-horizon execution with updated observations.Pose latents and action chunks are refreshed as new observations become available.Source: C. Stage 2: Pose-aware Post-training · pp. 5–6

IV. EXPERIMENTS

The experiments assess P2P-T’s performance on complex real-world tool manipulation and examine whether its pose prior transfers to low-level execution. They also test how model size affects performance.

IV. EXPERIMENTS

  • The evaluation asks how P2P-T compares with state-of-the-art vision-language-action baselines on complex real-world tool manipulation.The benchmark targets tasks requiring high-level functional-affordance understanding and delicate, long-horizon SE(3) execution.Source: §IV · p. 6
  • A second question tests whether the Stage 1 object-centric world model supplies a robust, transferable pose prior for the low-level policy.The evaluation frames bridging the morphological gap and precise tool manipulation without paired human-robot data as its primary goal.Source: §IV · p. 6
  • A third question examines how scaling model size affects the two-stage framework’s performance on complex tool manipulation tasks.These questions organize the experiments into baseline comparison, pose-prior transfer, and model-scaling evaluations.Source: §IV · p. 6

A. Tasks

The task suite tests tool manipulation across six real-world scenarios on a UR10e robot, with tools randomly placed in varied initial poses. Each task requires reorientation to a working pose followed by the tool operation.

A. Tasks

  • Six real-world tasks run on a UR10e with a Robotiq 2F-140 gripper and two D-435 RealSense cameras.The suite targets distinct facets of complex spatial manipulation beyond standard pick-and-place benchmarks.Source: A. Tasks · p. 6
  • The tasks are hammer striking, screwdriver rotation, cup pouring, knife slicing, brush sweeping, and spoon scooping.They respectively involve dynamic impacts, screw alignment, orientation control, downward force, sustained surface contact, and coordinated pitch adjustments.Source: A. Tasks · p. 6
  • Tools begin in varied random poses, and models must reorient each tool to its working pose before executing the operation.The diverse categories test translation of human-designed tool affordances into robust robot execution.Source: A. Tasks · p. 6

B. Implementation Details

P2P-T combines an object-centric world model with a pose-aware execution policy, using pose priors to condition low-level actions. Its implementation includes multimodal inputs and diffusion-based action prediction.

B. Implementation Details

  • The Stage-1 world model predicts next tool poses from RGB frames and FoundationPose-estimated 6D poses, while Stage 2 conditions an RDT-initialized policy on its pose priors.The policy fuses multi-view images, proprioception, and language to predict 64-step action chunks via diffusion.Source: B. Implementation Details · p. 6
  • Stage 1 uses frozen DINOv2, SigLIP, and T5-base encoders, and a transformer to predict next tool poses.The world model is pretrained on parsed TACO trajectories and human demonstrations.Source: B. Implementation Details · p. 6
  • Table I reports per-task and average success rates plus inference rate, using 100 demonstrations for fine-tuning and 50 trials per task.Inference rate is measured in Hz, with higher values better; best success rates are highlighted.Source: Table I · p. 7
  • Table II is identified as the framework-design ablation table.The supplied passage gives its scope but no further table encoding or values.Source: Table II · p. 7

C. Result Analysis

P2P-T outperforms the strongest reported baseline on real-world tool manipulation, while trajectory analysis highlights difficulties with functional geometry and precise tool interaction. The paper also contrasts its pose-prior approach with world action models and notes a grasping-instability trade-off.

C. Result Analysis

  • 73% average improvement: P2P-T achieves a 0.57 average success rate, versus 0.33 for the strongest baseline, π0.5 [13].The comparison concerns real-world tool manipulation tasks.Source: C. Result Analysis · p. 7
  • Generalist vision-language-action models such as π0.5 can grasp tools but lack explicit functional geometry knowledge and fail at precise SE(3) interaction maneuvers.Failures occur when orienting and translating tools to functional working positions, such as aligning a hammer head.Source: C. Result Analysis · p. 7
  • FastWAM [27] and Lingbot-VA [17] remain inadequate for tool-use scenarios despite generalization on pick-and-place and assembly, with reported success rates of 0.02 and 0.21, respectively.The passage attributes difficulty fine-tuning them with the same demonstrations to learning both interaction dynamics and dexterous tool motions; their inference rates are 10 Hz and 8 Hz versus P2P-T's 30 Hz.Source: C. Result Analysis · p. 7
  • Table III compares P2P-T-Small, P2P-T-Base, and P2P-T-Large variants for scaling verification.The supplied caption defines the table's scope but does not provide its cell values.Source: Table III · p. 7
  • P2P-T can show minor grasping instability at smaller pretraining scale, but its Stage-1 pose priors support spatial reasoning over tool geometry and continuous working orientations.The passage links these structured priors with successful execution of complex tasks.Source: C. Result Analysis · p. 7

D. Ablations

The ablations compare standard and pose-conditioned policies with the full two-stage framework. The results support combining Stage-1 pose priors with a pose-aware Stage-2 policy for tool manipulation.

D. Ablations

  • The ablation evaluates the necessity and transferability of Stage-1 pose priors by removing or changing framework components.The authors identify Table II as the framework-components ablation.Source: D. Ablations · p. 7
  • A standard DP2-DINOv2 policy attached to the Stage-1 world model reaches only 0.12 on Hammer and 0.14 on Cup.The authors suggest a standard policy cannot fully exploit the geometric priors for complex SE(3) maneuvers without a tailored pose-aware architecture.Source: D. Ablations · p. 7
  • Fine-tuned RDT conditioned on FoundationPose's current pose reaches 0.32 on Hammer and 0.36 on Cup, outperforming the DP2 baseline but remaining below the full framework.The authors state that implicit 2D features and current-pose information alone are insufficient for fine-grained tool alignment.Source: D. Ablations · p. 7
  • The full two-stage framework reaches 0.59 success on Hammer and 0.60 on Cup, exceeding the reported alternatives.The authors state that Stage-1 pose priors, when ingested by Stage 2, bridge functional understanding and low-level execution.Source: D. Ablations · p. 7

E. Scaling Potential

The scaling results show higher average success rates as P2P-T model capacity increases, alongside reported gains in robustness to geometric variation and disturbances. The authors also identify embodiment diversity and massive-scale training as unresolved evaluation boundaries.

E. Scaling Potential

  • Average success rises with model size: P2P-T-Small reaches 0.41, P2P-T-Base 0.60, and P2P-T-Large 0.66.The variants have 1.19B, 1.92B, and 2.83B parameters, respectively.Source: E. Scaling Potential · p. 8
  • Larger variants show better generalization to handle-length and weight variations and dynamic disturbances, with more robust, smoother trajectories.Smaller models occasionally require behavioral corrections under extreme variance.Source: E. Scaling Potential · p. 8

V. LIMITATIONS

  • P2P-T extracts 6D pose trajectories from human demonstrations to pretrain an object-centric world model, then integrates its priors into a pose-aware diffusion policy.The authors describe this design as bypassing paired human-robot data and costly domain alignment.Source: §V · p. 8
  • Evaluation is limited to one robot embodiment, excludes multi-fingered dexterous hands, and has not tested scaling under massive parameter regimes.The authors identify diverse embodiments, dexterous hardware, and larger-scale training as future directions.Source: §V · p. 8

Related papers

Loading 2609.35375v1…