PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectory-based Diffusion from One-shot Human Video for Generalizable Manipulation

Guoping Pan, Xinmiao Yu, Wannan Cao, Jincheng Li, Baolin Jiang, Yizhe Li, Yi Cheng, Houde Liu

arXiv:2610.21792v1cs.RO

TL;DR

Robot imitation learning commonly depends on teleoperated demonstrations, while human videos contain manipulation knowledge that is difficult to transfer because of visual-domain gaps and missing robot actions. TDV extracts task-relevant trajectories from human videos and trains a trajectory-based diffusion policy. It ranks first on 8 of 12 RLBench tasks, and its spatial-generalization success rate approaches 100% with 50 trajectories.

  • Problem

    Robot imitation learning often relies on labor-intensive teleoperated data, while human-video transfer faces visual-domain gaps and lacks explicit robot action information.

    Source: §I · p. 1

  • Method

    TDV extracts hand or object trajectories from human videos, augments the data, and trains a trajectory-based diffusion policy for closed-loop execution.

    Source: §I · p. 2

  • Results

    TDV ranks first in 8 of 12 RLBench tasks, and with 50 trajectories its spatial-generalization success rate approaches 100%.

    Source: 3) Results · p. 5, 2) Spatial Generalization · p. 6

  • Takeaways & Limitations

    The authors conclude that trajectories support efficient data generation and cross-embodiment manipulation from one-shot human demonstrations.

    Source: §V · p. 7

    TDV relies heavily on accurate 6D pose estimation, limiting applicability to objects with complex geometry or deformable structures that are difficult to track reliably.

    Source: §V · p. 7

Abstract

from arXiv · show

Imitation learning enables robots to perform complex tasks by leveraging human-collected demonstrations. However, most existing approaches require labor-intensive data collection with additional teleoperation systems, which hinders scalability. Human videos represent a natural source of data that implicitly contain knowledge of manipulation behaviors. However, transferring them to robots remains challenging due to the visual domain gap and the absence of explicit action labels. In this work, we introduce TDV, a trajectory-based framework that leverages data generation to enable generalizable manipulation. TDV focuses on task-relevant objects by leveraging plug-and-play 6D pose estimation techniques to extract object trajectories from human videos, thereby mitigating the impact of irrelevant backgrounds and varying viewpoints. As a modality-agnostic intermediate representation, trajectories are inherently independent of the visual domains and different embodiments, enabling efficient data generation. Our method outperforms previous approaches on RLBench simulation tasks, achieving near 100\% success rates across seven tasks. In real-world experiments, TDV enables cross-embodiment generalization manipulation from one-shot human demonstrations by leveraging efficient trajectory generation techniques.

I. INTRODUCTION

Human videos offer manipulation knowledge without robot teleoperation, but transferring that knowledge is difficult because of visual-domain gaps and missing robot actions. TDV extracts task-relevant trajectories from videos and trains a trajectory-based diffusion policy for generalizable, cross-embodiment manipulation.

I. INTRODUCTION

  • Human-video demonstrations could reduce reliance on labor-intensive robot data collection, but extracting useful perception and actions from them remains challenging.Existing imitation-learning approaches typically use robot-collected proprioceptive and visual data gathered through teleoperation.Source: §I · p. 1
  • Visual-domain differences and missing robot action information complicate transfer, while object-centric 6D pose trajectories capture task-relevant state changes across viewpoints.Unlike image editing, which depends on editing quality, object-centric representations focus on manipulated objects rather than the acting agent.Source: §I · pp. 1–2
  • TDV parses human videos into hand or object subtasks, extracts trajectories, augments them by scaling and interpolation, and trains a trajectory-based diffusion policy.Hand trajectories use hand-pose estimation; object trajectories use plug-and-play 6D pose estimation, represented relative to a reference-object frame.Source: §I · p. 2
  • At deployment, real-time moving-entity pose estimates enter the policy history for closed-loop execution and high-level-description-conditioned multi-procedure tasks.The overview also describes a switch policy for task execution.Source: §I · p. 2, Figure 2 · p. 2
  • TDV is presented as learning from human videos to support scalable policy learning, cross-embodiment generalization, spatial generalization, and robustness to complex backgrounds and camera viewpoints.Source: §I · p. 2

II. RELATED WORKS

Prior work transfers human-video knowledge through visual pretraining, in-domain videos, or transferable representations, but substantial domain gaps make transfer to specific robot manipulation tasks challenging.

A. Learning from Human Videos

B. Object-Centric Manipulation

Object-centric representations focus manipulation learning on task-relevant objects, and TDV builds training data from categorized trajectories. For hand tasks, it remaps selected human hand-joint positions to a gripper-aligned pose.

B. Object-Centric Manipulation

  • Among object-centric representations, 6D object pose is described as comparatively straightforward and efficient for tracking task-relevant objects during closed-loop execution.Prior work also uses point clouds, Gaussian splatting, and neural implicit fields.Source: B. Object-Centric Manipulation · p. 3

III. METHODS

  • TDV extracts trajectories from human videos, categorizes segments by moving entity, samples valid initial poses, and trains a trajectory-based diffusion transformer policy.A switch policy governs subtask transitions, while real-time pose estimates support closed-loop control.Source: §III · p. 3

1) Hand Pose Trajectory:

  • For hand tasks, TDV uses HaMeR hand reconstruction and remaps the thumb MCP, thumb IP, and index MCP positions to a pose aligned with the gripper end-effector.The setup uses a two-finger gripper.Source: 1) Hand Pose Trajectory · p. 3

2) Object Pose Trajectory:

The object-pose pipeline generates a scaled object mesh, tracks its image mask across video frames, and estimates its 6D pose from RGB-D inputs.

2) Object Pose Trajectory:

  • TRELLIS generates a 3D object model from a close-up image, and similarity ICP against a SAM2-segmented point cloud recovers its global scale.Source: 2) Object Pose Trajectory · p. 3
  • GroundingSAM uses the task description to create the initial object mask, which prompts SAM2 to propagate masks through subsequent video frames.Source: 2) Object Pose Trajectory · p. 3
  • FoundationPose estimates the object’s 6D pose from raw RGB-D inputs using the object mesh and corresponding mask.Source: 2) Object Pose Trajectory · p. 3

B. Trajectory Generation.

Trajectory generation augments demonstrations by changing initial poses and warping trajectories, while the diffusion transformer predicts future object motion from trajectory history, task language, and procedure signals.

B. Trajectory Generation.

  • Randomizing the target object's initial pose within the workspace generates augmented trajectories while preserving the original trajectory's feasibility.The method randomizes only the initial pose of the target object.Source: B. Trajectory Generation · p. 3
  • Because trajectories are referenced to the reference-object frame, changing that object's pose is equivalent to changing the target object's pose.Source: B. Trajectory Generation · p. 3
  • Position warping rescales and rotates XY-projected paths, then translates the start; orientations use Slerp while retaining the terminal region at ratio α.Source: B. Trajectory Generation · pp. 3–4

C. Trajectory-based Diffusion Transformer

  • The transformer predicts an n-step target-object trajectory from h-step history, conditioned on task language and procedure signals using a DDPM.Source: C. Trajectory-based Diffusion… · p. 4
  • Historical trajectory input supports revisiting poses; trajectory, BERT-tokenized task text, and a learned procedure embedding are fused before DiT decoding.Source: C. Trajectory-based Diffusion… · p. 4
  • An MLP switch classifier advances the procedure and gripper state at subtask completion, labeling terminal frames positive and preceding frames negative with 1:100 loss weighting.Source: C. Trajectory-based Diffusion… · p. 4

D. Close-loop Execution

Close-loop execution combines trajectory-policy control within subtasks with switch-policy transitions, using real-time pose estimates for object tasks and end-effector poses for hand tasks.

D. Close-loop Execution

  • Each subtask segment is executed by the trajectory-based diffusion transformer, while the switch policy governs transitions between subtasks.Source: D. Close-loop Execution · p. 4
  • For object tasks, deployment estimates moving- and reference-object poses in real time; for hand tasks, the moving entity is represented by end-effector pose Tee.Source: D. Close-loop Execution · p. 4
  • The pipeline runs at 30 Hz on one NVIDIA RTX 4090 GPU and supports execution under challenging conditions such as object drops.Source: D. Close-loop Execution · p. 4

IV. EXPERIMENTS

The experiments assess trajectories as an RGB-free policy representation, one-shot human-video generalization through data augmentation across embodiments, and robustness to visual domain shifts.

IV. EXPERIMENTS

  • The experiments test whether trajectories serve as an RGB-free intermediate representation for generalization across manipulation tasks.Source: §IV · p. 4
  • They examine whether efficient data augmentation across embodiments supports generalizable manipulation from one-shot human video.Source: §IV · p. 4
  • They also evaluate generalization under visual domain shifts, including changes in camera viewpoints and background scenes.Source: §IV · p. 4

A. Simulation Experiments

The simulation experiments use selected RLBench tasks to evaluate trajectories as policy observations and compare TDV with established manipulation baselines.

1) Benchmark:

  • The benchmark comprises 12 RLBench tasks with 2–60 variants each, including high-precision and multi-step tasks; training uses 100 demonstrations per task and evaluation uses 25.Source: 1) Benchmark · p. 4

2) Baselines:

  • TDV is compared with 3D-DA, RVT2, SPOT, and LoRA-fine-tuned OpenVLA, with prior official training pipelines followed where available.Source: 2) Baselines · p. 4
  • On unseen pouring scenarios, 50 augmented demonstrations yield reliable and efficient completion, unlike single-demo collisions and overfitting.Source: Figure 3 · p. 5

3) Results:

TDV leads the RLBench evaluation, while the real-world setup tests six object- and hand-centric tasks on a UR5 system.

3) Results:

  • TDV achieves a 91.8% average success rate, surpassing the previous state-of-the-art by 13.4% across RLBench tasks.It ranks first in 8 of 12 tasks and is competitive in the remaining four.Source: 3) Results · p. 5

1) Setup:

  • The real-world evaluation covers six tasks spanning object placement, pouring, drawer and charger manipulation, and tissue pulling.The setup uses a UR5 arm with a Robotiq 2F-85 gripper and a viewpoint-adjustable RealSense D435 camera.Source: 1) Setup · p. 5
  • Table II summarizes success rates across demonstration counts and object tasks, distinguishing position generalization (PG) from pose generalization (PoG).Source: Table II · p. 5

2) Spatial Generalization:

The spatial-generalization evaluation varies object positions and orientations in a 45 cm × 50 cm workspace, finding that larger trajectory sets improve performance.

  • The evaluation samples test cases in a 45 cm × 50 cm workspace, using position generalization (PG) and pose generalization (PoG) settings.PG samples 13 workspace points; PoG adds random orientation perturbations at each point to create configurations unseen in the demonstration.Source: 2) Spatial Generalization · pp. 5–6
  • Training from one generated trajectory shows some spatial generalization but tends to revert to the demonstration’s initial pose, reducing efficiency and increasing collision risk.The evaluation uses one human demonstration to create the initial trajectory set before expanding it to 15 or 50 trajectories.Source: 2) Spatial Generalization · p. 6
  • With 50 trajectories, spatial-generalization success approaches 100%, improving as the generated training set grows.The 15-trajectory set remains sparse and can fail when the initial pose is far from its data distribution.Source: 2) Spatial Generalization · p. 6

3) Visual Domain Generalization:

TDV is reported to remain robust across camera viewpoints and background conditions, but its dependence on accurate 6D pose estimation limits the objects it can reliably track.

3) Visual Domain Generalization:

  • TDV demonstrates robustness to camera-viewpoint and background variation, including deployment viewpoints distinct from data collection.Rare failures occur with steep top-down views that hinder localization of small objects or when object color resembles the background, causing tracking failure.Source: 3) Visual Domain Generalization · pp. 6–7

V. CONCLUSIONS

  • The conclusion identifies accurate 6D pose estimation as a limitation for objects with complex geometry or deformable structures that are difficult to track reliably.Source: §V · p. 7
Loading 2610.21792v1…