Source-linked AI summary
OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation
Jinhan Li, Yifeng Zhu, Yuqi Xie, Zhenyu Jiang, Mingyo Seo, Georgios Pavlakos, Yuke Zhu
TL;DR
Humanoid robots are difficult to teach manipulation skills from a single video because the video lacks action labels and the robot must adapt to new situations. OKAMI generates a plan from an RGB-D demonstration and retargets human motion to objects in the deployment scene. It reports 71.7% average task success, outperforming ORION [4] by 58.3%, and 79.2% average success for closed-loop policies trained on its rollouts.
Problem
Single-video humanoid imitation must infer actions without labels and adapt manipulation to new situations, while existing retargeting lacks object and interaction context.
Source: §1 · p. 2Method
OKAMI generates a reference plan from an RGB-D video, identifies task-relevant objects, and separately retargets body motions and hand poses to adapt to object locations.
Source: §1 · p. 2Results
Takeaways & Limitations
OKAMI demonstrates real-robot deployment with unseen object layouts, varying visual backgrounds, and new object instances.
Source: §1 · p. 3
Abstract
from arXiv · showhide
We study the problem of teaching humanoid robots manipulation skills by imitating from single video demonstrations. We introduce OKAMI, a method that generates a manipulation plan from a single RGB-D video and derives a policy for execution. At the heart of our approach is object-aware retargeting, which enables the humanoid robot to mimic the human motions in an RGB-D video while adjusting to different object locations during deployment. OKAMI uses open-world vision models to identify task-relevant objects and retarget the body motions and hand poses separately. Our experiments show that OKAMI achieves strong generalizations across varying visual and spatial conditions, outperforming the state-of-the-art baseline on open-world imitation from observation. Furthermore, OKAMI rollout trajectories are leveraged to train closed-loop visuomotor policies, which achieve an average success rate of 79.2% without the need for labor-intensive teleoperation. More videos can be found on our website https://ut-austin-rpl.github.io/OKAMI/.
1 Introduction
OKAMI addresses single-video humanoid manipulation by adapting retargeted human motions to objects in new environments. Its experiments report generalization across spatial and visual changes, with success rates of 71.7% for retargeting and 79.2% for policies trained on its rollouts.
1 Introduction
- Single-video imitation avoids the large demonstration collections collected through whole-body teleoperation, which require domain expertise and strenuous effort.The paper frames this setting as a step toward training robotic foundation models from Internet-scale human activity videos.Source: §1 · p. 2
- The method generates a reference manipulation plan from a single RGB-D video, then separately retargets body motions and hand poses to the humanoid.The plan adapts to object locations in target environments.Source: §1 · p. 2
- 71.7% task success rates averaged across all tasks, outperforming the ORION [4] baseline by 58.3%.OKAMI also trains closed-loop visuomotor policies on its generated trajectories, achieving an average success rate of 79.2%.Source: §1 · p. 2
- OKAMI uses open-world vision models to identify task-relevant objects without additional human input, including objects not directly contacting other objects or robot hands.This enables imitation of more diverse tasks than prior work.Source: §1 · pp. 2–3
- OKAMI enables real-robot deployment with unseen object layouts, varying visual backgrounds, and new object instances.The method’s object-aware retargeting adapts feasible humanoid motions to target object locations at test time.Source: §1 · pp. 2–3
2 Related Work
Prior humanoid manipulation approaches rely on modeling, simulation, or real-time human control, while imitation methods often remain costly to demonstrate or meta-train. OKAMI combines video-based vision with motion retargeting to adapt human demonstrations to object locations.
2 Related Work
- Humanoid teleoperation systems can generate diverse behaviors but require real-time human input, imposing cognitive and physical burdens.OKAMI instead requires only single RGB-D human videos to teach new skills, reducing human cost.Source: §2 · p. 3
- Single-video imitation methods avoid robot-action labels, but prior methods target single-arm manipulators or require excessive meta-training data.OKAMI exploits the kinematic similarity between humans and humanoids to retarget embodiment motions.Source: §2 · p. 3
- Existing robot motion-retargeting methods for manipulation have been used within teleoperation and lack automatic vision-based adaptation to object locations.OKAMI integrates retargeting with open-world vision to adapt motions at test time.Source: §2 · p. 3
3 OKAMI
OKAMI takes an RGB-D human video and produces a humanoid manipulation policy by generating a reference plan and retargeting motions to the observed objects. Its pipeline identifies task-relevant objects, reconstructs human body and hand motion, and adapts robot trajectories to object locations.
3 OKAMI
- OKAMI uses a two-stage pipeline: it generates a reference plan from an RGB-D video, then retargets human motion to the humanoid using updated object locations.The resulting trajectories are converted into joint commands for execution.Source: §3 · p. 3, Figure 2 · p. 4, §3.2 · p. 5
- The task is framed as finding a policy that maximizes expected success across a range of test-time initial configurations.The state comprises raw RGB-D observations of the robot and objects, actions are humanoid motion commands, and reward is one when the task is complete.Source: §3 · p. 4
3.1 Reference Plan Generation
- For reference-plan generation, OKAMI identifies task-relevant objects and reconstructs human motion, including optimized hand poses, from the video.Temporal segmentation uses keypoint velocity changes to identify subgoals; each plan step associates a target object, a reference object when needed, and a motion segment.Source: §3.1 · pp. 4–5
3.2 Object-Aware Retargeting
- At test time, OKAMI localizes task-relevant objects and factorizes retargeting into arm motions adapted to object locations and hand poses mapped to robot finger joints.Trajectory warping uses target-object locations when target-reference relations remain unchanged, and reference-object locations when those relations change; inverse kinematics produces arm joint configurations.Source: §3.2 · pp. 5–6
4 Experiments
The experiments test single-video imitation across varied manipulation tasks, the value of body-motion retargeting, robustness across demonstrators, and training visuomotor policies from OKAMI rollouts.
4.1 Experimental Setup
- Evaluation spans six tabletop behaviors, randomized object placements, and selected same-category object instances, with comparison against a minimally modified ORION baseline.Each task is run for 12 trials; the baseline uses SMPL-H palm trajectories warped to new object locations.Source: §4.1 · pp. 6–7
4.2 Quantitative Results
- Across six tasks with randomized object locations, OKAMI generalizes over varied visual and spatial conditions.The tasks include pick-and-place, pouring, and articulated-object manipulation.Source: Figure 4 · p. 7, §4.2 · p. 7
- OKAMI reaches 75.0% versus ORION’s 0.0% on Place-snacks-on-plate and 83.3% versus 41.2% on Close-the-laptop; in simulation, it reaches 82.0% versus 0.0% on Sprinkle-salt and 84.0% versus 10.0% on Close-the-drawer.ORION does not condition on human body poses in this comparison.Source: §4.2 · p. 7
- 16.7% separates the best and worst Place-snacks-on-plate policies across demonstrators, while Close-the-laptop shows no statistically significant performance change.The authors link noisier motion estimates to the relatively faster movements of demonstrator 2.Source: §4.2 · p. 8
4.3 Learning Visuomotor Policy With OKAMI Rollout Data
- OKAMI generates successful rollouts used to train ACT-based behavioral-cloning visuomotor policies for Sprinkle-salt and Bagging; Figure 5 shows their success rates, which improve as more rollouts are collected.Source: §4.3 · p. 8
5 Conclusion
OKAMI retargets human motions from a single RGB-D demonstration to a humanoid, and its experiments support systematic policy generalization and lower data-collection labor than teleoperation. The current scope is upper-body tabletop manipulation with RGB-D input, and robustness to large object-shape changes is limited.
5 Conclusion
- OKAMI generates a reference manipulation plan from one RGB-D human video, then retargets arm motions in task space and finger motions in joint-configuration space.The approach adapts retargeted motion to target-object locations.Source: §5 · p. 8
- OKAMI-based data collection reduces human cost relative to teleoperation, and Figure 5 reports learned-policy success rates for Sprinkle-salt and Bagging using 50 and 100 trajectories, respectively.The conclusion also reports systematic generalization of OKAMI policies.Source: §5 · p. 8, Figure 5 · p. 8
- Current retargeting focuses on upper-body tabletop tasks, requires RGB-D rather than in-the-wild RGB video, and has limited robustness to large object-shape variations.The authors identify lower-body retargeting, web-video use, and stronger object-interaction models as future directions.Source: §5 · p. 8
A Implementation Details
The appendix details how OKAMI reconstructs human body and hand motion from video and structures its GPT-4V prompts for object identification.
A.1 Human Reconstruction From Videos
- Human reconstruction initializes body pose from 4D Humans and hand pose from HaMeR, then jointly refines body location, body pose, and hand poses over the video sequence.The refinement addresses hand-arm inconsistency and encourages smooth motion; the optimization uses 2D joint reprojection error.Source: §A.1 · p. 14
- A 10-second video at 30 fps takes 10 minutes to process on a desktop with an RTX3090 GPU and 24 GB of memory.Source: §A.1 · p. 14
A.2 Prompts of Using GPT4V
- GPT-4V prompts identify task-relevant objects, manipulated objects, and per-step target and reference objects using typed, task-agnostic formats.The object prompts distinguish objects of interest from irrelevant scene objects and allow for no reference object.Source: §A.2 · pp. 14–15
A.3 Details on Factorized Process for Retargeting
OKAMI retargets body motion and hand poses separately, then segments demonstrations into subgoals and warps robot trajectories to new object locations. Its implementation combines inverse kinematics, hand-pose mapping, object-relation inference, and endpoint-aware trajectory warping.
A.3 Details on Factorized Process for Retargeting
- Body retargeting solves humanoid joints from SMPL-H shoulder, elbow, and wrist targets, while hand mapping transfers human poses through a canonical hand model.Inverse-kinematics weights prioritize wrist position and hand placement; hand mapping uses dex-retarget to compute robot finger angles.Source: §A.3 · p. 16
A.4 Additional Details of Plan Generation
- Keypoints tracked across video identify subgoal keyframes, while object contact is inferred from point-cloud distance and difficult non-contact relations from GPT4V.For example, GPT4V can identify a cup as the recipient in a pouring task without direct contact.Source: §A.4 · p. 16
A.5 Trajectory Warping
- Trajectory warping maps each SE(3) segment between transformed endpoint poses, adapting to a new target alone or to new target and reference locations, and assumes distinct endpoints.The segment’s start and end points are represented in SE(3), with transformations applied to those endpoints.Source: §A.5 · pp. 16–17
B.1 Success Conditions
Success is judged by whether each rollout meets its task-specific completion condition.
B.1 Success Conditions
- Sprinkle-salt succeeds when the bottle reaches a position where salt pours into the bowl.Source: §B.1 · p. 17
- Plush-toy-in-basket succeeds when more than 50% of the toy is inside the container.Source: §B.1 · p. 17
- Close-the-laptop succeeds when the display is lowered until it meets the base at the hinge.Source: §B.1 · p. 17
- Close-the-drawer succeeds when the drawer is pushed back into its containing region.Source: §B.1 · p. 17
- Place-snacks-on-plate succeeds when more than 50% of the snack package is on the plate.Source: §B.1 · p. 17
- Bagging succeeds when the chip bag is placed inside the initially closed shopping bag.Source: §B.1 · p. 17
B.2 Implementation of Baseline
The baseline adapts ORION’s palm-trajectory retargeting to the humanoid setting. A separate evaluation measures imitation success across videos from different human demonstrators.
B.2 Implementation of Baseline
- ORION is adapted by estimating palm positions from reconstructed fingers, warping the palm trajectory to test-time object locations, and using inverse kinematics for body joints.Source: §B.2 · p. 17
B.3 Details on Different Demonstrators
- Three human demonstrators perform Place-snacks-on-plate and Close-the-laptop, with imitation success rates reported in Figure 4(b).Source: §B.3 · p. 17
B.4 Simulation Evaluation
Simulation evaluation compares OKAMI variants with ORION on two tasks, and the reported results highlight the effect of perception uncertainty. The visuomotor-policy details specify ACT’s visual input and joint-position action representation.
B.4 Simulation Evaluation
- The simulation reproduces Sprinkle-salt and Close-the-drawer and compares OKAMI with vision, OKAMI without vision using ground-truth object poses, and ORION.The tasks use the same human videos as the real-robot experiments; Figure 7 shows their starting and ending frames.Source: §B.4 · p. 17, Figure 7 · p. 19
- Simulation results are generally better than real-robot results, while OKAMI without vision performs better than its vision-based version because perception uncertainty affects trajectory warping.The reported simulation success rates average 50 rollouts; partial point clouds and object-location estimation errors are identified as sources of uncertainty.Source: Table 1 · p. 18, §B.4 · p. 18
B.5 Visuomotor Policy Details
- Behavioral cloning uses ACT with pretrained DinoV2; the policy takes an RGB image and 26 joint positions and outputs 26 absolute joint positions.Table 2 lists the ACT hyperparameters.Source: §B.5 · p. 18, Table 2 · p. 19