PaperlayerView exact revision on arXiv

Source-linked AI summary

Monkey See, Can Monkey Do? A Benchmark for Evaluating Robot Skill Learning by Observation

Weiwei Gu, Anmol Gupta, Anant Sah, Ryan Varghese, Lalitha Shreya Vanam, Prabhath Adireddi, Peter Karkus, Nakul Gopalan

arXiv:2609.08209v2cs.RO

TL;DR

LfO progress is difficult to assess because methods use varied assumptions, hardware, and environments, limiting meaningful comparisons. RoboReel pairs human demonstrations with simulated robot trajectories and evaluation environments across ten tasks. Its analysis finds that long-horizon and precise manipulation remain challenging for evaluated methods.

  • Problem

    LfO methods vary in assumptions, hardware, and environments, while the field lacks benchmarks for systematic, reproducible comparisons.

    Source: §1 · p. 2

  • Method

    RoboReel pairs real-world human demonstrations with simulation and evaluates varied LfO models through suites that test distraction settings and calibrated keypoint annotations.

    Source: §1 · pp. 2–3, §3.1 · p. 4

  • Results

    Across evaluated methods, long-horizon tasks and tasks requiring precise manipulation remain challenging.

    Source: §4.3 · p. 8

  • Takeaways & Limitations

    RoboReel introduces a shared evaluation protocol for different LfO model families using four suites and paired human demonstrations and simulation.

    Source: §6 · p. 9

    The benchmark focuses on goal-based, single-arm tasks and excludes soft objects and bimanual tasks.

    Source: §5 · p. 9

Abstract

from arXiv · show

Learning from Observation (LfO) is a fundamental robotic capability that replicates how humans and animals socially learn from each other. Beyond its biological parallels, this modality provides a practical solution for data scaling in sample-inefficient and data-starved domains like robotics. Recent work has demonstrated promising results in learning manipulation skills from human videos, yet progress in this area remains difficult to assess. Existing methods vary widely in assumptions, hardware choices, and environment setups making it difficult to draw meaningful comparisons and identify advances in the field. To address these challenges, we introduce RoboReel: a unified benchmark for evaluating models that learn policies from human videos. RoboReel consists of bundled real-world human demonstration videos, simulated robot trajectories, and evaluation environments on ten manipulation tasks. We develop four test suites to evaluate the models' performance on multiple axes, including the robustness to visual distractors and the ability to complete long-horizon tasks. Our benchmark covers learning-from-observation models from different categories, and studies the effectiveness of multiple representation choices in our benchmark evaluation that covers over seven state-of-the-art algorithms (including our VLA based variants) in the field of LfO. Finally, we present an analysis of the different types of algorithms showing that long-horizon tasks and tasks with low tolerances are still challenging for current models. Webpage: https://roboreel.github.io

1 Introduction

RoboReel addresses the lack of systematic, reproducible evaluation that makes comparisons among Learning from Observation methods difficult. It pairs real-world human demonstrations with simulation and evaluates varied LfO approaches across multiple tasks and settings.

1 Introduction

  • The field lacks benchmarks for systematic, reproducible LfO evaluation, making apples-to-apples method comparisons difficult.Existing work also varies in demonstration assumptions and training and testing protocols.Source: §1 · p. 2
  • RoboReel pairs a calibrated real-world demonstration environment with a matched simulation environment to train and evaluate LfO models.The benchmark uses reconstructed 3D objects and multiple test suites.Source: §1 · p. 2
  • The benchmark provides over seven hours of paired, calibrated human data across ten tabletop tasks and evaluates more than seven LfO models.The evaluated models use varying representations, and the analysis examines their effectiveness and existing gaps.Source: §1 · p. 3

2 Related Work

Prior datasets and benchmarks often omit either real-world human video demonstrations or simulated evaluation environments. RoboReel situates its contribution among diverse LfO representations and tabletop manipulation tasks.

2 Related Work

  • RH20T pairs robot trajectories with human videos but lacks simulated evaluation environments, while MimicDroid pairs robot videos with evaluation environments but lacks real-world human demonstrations.RoboReel addresses this gap with paired real-world human videos and simulated environments.Source: §2 · p. 3
  • LfO methods use explicit language, implicit latent representations, or keypoints to extract robot actions from human videos.The methods differ in their assumptions about how demonstrations are represented and used.Source: §2 · p. 3
  • RoboReel's ten tabletop tasks vary in horizon, object interactions, and required manipulation tolerances.The task set includes pushing, articulated-object interactions, pick-and-place, and cup stacking.Source: Figure 2 · p. 4

3 RoboReel - A Learning from Observation Benchmark

RoboReel provides a shared evaluation benchmark for LfO methods, with paired human observations and simulation. Its suites, tasks, collected demonstrations, and generated robot trajectories support evaluation across varied settings.

3 RoboReel - A Learning from Observation Benchmark

  • RoboReel provides real-world human observations paired with simulated evaluation environments for systematic assessment of LfO methods.The benchmark targets a requirement missing from the closest compared benchmarks.Source: §3 · p. 4

3.1 Research Categories addressed by the RoboReel

  • The four suites test clean and distracted settings, evaluation-time observational shifts, and whether calibrated keypoint annotations benefit LfO.Distractors are absent during training but appear in demonstrations and simulation at evaluation in the Evaluation Distraction Suite.Source: §3.1 · p. 4

3.2 Task Curation

  • Ten tasks vary object type, task horizon, and manipulation precision; the benchmark restricts scope to goal-based tasks.The authors cite inaccurate soft-body simulation and difficulty validating non-goal-oriented tasks as reasons for this scope.Source: §3.2 · pp. 4–5

3.3 Human Demonstration Collection

  • The dataset contains 2,000 human demonstration videos recorded from four camera perspectives, with 100 demonstrations per task in clean and distracted conditions.RGB and depth images are collected from all sensors with timestamps for synchronization.Source: §3.3 · p. 5

3.4 Constructing a Corresponding Simulation Environment and Generating Paired data

  • Object-matched simulation aligns camera poses, while grasp planning and waypoint-based motions generate collision-free robot trajectories labeled against predefined goals.Pick-and-place trajectories use five phases: pre-grasp, grasp, transport, pre-place, and drop.Source: §3.4 · p. 5

4 Experiments

The experiments compare learning-from-observation baselines grouped by how they represent human videos, then assess their performance across multiple test suites and task types. Video-conditioned π0.5 variants perform strongest overall, while long-horizon and precise manipulation remain difficult.

4.1 Baselines

  • The baselines cover three video-representation pathways: explicit language followed by planning or a VLA, latent representations guiding a policy, and imitation from videos paired with 3D hand poses.The evaluated baselines include SeeDo and explicit π0.5 variants, UniSkill and Vid2Robot plus video-conditioned π0.5 variants, and PointPolicy, respectively.Source: Figure 3 · p. 6, §4.1 · p. 7

4.2 Results

  • In the no-distraction suite implicit video-conditioned policies perform strongest overall; in the with-distraction suite π0.5 variants lead, while the evaluation-distraction suite favors keyframe-based π0.5 on most tasks.Most models perform worse in the with-distraction setting, and the evaluation-distraction suite is more challenging than the other suites.Source: §4.2 · p. 7
  • PointPolicy performs well on Push Cube and Close Drawer and remains similarly robust with and without distractions, but fails on the three tasks requiring grasping.A different PointPolicy model is trained for each task.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3 Analysis

  • Across all ten tasks and test suites, video-conditioned π0.5 variants perform strongest overall, with keyframe sampling outperforming uniform sampling on 7 of 10 tasks in the evaluation-distraction suite.The authors attribute the keyframe advantage to capturing task-relevant moments that improve robustness to evaluation-time visual domain shift.Source: §4.3 · pp. 7–8
  • Long-horizon tasks have the lowest success rates across model families because every sequential sub-goal must succeed, and even π0.5 variants remain below 25% success.Short-horizon tasks such as Push Cube, Pick Cube, and Close Drawer achieve substantially higher success rates.Source: §4.3 · p. 8
  • Implicit representations outperform explicit language representations across all three test suites, despite the expectation that language guidance would benefit pretrained VLMs and VLAs.The authors hypothesize that additional robot-trajectory exposure helps VLAs overcome VLMs’ weakness in visual-temporal reasoning [60].Source: §4.3 · p. 8
  • Press Toaster performance degrades from the no-distraction to the with-distraction and evaluation-distraction suites, while Stack Cups remains uniformly difficult across suites and models.The authors attribute Press Toaster’s degradation to distractors obscuring its small target area, and Stack Cups’ difficulty to the combination of precise placement and a long horizon.Source: §4.3 · p. 8
  • Keypoint annotations capture overall trajectory behavior but do not yield consistent gains, with performance dropping on tasks that require precise grasping.The authors attribute grasp failures to the difficulty of deriving grasp configurations from keypoints alone, especially for bowls and handles.Source: §4.3 · p. 8

5 Limitations

RoboReel is limited to goal-based tasks, excluding dynamic skills whose goals cannot be assessed at a single time point. The benchmark also omits soft objects and bimanual tasks.

5 Limitations

  • The benchmark uses goal-based tasks, so it does not test dynamic, non-Markovian skills whose goals cannot be checked at one time point.The paper describes this as a broader robotics challenge.Source: §5 · p. 9
  • Soft objects are absent because their physics are harder to simulate.Source: §5 · p. 9
  • The task set uses a single arm; the authors note that multiple arms would make action generation significantly more challenging for existing models.They restrict the benchmark because most LfO approaches work in single-arm domains.Source: §5 · p. 9

6 Conclusion

RoboReel pairs real-world human demonstrations with simulated evaluation environments to assess Learning-from-Observation frameworks. The authors use four task suites to evaluate more than seven models and plan to expand the benchmark over time.

6 Conclusion

  • RoboReel provides a shared evaluation protocol for different LfO model families, which the authors identify as the benchmark’s first.Source: §6 · p. 9
  • The authors manually curate 10 tasks, collect 2,000 real-world human demonstrations from multiple angles, and build paired simulation evaluation environments.Source: §6 · p. 9
  • The evaluation uses four task suites to study more than seven state-of-the-art models from varied model families under different settings.Source: §6 · p. 9
  • The authors plan to increase RoboReel’s data scale and task variety over time.Source: §6 · p. 9

Contributions

The listed contributors cover benchmark and simulation development, model evaluation, real-world experiments, and data collection, alongside project guidance and supervision.

Contributions

  • Weiwei Gu implemented models, evaluated baselines, organized real-world data collection, and supported real-robot experiments.Source: Contributions · p. 17
  • Anmol Gupta primarily developed the benchmark, designed simulated tasks, built the simulation data-collection pipeline, and served as primary writer.Source: Contributions · p. 17
  • Anant Sah conducted real-world robot experiments and supported data collection and baseline implementation.Source: Contributions · p. 17
  • Ryan Varghese and Lalitha Shreya Vanam collected real-world human demonstrations; Prabhath Adireddi also assisted with real-world robot data.Source: Contributions · p. 17
  • Peter Karkus advised on problem formulation and experimental design, while Nakul Gopalan supervised the project and contributed to writing.Source: Contributions · p. 17

A Additional Results for PointPolicy

Point Policy achieved no simulator-defined successes across five extended tasks, despite some directionally correct behavior. The authors attribute the gap to longer task horizons and keypoint/action-recovery precision limits.

A Additional Results for PointPolicy

  • 0% success: Point Policy achieved 0% under the simulator’s native success criterion across five extended tasks.The tasks were empty basket, stack cups, toaster, toy drawer close, and toys in basket.Source: A Additional Results for PointPolicy · p. 17
  • The extended tasks require sequential approach, grasp, lift, transport, and release stages, while previously evaluated tasks were largely single-stage motions.The policy often reproduces the initial approach but fails at later manipulation stages needed for success.Source: A Additional Results for PointPolicy · p. 17
  • Keypoint identity drift and an approximately 2–3 cm action-recovery precision floor make final contact unreliable for graspable surfaces only a few centimeters wide.The policy uses 3D keypoint relationships; the cited passage contrasts these precision demands with more tolerant pushing tasks.Source: A Additional Results for PointPolicy · p. 17

B Model Architecture for Video-Conditioned π0.5

Video-Conditioned π0.5 adds a fixed-length visual encoding of a human demonstration to the standard Pi-0.5 inputs, then predicts robot action chunks from the combined context. Its underlying Pi-0.5 architecture and action-prediction procedure remain unchanged.

B.1 Video-Conditioned Policy Formulation

  • The policy predicts an action chunk from a human demonstration, multi-view robot observation history, and a fixed task prompt.The prompt is “complete the task” for every task, leaving the demonstration as the only task-specific conditioning signal.Source: §B.1 · pp. 18–19

B.2 Base Pi-0.5 Components

  • Pi-0.5’s VLM expert fuses visual and language prefix tokens, while its action expert predicts robot action chunks from VLM features and the denoising timestep.The two experts are jointly fine-tuned with the demonstration encoder.Source: §B.2 · p. 19

B.3 Human Demonstration Encoder

  • The demonstration encoder builds a fixed-length 512-token visual prefix from selected front-view video frames, using frozen SigLIP-So400m/14 image features augmented with 3-D sinusoidal spatiotemporal positional embeddings.Uniform sampling selects 32 frames evenly, with 16 visual tokens per frame. BPP includes annotated keyframes when they fit within the frame budget, then samples remaining non-keyframes with probability inversely related to distance from the nearest keyframe.Source: §B.3 · pp. 19–20

B.4 Demonstration-as-Context Integration

  • The demonstration prefix is processed alongside robot observations and language tokens, letting the model compare the human video with the current robot scene before predicting actions.The action head, action horizon, denoising procedure, and flow-matching objective remain unchanged from Pi-0.5.Source: §B.4 · p. 20

B Model Architecture for Video-Conditioned π0.5

  • A lightweight demonstration encoder turns a human video into a 512-token visual prefix prepended to Pi-0.5’s standard input sequence.The encoder adds approximately 6.5M parameters and no transformer blocks, attention heads, or expert branches.Source: §B.3 · p. 20, §B.4 · p. 21

C Training Details

This section specifies training configurations for the evaluated baselines, video-conditioned Pi-0.5 variants, and PointPolicy. It also points to RoboReel’s paired-data comparison with existing benchmarks.

C Training Details

  • Each model is trained with identical hyperparameters on both no-distraction and with-distraction training sets, with final-checkpoint results reported.Most runs use 50k steps; the language-conditioned LeRobot Pi-0.5 baseline uses 30k.Source: C Training Details · p. 21

C.1 Vid2Robot

  • Vid2Robot uses 16 prompt frames, 8 state frames, and predicts chunks of four actions, with four losses averaged for training.Its objective combines action-token cross-entropy, temporal cycle consistency, video-video SigLIP, and video-text SigLIP losses.Source: §C.1 · p. 21

C.2 UniSkill

  • UniSkill’s released pretrained ISD encoder is used without fine-tuning; only the downstream diffusion policy is trained.The encoder precomputes a 64-dimensional skill-goal embedding for each robot frame and human demonstration.Source: §C.2 · p. 21

C.3 Video-Conditioned π0.5 (Uniform Frame Sampling)

  • The video-conditioned Pi-0.5 variant fine-tunes the public Pi-0.5 base checkpoint with a demonstration adapter, representing demonstrations with 32 uniformly sampled frames and a 512-token budget; robot observations use front and wrist cameras.The text prompt is fixed to complete the task, so the model receives no language information beyond the demonstration video.Source: §C.3 · p. 22

C.4 Video-Conditioned π0.5 (BPP keyframe sampling)

  • The BPP Pi-0.5 variant shares the uniform variant’s architecture and optimization settings, differing only in demonstration-frame sampling.A VLM selects keyframes from uniformly downsampled videos, and frames closer to selected keyframes receive higher sampling weights.Source: §C.4 · pp. 22–23

C.5 Language-conditioned π0.5(VLM Upstream)

  • The language-conditioned Pi-0.5 baseline omits video input and fine-tunes the action expert while keeping the VLM expert frozen.Qwen3-VL-4B generates each robot sample’s language instruction from a human demonstration of the same task.Source: §C.5 · p. 23

C.6 PointPolicy

  • PointPolicy retargets keypoint trajectories from human demonstrations into robot end-effector trajectories and trains a separate policy for each task.A waypoint controller executes predicted end-effector targets; evaluation uses simulated environments with and without distraction objects.Source: §C.6 · pp. 23–24

D Details of the Benchmark

  • RoboReel pairs real-world human videos with simulation trajectories and task environments for comparing learning-from-observation models across model families.Table 5 compares benchmark statistics; its caption highlights paired data and portable, scalable evaluation environments.Source: Table 5 · p. 24, D Details of the Benchmark · p. 25

D.1 Dataset Statistics

RoboReel pairs human demonstrations with robot trajectories across ten tabletop tasks and builds matched simulation environments for those tasks. The section describes the dataset, task annotations, and simulation pipeline.

D.1 Dataset Statistics

  • Across ten tasks, RoboReel contains 3,000 human demonstrations recorded from four cameras and 2,000 robot trajectories collected in matched ManiSkill 3 / SAPIEN scenes.Each task-condition pair includes 100 human demonstrations and 100 robot trajectories; the human demonstrations span roughly nine hours of front-camera footage.Source: §D.1 · p. 25

D.2 Additional Descriptions on the Benchmark

  • The benchmark describes task success criteria and shows task configurations, real-world objects, and corresponding real-world and simulation camera views.Human-video keypoints are localized with DIFT and propagated by CoTracker; filtered 3D keypoint trajectories can serve as action annotations in the calibrated setting.Source: §D.2 · pp. 25, 27, 28, Figure 10 · p. 26, Figure 11 · p. 27

D.3 Creating Paired Task in Simulation.

  • The paired simulation pipeline targets visual, geometric, and kinematic fidelity alongside procedural reproducibility, using reconstructed assets and calibrated cameras.Task-specific planners generate robot trajectories and verify them against the environment’s success predicate; recorded trajectories include observations, actions, simulator state, and synchronized video streams.Source: §D.3 · pp. 28–29, Table 8 · p. 29

D.4 Evaluation Protocol

RoboReel’s evaluation protocol uses reproducible starting states and task-specific evaluation inputs. The related-work discussion organizes LfO methods by transfer pathway and motivates a unified cross-embodiment benchmark.

D.4 Evaluation Protocol

  • Each reported success rate averages three evaluation runs of 20 rollouts each, or 60 total rollouts per model and table cell.The tables report means and standard errors of means across runs.Source: §D.4 · p. 29
  • Pre-recorded evaluation banks replay solvable starting configurations and fix distractor identities, giving different models identical rollout states.Each bank entry stores the initialization seed and full state snapshot; oracle-failed configurations are discarded.Source: §D.4 · p. 29
  • ND, WD, and ED evaluations use 80 human videos for training and 20 held-out videos for evaluation; PD models train on all 100 annotated videos and infer from the environment alone.The same 20 evaluation videos are used across the three runs, paired with differently seeded environments.Source: §D.4 · pp. 29–30

E Extended Related Work

  • LfO methods are grouped as task-oriented, observation-oriented, and action-oriented transfer, with different assumptions, architectures, and evaluations motivating a unified benchmark.Action-oriented methods that transfer explicit affordances require calibration between human and robot observation and action spaces.Source: E Extended Related Work · pp. 30–31
Loading 2609.08209v2…