PaperlayerView exact revision on arXiv

Source-linked AI summary

The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction

Xunzhe Zhou, Yiyang Cai, Fengyi Wang, Ran Ju, Hanxiang Ren, Ruizhe Liu, Yu Zhang, Qian Luo, Feng Chen, Pei Zhou, Yi Ma, Yanchao Yang

arXiv:2608.22301v1cs.ROcs.AI

TL;DR

Robot imitation from human video often replays trajectories and therefore lacks explicit task inference under scene changes. The paper introduces a four-level benchmark with paired data and unified evaluation, finding that current models remain stable through L2 but collapse at L3 functional substitution; pretraining and few-shot paired fine-tuning improve adaptation, while broader transfer remains limited.

  • Problem

    Robot policies generally learn observation-to-action mappings or trajectory remappings without explicitly inferring the demonstrated task, limiting imitation when execution scenes differ.

  • Method

    The paper introduces the Imitator Game’s four-level mismatch hierarchy, IG-10K paired human-robot data, and Imitator Arena for unified automated and human evaluation.

  • Results

    Performance is nearly stable from L0 to L2 but drops at L3, where functional substitution requires achieving the same intent through a different object function.

  • Takeaways & Limitations

    Intent-level imitation remains a distinct barrier: current systems transfer poorly to unseen tasks, while paired pretraining and few-shot fine-tuning improve adaptation.

  • Takeaways & Limitations

    IG-10K uses one substitution pattern per task-level pair, finite assets, and a hierarchy that approximately fixes the demonstrator-robot embodiment gap.

Abstract

from arXiv · show

Humans imitate at the level of intent: given a demonstration, we infer its goal and carry it out with whatever tools, objects, and layouts are at hand. Current robot policies instead learn observation-to-action mappings from visual inputs and language instructions, without explicitly inferring the demonstrated task. Learning from human video thus remains largely trajectory-level: models can replay motions in near-identical scenes, but still struggle to imitate what the demonstrator intends rather than merely what they do. We introduce The Imitator Game, a four-level benchmark (L0-L3) that progressively widens the gap between the human demonstration and the robot's own scene, isolating where trajectory replay ceases to suffice and task understanding becomes necessary. We pair it with IG-10K, the largest environment-aligned paired human-robot dataset to date and the only one instantiated across all four levels in both real and simulated settings (20,000+ paired episodes, 50+ tasks, 6 domains), and Imitator Arena, an open platform for blind A/B human evaluation. Across nine state-of-the-art models, performance is stable from L0 to L2 but collapses at L3, identifying functional substitution - achieving the same intent through a different object affordance - as the decisive barrier to intent-level imitation. Human-video-conditioned models outperform caption-conditioned ones, yet every model falls below 13% zero-shot success on unseen tasks; fine-tuning IG-10K-pretrained models with only $10$ paired human-robot demonstrations yields large gains that grow with pretraining scale. The project website and access to Imitator Arena are available at https://imitator-game.github.io.

1 Introduction

Humans imitate goals flexibly across changed tools, objects, and layouts, whereas robot policies largely replay trajectories and remain dependent on similar scenes. The Imitator Benchmark measures this gap through four progressively mismatched execution levels.

  • Humans preserve an action’s purpose while adapting to available tools, objects, and layouts.The paper contrasts goal-directed imitation with memorizing an exact motion sequence.
  • Robot video-learning methods commonly extract and retarget motion, making them vulnerable when scene properties change despite an unchanged task.Small changes in placement, geometry, or available tools can invalidate the demonstrated motion.
  • The benchmark tests whether policies reproduce demonstrated intent in unseen scenes with substantially different object placement, geometry, or function.Its focus is what behavior should be preserved under scene change, rather than trajectory reproduction alone.
  • L0 requires trajectory replay, L1 spatial adaptation, L2 generalization across same-category objects, and L3 intent transfer through a different affordance.The levels increase mismatch from scene-identical execution to functional substitution.
  • The work introduces a four-level evaluation framework, IG-10K, Imitator Arena, and systematic evaluation of nine state-of-the-art models.The evaluation spans 15 trained variants, multiple model families, training scales, and regimes.

2 Related Works

Prior manipulation research learns from human video through motion retargeting, correspondence, and video conditioning, typically assuming similar demonstration and robot scenes. Existing benchmarks and datasets study other forms of robustness, diversity, or language-conditioned composition rather than graded intent-level mismatch.

  • Human-video robot learning commonly retargets hand or object poses, optical flow, or cross-embodiment actions to robot behavior.These approaches include hand inpainting, motion cues, filtering, and learned human-robot correspondences.
  • Skill-representation and video-conditioned methods generally assume the robot scene is largely congruent with the demonstration and are evaluated in-distribution.They do not impose a graded mismatch between demonstration and execution environments.
  • Existing manipulation benchmarks perturb robustness, embodiment, environment, or visual factors, while others emphasize compositional language-conditioned tasks.The cited benchmark families include COLOSSEUM, RoboTwin 2.0, REALM, CALVIN, LIBERO, and VLABench.
  • Large-scale manipulation datasets emphasize diversity across tasks, scenes, and embodiments rather than environment-aligned human-robot pairing for graded imitation.The comparison includes Open X-Embodiment, DROID, BridgeData V2, and FMB.
  • Robot policies span specialized visuomotor methods and large-scale VLA or diffusion foundation models with language conditioning, action generation, and few-shot adaptation.The paper situates ACT, Diffusion Policy, VQ-BeT, OpenVLA, π0.5, GR00T, RDT-1B, and H-RDT in this landscape.

3 The Imitator Game

The Imitator Game formalizes imitation as reproducing a human-demonstrated task in a separate robot scene whose mismatch determines the highest achievable fidelity. It combines a four-level hierarchy, paired data, unified evaluation, and model-agnostic interfaces to study this capability.

  • 3.1.1 Problem setting and scene decomposition: The problem setting gives an imitator only a human video and robot observations, with the task defined implicitly by the demonstrated behavior.There is no separate task description, goal predicate, or symbolic specification as policy input.
  • 3.1.1 Problem setting and scene decomposition: Scenes are decomposed into object appearance, geometry, semantic category, and spatial configuration, separating intrinsic properties from object placement.Changing configuration preserves object identity, while changing object properties can alter the behavior logic required for the task.
  • 3.1.2 Four levels of imitation: The hierarchy progressively reduces copyable fidelity from exact motion to final object state, ordinary task completion, and underlying purpose.The progression follows increasing divergence between demonstration and imitator scenes.
  • 3.2 The Imitator Benchmark: The benchmark combines automated simulation scoring, structured human evaluation, and a shared demonstration-video plus robot-observation interface across policy families.Language-conditioned models receive deterministic captions generated from the demonstration video, enabling comparison with video-conditioned policies.
  • 3.2.1 IG-10K: Each paired episode links one human demonstration scene to robot scenes instantiating L0 through L3, so level is determined by the demonstration-execution pair.The same human clip can support four levels by changing only the robot-side scene, with one substitution pattern per task-level pair.

4 Experiments

The experiments compare imitation interfaces, transfer settings, and progressively mismatched scenes across simulation, real-world rollouts, and human judgments. Video-based methods are strongest for seen-task imitation, paired pretraining mainly helps few-shot adaptation, and L3 functional substitution is the clearest difficulty.

  • Experimental setup: The evaluation covers five seen and five disjoint unseen manipulation tasks across L0–L3, testing seen-task imitation, zero-shot transfer, few-shot adaptation, and human judgments.Both task groups are evaluated at all four levels, yielding 40 task–level combinations.
  • Experimental setup: Simulation compares language-conditioned VLAs, video-skill methods, and video-conditioned visuomotor models using success rate and sub-goal success rate.Real-world evaluation uses four representative models and human judgments because hardware lacks an automated metric.
  • Q1: Which imitation interface is strongest?: Video-based pipelines are strongest for seen-task imitation, while π0.5 leads few-shot adaptation in simulation with SR = 0.80 in Scr. and SR = 0.85 in P+FT.ACT/DINOv2 achieves the highest simulation seen-task performance with SR = 0.81 and Sub-SR = 0.93; XSkill is strongest in the real-world Arena across settings.
  • Q2: Does scale improve zero-shot transfer or few-shot adaptation?: Zero-shot performance remains near the floor, with the best simulation result reaching only SR = 0.13.Current frameworks struggle to generalize to unseen tasks without task-specific robot data.
  • Q2: Does scale improve zero-shot transfer or few-shot adaptation?: From 15 to 45 pretraining tasks, P+FT success improves for 14 of 15 simulation variants and beats Scr. for 12 of 15 variants.Real-world representative models show the same few-shot improvement pattern, while only video-conditioned models gain some zero-shot ability as the corpus grows.
  • Q3: Where does the hierarchy become hard?: Real-world success is nearly stable from L0 to L2 at around 0.4 but drops to 0.29 at L3, while human imitation scores fall from about 6.2 to 5.62.XSkill maintains SR ≈0.53–0.57 through L2 but falls to 0.29 at L3, indicating difficulty with functional substitution.

5 Conclusion

The paper frames robot video imitation as a hierarchy from trajectory copying to intent-level transfer and releases a benchmark, paired dataset, and human-evaluation platform. Its conclusion is that functional substitution remains difficult, while finite designed data and fixed substitution patterns constrain the benchmark’s scope.

  • 5 Conclusion: The benchmark measures imitation under progressively increasing mismatch between the human demonstration scene and the robot’s own scene.The hierarchy moves from exact motion and final object states toward semantic task completion and purpose-preserving affordance adaptation.
  • 5 Conclusion: IG-10K provides 20,000+ paired episodes across 50+ base tasks, six domains, simulation, and the real world, alongside Imitator Arena for structured human evaluation.The dataset uses VR teleoperation for real-robot collection and OMPL-planned trajectories from hand-crafted waypoints in simulation.
  • 5 Conclusion: The benchmark remains finite and manually designed, while simulated L3 results are biased by finite assets and rule-based planned motion.The evaluated policies were adapted to the Imitator Game rather than designed specifically for intent-level human-video imitation.
  • 5 Conclusion: The hierarchy distinguishes trajectory imitation at L0, object end-state imitation at L1, semantic-task imitation at L2, and affordance-adapted imitation at L3.L3 requires selecting an object whose affordance can be adapted to achieve the same purpose after a semantic-category change.
  • 5 Conclusion: The benchmark holds the demonstrator–robot embodiment gap approximately fixed while grading scene-level mismatch between demonstration and execution scenes.This makes the hierarchy orthogonal to embodiment-gap hierarchies that vary the bodies while holding the scene fixed.

B.3 Imitator Arena: Scoring Procedure and Metric Definitions

Imitator Arena combines level-sensitive human judgments with automated simulation metrics, while defining binary task success and continuous sub-goal completion measures. Its scoring protocol standardizes evaluation across models, levels, and domains, but automated metrics remain limited to simulation.

  • Platform and annotation protocol: Arena presents synchronized demonstration and rollout videos, collects absolute success, quality, and pairwise preference judgments, and hides the evaluation level from annotators.Screens use one demonstration and two anonymized, randomly ordered rollouts on the same imitator scene.
  • Quality rubric: Human evaluators assign imitation-quality scores q from 0 to 10 using five fixed anchors, with intermediate integers interpolating between adjacent anchors.The same anchored rubric is applied across human evaluators, models, and levels.
  • Automated metrics: Final success rate is binary: an episode succeeds only when all hand-crafted, task-specific goal-state predicates hold at termination.The predicates reflect each task’s goal states and intermediate milestones.
  • Automated metrics: Sub-SR measures the fraction of ordered task phases completed when each phase’s peak shaped sub-reward exceeds a fixed threshold.It provides a continuous proxy for imitation depth when final success is near zero.
  • Metric scope: Automated SR and Sub-SR apply uniformly in simulation, whereas real-hardware absolute success is determined solely through Arena human evaluation.The benchmark omits trajectory similarity because embodiment differences make a trajectory-level predicate ambiguous.
  • Baseline adapters: The benchmark freezes video or vision-language encoders and trains only action heads, enabling cached representations and comparable trainable-parameter counts across families.The authors report comparable gains when encoders were unfrozen but retain freezing to make the full comparison computationally feasible.

C Additional Experimental Results

Additional experiments compare imitation interfaces, frozen video encoders, and automated versus human scoring. They show substantial within-family variation, agreement between scoring channels, and family-level rankings that depend on the evaluated variants.

  • Q1: imitation interface: Video-conditioned families cluster at the upper-right of the seen-SR versus P+FT-SR plane, while VLA variants range from OpenVLA (0.29) to π0.5 (0.73).Seen-task strength and few-shot adaptability are correlated but not interchangeable; GR00T and π0.5 adapt better than their seen-task scores predict.
  • Q1: imitation interface: Within-family seen-task SR spans 0.29–0.73 for VLA and 0.14–0.81 for Video-VA, exceeding the widest between-family gap of 0.28.Video-Skill is the tightest family, with similar behavior across its two members.
  • Q1: imitation interface: Simulation Arena win rates reproduce the automated family ranking: Video-Skill 0.64, Video-VA 0.50, and VLA 0.42.These blind A/B judgments use the same rollouts as the automated comparison.
  • Q1: frozen video encoder: Figure 9 compares three Video-VA action heads crossed with three frozen encoders across automated simulation panels and a real-world Arena panel.Real-world ACT runs for video encoders use the 45-task pre-training scale.

C.2 Q1: does the frozen video encoder matter?

The frozen video encoder materially affects performance, but its impact depends on the action head. Encoder ordering in simulation is largely reproduced on hardware.

  • Encoder–action-head interaction: 0.53 SR separates VideoMAE from DINOv2 for Diffusion Policy, showing that encoder choice can determine near-failure for a given action head.VideoMAE changes Diffusion Policy from 0.67 to 0.14 SR, while its ACT penalty is 0.09.
  • Few-shot adaptation: Ten demonstrations nearly close ACT’s encoder gap but do not similarly repair Diffusion Policy or VQ-BeT.Few-shot adaptation compensates for a poor task embedding only when the action head is already strong.
  • Cross-domain validation: At the matched 45-task scale, hardware reproduces the simulation encoder ordering across every regime: VideoMAE is last and DINOv2 and SigLIP2 are close.This cross-domain agreement supports the robustness of the encoder comparison.

C.3 Q2: pre-training scale, zero-shot transfer, and few-shot adaptation

Larger paired pre-training corpora improve fine-tuned transfer across domains and, generally, hierarchy levels. However, scaling improves intent-level transfer without eliminating its gap, while simulation zero-shot remains unreliable.

  • Corpus scaling: 18 of 19 trained variants gain ∆SR when pre-training expands from 15 to 45 tasks.The gains occur in 14/15 simulation variants and all 4/4 hardware models.
  • Corpus scaling: By 45 tasks, every paradigm’s P+FT curve exceeds its domain-specific SCRATCH baseline.The baselines are simulation 0.35 and real world 0.25; both conditions receive the same ten few-shot demonstrations.
  • Hierarchy-level scaling: Real-world unseen-task P+FT success at L3 rises from 0.23 to 0.36 with scale but remains below L0–L2 at every scale.Scaling helps intent-level transfer without closing the intent-level gap.
  • Hierarchy-level scaling: Success improves monotonically or near-monotonically from 15 to 45 pre-training tasks at nearly every level and in every setting except simulation zero-shot.Simulation zero-shot stays at SR ≤0.14 throughout, so it is not used for ranking.

C.5 Q3: where the hierarchy becomes hard

Real-world success is stable through L2 but drops at L3, where functional substitution requires handling an object differently. Simulation shows a construction-related L2 dip and an L3 rebound.

  • Real-world results: 0.54/0.55/0.54 real-world seen-task SR across L0–L2 falls to 0.39 at L3.Completion quality follows the same pattern, declining from 6.97/7.12/7.07 to 6.09.
  • Real-world results: 0.42/0.42/0.39 real-world P+FT SR across L0–L2 drops to 0.29 at L3.For XSkill, SR falls from approximately 0.53–0.57 through L2 to 0.29 at L3, while Q declines from 7.2 to 5.6.
  • Simulation results: Simulation seen-task SR dips at L2 and rebounds at L3, a pattern reproduced by Arena scoring on the same rollouts.The per-family Sub-SR curves show the same dip-then-rebound shape, so the effect is not confined to one paradigm or binary predicate.
  • Interpretation: The simulated L2 dip reflects deliberate arm swapping and a finite, rule-based asset pool, while hardware lacks this shortcut for functionally different objects.Hardware substitutions differ in mass, shape, and gripping requirements, making adaptation necessary.
  • Sanity check: The demonstration-swap sanity check tests whether models rely on the correct conditioning video while keeping the imitator scene unchanged.It uses 45-task checkpoints and compares original demonstrations with similar or unrelated task videos.

C.7 How far do the two evaluation channels, and the two domains, agree?

Automated and Arena evaluation agree closely on identical simulation rollouts and reproduce the same hierarchy-level pattern. A full-coverage check further tests whether the selected tasks represent the wider pool.

  • Channel agreement: r = 0.858 for SR versus SRhuman and r = 0.861 for Sub-SR versus Q indicate close agreement between automated and Arena scoring.This agreement supports using automated scoring for the full simulation model grid.
  • Channel agreement: Both evaluation channels reproduce the simulation dip at L2 and rebound at L3 on seen-task rollouts.The agreement holds for both success metrics and completion-quality metrics when resolved by hierarchy level.
  • Task coverage: DP and ACT trained on all 50 simulation tasks are evaluated across every task to check whether the ten-task comparison set represents the wider pool.The sanity check is intended as coverage validation rather than a cross-model comparison.

F.5 RDT-1B implementation details

The implementations adapt several robot policies to human-video or caption conditioning while freezing selected pretrained components and evaluating them under standardized rollout protocols in simulation and the real world.

  • RDT-1B: RDT-1B uses frozen SigLIP-SO400M/14 and T5-v1.1-XXL components with LoRA adapters for language, image, state, and action conditioning.Actions use the original unified action space with a validity mask, a 16-step horizon, and 30 Hz control-frequency tokens.
  • GR00T-N1.6-3B: GR00T-N1.6-3B freezes its Eagle vision-language backbone while training the flow-matching action expert and embodiment-specific state/action components.The pretraining configuration uses 10 epochs with AdamW, learning rate 1×10−4, weight decay 1×10−5, and a cosine schedule.
  • Video-conditioned policies: ACT, Diffusion Policy, and VQ-BeT condition on a 256-dimensional task embedding extracted from four uniformly sampled human-video frames.The task embedding is produced using frozen DINOv2-ViT-L/14, SigLIP2-SO400M, or VideoMAE-Large encoders with a trained one-layer adapter.
  • VQ-BeT: VQ-VAE is trained for 100 epochs on robot action chunks, followed by 100 epochs of GPT-policy training with batch size 256 and learning rate 1×10−4.The GPT policy is trained after the action-chunk representation has been learned.
  • Evaluation: Simulation evaluates 10 trials per task–level pair across 15 variants, while real-world evaluation uses 5 trials per task–level pair across the same 40 task–level pairs.At inference, video-conditioned policies encode the video once at episode start, whereas VLA policies reuse the fixed caption T(V ).

G Failure Modes and Task-Level Diagnostics

Real-world and simulation diagnostics show that failures often arise from contact precision and affordance adaptation rather than complete task misunderstanding, with L3 exposing changed target relations most clearly.

  • Real-world failure modes: Plausible task behavior does not guarantee reliable execution when precise grasping or release is required.Observed failures include insufficient lift, premature gripper closure, failure to release, and unstable grasps.
  • Task-level patterns: STIR SOUP IN BOWL, DISCARD FOOD WASTE, and SCAN BEVERAGE BARCODE are especially vulnerable to thin tools, deformable objects, bimanual coordination, and grasp instability.These tasks frequently fail after small contact errors or incomplete gripper closure.
  • Unseen transfer: L3 reveals missing affordance adaptation in unseen tasks even when policies can localize and grasp the demonstrated object.For RETURN REMOTE TO BOX, failures include wrong placement, insufficient lift, or premature release in the substitute-container setting.
  • Unseen transfer: Few-shot real-world data remain too small to calibrate precise grasp points, release timing, and level-specific affordance substitutions.The released rollout metadata supports tracing these qualitative failure modes to representative benchmark episodes.
  • Simulation per-task view: PICKREMOTECONTROL remains strong through L0–L2 for DP and ACT but drops at L3, matching failures to execute changed target relations.PLACEFOODSCALE and PLACECOMMODITYRACK instead stress spatial alignment and contact geometry, indicating manipulation coverage and calibration issues.

H Per-Task Reference Results

Table 11 reports per-task simulation reference results across the benchmark’s imitation levels using success rates measured over repeated evaluation episodes.

  • Reference results: Table 11 provides per-task reference results for simulation success rate.The table organizes results by task and imitation level.
  • Reference results: Each table cell reports success rate (%) for its corresponding task and imitation level.The metric is computed over 10 evaluation episodes per cell.
  • Reference results: The per-task and per-level layout enables comparisons of simulation performance across imitation conditions.The caption specifies the task–level correspondence for every cell.
Loading 2608.22301v1…