PaperlayerView exact revision on arXiv

Source-linked AI summary

Track-and-Complete: Learning Humanoid Skills from a Single Failed Human Video

Sarmad Idrees, Jongeun Choi

arXiv:2609.36924v1cs.ROcs.CV

TL;DR

Successful-video imitation often requires costly custom collection, while failed videos can contain a useful motion prefix and intended outcome. TRACC tracks that prefix and uses inferred intent to complete tasks; experiments demonstrate learning from six failed-action videos without successful completion trajectories.

  • Problem

    Successful-video imitation often requires tedious, expensive custom collection, while treating failures only as negative examples discards useful prefixes and task intent.

    Source: §I · p. 1

  • Method

    TRACC retargets and tracks a failed video's useful prefix, then uses inferred intent and task-completion reward after the Point-of-No-Return transition.

    Source: §I · p. 2, §III · p. 3

  • Results

    Across six Oops! failed-action videos, TRACC tracked initial guidance and subsequently completed tasks without successful task-completion trajectories.

    Source: §VI · p. 7

  • Takeaways & Limitations

    Failed human videos can provide useful information for humanoid skill learning and establish a setting for future research on unsuccessful demonstrations.

    Source: §VI · p. 7

    The current approach is limited to one humanoid embodiment and non-dexterious humanoid-object interaction tasks.

    Source: §V · p. 7

Abstract

from arXiv · show

Learning humanoid skills from videos typically requires a successful human demonstration, which often demands custom data collection. Although failures have traditionally been treated only as negative examples in robot learning, they can still reveal a usable trajectory prefix before the task fails, as well as the intended outcome. To leverage this information from a failed-attempt video, we propose TRACC, a pipeline that imitates the useful portion of the motion trajectory and then completes the task based on the inferred task outcome. The usable motion prefix serves as prior knowledge until the failure occurs, after which the task-completion reward guides the policy to learn the intended task goal without requiring a successful task trajectory. We evaluate our method on six in-the-wild failed human tasks from the Oops! dataset. Our experimental results demonstrate the effectiveness of the proposed approach for learning from failed attempts when no successful demonstration is available. Thus, these findings establish failed human videos as a viable source of supervision for humanoid skill learning.

I. INTRODUCTION

Successful-video imitation can require tedious, expensive custom data collection, while failed attempts contain both useful motion before failure and clues to the intended outcome. TRACC uses these signals to track a viable prefix and then learn task completion; the paper evaluates this approach on six failed-action videos.

I. INTRODUCTION

  • Relying on successful attempts for video-based humanoid learning often requires tedious, expensive custom data collection.Source: §I · p. 1
  • Failed attempts can contain useful preparation before failure and reveal intended goals, but whole-video imitation includes the failed ending and negative-only treatment discards the useful prefix.Source: §I · p. 1
  • TRACC treats a failed video as motion guidance for how a task begins and inferred intent for how it should end, using retargeted motion and a task-completion reward.Source: §I · p. 2
  • PoNR marks the last trajectory point still useful before failure, enabling tracking reward to fade as task-completion reward takes over.Source: §I · p. 2
  • The authors present TRACC as the first method, to their knowledge, to learn whole-body humanoid skills from one failed human video using pre-failure motion and an inferred outcome.Source: §I · p. 2
  • On six failed-action videos, TRACC tracked initial guidance and subsequently completed tasks without successful task-completion trajectories; the selected videos had usable prefixes and simulation-feasible outcomes.The videos came from the Oops! dataset.Source: §I · p. 2

II. RELATED WORK

Prior video-based humanoid learning recovers and imitates demonstrated motion, generally requiring a successful task demonstration. Work on failed demonstrations has used them for avoidance, confidence weighting, rankings, or goal inference; TRACC instead uses pre-failure motion as positive guidance and infers a completion reward.

a) Video-Based Humanoid Learning

  • Video-based imitation recovers reference motion from monocular video and imitates demonstrated actions,,; transferred humanoid methods likewise learn from human-video demonstrations,, [20], [2].These methods require a successful task demonstration as behavior to reproduce.Source: §II · p. 3

b) Learning From Failed Demonstrations

  • Prior failed-demonstration methods use failures for avoidance, confidence weighting [21], suboptimal-behavior rankings, or visual-planning goal inference.Source: §II · p. 3

c) Language Models for Reward Generation

  • Language models can generate executable reinforcement-learning rewards, [24],, while TRACC uses a VLM to infer a failed action's intended outcome and generate a task-completion reward.Related reward work includes sim-to-real transfer [26], video imitation rewards, and VLM reward or preference models [28],.Source: §II · p. 3

III. TRACC: TRACK-AND-COMPLETE

TRACC treats a failed-attempt video as partial supervision: it tracks the useful motion prefix, then learns to complete the intended outcome. Its unified policy samples a release time and shifts from tracking guidance to task reward.

III. TRACC: TRACK-AND-COMPLETE

  • TRACC recovers and aligns human motion, infers task intent and reward candidates, trains a prefix tracker, then fine-tunes a unified policy to track and complete the task.The pipeline uses a separate task-specific unified policy for each task.Source: §III · p. 3

A. Problem Formulation

  • The policy aims to reproduce the useful trajectory from a failed video and complete the intended task, using retargeted motion as its reference.The formulation defines the video as a sequence of frames and the retargeted reference as a sequence of states.Source: A. Problem Formulation · p. 3
  • During each episode, the policy samples a release time from the VLM-proposed PoNR window, uses tracking guidance before release, and task reward alone afterward.The release time is included in the policy observation, and the reference trajectory is masked after release.Source: A. Problem Formulation · p. 3

B. Motion Reconstruction

Motion reconstruction converts the monocular human video into a scene-aligned humanoid reference trajectory, with the human motion retargeted to the robot and aligned to task objects.

B. Motion Reconstruction

C. Prefix Reference Tracking

Prefix tracking uses only the usable portion of the failed motion as a positive imitation target. The tracker is first trained with varied reference-state resets, then fine-tuned from the initial frame to address errors that accumulate during full execution.

C. Prefix Reference Tracking

  • Only the usable prefix of the retargeted motion is used as a positive imitation target, with its cutoff determined by the VLM-estimated PoNR window.Source: C. Prefix Reference Tracking · p. 4
  • The tracking reward combines pose, joint-velocity, end-effector, and root tracking terms, weighted by their contributions and sensitivities to mean-squared tracking error.Source: C. Prefix Reference Tracking · p. 4
  • Reference-state initialization exposes the policy to every motion phase, but initial-frame fine-tuning makes it reach later prefix states through its own dynamics.The initial-frame reset distribution matches unified-policy training and addresses errors that reference-state initialization alone can hide.Source: C. Prefix Reference Tracking · p. 4

D. Intent Inference and Reward Generation

A VLM infers the intended outcome, proposes task-completion rewards, and supplies a PoNR window for release-time sampling. The unified policy keeps task reward active while tracking reward fades before release, then completes the task without reference guidance.

D. Intent Inference and Reward Generation

  • The VLM infers the intended outcome, generates three task-reward candidates, and proposes a PoNR interval rather than a single release timestamp.The interval is intended to accommodate uncertainty about when demonstrated behavior begins to deteriorate.Source: D. Intent Inference and Reward… · p. 4
  • The VLM-proposed PoNR window is a visual training prior; measured PoNR is the latest tested release time with the best terminal success rate.The trained policy is evaluated at fixed release times across the video timeframe.Source: D. Intent Inference and Reward… · p. 4

E. Unified Policy Training

  • The unified policy starts from the prefix tracker, and each VLM reward proposal receives the same training budget before the best overall success proposal is selected.Source: E. Unified Policy Training · p. 4
  • Task reward remains active throughout training while tracking reward fades to zero before release; afterward, the reference is masked and the policy uses robot-state and task observations.The release time is provided as an observation so the policy can anticipate the disappearance of reference guidance.Source: E. Unified Policy Training · p. 5

A. Experimental Setup

The experiments use six failed human demonstrations spanning varied humanoid tasks, evaluated in simulation. The setup specifies the robot, policy, and training procedure used across tasks.

A. Experimental Setup

  • Six failed Oops! videos cover box jumping, log walking, target kicking, football kicking, backflip, and handstand, spanning impulsive motion, narrow-support control, and object contact.Experiments use a 29-DoF Unitree G1 in Isaac Gym.Source: A. Experimental Setup · p. 5
  • The policy outputs normalized joint-position targets at 30 Hz and trains in 4,096 parallel environments, with GPT-5 used as the VLM and task success criteria defined manually.Manual success criteria are used instead of VLM-generated criteria.Source: A. Experimental Setup · p. 5
  • Separate MLP encoders map proprioception, task, and reference-motion states to 64-dimensional tokens for a four-layer, two-head Transformer and action head.The feed-forward layer is 512-dimensional and the action head is [1024,512].Source: A. Experimental Setup · pp. 5–6
  • Policies use PPO with a 32-step rollout horizon, γ = 0.99, GAE 0.95, clipping ratio 0.2, and learning rate 2×10−5.The prefix tracker receives 1,500 additional training iterations and 800 fine-tuning iterations; task-reward candidates receive equal 300-iteration budgets before the best is further fine-tuned.Source: A. Experimental Setup · p. 6
  • The 18-dimensional reference block is masked to zero at release time τ, while task-observation dimensions vary with the task.The task block contains the object, goal, and scene quantities required by each task.Source: A. Experimental Setup · p. 6

B. Evaluation Results

Across the evaluated tasks, policy behavior depends on how much useful reference motion is retained before release. The useful boundary can differ from the VLM-proposed window, and task-specific outcomes show how the policy completes or fails after release.

Kick target

  • For target kicking, recovered motion supplies the step-in, support-leg alignment, and leg swing; task reward teaches reaching the target, balancing, and recovery.Success remains high beyond the visual window, but recovery fails when the reference extends into purely unusable motion; measured τ* falls after the visual window but before motion becomes harmful.Source: B. Evaluation Results · p. 6

Football

  • For football, success follows once guidance reaches a useful approach and leg-swing state, while early release timing is sensitive to the state at removal.The unified policy outperforms the task-reward-only policy in success rate.Source: B. Evaluation Results · p. 6

Backflip

  • In backflip, inversion contributes only 0.05 to reward although success requires full inversion before landing, so early release can produce high survival but low task success.Once reference motion provides the inverted state, the policy only needs to land and settle rather than learn the full backflip.Source: B. Evaluation Results · pp. 6–7

Box jump

  • For box jump, intermediate releases expose a useful state for completing the jump and landing, while later releases sharply reduce success as reference enters the failed continuation.The approach and crouched loading motion prepare the humanoid for takeoff.Source: B. Evaluation Results · p. 7

Handstand

  • For handstand, the VLM window is near early visual failure at approximately 2.5–2.8 s, but greatest success occurs near 6.7 s because the later motion already produces an inverted hand-supported state.The policy then stabilizes the pose for the remaining hold duration.Source: B. Evaluation Results · p. 7

Log walk

  • Log walking uses an approach and initial balance-oriented steps as prefix, then requires maintaining balance and walking at least 1.0 m for success.Source: B. Evaluation Results · p. 7

Overall

  • The VLM-proposed PoNR window is a training prior, while the useful release boundary depends on the task and trained policy and is determined from the release-time curve.Release time is an observation, and success outside the sampled window indicates temporal generalization rather than reliance on one hand-selected release time.Source: B. Evaluation Results · pp. 6–7

C. Ablation Studies

The ablations examine the value of prefix guidance and of retaining one policy across reference release. Unified training provides task success where reward-only training does not, and avoids the release discontinuity of policy chaining.

Prefix guidance vs. task reward only

  • Unified policies produce non-zero success on all six tasks, whereas task-reward-only policies fail the terminal success criteria despite high survival success.The comparison evaluates prefix guidance by removing the prefix trajectory and training only with selected task rewards.Source: C. Ablation Studies · p. 7

Unified policy vs. policy chaining

  • Policy chaining is tested on log walking because narrow-support locomotion makes control discontinuities immediately visible; rollouts are aligned at τ* and jerk is measured over the complete episode.Source: C. Ablation Studies · p. 7
  • At release τ, the unified policy has 2.1× lower normalized joint jerk than a two-policy chain, avoiding the sharp discontinuity caused by switching independently trained policies.The chain rises from around 0.17 before release to 0.90 at the switch; the unified policy changes from 0.30 to 0.42.Source: C. Ablation Studies · p. 7

V. DISCUSSION AND FUTURE WORK

The discussion finds failed-video supervision useful when motion prefixes and intended outcomes are combined, while identifying scope limits and future extensions. The conclusion reports that prefix guidance matters and a unified policy reduces release discontinuity from policy chaining.

V. DISCUSSION AND FUTURE WORK

  • Visual failure does not always coincide with when recovered motion becomes dynamically harmful, supporting policy-conditioned evaluation of the Point-of-No-Return.Source: §V · p. 7
  • The current approach is limited to one humanoid embodiment and non-dexterous humanoid-object interaction tasks.Source: §V · p. 7
  • Future work targets dexterous hand-motion reconstruction and retargeting for precise hand-object manipulation, as well as failed demonstrations in multi-humanoid collaboration.Source: §V · p. 7

VI. CONCLUSION

  • Prefix guidance matters, and a unified release-time-conditioned policy reduces the release discontinuity introduced by policy chaining.Source: §VI · p. 7

Related papers

Loading 2609.36924v1…