Source-linked AI summary
Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics
Peiyan Tu, Cong Wang, Hanxin Zhu, Xin Jin, Zhibo Chen
TL;DR
Embodied world models face limited interaction coverage, a human–robot embodiment gap, and sparse failure examples. HF-WM combines stabilized pointflow learning from human interactions with reward-guided post-training on perturbed actions. It achieves state-of-the-art action-conditioned video generation and accurately estimates real-world policy performance.
Problem
Embodied datasets offer limited interaction and failure coverage, while human egocentric data presents a human–robot manipulation gap and viewpoint instability.
Source: §1 · pp. 1–2Method
HF-WM combines a dual-branch video and pointflow framework for human–robot motion representation with GRPO post-training on perturbed successful actions, guided by action-alignment and physical-plausibility rewards.
Source: §2 · p. 2, §3.2 · p. 6, §3.3 · p. 6Results
HF-WM achieves state-of-the-art action-conditioned video generation and accurately estimates real-world policy performance.
Source: §5 · p. 10Takeaways & Limitations
The results highlight the value of learning from human interactions and modeling failure dynamics to extend world models beyond successful robot demonstrations.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Action-conditioned world models offer a promising foundation for generalist robotic intelligence by predicting the consequences of actions. However, existing robotic datasets provide limited coverage of the diverse interaction dynamics required for generalizable world modeling. Large-scale egocentric human videos offer complementary interaction experience, yet the embodiment gap between human hands and robotic manipulators hinders effective cross-embodiment learning. Meanwhile, the predominance of successful demonstrations leaves failure dynamics substantially underrepresented, limiting the ability of world models to anticipate unsuccessful executions. To address these limitations, we introduce HF-WM, an action-conditioned world model that combines Human interaction learning with Failure-aware post-training. To bridge the embodiment gap, we develop a dual-branch framework that couples video generation with auxiliary pointflow prediction, using stabilized pointflow as a unified motion representation across human and robot interactions while suppressing egocentric camera motion. To improve failure modeling, we further perform Group Relative Policy Optimization (GRPO) post-training on perturbed actions derived from successful demonstrations, guided by action alignment and physical plausibility rewards, enabling the model to learn plausible failure dynamics without paired ground-truth videos. Extensive experiments demonstrate that HF-WM achieves state-of-the-art performance in action-conditioned video generation and exhibits strong utility for policy evaluation in downstream robotic tasks.
1 Introduction
Action-conditioned world models predict future states from robot actions and are viewed as a foundation for robot planning, but embodied data remain limited in scale and coverage. Human egocentric videos and underrepresented failure outcomes pose complementary challenges for learning interaction dynamics.
1 Introduction
- Action-conditioned world models predict state transitions from robot action sequences, supporting scalable data generation, policy evaluation, and reinforcement learning in generated environments.The paper presents this paradigm as a foundation for learning and improving robot action planning systems.Source: §1 · p. 1
- Limited scale and coverage of high-quality embodied data constrain world models’ ability to capture interaction dynamics and execution outcomes.Recent studies have therefore turned to large-scale egocentric human-object interaction datasets.Source: §1 · p. 1
- Human egocentric recordings retain a human–robot manipulation gap, while camera jitter and unstable viewpoints can obscure interaction regions and hinder physically meaningful representations.Effectively exploiting this data for embodied world-model learning remains an open challenge.Source: §1 · p. 1
- Embodied datasets emphasize successful demonstrations and offer limited failure coverage, leaving the synthesis of failure scenarios from erroneous or suboptimal actions an open problem.World models are expected to capture both successful trajectories and failure dynamics or counterfactual outcomes.Source: §1 · p. 2
2 Related Work
Prior work connects controllable video generation with action-conditioned dynamics prediction and policy evaluation, but embodied data and failure coverage remain limited. HF-WM combines cross-embodiment motion representations with reward-guided post-training on perturbed actions.
2 Related Work
- HF-WM uses a skeleton-grounded shared spatial formulation and stabilized pointflow supervision to align human and robot motion while suppressing viewpoint variation (Wu and Gao, 2026; Huang et al., 2026; Lee et al., 2026).Point trajectories track interaction-driven dynamics rather than camera-induced motion.Source: §2 · p. 2
- HF-WM’s second-stage GRPO framework perturbs successful action trajectories and optimizes generated videos using action-following and physical-plausibility rewards, without paired failure-video supervision (Liu et al., 2026; Chen et al., 2026).The two rewards assess action-condition consistency and physically consistent predicted dynamics.Source: §2 · p. 2, Figure 1 · p. 2
- HF-WM’s dual-branch framework learns from egocentric human interactions and counterfactual action perturbations, with auxiliary pointflow supervision for a shared human–robot motion representation.The paper reports state-of-the-art action-conditioned video generation and effectiveness as a policy evaluator for downstream robotic tasks.Source: §2 · p. 3
- Controllable video-generation methods condition pretrained diffusion models on spatial structure, temporal dynamics, or multimodal inputs through encoders, cross-attention, or ControlNet (Zhen et al., 2025; YU et al., 2025; Yang et al., 2026).Examples of conditioning signals include poses, depth maps, skeletons, optical flow, motion trajectories, camera parameters, reference images, and audio.Source: §2 · p. 3
- IRASim generates future visual states under robot-action conditions, Ctrl-World extends this to multi-view embodied generation, and SC3-Eval uses world models for offline policy evaluation (Zhu et al., 2025; Guo et al., 2025; Tseng et al., 2026).These works position action-conditioned world models at the intersection of video generation, dynamics prediction, and policy evaluation.Source: §2 · p. 3
- Existing action-conditioned world models remain constrained by limited embodied data and failure coverage, motivating investigation of human-object interactions and counterfactual failure samples (Gao et al., 2026; Wu and Gao, 2026).The related work frames these samples as a route to improving action-consequence and physical-dynamics modeling.Source: §2 · p. 3
3 Method
HF-WM combines shared human–robot action representations and stabilized pointflow supervision with GRPO post-training on counterfactual actions. Its auxiliary pointflow branch learns interaction motion from video representations, while failure-aware optimization uses action alignment and physical plausibility rewards.
3.1 Preliminaries
- Rectified-flow generation predicts future visual latents conditioned on text, visual observations, and action tokens; GRPO uses a marginal-preserving SDE formulation for reward-guided denoising.The generator is trained with a rectified-flow matching objective, and the GRPO formulation permits pixel-level rewards without paired video ground truth.Source: §3.1 · pp. 3–4
3.2 Learning Across Embodiments: From Human to Robot
- Human hand and robot arm actions are mapped into a shared 2D skeleton representation, while auxiliary pointflow supervision guides video representations toward interaction-relevant motion.Stabilized pointflow compensates for camera motion, and the auxiliary branch is used only during training, adding no inference overhead.Source: §3.2 · pp. 5–6
- The pointflow branch predicts interaction motion from intermediate video representations and avoids direct access to action-sketch tokens, encouraging supervision from generated visual dynamics.The branch uses intermediate latents from layers 4, 6, 8, and 10 and adds no inference overhead.Source: Figure 2 · p. 4, §3.2 · p. 5
3.3 Learning Across Outcomes: From Success to Failure
- GRPO post-training learns failure dynamics from counterfactual action perturbations, addressing the limited failure supervision in successful-demonstration datasets.For trajectories with clear gripper transitions, perturbations target gripper state, transition timing, or end-effector pose; otherwise, pose perturbations focus on action following and motion consistency.Source: §3.3 · p. 6
- GRPO generates multiple candidate futures for each counterfactual action and optimizes them with action alignment and physical plausibility rewards, without paired future videos.The physical checklist assesses contact, object motion, and robot structure; critical physical violations cap physical reward and remove the action alignment reward.Source: §3.3 · p. 6, Action Alignment Reward · p. 7
Action Alignment Reward.
- The action alignment scorer compares video–action samples sharing the same video but using different skeleton actions, learning preference direction and gap magnitude.The ranking term favors the better-aligned sample; target gaps vary with perturbation type, severity, and visual discernibility.Source: Action Alignment Reward · p. 6
- Figure 3 compares HF-WM with six models on successful and failed executions, with its caption reporting better visual quality and precise action following.The caption identifies a qualitative comparison rather than reporting numerical results.Source: Figure 3 · p. 7
4 Experiments
HF-WM is evaluated through baseline comparisons, component ablations, and policy evaluation, using robot and egocentric human interaction data in a two-stage training setup. It performs strongly across action-conditioned modeling outcomes and its estimated policy success rates closely track real-world performance.
4.1 Experimental Setup
- Training proceeds in two stages on 8 NVIDIA A100 GPUs, using about 45K robot trajectories and 83K human interaction trajectories.Stage one uses supervised fine-tuning with stabilized pointflow; stage two applies GRPO to improve failure-outcome modeling.Source: §4.1 · p. 7
4.2 Comparison with Baselines
- HF-WM achieves state-of-the-art performance across visual quality, action following, and dynamics modeling, and maintains strong results on successful and failed executions.The comparison uses held-out Franka trajectories and real-world failed trajectories.Source: Table 1 · p. 8, §4.2 · p. 8
4.3 Ablation Studies
- Human interaction data improves performance over embodied data alone, and stabilized pointflow with human data yields larger gains across most metrics than human data alone.The results suggest that shared motion representations and camera-motion suppression facilitate cross-embodiment transfer.Source: Figure 4 · p. 8, Table 2 · p. 9, §4.3 · p. 9
- Action alignment and physical plausibility rewards complement one another: their combination improves action adherence while maintaining physically coherent scene dynamics.Physical-plausibility-only training can produce static or action-contradictory motion; alignment-only training can compromise background temporal consistency and interaction plausibility.Source: Table 3 · p. 9, §4.3 · p. 9
4.4 Policy Evaluation
- Predicted success rates across six policies strongly correlate with real-world rates (Pearson r = 0.983), while MMRV = 0.002 measures preservation of relative policy performance.HF-WM evaluates the policies on 90 matched sessions each.Source: Figure 5 · p. 9, §4.4 · pp. 9–10
5 Conclusion
HF-WM combines egocentric human interaction learning with failure-aware post-training. Its experiments show state-of-the-art action-conditioned video generation and accurate estimation of real-world policy performance.
5 Conclusion
- HF-WM combines stabilized pointflow supervision with GRPO post-training on perturbed actions to model human and robot interactions and failure dynamics without paired failure videos.The GRPO rewards target action alignment and physical plausibility.Source: §5 · p. 10
- Experiments demonstrate state-of-the-art performance in action-conditioned video generation and accurate estimation of real-world policy performance.The conclusion identifies learning from human interactions and modeling failure dynamics as valuable for extending world models beyond successful robot demonstrations.Source: §5 · p. 10
- Stabilized pointflow provides a shared motion representation across human and robot interactions while reducing egocentric camera-motion effects.This is the conclusion's stated approach to cross-embodiment learning.Source: §5 · p. 10
A Implementation Details
The appendix details the attention constraints and pointflow stabilization used for cross-embodiment learning, along with the event-based action perturbations and reward procedures used for failure-aware post-training.
A.1 Learning Across Embodiments: From Human to Robot
- Pointflow queries read only generated-video features, while camera-motion compensation reduces background drift and preserves hand-object motion.The pointflow trajectories are stabilized before supervision is temporally aligned with video latents.Source: §A.1 · p. 13, Figure 6 · p. 13, Figure 7 · p. 14
A.2 Learning Across Outcomes: From Success to Failure
- GRPO clips are anchored to gripper-state transitions, classified into seven event patterns, and routed to no, gripper, or geometric perturbations.Half of sampled windows retain original actions; geometric candidates are accepted only after visual and kinematic checks.Source: §A.2 · pp. 14–15, Table 4 · p. 15
A.2.1 Action Alignment Reward
- The action-alignment reward compares generated video with conditioned actions using global robot-motion and local end-effector views, with graded preference pairs spanning perturbation severity.Pairs can distinguish mild from severe misalignment without requiring the preferred action to be fully correct.Source: §A.2.1 · pp. 15–16, Figure 8 · p. 16
A.2.2 Physical Plausibility Reward
- Physical plausibility is assessed with scene-specific checks of physical consistency, robot structure, contact causality, and visual quality; any Tier-1 failure triggers hard rejection.The score is the fraction of eight checks passed, but Tier-1 failures cannot be offset by Tier-2 passes.Source: §A.2.2 · pp. 16, 18
B.1 Training Details
Training uses staged fine-tuning and GRPO with compact video rollouts and action-alignment and physical-plausibility rewards. The baseline comparison reports a marked distinction between existing methods and HF-WM on failed executions.
B.1 Training Details
- First-stage supervised fine-tuning warms up on robot-only embodied data, then jointly trains with egocentric human interaction data; GRPO generates four 256 × 256 candidates using 12 denoising steps and combines action-alignment and physical-plausibility rewards at 0.4 and 0.6, respectively.Each candidate uses the first observation frame as the visual condition and predicts 40 future frames.Source: §B.1 · p. 18
B.2 Comparison with Baselines
- HF-WM maintains closely comparable performance on successful and failed executions, while existing methods show a clear performance drop on failures, especially IDM.The authors attribute the existing methods' degradation to bias toward successful trajectories under failure-inducing actions.Source: §B.2 · p. 18
B.3.1 Ablation on PointFlow Feature Layers
The ablation compares pointflow supervision attached to features from different backbone depths. Front-layer features perform best, so the final model uses layers [4, 6, 8, 10].
B.3.1 Ablation on PointFlow Feature Layers
- The ablation compares pointflow supervision using front layers [4, 6, 8, 10], middle layers [12, 14, 16, 18], back layers [20, 22, 24, 26], or uniform layers [8, 14, 20, 26].The configurations use intermediate video features extracted from different backbone depths.Source: §B.3.1 · pp. 18–19
- Front-layer features [4, 6, 8, 10] perform best, followed by uniform sampling, while middle- and back-layer features perform progressively worse.The final model uses [4, 6, 8, 10] for the pointflow branch.Source: §B.3.1 · p. 19, Table 6 · p. 19
- The authors attribute the front-layer advantage to shallower representations retaining fine-grained spatial structure and motion cues.Uniform sampling captures complementary information across feature hierarchies, whereas deeper features provide less direct motion information.Source: §B.3.1 · p. 19
B.3.2 Ablation on GRPO Rewards
The reward ablation finds that action alignment and physical plausibility rewards each have shortcomings alone, while combining them improves performance on successful and failed executions.
B.3.2 Ablation on GRPO Rewards
- Combining action alignment and physical plausibility rewards improves performance on both successful and failed executions.The accompanying qualitative results show more physically plausible object dynamics and failure outcomes under perturbed actions.Source: §B.3.2 · p. 19, Table 7 · p. 19
- Either reward alone improves some metrics but does not consistently improve both generation quality and action following.Action alignment alone can yield implausible dynamics, while physical plausibility alone can deviate from the conditioning actions.Source: §B.3.2 · p. 19
- Under perturbed actions, GRPO yields more physically plausible predictions of object dynamics and failure outcomes.The comparison is shown in Figure 11.Source: §B.3.2 · p. 19
C.1 Additional Analysis of Policy Evaluation
The additional analysis assesses HF-WM’s success-rate estimates and policy rankings against real-world outcomes. It reports a 2.41-percentage-point mean absolute deviation and strong agreement in policy ordering.
C.1 Additional Analysis of Policy Evaluation
- The evaluation covers six policies on 90 matched execution sessions, using predicted outcomes from HF-WM-generated videos.For each session, HF-WM receives the initial observation and policy action sequence; GPT-5 evaluates task success from the generated video.Source: §C.1 · p. 19
- 2.41 percentage points is the mean absolute deviation between predicted and real-world policy success rates.Table 8 reports the policy evaluation on 90 matched sessions.Source: §C.1 · p. 20, Table 8 · p. 22
- Both rank correlations indicate strong agreement between the predicted and real-world policy ordering.The analysis uses Spearman’s ρ and Kendall’s τb, with ties assigned average rank for the policy rankings.Source: §C.1 · pp. 20–21
- MMRV is nonzero because HF-WM predicts a tie between PG-FAST and PG-FSQ, whose real-world success counts differ by one.MMRV weights pairwise ordering violations by real-world success-rate gaps and averages each policy’s largest violation.Source: §C.1 · pp. 21–22
C.2 Test-Time Scaling for Policy Inference
For closed-loop LIBERO-90 manipulation, HF-WM predicts futures for candidate action chunks and RoboMeter selects one for execution. This test-time selection improves success rate and reduces episode length.
C.2 Test-Time Scaling for Policy Inference
- The experiments cover all 90 LIBERO-90 tasks with a shared π0.5 base policy, fine-tuning HF-WM on LIBERO training data before testing.Training uses 24-frame clips, while testing evaluates the first 16 future steps.Source: §C.2 · p. 22
- At each decision step, HF-WM predicts futures for four action chunks sampled from the same π0.5 policy, and RoboMeter selects the highest-scoring candidate.The comparison uses the same policy and closed-loop protocol, differing only in whether HF-WM prediction and RoboMeter scoring select candidates.Source: §C.2 · p. 22
- Success rate rises from 27.8% to 30.0%, while average episode length falls from 349.39 to 344.64 steps across 90 LIBERO-90 tasks.Figure 12 presents the test-time scaling comparison.Source: §C.2 · p. 22, Figure 12 · p. 22
C.3 Additional Visualization Results
The appendix adds HF-WM rollouts with pointflow visualizations and qualitative comparisons with representative baselines. Two real-robot task figures juxtapose world-model verdicts with human success labels.
C.3 Additional Visualization Results
- Additional HF-WM rollouts include corresponding pointflow visualizations.Figure 14 presents these additional qualitative results.Source: §C.3 · p. 23, Figure 14 · p. 24
- Additional qualitative comparisons show HF-WM alongside representative baselines.Figure 15 contains the comparisons; the supplied caption does not state their outcomes.Source: §C.3 · p. 23, Figure 15 · p. 25
- For the Uncover the wooden block task, 1 of 6 policies succeeds on the real robot; each strip shows world-model rollout and real-robot video at six time steps, with human success labels and world-model verdicts.A red cell marks disagreement.Source: Figure 16 · p. 26
- In the depicted task, 5 of 6 policies succeed on the real robot; each strip shows world-model rollout and real-robot video at six time steps, while columns give human success labels and world-model verdicts.Red cells mark disagreement.Source: Figure 17 · p. 27