PaperlayerView exact revision on arXiv

Source-linked AI summary

Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Training

Yaxuan Li, Zhongyi Zhou, Yefei Chen, Yanjiang Guo, Jiaming Liu, Shanghang Zhang, Jianyu Chen, Yichen Zhu

arXiv:2604.21741v2cs.RO

TL;DR

Pretrained generalist robot policies can still fail in concrete scenes, while physical corrective post-training carries robot, setup, reset, and supervision costs. Hi-WM uses a learned world model for human-guided, failure-targeted correction and reusable corrective trajectories. Across real-world manipulation tasks, it improves success over both the base policy and a world-model closed-loop baseline.

  • Problem

    Generalist policies can fail under contact-rich dynamics, scene variation, and long-horizon error accumulation, while physical corrective post-training requires costly robot execution, scene setup, and resets.

    Source: §1 · p. 2

  • Method

    Hi-WM rolls a pretrained policy out in an action-conditioned world model, lets humans correct failure-prone behavior, and reuses cached states to collect corrective trajectories for post-training.

    Source: §1 · p. 2

  • Results

    Across real-world manipulation tasks, Hi-WM consistently improves real-world success over both the base policy and a world-model closed-loop baseline.

    Source: §1 · p. 2, §5 · p. 11

  • Takeaways & Limitations

    Hi-WM presents an economically efficient approach to scaling post-training, reserving expensive real-world deployments for sparse but critical final validation.

    Source: §5 · p. 11

    Corrective trajectories are unreliable for post-training if the world model diverges substantially from real execution in robot motion or object interaction.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Post-training is essential for turning pretrained generalist robot policies into reliable task-specific controllers, but existing human-in-the-loop pipelines remain tied to physical execution: each correction requires robot time, scene setup, resets, and operator supervision in the real world. Meanwhile, action-conditioned world models have been studied mainly for imagination, synthetic data generation, and policy evaluation. We propose \textbf{Human-in-the-World-Model (Hi-WM)}, a post-training framework that uses a learned world model as a reusable corrective substrate for failure-targeted policy improvement. A policy is first rolled out in closed loop inside the world model; when the rollout becomes incorrect or failure-prone, a human intervenes directly in the model to provide short corrective actions. Hi-WM caches intermediate states and supports rollback and branching, allowing a single failure state to be reused for multiple corrective continuations and yielding dense supervision around behaviors that the base policy handles poorly. The resulting corrective trajectories are then added back to the training set for post-training. We evaluate Hi-WM on three real-world manipulation tasks spanning both rigid and deformable object interaction, and on two policy backbones. Hi-WM improves real-world success by 37.9 points on average over the base policy and by 19.0 points over a world-model closed-loop baseline, while world-model evaluation correlates strongly with real-world performance (r = 0.953). These results suggest that world models can serve not only as generators or evaluators, but also as effective corrective substrates for scalable robot post-training.

1 Introduction

Hi-WM moves failure-targeted human correction from physical execution into an action-conditioned world model, where corrective trajectories can be reused for post-training. Across real-world manipulation tasks, it improves success over both the base policy and a world-model closed-loop baseline.

1 Introduction

  • Physical corrective post-training requires robots, real scenes, human monitoring, and often manual resets, making its marginal cost grow as coverage expands.The stated coverage includes more tasks, scenes, checkpoints, and long-tail failure modes.Source: §1 · p. 2
  • Hi-WM rolls a pretrained policy out in an action-conditioned world model and has a human intervene when behavior becomes incorrect, uncertain, or failure-prone.The resulting corrective trajectories are used for post-training, including through imitation learning or reinforcement learning.Source: §1 · p. 2
  • Hi-WM consistently improves real-world success over both the base policy and a world-model closed-loop baseline, with gains attributed to failure-targeted human correction rather than simulated rollouts alone.The evaluation spans three real-world manipulation tasks and two policy backbones, with increasingly favorable economics as virtual intervention data scale.Source: §1 · p. 2
  • Caching, rewinding, and branching failure states yields dense recovery supervision without repeated real-world execution.The approach targets short corrective continuations near the base policy’s failure states rather than generic successful rollouts.Source: §1 · p. 2

2 Related Work

Prior work studies human corrective learning during physical execution and uses world models for generation, optimization, or evaluation. Hi-WM combines human-guided correction with a reusable world-model workspace for post-training around failure-prone states.

2 Related Work

  • Human corrective learning adapts policies with targeted supervision near policy failures, but interventions remain tied to physical execution and its robot, reset, and operator costs [54].Hi-WM moves this corrective interaction from the robot to a learned environment.Source: §2 · p. 3
  • World-model research uses learned dynamics for synthetic trajectory generation and policy optimization, typically treating the model as an autonomous generator, simulator, or optimizer.Hi-WM instead uses human-guided interaction to collect post-training trajectories around failure-prone states.Source: §2 · p. 3
  • World models have also been studied as policy-evaluation substrates for ranking, checkpoint selection, and pre-deployment screening [13] [60].Hi-WM couples evaluation with intervention and corrective-data collection in the same learned environment.Source: §2 · p. 3
  • Hi-WM’s distinguishing feature is caching, rewinding, and branching learned-environment failure states into multiple corrective continuations for post-training.This converts one-shot physical correction into reusable recovery supervision.Source: §2 · p. 3

3 Methodology

Hi-WM shifts human intervention into an interactive world model, where policies roll out, humans correct failure-prone behavior, and corrective trajectories are collected for post-training. State reuse, reset-free virtual interaction, and a hardware-agnostic interface support this workflow.

3.1 The Human-in-the-World-Model Paradigm

  • Hi-WM shifts the intervention loop from physical execution to an interactive world model, where intermediate states can be cached and branched into alternative recovery actions.Virtualizing interaction removes manual scene resets, and the approach also supports remote, parallel collection without hardware or location restrictions.Source: §3.1 · p. 3

3.2 System Overview and Design Requirements

  • The system runs a policy in closed loop inside the world model and lets a human provide corrective actions when behavior deviates from the desired trajectory.The design requires faithful model response, a shared continuous action space for policy and human control, and corrective segments usable for post-training.Source: §3.2 · p. 4

3.3 Interactive World Model for Hi-WM

  • The interactive world model uses 14-dimensional continuous action inputs and training data that include failures, off-task states, and edge cases such as workspace boundaries and precise gripper alignment.These data strategies are intended to distinguish successful from failing actions and keep model trajectories aligned with real-world execution.Source: §3.3 · p. 5

3.4 Hardware-Agnostic Intervention Interface

  • The intervention interface maps keyboard, calibrated robot-arm, or VR-controller inputs into the policy’s standard continuous action representation.This avoids dependence on a single input setup.Source: §3.4 · p. 5

3.5 Collecting Corrective Trajectories for Post-Training

  • During closed-loop collection, a human intervenes in unfamiliar or failure-prone states, and control can return to the policy once behavior is guided toward success.Cached states can be revisited to branch alternative corrections; the resulting segments are merged with real-world demonstrations for post-training, reducing repeated deployment, manual resets, and hardware use.Source: §3.5 · pp. 5–6, Figure 3 · p. 6

4 Experiments

The experiments are organized around real-world alignment, post-training effectiveness, scalability, and generalization. The supplied passages state these research questions but do not report their experimental outcomes.

4 Experiments

  • RQ1 asks whether Hi-WM is sufficiently aligned with real-world execution for reliable human intervention and data collection.The experiments are described as real-robot experiments.Source: §4 · p. 6
  • RQ2 asks whether Hi-WM can serve as an effective, policy-agnostic post-training framework.Source: §4 · p. 6
  • RQ3 asks how the cost advantage changes with scene coverage and how real-world performance scales with more virtual intervention data.Source: §4 · p. 6
  • RQ4 asks whether intervening on novel scenarios enhances real-world policy generalization.Source: §4 · p. 6

4.1 Experimental Setup

The experiments cover three tabletop manipulation tasks, including rigid and deformable object interaction, on a dual-arm robot. Policies are evaluated in closed-loop world-model rollouts against an unmodified base policy and a human-free world-model baseline.

Real robot tasks

  • The task suite spans towel folding, planar T-object pushing, and rope routing, covering deformable and rigid-object manipulation.The towel and rope tasks involve flexible objects; Push-T requires pushing and rotating a rigid object to a target pose.Source: §4.1 · p. 6, Figure 4 · p. 7

Robot setup

  • Real-world experiments use a dual-arm YAM-Ultra with 6-DoF control, a top-view camera, and actions executed at 15 Hz.Each arm also provides a gripper state.Source: §4.1 · p. 7

Policy eval initialization

  • World-model evaluation starts from a real task observation and iteratively feeds predicted actions and generated observations back to the policy.The baselines are the offline-data-only Base Policy and a human-free World Model Closed-Loop policy fine-tuned on its generated successful trajectories.Source: §4.1 · p. 7

4.2 Real-World Alignment of Hi-WM

The alignment evaluation considers whether model-specified positions can be physically reproduced, whether edge-case coverage improves alignment, and whether world-model task success tracks real-world success. Results include improved position realization with edge-case augmentation and a strong task-level correlation.

4.2 Real-World Alignment of Hi-WM

  • The alignment evaluation tests position reproduction, interpolation within the covered workspace, and agreement between model and real-world task performance.The authors note that visual plausibility alone is insufficient; model outcomes must also be realizable by the robot.Source: §4.2 · p. 7

4.2.1 Repeated Positioning Accuracy

  • Edge-case augmentation substantially reduces boundary-region positioning errors and improves physical realization of world-model target positions.About 30% of the regular-scene data volume in edge-case data already yields clear alignment gains.Source: §4.2.1 · p. 8, Figure 5 · p. 8

4.2.2 Interpolation within the Covered Workspace

  • Increasing edge-case data consistently raises PSNR and SSIM and lowers LPIPS, with full augmentation producing the strongest overall visual fidelity.The percentages denote fractions of the complete edge-case trajectory dataset.Source: Table 1 · p. 9

4.2.3 Task Level Evaluation Consistency

  • World-model and real-world success rates are strongly correlated, with Pearson r = 0.953.The authors describe the world model as a proxy for policy evaluation and a signal for when and where intervention is needed.Source: §4.2.3 · p. 9

4.3 Policy Performance via Hi-WM

Hi-WM improves real-world success over both the base policy and a world-model closed-loop baseline across two policy backbones and three tasks. The authors attribute the advantage to corrective supervision concentrated near failure-prone states.

Policy performance

  • Hi-WM improves success by 37.9 percentage points over the base policy and 19.0 points over the world-model closed-loop baseline, averaged across settings.It achieves the best result in all six task–policy settings under matched post-training budgets; the closed-loop baseline improves over the base policy in all six but remains below Hi-WM.Source: §4.3 · p. 9
  • The comparison spans DP and π0, indicating that the reported advantage is not tied to one policy architecture.The six settings comprise two policies evaluated on three tasks.Source: §4.3 · p. 9
  • The authors attribute the gap to Hi-WM collecting human corrections near failure-prone states and branching multiple continuations from cached pre-failure states.In contrast, the closed-loop baseline mainly adds successful trajectories that reinforce behavior already available to the policy.Source: §4.3 · p. 9

4.4 Efficiency and Scaling of Hi-WM

Hi-WM’s cost savings grow with scene count, while real-world success improves as more corrective data are added across the tested tasks. The reported cost comparison includes fixed setup and time-dependent operating costs.

Cost scaling with virtual intervention data

  • Absolute cost savings widen as the number of scenes grows, indicating a more favorable marginal cost under increasing scene diversity.The gap is smaller at low scene counts.Source: §4.4 · p. 9, Figure 6 · p. 10
  • Replacing real-world collection with Hi-WM yields nearly $45,000,000 USD in savings.The analysis includes fixed setup costs and time-dependent costs such as equipment deployment, human labor, site overhead, and inference compute.Source: §4.4 · pp. 9–10, Figure 6 · p. 10

Scaling with virtual intervention data

  • Real-world success rates improve monotonically across all three tasks as corrective data increase from roughly 20% to approximately 35% of the original dataset size.The gains are reported across the tested data range, from the base policy through two intervention-data iterations.Source: §4.4 · p. 10, Figure 6 · p. 10

4.5 Policy Generalization via Hi-WM

RQ4 tests whether world-model corrective data transfer to held-out Push-T scene variations. The post-trained policy outperforms the base policy across appearance, background, and distractor settings.

4.5 Policy Generalization via Hi-WM

  • The study evaluates Push-T under three controlled settings: appearance variation, background variation, and distractor variation.Source: §4.5 · p. 10
  • Corrective data collected under these scene variations in the world model are used to post-train a policy, which is then evaluated in the real world.The world model is initialized from varied observations, enabling interventions on states beyond those covered by the original real-world dataset.Source: §4.5 · pp. 10–11
  • The post-trained policy consistently outperforms the Base Policy across appearance, background, and distractor variations in real-world evaluation.These results indicate improved robustness under the three evaluated held-out conditions.Source: §4.5 · p. 11

5 Conclusion

Hi-WM shifts human intervention into an interactive world simulator to address the hardware and reset costs of robot post-training. Across real-world manipulation tasks with rigid and deformable objects, the evaluation reports improved policy success rates.

5 Conclusion

  • State caching, trajectory branching, and reset-free data collection make corrective data acquisition an amortized interactive process rather than a hardware-bound procedure.Source: §5 · p. 11
  • Evaluation across real-world manipulation tasks involving rigid and deformable objects shows that interactive targeted correction within the world model improves real-world policy success rates.Source: §5 · p. 11
  • Hi-WM uses the world model as the primary workspace for frequent evaluation and correction, reserving expensive real-world deployments for sparse final validation.Source: §5 · p. 11
Loading 2604.21741v2…