PaperlayerView exact revision on arXiv

Source-linked AI summary

XPACE: Joint World and Action Modeling from Heterogeneous Experience

Jiacheng Wei, Jerry Bai, Xiaoyu Yue, Zidong Wang, Xiaoyang Guo, Cheng Chen, Fanqi Pu, Fan Wu, Zhixu Yue, Yizhuo Li, Feng Qiu, Bo Liu, Yuying Ge, Hui Zhou, Chenyi Chen, Yixiao Ge

arXiv:2609.17372v1cs.RO

TL;DR

Robots need to learn from diverse experience while predicting actions and their visual consequences, but human and robot data provide different kinds of supervision. XPACE combines them through shared video–action modeling and uses its simulator to generate recovery training examples; experiments report transfer to robot-absent tasks and improved real-world task completion after recovery fine-tuning.

  • Problem

    Human videos offer broad behavioral coverage but may lack robot-control supervision, while robot demonstrations provide executable actions but are costly and limited corrective supervision is available for unfamiliar states.

    Source: §1 · p. 2

  • Method

    XPACE shares a video backbone for action generation and simulation, progressively emphasizes robot data while retaining human experience, and fine-tunes the policy on filtered simulator-generated recovery trajectories.

    Source: §1 · p. 2

  • Results

    Experiments report improved robustness, transfer of human-observed skills to tasks absent from robot teleoperation, and higher real-world completion after recovery fine-tuning.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Human experience can broaden behaviors learned during robot adaptation, while the learned simulator can provide recovery experience for policy improvement.

    Source: §5 · p. 22

    The reported before-versus-after fine-tuning comparison does not separate recovery-data effects from the additional optimization budget.

    Source: §4.3.6 · p. 22

Abstract

from arXiv · show

A general-purpose robot needs to draw on diverse experience, choose actions, and anticipate how those actions will change the world. We introduce XPACE, a unified embodied world model that serves as both a world action model, jointly predicting executable robot actions and future video, and a world simulator, predicting the visual consequences of prescribed actions. Our key insight is that video prediction can both connect heterogeneous experience to action learning and generate new experience for policy improvement. With a shared video backbone between the policy and simulator, we use action-unlabeled video to learn visual dynamics and action-labeled human and robot demonstrations to jointly learn video and action prediction. Building on this architecture, a coarse-to-fine training curriculum progressively emphasizes robot control while retaining human experience, allowing the policy to learn behaviors beyond those covered by robot demonstrations. Beyond learning from recorded experience, XPACE uses its simulator to create additional recovery supervision for the policy. Specifically, we adapt the simulator to its own generated context, synthesize deviation-recovery trajectories around expert demonstrations, and fine-tune the policy on filtered recovery examples. Experiments on XPENG's IRON humanoid robot show that heterogeneous training improves robustness and enables transfer of human-observed skills to tasks absent from robot demonstrations, while recovery data generated by the model's own simulator further improves real-world task completion. Together, these results demonstrate how joint world and action modeling connects learning from heterogeneous experience with simulation-driven policy self-improvement.

1 Introduction

XPACE addresses how a robot can learn from heterogeneous experience, jointly predict actions and visual futures, and improve from simulated experience. Its experiments report improved performance and robustness, transfer to robot-absent tasks, and better real-world completion after recovery fine-tuning.

1 Introduction

  • The motivating gap is how to combine human video’s broad coverage with robot demonstrations’ executable actions and provide corrective supervision for unfamiliar execution states.The introduction frames this as questions about training experience, joint action and simulation modeling, and converting those capabilities into robot control and self-improvement.Source: §1 · p. 2
  • XPACE uses a shared video backbone for task-conditioned action generation and action-conditioned simulation, then generates recovery supervision to fine-tune the policy.The simulator is adapted to self-generated visual context before synthesizing deviation–recovery trajectories around expert demonstrations.Source: §1 · pp. 2–3
  • The framework combines action-unlabeled egocentric video, kinematically labeled human demonstrations, robot teleoperation, and task- and appearance-aligned bridge data.Each source contributes distinct visual, manipulation, robot-control, or alignment supervision.Source: §1 · p. 2
  • Experiments on IRON report improved performance and robustness, transfer of human-observed skills to tasks absent from robot teleoperation, and improved real-world completion after recovery fine-tuning.Simulator adaptation also improves autoregressive video prediction while reducing inference cost.Source: §1 · p. 3

2 Related Work

Related work spans transfer from human experience, policy and simulator interfaces, video-based training, and simulation-driven policy improvement. XPACE positions its approach at the intersection of these lines through shared video–action learning and simulator-generated recovery supervision.

2.1 Data: human experience and robot grounding

  • Human-video transfer methods include affordance learning [7], demonstration transfer [12] [13], aligned adaptation [14], and predictive modeling [16]; XPACE jointly learns across video-only, human-action, and robot data.Generated-experience methods expand behavioral or visual coverage [17] [18], whereas XPACE targets off-trajectory recovery around demonstrations.Source: §2.1 · p. 3

2.2 Model: policies, simulators, and shared predictive representations

  • Robot policies predict actions directly [19] [20] [21] [22] [24], while simulators predict visual outcomes under controls [3] [4] [25] [26] [27]; unified models combine video and action interfaces [5] [6] [34] [35].XPACE shares a causal video backbone between policy and simulation, with an asymmetric action Transformer reading multi-level visual features and skeleton controls and camera poses conditioning simulation.Source: §2.2 · pp. 3–4

2.3 Training: self-supervised video learning and supervised action grounding

  • Self-supervised video objectives make action-unlabeled data useful without treating it as executable robot commands [48] [49] [50] [51] [52] [53]; XPACE combines video pretraining with joint video–action learning.Simulator adaptation also addresses the shift from recorded to generated visual context, following context-training approaches [54] [55].Source: §2.3 · p. 4

2.4 Simulation-based policy improvement and evaluation

  • Simulation-based policy improvement includes imagined rollouts [57] [58], recovery aggregation [9] [59], simulated corrections [60], and learned-simulator refinement [61] [62] [63] [65].XPACE uses its jointly trained simulator to render and filter deviation–recovery trajectories, then fine-tunes a separate policy rather than querying an expert on-policy.Source: §2.4 · pp. 4–5
  • Policy-conditioned simulated rollouts support qualitative diagnosis, while quantitative evidence for XPACE policy self-improvement comes from real-robot trials.The related-work section contrasts this evaluation scope with WorldGym’s tests of whether simulated rollouts predict real-robot policy performance.Source: §2.4 · p. 5

3 Method

XPACE integrates heterogeneous data, shared video-based action and world modeling, and staged robot adaptation with simulator-generated recovery training.

3 Method

  • A four-layer data pyramid combines human experience and robot demonstrations to connect broad interaction coverage with robot-executable control.The layers progress from egocentric video through human video–action and bridge data to robot teleoperation.Source: §3 · p. 5
  • A shared video backbone connects the simulator and policy so both learn from a common representation of visual dynamics.The architecture supports video prediction alongside executable-action prediction.Source: §3 · p. 5
  • A coarse-to-fine curriculum progressively emphasizes robot supervision while retaining human experience during adaptation to the deployment embodiment.After simulator adaptation, generated recovery trajectories provide supervision for policy fine-tuning.Source: §3 · p. 5

3.1 Data

XPACE organizes 5,000 hours of embodied video into complementary layers that connect visual dynamics learning to robot control. Multi-level captions and a shared human–robot action schema support learning across clips and embodiments.

3.1 Data

  • Bridge data aligns human experience with robot deployment through task-family matching or similar camera viewpoints, backgrounds, and object arrangements; IRON teleoperation supplies native robot actions.The bridge encourages cross-embodiment representations through task and visual overlap without an explicit feature-alignment loss.Source: §3.1 · pp. 5–6
  • The data pyramid progresses from unlabeled egocentric video for visual dynamics to human video–action data with shared-schema pose supervision.L1 supplies broad scene, object, and dynamics coverage; L2 adds manipulation behavior labels.Source: §3.1 · p. 5, Figure 2 · p. 6
  • Failure, low-progress, and recovery trajectories train video prediction on successful and unsuccessful motions, but their actions are excluded from policy imitation.This supplies action-dependent simulation supervision for evaluation and recovery-data synthesis.Source: §3.1 · p. 6
  • Each clip retains its episode-level task instruction and adds dense descriptions, atomic action commands, and task paraphrases, with sampling shifting toward deployment-style instructions.L1 clips without action labels retain descriptive captions throughout training.Source: §3.1 · p. 6
  • Human and robot demonstrations use a common kinematic schema for end-effector poses, hand articulation, and torso or camera poses, with source-specific normalization and embodiment-specific projections.The shared action Transformer preserves hand articulation alongside wrist motion while accommodating embodiment-specific kinematics and action dimensions.Source: §3.1 · p. 7

3.2 Model Architecture

XPACE shares a video backbone between an action-generating policy and a visual simulator, with video features flowing into action prediction. The modes use different conditions to predict future video and actions or the visual consequences of prescribed motion.

3.2 Model Architecture

  • The asymmetric architecture shares a video Transformer between policy and simulation, while an Action Transformer reads its multi-level features to predict executable actions.Action-loss gradients also update the shared video backbone, refining the representation used by both modes.Source: §3.2 · p. 7
  • Both modes use chunk-wise causal video prediction, with each chunk conditioned on an anchor and preceding visual chunks.Each video chunk contains four latent frames, corresponding to 16 video frames; the policy action horizon is 16 steps.Source: §3.2 · pp. 7–8
  • In simulator mode, the video Transformer predicts future video from visual history, prescribed skeleton controls, and camera poses, with the Action Transformer inactive.Camera poses are explicit simulator inputs, and skeleton tokens are added to video tokens without increasing sequence length.Source: §3.2 · pp. 8–9
  • In policy mode, visual history and a task instruction condition future-video prediction, while the Action Transformer also receives the current state to predict an action chunk.The policy does not receive prescribed future skeleton or camera controls.Source: §3.2 · p. 8
  • At deployment, streaming visual encoding and overlapping-horizon stitching let new action plans enter ongoing robot execution.The publisher interpolates commands at approximately 10 Hz and smooths transitions between successive plans.Source: §3.2 · p. 9

3.3 Training Paradigm

Training first adapts video prediction to embodied video, then adds joint action prediction and simulation. A coarse-to-fine curriculum increases robot and bridge supervision while retaining human and video experience.

3.3 Training Paradigm

  • Stage II adds the action branch and jointly trains policy and simulation modes, sampling the two modes with equal probability.Simulation updates optimize video loss; video–action updates optimize video and action losses, with action loss zero when no valid labels are present.Source: §3.3 · p. 9, §3.3.2 · p. 10

3.3.1 Stage I: Video-only pretraining

  • Stage I trains the video backbone on embodied human and robot video as video-only data, before introducing action supervision.The stage uses multi-level captions and excludes F and DAgger data.Source: §3.3.1 · pp. 9–10

3.3.2 Stage II: Coarse-to-fine joint training

  • Across Stage II, the curriculum shifts from broad human and action-unlabeled video toward bridge and robot supervision, with target-robot data dominant in Phase II-c.Phase II-b introduces F only in simulation updates; exact source-mixture proportions are not specified.Source: §3.3.2 · p. 10
  • Phase II-c retains a small replay mixture of human demonstrations, bridge data, and action-unlabeled video rather than switching to robot-only fine-tuning.Replay maintains human action supervision and video prediction during robot adaptation to limit overfitting and forgetting.Source: §3.3.2 · p. 10

3.4 Post Training

Post-training uses the world model to add corrective supervision beyond successful expert demonstrations. The simulator is first adapted to generated visual context, then renders and filters deviation–recovery examples for policy fine-tuning.

3.4 Post Training

  • The simulator is adapted with self-gradient forcing before it generates recovery examples, addressing the limited recovery supervision in successful expert demonstrations.The method uses a DAgger-style process that aggregates recovery examples with expert data for policy fine-tuning.Source: §3.4 · pp. 10–11

3.4.1 Stage III-sim: Chunk-wise self-gradient forcing

  • Stage III-sim adapts a separate Phase II-c simulator copy to self-generated visual context while retaining paired skeleton- and camera-conditioned flow-matching supervision.The action Transformer is inactive, and the method retains ℓ2 loss so simulation reproduces consequences of supplied controls rather than only plausible continuations.Source: §3.4.1 · p. 11

3.4.2 Stage III-policy: Improve from aggregated corrections

  • Recovery trajectories perturb an expert end effector away from its path for k frames, then return it to the pivot pose over the next k frames.Directions are limited to angles of 90°–150° relative to subsequent expert motion; the world model renders only the inserted 2k-frame segment, while the original expert tail supplies the continuation.Source: §3.4.2 · p. 12
  • Filtering requires recovery endpoints to be closer to the expert pivot than deviation endpoints and to meet absolute similarity thresholds, then ranks eligible candidates across three metrics.The retained clips form a task-balanced mixture across k = 8 and k = 16 horizons; DAgger recovery examples comprise 8% of policy fine-tuning data, with the SGF simulator fixed.Source: §3.4.2 · pp. 12–13

4.1 Experimental Setup

The evaluation combines held-out simulation and offline policy benchmarks with closed-loop trials on the IRON humanoid. Its test subsets span in-distribution interactions and varied out-of-distribution conditions, including behaviors absent from robot demonstrations.

4.1 Experimental Setup

  • Evaluation covers video prediction quality, offline action prediction, real-robot policy performance, and effects of training data and recovery supervision.External baselines assess overall policy performance, while internal ablations test conditioning, simulator adaptation, and training-data choices.Source: §4.1 · p. 13
  • Simulation and offline policy evaluation use 19 held-out subsets, with ID pick-and-place interactions and OOD changes in appearance, objects, instructions, arrangements, task composition, and action primitives.Some OOD behaviors absent from robot demonstrations are represented in human training data; subsets retain their original collection-based grouping.Source: §4.1 · p. 13
  • Real-robot evaluation measures task-progress scores and success rates on selected tasks, with 20 trials per task per method under matched resets.Tests use the IRON-R01-1.11 humanoid.Source: §4.1 · p. 13
  • The experiments separately ask whether simulation remains stable under autoregressive rollout, whether policy skills transfer from human data, and whether recovery supervision improves policy development.The questions also examine ID, task-level OOD, distractor robustness, and effects of data coverage.Source: §4.1 · p. 13

4.2 Simulation

The simulation evaluation tests skeleton-conditioning strategies and self-gradient forcing for video prediction under prescribed controls. Token addition leads short-horizon quality, while SGF improves long-horizon prediction with lower inference cost.

4.2 Simulation

  • Evaluation uses prescribed skeleton controls and camera poses on ID and OOD videos, assessing image fidelity, motion consistency, feature agreement, latency, and memory.Source: §4.2 · pp. 13–14

4.2.1 Short-horizon prediction: skeleton conditioning

  • Token addition achieves the highest PSNR, SSIM, and DINO scores on both splits and highest OOD OFS; its PSNR exceeds AdaLN by 0.95 dB on ID and 1.11 dB on OOD without added inference cost.ID OFS is slightly below AdaLN.Source: Table 1 · p. 14, §4.2.1 · p. 14

4.2.2 Long-horizon rollout: self-gradient forcing

  • The final SGF-adapted XPACE simulator predicts well on ID and OOD videos and supplies DAgger data for the real-robot self-improvement experiment.Source: §4.2.2 · p. 14
  • At 193 frames, Baseline + SGF raises PSNR by 0.89 dB on ID and 1.01 dB on OOD, with 3.00× and 3.30× speedups and essentially unchanged memory.SGF improves all four quality metrics on both splits at both horizons.Source: §4.2.2 · p. 14, Table 2 · p. 15

4.3 Policy

The policy evaluation measures real-robot performance, robustness, human-to-robot transfer, training-data contributions, and simulator-driven self-improvement. XPACE leads external baselines on the primary benchmark, while broader supervision and recovery fine-tuning improve performance across the evaluated settings.

4.3.1 Real-robot performance and robustness

  • 68.3% average SR vs. 40.0% for DreamZero and 6.7% for GR00T; average progress is 0.84 vs. 0.68 and 0.36, with XPACE leading every task on both metrics.This is an end-to-end comparison: XPACE alone receives Stage I video adaptation; bowl stacking is absent from L4 but present in human and bridge data.Source: §4.3 · p. 15, Figure 7 · p. 16, §4.3.1 · p. 16
  • 80% vs. 50% success and 0.93 vs. 0.70 progress across five banana positions; with unseen distractors, success is 50% vs. 30% and progress 0.74 vs. 0.48 for full-data vs. robot-only.The gains extend to familiar manipulation under spatial and visual variation, not only skills absent from robot demonstrations.Source: Figure 8 · p. 16, §4.3.1 · p. 17

4.3.2 Human-to-robot skill transfer

  • XPACE transfers approach and contact to drawers absent from robot data, toaster-to-plate bread handling absent as a task, and sequential bowl placements under one instruction.The examples progress from acting on an unseen object to changed object-receptacle configurations and multi-step composition.Source: Figure 9 · p. 17, §4.3.2 · p. 18

4.3.3 Contribution of video pretraining

  • 12.5% loss reduction with full Stage I exposure versus 8.1% at one-eighth exposure; every nonzero exposure reduced loss under identical action fine-tuning.The first eighth captured approximately two-thirds of the full-exposure reduction, while exposure and optimization compute increased together.Source: Figure 10 · p. 18, §4.3.3 · p. 18

4.3.4 Human supervision during robot adaptation

  • 14.0% vs. 8.5% mean benchmark action-loss reduction for human co-training vs. human mid-training followed by robot-only fine-tuning, relative to robot-only.The human-data arms share initialization and final-stage budget, so the comparison tests whether human data remains in the final mixture.Source: §4.3.4 · pp. 18–19, Table 4 · p. 19

4.3.5 Cross-embodiment generalization

  • Co-training reduces action loss by 22.7% on robot-absent, human-dense behaviors and 10.0% on robot-dense, human-dense behaviors.The results support retaining human supervision for both coverage strata, but do not isolate bridge-data contributions or establish a feature-alignment mechanism.Source: §4.3.5 · pp. 19–20, Figure 11 · p. 20

4.3.6 Policy self-improvement

  • Mean task progress rises from 0.81 to 0.93 and mean success rate from 61.7% to 86.7% after DAgger fine-tuning with simulator-generated recovery data.Fine-tuning uses an 8% recovery-data mixture with the simulator fixed; the comparison does not separate recovery-data effects from additional optimization.Source: Figure 12 · p. 20, §4.3.6 · pp. 21–22
  • Simulated policy rollouts expose target-selection errors despite plausible generated motions, supporting lower-cost qualitative diagnosis before real-robot testing.The closed loop feeds simulator-generated frames back to the policy as subsequent observations.Source: Figure 13 · p. 21, §4.3.6 · p. 22

5 Conclusion

XPACE combines human and robot experience in a shared embodied world model and uses simulator-generated recovery experience for policy improvement. The conclusion highlights robustness and skill transfer from retaining human experience during robot adaptation, while identifying requirements for moving beyond demonstration-centered recovery.

5 Conclusion

  • Human experience broadens interaction knowledge, while robot demonstrations ground it in executable control; retaining human data during robot adaptation supports robustness and skill transfer.The paper suggests robot demonstrations need not define the full range of behaviors a policy can learn.Source: §5 · p. 22
  • The learned simulator provides recovery experience for policy improvement and a setting for inspecting policy behavior.Generated experience connects prediction and control through subsequent action learning, beyond shared representations alone.Source: §5 · p. 22
  • Moving beyond demonstration-centered recovery requires models to represent uncertainty, remain reliable over longer contact-rich interactions, and incorporate real-world feedback without forgetting acquired skills.The paper frames actively seeking experience and testing imagined rollouts as a longer-term opportunity.Source: §5 · p. 22

Related papers

Loading 2609.17372v1…