PaperlayerView exact revision on arXiv

Source-linked AI summary

Motus2: A Self-Evolving General World Model for Dexterous Manipulation

Hongzhe Bi, Zihao Zhou, Yihang Tang, Jingrui Pang, Shuhe Huang, Haitian Liu, Runqing Wang, Shuai Huang, Yichen Wang, Yiming Cheng, Ruowen Zhao, Zhenghua Li, Hengkai Tan, Xiaolong Liu, Jinhui Wan, Jiabao Liu, Min Zhao, Fan Bao, Jun Zhu

arXiv:2608.30237v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Existing robot world models generally do not couple action generation, future simulation, and evaluation into a closed policy-improvement loop. Motus2 builds a shared-parameter model with policy, simulator, and evaluator interfaces, scales training from egocentric human data to robot-domain data, and reports higher matched-task success after robot-domain mid-training.

  • Problem

    Existing robot foundation models are trained predominantly on curated action-supervised data, while imitation alone lacks action-quality evaluation and policy improvement from outcomes.

  • Method

    Motus2 uses one shared-parameter model for world–action policy generation, action-conditioned visual simulation, and value evaluation, with trajectory-dependent supervision and model-based reinforcement learning.

  • Results

    Motus2 raises matched-task average success to 84%, a 33-point gain over Pretrain-SFT after egocentric pretraining.

  • Takeaways & Limitations

    Egocentric data scaling and closed-loop world modeling provide a path toward self-evolving dexterous manipulation.

  • Takeaways & Limitations

    Wearable tactile supervision does not scale directly across embodiments because human and robotic hands differ geometrically and require embodiment-specific glove patterns.

Abstract

from arXiv · show

General embodied agents should perceive, predict, act, evaluate, and improve within a unified system. World models have shown great promise in building such agents, yet existing models typically append an action output head to a world simulator, without coupling them into a closed decision-and-learning loop for policy improvement. We present Motus2, a self-evolving general world model for dexterous manipulation. Motus2 advances world modeling through model scaling and data scaling. For model scaling, a single model with shared weights exposes three control interfaces: a policy (world-action model), a simulator (action-conditioned world model), and an evaluator (value model). The policy proposes candidate action chunks, the simulator predicts their visual consequences, and the evaluator assesses the predicted outcomes. Their coupling forms a closed decision-and-learning loop for policy improvement. This formulation uses curated expert demonstrations for action learning, while failed and suboptimal interactions provide valuable evidence for dynamics modeling and value learning. For data scaling, Motus2 progresses from large-scale monocular egocentric data to synchronized stereo egocentric data, followed by robot-domain adaptation with robot trajectories and supplementary human-robot alignment data. Motus2 further studies global-autoregressive and hybrid-memory extensions of its sliding-window context, adds tactile feedback for contact-aware control, and is instantiated on a fully biomimetic platform with stereo vision, dual arms, dual dexterous hands, and tactile sensing. Together, egocentric data scaling and closed-loop general world model scaling provide a general path toward self-evolving dexterous manipulation.

1. Introduction

Motus2 addresses the limits of imitation-centered robot foundation models by coupling action proposal, consequence prediction, and outcome evaluation in a self-evolving loop. It also scales data from egocentric human observations to robot-domain training while routing different interaction evidence to appropriate learning objectives.

  • Motivation: Imitation alone provides no notion of action quality or mechanism for improving a policy from its own outcomes.Collecting embodiment-aligned robot demonstrations at scale is expensive.
  • Data scaling: Motus2 scales data from monocular egocentric observations to synchronized stereo data, then robot trajectories with supplementary human–robot alignment data.Stereo data add implicit depth cues and support more accurate 3D hand-pose estimation.
  • Model scaling: One shared-parameter model exposes a policy, simulator, and evaluator that propose actions, predict visual consequences, and estimate task progress.Best-of-N planning selects candidates at test time, while model-based reinforcement learning converts scores into policy updates.
  • Trajectory-dependent supervision: Curated demonstrations supervise action learning, whereas failed and suboptimal interactions provide evidence for dynamics modeling and value learning.A lightweight tactile expert supports tactile-conditioned action refinement and tactile prediction for contact-sensitive execution.
  • Temporal and contact context: Motus2 studies global autoregression and hybrid memory as extensions of bounded sliding-window context for long-horizon partial observability.These extensions address histories affected by hand occlusion and delayed consequences.

2. Related Work

Related work organizes robot world models around policy and simulator interfaces, while Motus2 adds value-based evaluation and model-based reinforcement learning. The surrounding literature also emphasizes dexterity-specific control, scalable egocentric priors, and world-model-based policy improvement.

  • General World Models: General World Models integrate current-world understanding, imagination of possible and action-conditioned futures, and action grounding.The term spans latent dynamics, generative observable-future prediction, and interactive generation conditioned on external inputs.
  • Functional interfaces: Robot control systems are organized around policy interfaces that generate actions and simulator interfaces that predict future observations under proposed actions.These interfaces correspond to world–action models and action-conditioned world models.
  • Motus and Motus2: Motus2 extends Motus by adding a value-based evaluator and model-based reinforcement learning to couple policy, simulation, and evaluation.Motus already implemented policy and simulator functions as conditional query modes of one shared video–action model.
  • Dexterous policies: Dexterous manipulation requires high-DoF hands to coordinate discontinuous contact dynamics, motivating hand-specific representations, retargeting, tactile-reactive control, and scalable data generation.Small pose or timing errors can alter grasp mode, induce slip, or prevent recovery.
  • Egocentric and human priors: Monocular egocentric data offer broad interaction coverage, while stereo data add implicit depth cues and more accurate 3D hand-pose estimates for robot policy learning.Related systems use human interaction data to strengthen robot policy learning.
  • Policy improvement: World-model-based policy improvement methods extend beyond action imitation by using predicted outcomes, future-conditioned feedback, or preference rewards.The related approaches include WMPO, RISE, Reinforcing Action Policies by Prophesying, and NORA-1.5.

3. Method

Motus2 builds a shared-parameter video–action model whose policy, simulator, and evaluator form a sequential control chain and closed policy-improvement loop. Robot-domain adaptation adds trajectory-dependent supervision, planning, memory extensions, and tactile refinement while preserving the shared backbone.

  • General World Model: Motus2 factorizes one shared-parameter model into a policy, simulator, and evaluator for action generation, future prediction, and value assessment.The policy models action chunks, the simulator models action-conditioned future observations, and the evaluator models task-progress values.
  • Trajectory-Dependent Supervision: Trajectory-dependent gates use successful trajectories for action supervision while routing failed, suboptimal, and irrelevant interactions to simulation or evaluation learning.Recorded actions from undesirable trajectories remain conditioning variables rather than imitation targets.
  • Stage-Specific Information Flow: Stage-specific masks preserve bidirectional video–action interaction during joint pre-training and enforce action-first, causal information flow during robot-domain training.Action tokens cannot read current future-video or value tokens; future-video tokens read actions, and value queries read both within each chunk.
  • Data, Memory, and Tactile Extensions: A hierarchical data curriculum progresses from monocular to synchronized stereo egocentric data before robot-domain adaptation, while working memory and tactile expertise address partial observability and contact ambiguity.The tactile expert reuses intermediate action-chunk representations and detached backbone KV caches for short-horizon tactile refinement.
  • Action-First Control: The action-first dependency makes candidate generation, simulation, and evaluation executable in sequence for planning and policy optimization.Ordinary control queries only the action factor, while planning and policy improvement activate the full chain.
  • Closed-Loop Self-Evolution: Value-guided self-evolution uses predicted consequences and progress values to support Best-of-N selection and MBRL updates to the action pathway.The evaluator receives positive progress targets from successful segments and negative supervision from failed and irrelevant interactions.

4. Large-Scale Egocentric Dataset

Motus2’s egocentric corpus combines large-scale monocular and stereo recordings with human–robot alignment data, then standardizes them into annotated synchronized episodes. The dataset spans diverse manipulation actions, tasks, objects, and scenes while preserving stereo-specific depth and 3D hand-pose benefits.

  • Approximately 130K hours of open-source and procured egocentric recordings span monocular and stereo cameras, resolutions, manipulation tasks, and environments.
  • Stereo recordings add implicit depth cues and more accurate 3D hand-pose estimates, complementing monocular data’s broad visual and semantic coverage.
  • Tens of hours of human–robot alignment data supplement mid-training, which uses more than 100 hours of data in total.
  • The corpus hierarchy progresses from web and monocular egocentric video through stereo egocentric data to human–robot alignment and robot interaction data.
  • The annotated corpus covers manipulation primitives, objects, tasks, and scenes, including frequent actions such as grasp, pick, lift, press, and adjust.
  • Recordings become synchronized episodes through a common 134-D hand-pose representation, quality-aware temporal segmentation, and vision–language annotation.

5. Experiments

Experiments evaluate Motus2’s data scaling, closed-loop policy improvement, context mechanisms, and tactile feedback across simulation and real-robot manipulation tasks. Results show gains from egocentric pretraining and robot-domain mid-training, MBRL and planning, global autoregression, and tactile feedback.

  • Main Results: Under matched target-task SFT, macro-average success rises from 0% for WAN-SFT to 51% for Pretrain-SFT and 84% for Motus2 Midtrain-SFT.The latter is a 33-point gain over Pretrain-SFT; task success ranges from 60% to 100%.
  • Stereo Egocentric Data Scaling: Stereo human-data scaling lowers optimal held-out action-prediction error monotonically, approximately linearly with the logarithm of corpus size from 2k to 20k hours.The plotted hours are raw recording hours before quality filtering and temporal segmentation.
  • Policy Improvement and Planning: Planning increases average success from 65.0% to 67.5%, MBRL reaches 72.5% under direct inference, and combining both reaches 75.0%.Planning and MBRL each retain an observed 2.5- or 7.5-point gain when the other component is enabled.
  • Policy Improvement and Planning: Value trajectories rise from 0.34 to approximately 0.81 on a successful Screw Bulb rollout and decrease after progress is lost on failed rollouts.These predicted values provide the signal used for candidate ranking.
  • Long-Horizon Context Mechanisms: Global autoregression outperforms hybrid memory in both simulation and real-robot long-horizon probes, with macro-averages of 78% versus 52% and 57.5% versus 25%.The comparison uses Find Square and Press Button under matched training and paired evaluation configurations.
  • Tactile Feedback: Tactile feedback raises macro-average success from 60.0% to 72.5% across Pull Out the Paper Cup and Tear Paper.Success improves from 65% to 75% on the former and from 55% to 70% on the latter.

6. Conclusion

Motus2 unifies action generation, future simulation, and outcome evaluation within one shared physical-world model. Their coupling forms a closed loop for model-based policy improvement, while scaled egocentric data and differentiated interaction evidence support physical grounding.

  • Closed-loop world modeling: Motus2 uses shared weights for a world–action policy, an action-conditioned simulator, and a value-based evaluator.The policy proposes executable actions, the simulator predicts visual consequences, and the value model evaluates predicted outcomes.
  • Closed-loop world modeling: Coupling these interfaces enables candidate generation, consequence prediction, value-based selection, and policy updates in a closed decision-and-learning loop.
  • Data scaling: Motus2 scales physical experience from egocentric human interaction through monocular and stereo data to robot-domain grounding.
  • Data scaling: Curated successful demonstrations supervise action learning, whereas failed and suboptimal interactions inform dynamics and outcome evaluation.
  • Data scaling: Tactile feedback strengthens physical grounding during contact-sensitive manipulation.

7. Limitations and Future Work

Motus2’s main limitation is cross-embodiment tactile supervision: human and robot hands differ morphologically, and glove signals are noisy. Future work targets scalable tactile sensing, more anthropomorphic hands, and stronger multimodal world models.

  • Limitations: A single tactile glove pattern cannot fit both human and robotic hands because current dexterous hands are not geometrically isomorphic to human hands.Robot fingers may be elongated, thicker, or larger than human fingers.
  • Limitations: Glove deformation, material strain, and internal fabric contact create noisy tactile signals that are difficult to distinguish from meaningful physical contact.
  • Limitations: The morphology-induced gap limits the scale and reliability of cross-embodiment tactile learning.
  • Future Work: Advances in wearable tactile materials and fabrication could enable scalable whole-hand tactile sensing for multimodal egocentric human data collection.
  • Future Work: More anthropomorphic hands, stronger video models, and unified multimodal models could narrow embodiment gaps and improve long-horizon multimodal simulation.

Core Contributors

The project’s core contributors cover base modeling, reinforcement learning, memory, simulation, tactile sensing, hardware, data collection, robot post-training, and writing. The work uses three bimanual robot configurations spanning Wuji and Sharpa hands on Tianji platforms.

  • Contributors: Hongzhe Bi contributed to the base model, pre-training, MBRL, memory, tactile sensing, hardware, robot post-training, egocentric data, and writing.
  • Contributors: Zihao Zhou contributed to MBRL, memory, simulation, robot post-training, hardware, and writing.
  • Contributors: Yihang Tang contributed to Sharpa and Wuji-2 hardware, tactile sensing, and real-robot post-training.
  • Contributors: Jingrui Pang contributed to Wuji-1 hardware, human-data collection, and real-robot post-training.
  • Contributors: Shuhe Huang contributed to MBRL, simulation, and writing.
  • Robot platforms: The data-collection platforms combine Tianji Marvin or Gento Luna dual arms with Wuji or Sharpa dexterous hands.The configurations include two 20-DoF Wuji hands, 20-DoF Wuji Hand 2, and 22-DoF Sharpa Wave.

B. Human–Robot Alignment Data Collection

Human–robot alignment data are collected from an operator wearing Wuji Human Gloves and wrist trackers while a stereo camera records first-person views. A table-fixed tracker provides a stable workspace-centered reference for wrist motion.

  • Collection setup: Operators perform manipulation tasks while wearing Wuji Human Gloves and one HTC VIVE Tracker on each wrist, without commanding a robot.
  • Collection setup: A third tracker is fixed to the table as a stationary workspace reference, while a ZED 2 stereo camera records first-person left and right RGB views.
  • Reference-frame alignment: The alignment representation expresses each human wrist pose relative to the fixed tracker in the VIVE Lighthouse frame.
  • Reference-frame alignment: This representation removes the arbitrary Lighthouse origin and expresses human wrist motion in a stable workspace-centered frame.

C. Egocentric Data Curation and Offline Annotation

The curation pipeline standardizes multimodal egocentric recordings, removes unreliable temporal regions, and preserves synchronization across available signals. Hand observations are normalized into a robot-compatible representation and enriched with structured language annotations.

  • Annotation: Each processed episode contains synchronized RGB streams, hand poses, and natural-language descriptions of the environment and manipulation behavior.For multiple views, one selected egocentric stream generates captions while all synchronized views remain available.
  • Curation: BoxTrim removes unreliable temporal regions and converts recordings into compact, action-centric clips while preserving synchronization across modalities.The pipeline targets viewpoint changes, wrist-trajectory discontinuities, and intervals where both hands are unobserved.
  • Curation: Temporal masks remove corrupted intervals consistently from RGB, poses, depth, calibration, and auxiliary motion signals.Semantic annotations can define preferred clip boundaries, while overlapping or long intervals are merged or split near low-motion frames.
  • Curation: Motion-discontinuity thresholds are adapted to each recording rather than imposed universally across corpora.Thresholds depend on frame rate, annotation density, and tracking characteristics.
  • Hand representation: Hand keypoints are transformed into wrist-centered, side-specific Wuji coordinates before inverse-kinematics optimization.The representation uses wrist-relative geometry, a palm frame estimated from hand landmarks, and Wuji joint limits.
  • Hand representation: The retargeting objective combines full-finger configuration and fingertip geometry losses, with proximity-dependent weighting and trajectory filtering.The resulting action representation contains 20 finger joint angles per hand; robot post-training data instead retains native recorded joint angles.

E.1. Pre-training

Pre-training progresses from video-only monocular learning to joint stereo video–action learning, followed by robot-domain adaptation. The resulting system supports sliding-window control, longer-context variants, model-based policy improvement, and tactile refinement.

  • Pre-training: Stage 1 trains the video pathway on low- and high-resolution monocular egocentric clips before action and value tokens are introduced.The two phases use 500K and 340K training steps, respectively.
  • Pre-training: Stage 2 jointly trains stereo egocentric video and human actions with shared video–action supervision.The maximum window contains W = 12 latent frames, with κ = 2 frames per prediction chunk.
  • Robot adaptation: Mid-training adapts the model using more than 100 hours of robot trajectories plus human–robot alignment data.Policy, simulation, and evaluation modes are mixed during robot-domain adaptation.
  • Context: The default target-robot policy uses a sliding window with W = 8, κ = 2, one-to-three-chunk history, and a two-chunk streaming cache.Deployment samples action chunks with five flow-matching denoising steps.
  • Context: Global autoregression removes the binding recent window so each prediction can attend to all preceding clean observations.The hybrid-memory branch instead combines full-resolution anchor and recent frames with memory tokens.
  • Self-evolution: MBRL generates eight policy candidates plus one ground-truth candidate for each sampled prefix and rolls out one imagined chunk.The rollout pipeline uses separate denoising schedules for action and future-video sampling.
  • Tactile control: The tactile expert refines eight sequential six-action sub-chunks using preceding tactile windows at a 30 Hz action rate.Force signals are sampled at 90 Hz, and the expert shares cached backbone states across calls.

F. Qualitative Value-Model Trajectories

The value model produces progress estimates that track both successful execution and failure development along recorded trajectories. Successful episodes reach similar terminal values, whereas failed episodes can rise during correct partial execution before declining after task-specific errors.

  • Evaluation protocol: Values are read from context, executed actions, and observed video without access to eventual episode success.One value is plotted per latent frame during replay through the training-time streaming window.
  • Successful executions: 0.63–0.65: successful executions terminate at similar predicted progress levels despite distinct task-specific profiles.Cut Reeds remains near zero before its cuts, while Make Tea, Multi-Finger, and Put Phone rise more smoothly.
  • Failed executions: 0.27 to −0.51: Cut Reeds rises through the cutting phase, then drops after the first cut fails and the rack is misaligned.The prediction falls to −0.39 within two seconds of the failed cut.
  • Failed executions: 0.60 to −0.26: Put Phone peaks after placement succeeds, then declines when repeated lid-closing failures prevent completion.The task's required lid closure is not achieved despite earlier progress.
  • Failed executions: 0.14 to −0.47: Make Tea rises during partial execution and declines as the robot fails to lift the lid.The decline continues over the remaining 80 seconds.
  • Failed executions: 0.18 to −0.53: Multi-Finger falls after objects intended for separate containers are released into the same container.Its prediction rises near 50 seconds before declining to the episode's end.

G. Evaluation Tasks

The evaluation comprises nine unique physical tasks organized into main-suite, working-memory, and tactile groups. The MBRL study reuses two main-suite tasks under altered optimization and inference settings.

  • Task suite: Nine unique physical tasks are summarized in the evaluation suite, with the MBRL study reusing two existing tasks rather than adding new definitions.The reused tasks are Put Phone and Multi-Finger.
  • Task suite: Rows in the task figure show eight sampled frames from representative episodes across main-suite, working-memory, and tactile task groups.The groups occupy panels (a)–(e), (f)–(g), and (h)–(i), respectively.
Loading 2608.30237v1…