PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence

Thinker Team, Peng Gao, Jichao Jiao, Chen Liao, Daqin Luo, Yiming Ma, Baiyu Pan, Hailin Shi, Jiyuan Wang, Xiangming Xu, Xiongwei Zhao

arXiv:2610.18447v2cs.AIcs.CVcs.RO

TL;DR

Existing world-action models can keep visual dynamics, geometry, and executable actions in separate interfaces or modules rather than evolving them in one shared representation. Thinker-WM jointly models RGB, depth, and robot action through shared dynamics with modality-specific encoders and decoders, and ranks first across three embodied benchmarks. These results support representation-level unification as a viable foundation for embodied world-action modeling.

  • Problem

    World-action models may leave executable actions outside the generative process, while visual dynamics, geometry, and action are not jointly evolved in one shared representation.

    Source: §1 · p. 1, §2.2 · p. 4

  • Method

    Thinker-WM encodes RGB, metric depth, and robot action into a shared intermediate representation for joint evolution by a Unified Dynamics Transformer, with modality-specific decoding.

    Source: §1 · p. 3, §3.1 · p. 6

  • Results

    Thinker-WM ranked first on EWMBench, WorldOlympiad, and WorldArena, achieving an Overall score of 4.66 on EWMBench.

    Source: §1 · p. 3, §9 · p. 35

  • Takeaways & Limitations

    The results support representation-level unification as a viable foundation for embodied world-action modeling.

    Source: §9 · p. 35

    Thinker-WM’s effectiveness as a closed-loop policy on real robots remains to be established.

    Source: §9 · p. 35

Abstract

from arXiv · show

World models have emerged as a promising paradigm for embodied AI, yet existing systems often remain fragmented in the modalities they model and the functions they support. In particular, visual dynamics, geometric structure, and executable action are commonly coupled through separate interfaces or downstream modules rather than being jointly represented and evolved within a shared dynamics model. In this work, we present Thinker-WM, a unified multimodal world model that jointly models and generates RGB, metric depth, and robot action through a shared intermediate Unified Dynamics Transformer with modality-specific encoders and decoders. The model performs multimodal unification at the representation level: appearance, geometry, and action are transformed into a common dynamic representation and jointly evolved by shared parameters, while their distinct output structures are preserved. The same parameterization supports multiple functions, including joint video-action generation, world prediction, policy inference, action-conditioned generation, video-to-action generation, and inverse dynamics. To examine whether the shared representation captures action-related information, we introduce a lightweight motion probe that shows the frozen visual representation contains recoverable motion information. Thinker-WM is trained on a heterogeneous corpus comprising approximately 20K hours of egocentric video, 10K hours of real-robot data, and 1K hours of simulation data, including approximately 4K hours of real-robot data with aligned depth supervision. Extensive experiments demonstrate strong performance across embodied world-model evaluations. Thinker-WM ranks first on EWMBench, WorldOlympiad and WorldArena, while controlled studies further demonstrate the benefits of action conditioning, geometric interaction, and multi-function training within the shared representation.

1 Introduction

Thinker-WM addresses the challenge of jointly modeling appearance, geometry, and executable action within a shared dynamics representation. It combines modality-specific interfaces with shared dynamics, native depth generation, heterogeneous training data, and evaluation across embodied benchmarks and generative settings.

1 Introduction

  • The paper asks where multimodal unification should occur because prior approaches often connect modalities through interfaces or conditioning rather than a common representation carrying future-world dynamics.For embodied intelligence, future evolution involves visual observations, physical geometry, executable actions, and task intent.Source: §1 · pp. 1–2
  • Thinker-WM encodes RGB, depth, and action into a shared intermediate representation, where a common Transformer models their joint evolution before modality-specific decoding.Task-level representations provide structured guidance for future prediction.Source: §1 · pp. 2–3
  • Depth is a native generative modality whose supervision shapes the shared representation, making physical structure explicit without claiming to solve physical reasoning.Depth shares the latent and spatiotemporal organization of visual generation.Source: §1 · pp. 2–3
  • Thinker-WM combines general video, public and in-house robot data, and egocentric human video, mapping heterogeneous robot actions into a common representation while retaining robot-specific decoding.These sources contribute visual and temporal priors, action-grounded supervision, and broader coverage of real-world interactions and human motion [7] [10] [13] [43].Source: §1 · p. 3
  • Thinker-WM ranked first on EWMBench, WorldOlympiad, and WorldArena while supporting multiple world-model and world-action functions within one parameterization.Evaluations include action-conditioned future generation, depth generation, long-horizon rollout, trajectory transfer, video-to-action generation, and instruction-conditioned generation.Source: §1 · p. 3

2 Related Work

Related work progresses from compact predictive simulators to large-scale generative models and then to models that jointly generate future observations and actions. Thinker-WM positions itself within this last direction while extending shared modeling to metric geometry and heterogeneous robot actions.

2 Related Work

  • Related work frames world models as evolving from compact simulators through large-scale generative models to systems that also generate executable actions.The review also discusses explicit geometric structure as context for Thinker-WM.Source: §2 · p. 3

2.1 Simulator-oriented World Models

  • Simulator-oriented world models use latent transition models to predict plausible future states from a current state and action, supporting planning or control.This formulation is computationally efficient, but its latent space is usually optimized for encountered environments and tasks rather than inheriting broad visual priors from large image and video corpora.Source: §2.1 · pp. 3–4

2.2 Latent and Generative World Models

  • Large-scale generative world models inherit broad visual priors, but their action interfaces may remain indirect or outside the generative process.The cited examples include GAIA-1 [11], Genie [5], UniSim [38], iVideoGPT [33], and Pandora [35].Source: §2.2 · p. 4

2.3 World Models with Action

  • World-action models seek to predict future observations and executable actions jointly, with prior approaches using expert pathways, shared transformers, or diffusion configurations.Motus combines pretrained experts in a Mixture-of-Transformers architecture [3]; MotuBrain uses three streams and a data pyramid [21]; UWM supports policy, world-model, and joint video-action functions through masking [45].Source: §2.3 · pp. 5–6

2.4 Position of Thinker-WM

  • Thinker-WM extends joint world-action modeling by co-generating appearance, metric geometry, and action from a shared representation, while registering heterogeneous actions through a common relative end-effector representation.It preserves a pretrained video-generation backbone and robot-specific action decoding.Source: §2.4 · p. 6

3 Model Architecture

Thinker-WM models RGB video, metric depth, and robot action as a shared evolving world state while preserving modality-specific interfaces. Its architecture combines structured task conditioning and a unified action representation with a Unified Dynamics Transformer whose depth interactions differ between pretraining and post-training.

3.1 Overview: Representation-Level Unified World Model

  • RGB, metric depth, and robot action are encoded into an intermediate dynamic representation, jointly evolved by the Unified Dynamics Transformer, then decoded into modality-specific outputs.The model preserves separate modality interfaces while sharing the dynamics stage.Source: §3.1 · p. 6

3.2.1 Language Representation

  • Task instructions are normalized into Task, Scene, and Action fields, with field order shuffled and fields randomly dropped during training.Evaluation uses a fixed Task–Scene–Action order; a frozen Qwen3-VL-2B produces the conditioning sequence.Source: §3.2.1 · p. 7

3.2.2 Visual Representation

  • RGB and depth share a latent topology, but use separate input and output projections to preserve their distinct statistics and semantics.At 640×480 resolution, 49 frames map to a 13×30×40 latent grid, with 3900 tokens per visual stream after patchification.Source: §3.2.2 · p. 7

3.2.3 Unified Action Representation

  • Heterogeneous robot actions are mapped into a unified 20-dimensional space, while robot identity is supplied through conditioning tokens rather than the generative sequence.The shared boundary also aligns visual and action representations for joint temporal modeling.Source: §3.2.3 · pp. 8–9

3.3 Unified Dynamics Transformer

  • The UDT models RGB, depth, and action together; pretraining lets depth read from RGB and action, while post-training opens bidirectional interaction under reliable depth supervision.Task conditioning enters through cross-attention, separate from cross-modal self-attention.Source: §3.3 · pp. 9–10, Figure 3 · p. 9, Figure 4 · p. 10

3.4 Function-Oriented Training

Thinker-WM uses explicit function modes to make one shared model handle different combinations of generated, conditioned, and omitted modalities. A shared flow-matching objective and modality-specific schedules support these modes at training and inference.

3.4 Function-Oriented Training

  • One shared UDT supports joint generation, visual world modeling, policy inference, and cross-modal generation by generating, conditioning, or omitting RGB, depth, and action streams.The six modes are WAM, WM, VLA, A2V, V2A, and unconditional generation.Source: §3.4 · pp. 10–11

3.5 Flow-Matching Parameterization

  • RGB and depth share a flow trajectory and noise schedule, while action uses a separate schedule retained at inference.Inference integrates the visual velocity fields for 50 steps and action for 10; classifier-free guidance is used only for visual streams.Source: §3.5 · p. 12

3.6 Training Objective

  • Each mode uses one loss form: conditioned modalities remain clean without prediction loss, omitted modalities leave computation and loss, and generated modalities receive supervision.The mode partitions modalities into generated, conditioned, and omitted sets; modality-specific losses use weights to balance scales or effective token counts.Source: §3.6 · p. 12

4 Data

Thinker-WM combines egocentric, robot, and internet data to provide varied visual, action, and geometric supervision. Its depth pipeline aligns filtered depth labels with RGB sequences for training.

4 Data

  • The corpus includes about 20,000 hours of egocentric video and 10,000 hours of robot data, including around 4,000 hours with aligned depth.Figure 6 shows example episodes with video frames, aligned depth labels, action curves, and task text.Source: §4 · p. 12, Figure 6 · p. 13

4.1 In-House Data

  • In-house egocentric recordings span household, commercial, retail, and industrial settings, while real-robot data pairs visual observations with executable actions.The egocentric recordings use head-mounted stereo devices and include hand keypoints and head pose.Source: §4.1 · p. 14

4.2 Internet Data

  • Internet sources combine general video for visual and temporal priors, robot datasets for paired observations and actions, simulation, and egocentric video for activity diversity.The listed datasets are representative rather than exhaustive.Source: §4.2 · p. 14

4.3 Depth Data

  • Depth labels come from stereo matching for egocentric video and monocular prediction for robot video; quality filtering removes unreliable clips before aligned RGB-depth training.The depth stream uses logarithmic encoding and the same frozen VAE as RGB, then is learned as an additional generative stream using flow matching.Source: §4.3 · p. 14, Figure 7 · p. 14

5 Infrastructure

The Data Factory separates heterogeneous data records from training-time sampling and preparation. Quality gates, batch reuse, deterministic rank coordination, and cached evaluation are designed to keep training supplied with valid data without stalling.

5.1 Overview

  • The Data Factory separates factual episode records from sampling policies, allowing either layer to evolve independently while materializing the samples requested by the model.Episodes carry modalities, independent index spaces, schemas, and physical locations; absent modalities are declared rather than imputed.Source: §5.1 · p. 15

5.2 Quality Control and Gating

  • Quality gates reject invalid or unsuitable episodes and windows, validate loaded actions and states, and mark missing or unreadable depth absent rather than filling it.Batch checks validate shapes, ranges, finiteness, and the model input contract.Source: §5.2 · pp. 15–16

5.3 Data Preparation and Throughput Matching

  • Each encoded batch is reused across N optimization steps with fresh noise and timesteps, while a bounded producer queue keeps prepared batches ahead of consumption when preparation fits within that window.Figure 9 illustrates preparation staying ahead of consumption within a VAE reuse window.Source: §5.3 · p. 16, Figure 9 · p. 16

5.4 Load Balancing across Ranks

  • Function modes are determined by the global step and shared across ranks and restarts, keeping branch subsets consistent; fixed token lengths also ensure matching rank layouts.Source assignment can use the same source on every rank or distribute rank slots across sources, with the latter carrying a risk of stragglers.Source: §5.4 · p. 16, Figure 10 · p. 17

5.5 Evaluation Orchestration

  • Evaluation uses deterministic samples and a shared-storage cache prepared in the background, so foreground evaluation reads cached clips without occupying training GPUs.Evaluations with unready caches are skipped rather than stalling training; out-of-memory shards rerun with smaller micro-batches.Source: §5.5 · p. 17

6 Training Strategy

Thinker-WM trains on varied video and robot sources through a staged curriculum that introduces capabilities while replaying earlier data. The schedule progressively integrates geometry and action into the shared model, with function modes and noise schedules configured for training.

6.1 Staged Training Curriculum

  • The curriculum addresses competing heterogeneous objectives and forgetting by progressively introducing data and capabilities while retaining a small replay stream from earlier stages.Stages progress from the pretrained visual prior, through video-only adaptation and joint video-action-geometry training, to target-robot specialization.Source: §6.1 · pp. 17–18, Table 3 · p. 18

6.2 Stage-wise Parameter and Geometry Schedule

  • Geometry enters in Stage 2, first shaping shared weights through its own loss; with accurate target-robot depth, appearance and action also read the geometric stream.Stage 1 updates the visual pathway only, while Stage 2 onward jointly optimizes trainable parameters.Source: §6.2 · p. 18

6.3 Function Mode and Noise Scheduling

  • Six function modes are sampled in a fixed ratio, with WAM receiving half of optimization steps; joint video-action modes use a shared noise base with independent jitter.At inference, the two streams are denoised in lockstep; single-modality modes retain independent per-modality sampling.Source: §6.3 · p. 19

7 Motion Probe

Motion Probe tests whether action-relevant displacement can be recovered from frozen UDT video representations, offering evidence about the shared representation rather than improving model performance. Its lightweight head is evaluated against a fixed visual motion teacher.

7.1 Probe Design and Offline Motion Teacher

  • A frozen visual teacher is built by tracking RGB trajectories, filtering and aggregating them, then aligning each window to recorded end-effector motion with a Sim(3) transform.Teacher signals are cached; windows failing reliability thresholds are excluded.Source: §7.1 · pp. 19–20, Figure 11 · p. 20

7.2 Probe Objective and Interpretation

  • A small Motion Head predicts left- and right-hand displacement from frozen video tokens using a loss combining endpoint error and motion-magnitude discrepancy.Only the head is trained; low probe error is interpreted as evidence that appearance and action are coupled in the shared representation.Source: §7.2 · pp. 20–21

7 Motion Probe

  • Motion Probe measures whether a lightweight head can recover end-effector displacement from frozen UDT video tokens, testing whether the representation carries action-relevant motion.The UDT is not modified, and the probe adds no training stage or generation branch.Source: §7 · p. 19, §7.1 · p. 19, §7.2 · p. 21

8.1 Results on Benchmarks

Thinker-WM ranks first on the reported overall metrics for EWMBench, WorldOlympiad Robotics, and WorldArena Track-1. The evaluations cover real-robot and simulated settings, with EWMBench also showing gains through the training stages.

8.1.1 EWMBench Evaluation

  • On EWMBench, Thinker-WM reaches nDTW 0.7102 and Dyn 0.3630, exceeding Qwen-RobotWorld and LVP on both motion-related metrics.Overall rises from 2.63 at the pretrained backbone to 3.34 after video training, 4.44 after joint video–action training, and 4.66 after robot-specific fine-tuning (Figure 12).Source: §8.1.1 · pp. 21–22, Figure 12 · p. 22

8.1 Results on Benchmarks

  • Thinker-WM achieves the highest EWMBench Overall score (4.66), where Overall sums eight normalized dimensions, and the highest WorldOlympiad Robotics Subdomain leaderboard overall score (0.740) among compared methods.Source: §8.1.1 · p. 21, Table 4 · p. 22, Table 5 · p. 23

8.1.2 WorldOlympiad Evaluation

  • On WorldOlympiad Robotics, Thinker-WM achieves the highest All score (0.740), 3D Consistency (0.531), and Interaction score (0.765) among compared methods.The leaderboard covers 400 manipulation episodes across more than ten physical robot embodiments and multiple camera viewpoints.Source: §8.1.2 · pp. 22–23, Table 5 · p. 23, Figure 13 · p. 23

8.1.3 WorldArena Track-1 Evaluation

  • On WorldArena Track-1, Thinker-WM achieves the highest 12-metric overall score, 71.73, in the simulation-based evaluation.The comparison excludes three motion metrics because of frame-sampling exploitation bugs in those metrics.Source: Table 6 · p. 24, §8.1.3 · p. 24

8.2 Ablation and Analysis

Controlled studies test action conditioning, depth interaction, shared-mode optimization, and motion information in frozen video features. They report gains from explicit action conditioning and bidirectional RGB–depth interaction, non-conflicting mode gradients, and progressively more recoverable motion information, with limits on what the probe and gradient analysis establish.

8.2.1 Action Conditioning Controls Future Generation

  • Ground-truth action conditioning raises EWMBench Overall from 4.66 to 5.03, with HSD rising from 0.5638 to 0.7369 and nDTW from 0.7102 to 0.8736.The comparison uses the same model, and the study interprets the motion-sensitive metric changes as evidence that generated futures respond to the supplied trajectory.Source: Table 7 · p. 25, §8.2.1 · p. 25

8.2.2 Depth as a Co-Evolving Generative State

  • On 48 AgiBot validation sequences, Thinker-WM’s depth AbsRel is 0.1360 and RMSE is 0.1790 m, versus 0.1625 and 0.2075 m for RynnWorld-4D.Restricting RynnWorld-4D to Thinker-WM’s valid depth range changes its AbsRel only from 0.162517 to 0.162474.Source: Table 8 · p. 25, §8.2.2 · p. 25
  • On RoboTwin turn_switch, bidirectional RGB–depth interaction achieves 75.0% success, versus 63.3% with one-way interaction and 60.0% for RGB-only.The full model outperforms both variants at every evaluated checkpoint.Source: Table 9 · p. 26, §8.2.2 · p. 26

8.2.3 Gradient Compatibility across Function Modes

  • Across six function modes, shared-UDT gradient cosine similarity has a minimum of approximately −0.005, while auxiliary-mode gradient norms range from 0.34 to 0.50 versus 0.68 for joint WAM.The gradients are measured on the same batches and noise realization; the analysis does not determine whether the mode-sampling ratio is optimal under a fixed training budget.Source: §8.2.3 · pp. 26–27, Figure 14 · p. 27

8.2.4 Motion Probe Evaluation

  • Action MSE decreases monotonically across evaluated checkpoints, indicating that end-effector displacement becomes increasingly recoverable from frozen UDT video representations.The probe uses 96 held-out AgiBot windows and relies on an imperfect visual teacher, so the result is evidence of recoverability under the probe protocol rather than exact reconstruction across all candidate windows.Source: §8.2.4 · pp. 27–29, Figure 15 · p. 28

8.3 Qualitative Capabilities and Generalization

The section uses controlled qualitative probes rather than quantitative comparisons to test long-horizon generation, bidirectional trajectory/video interaction, and zero-shot execution of unseen instructions in unseen scenes.

8.3.1 Training-Free Long-Horizon Generation

  • With only the first frame as ground truth, chained rollouts span 96 seconds on EWMBench and about 115 seconds on real-robot episodes without test-time supervision.The model preserves object layout and robot configuration; each segment is conditioned on a clean generated frame and the instruction for its time window.Source: §8.3.1 · pp. 29–30, Figure 16 · p. 29

8.3.2 Trajectory Transfer and Action Recovery

  • Inverse video-to-action recovery matches recorded actions at RMSE 0.0540 in native reader space and 0.0293 in unified action space.The largest residuals occur at gripper open/close transitions.Source: Table 11 · p. 30, §8.3.2 · p. 30
  • Across recordings, supplied real end-effector trajectories steer generated motion along the requested path rather than the motion originally performed in the scene.The scene and trajectory come from different recordings, testing whether generation follows the supplied trajectory.Source: §8.3.2 · p. 30, Figure 17 · p. 30

8.3.3 Zero-Shot Generalization

  • From the same unseen conditioning frame, the model follows two different handwritten instructions while leaving the uncommanded arm in place.One instruction requests lifting a spoon from a cup; the other requests stirring its contents.Source: §8.3.3 · pp. 31–32

8.3.4 Zero-Shot Manipulation across Diverse Scenes

  • After clean fixed-view RoboTwin training, the model executes new clock-pressing and roller-grasp instructions in randomized scenes, including an unseen 65° camera view, without fine-tuning.The examples show joint scene and viewpoint generalization, not an isolated viewpoint effect.Source: §8.3.4 · p. 33, Figure 23 · p. 34
  • Across all 50 RoboTwin 2.0 tasks, under standard Clean and Randomized settings and the benchmark’s closed-loop protocol, the model achieves average success rates of 92.6% and 91.4%, respectively—a 1.2 percentage-point drop.Source: §8.3.4 · p. 33

9 Conclusion and Future Work

Thinker-WM unifies RGB, depth, and robot action through a shared Unified Dynamics Transformer while retaining modality-specific encoders and decoders. It leads three benchmarks, while real-world closed-loop validation remains open.

9 Conclusion and Future Work

  • Thinker-WM shares one dynamics transformer across RGB, depth, and robot action, supporting six world-action functions with modality-specific encoders and decoders.Training uses approximately 20K hours of egocentric video, 10K hours of robot data, and 1K hours of simulation, including about 4K hours with aligned depth; staged curriculum and replay stabilize shared representations and specialize to the deployed robot.Source: §9 · p. 34
  • EWMBench Overall reaches 4.66, WorldOlympiad Robotics All scores 0.740, and WorldArena Track-1 EWMScore reaches 71.73.Source: §9 · p. 35
  • Recorded-action conditioning raises EWMBench Overall from 4.66 to 5.03, while bidirectional RGB-depth interaction raises RoboTwin closed-loop success from 60.0% to 75.0%.A frozen video representation also contains recoverable action-relevant motion information.Source: §9 · p. 35
  • Real-robot closed-loop efficacy remains unestablished; depth supervision covers only part of training data, and long-horizon drift lacks quantitative evaluation.Long-horizon generation currently uses inference-time autoregressive rollout; scaling and choosing function-mode sampling ratios also remain open.Source: §9 · p. 35

Related papers

Loading 2610.18447v2…