X-World: Transferring Video Priors to Variable-View Robot Control

Introduction

• A fixed-length token sequence separates variable camera processing from action decoding, with cross-view attention added through a residual branch that retains pretrained self-attention.A layer-aligned Q-Former aggregates features across backbone layers, while dataset-specific projections map native state and action representations to and from the shared action expert.Source: §1 · pp. 1–2
• Training adapts the video backbone first, freezes it for action-only pretraining, then jointly post-trains with video and action supervision; deployment uses one backbone pass and asynchronous action chunking.On one H100, cached model response is 46 ms and deployed refresh is 91 ms, while queued actions execute at 30 Hz.Source: §1 · p. 2
• X-World achieves 98.5% mean success on LIBERO, 76.5% on zero-shot LIBERO-Plus, and 81.7% on six dual-arm tasks versus X-VLA’s 58.3%.The real-world result averages 20 trials per task; LIBERO is nearly saturated, motivating analysis beyond its average ranking.Source: §1 · p. 2
• Removing cross-view attention lowers LIBERO success to 83.1% and Fold Towel success from 12/20 to 0/20.The paper analyzes transfer through training-stage ablations, seven distribution shifts, and dual-arm experiments, with separate model and system latency measurements.Source: §1 · p. 2

From video models to robot actions

• X-World aggregates video-backbone features into fixed-length tokens for action generation, using one backbone pass rather than decoding future observations as in rollout-based approaches (Du et al., 2023b,a; Wen et al., 2024).Prior work also uses internal video representations as action features (Hu et al., 2024; Yang et al., 2025), jointly models video and actions (Bi et al., 2025; Liao et al., 2025; Ye et al., 2026; Li et al., 2026; Pai et al., 2025), or encodes actions in video generation (Kim et al., 2026).Source: §2 · p. 2

Multi-view adaptation

• Unlike Genie-Envisioner (Liao et al., 2025), X-World retains per-view pretrained self-attention and adds residual cross-view fusion, initialized to contribute zero.The work separately evaluates residual fusion and view identity, and distinguishes missing-view evaluation from camera-pose robustness.Source: §2 · p. 2

Heterogeneous action learning and continuous execution

• X-World uses dataset-specific input and output modules around a shared action expert, building on prior work addressing heterogeneous robot interfaces (Wu et al., 2025; Jiang et al., 2025; Wu et al., 2024).It adopts asynchronous chunking inspired by RTC (Black et al., 2025a,b), and reports the system’s latency budget separately from task success.Source: §2 · pp. 2–3

3.1 PROBLEM AND DESIGN

• The policy maps RGB views, language, and proprioceptive state to 16-step action chunks, separating variable-view visual encoding from native action decoding with a fixed-length token sequence.The 24-frame video-adaptation horizon is distinct from the executable 16-action horizon.Source: §3.1 · p. 3

3.2 VARIABLE-VIEW VIDEO REPRESENTATIONS

• A zero-initialized residual cross-view branch preserves pretrained per-view self-attention while enabling fusion across one to three views, and a layer-aligned Q-Former outputs fixed-length tokens.View masks isolate replicated slots, while view embeddings and randomized view order support heterogeneous camera inputs.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 SHARED ACTION EXPERT AND TRAINING

• Training adapts the video backbone on robot and human videos, freezes it during action-only pretraining, then jointly updates all modules on target-domain demonstrations.A shared DiT-style action expert attends to the visual tokens, while dataset-specific interfaces encode state and decode native actions.Source: §3.3 · pp. 4–5, Table 1 · p. 5

3.4 FEATURE EXTRACTION AND ASYNCHRONOUS EXECUTION

• At control time, one video-backbone pass supplies features to the Q-Former and four-step action expert; asynchronous chunking executes queued actions while refreshing predictions.When the remaining queue reaches threshold P, the policy reuses committed actions as a prefix and appends up to E new suffix actions.Source: §3.4 · p. 5

4.1 SIMULATION RESULTS

• X-World reaches 98.5% mean success on LIBERO; it exceeds X-VLA on Spatial (99.6% versus 98.2%), while X-VLA leads on Long (97.6% versus 95.8%).The reported average gap between X-World and X-VLA is 0.4 points at this saturation level.Source: §4.1 · p. 6
• 76.5% total success on zero-shot LIBERO-Plus exceeds OpenVLA-OFT’s 69.6%, with gains under robot, language, and layout shifts but lower success under camera, background, and noise shifts.The robot results are 72.5% versus 31.9%; language and layout improve by 14.8 and 10.5 points, while camera, background, and noise success are lower by 3.8, 7.5, and 16.1 points. The LIBERO-trained checkpoint is evaluated without further adaptation.Source: §4.1 · p. 6, Table 3 · p. 7

4.2 REAL-WORLD SETUP AND TASKS

• Real-world evaluation uses synchronized three-view RGB inputs, 30 minutes of demonstrations per task, and 20 trials per task under shared placements, views, control frequency, and task definitions.The dual-arm platform has two wrist cameras and a shared third-person camera; actions are relative joint position targets and proprioceptive inputs are absolute joint positions.Source: §4.2 · p. 7

4.4 ABLATION STUDIES

• Video adaptation and action pretraining interact in the reported results, while removing residual cross-view attention drops LIBERO success from 98.5% to 83.1%.With video adaptation, action pretraining raises mean success from 97.4% to 98.5%; without it, mean success falls from 92.1% to 89.5%. Removing view embeddings yields 93.3% success; on Fold Towel, full-model success is 12/20 versus 0/20 without residual cross-view attention and 2/20 without view embeddings.Source: §4.4 · pp. 8–9, Table 4 · p. 9
• The Q-Former reaches 98.5% mean success versus 97.9% with direct last-layer features, while shared state and action projections yield 97.4%.Removing video loss yields 98.3% mean success and a higher Long score (96.2% versus 95.8%); the small average difference offers limited evidence for a control benefit from the joint video objective. The Q-Former takes 5 ms, but matched latency for direct features is unavailable.Source: §4.4 · pp. 8–9

4.3 REAL-WORLD RESULTS

• 81.7% mean success across six real-world tasks (98/120 trials) is the highest point estimate on every task, compared with 58.3% for X-VLA, 51.7% for Genie-Envisioner, and 45.0% for π0.5.The evaluation uses 20 trials per task; aggregate counts do not determine training-seed variability or paired trial outcomes.Source: §4.3 · pp. 7–8, Figure 5 · p. 9

4.5 VIEW AVAILABILITY AND REAL-TIME EXECUTION

• On a single H100, the cached response is 46 ms, including 14 ms of model-side overhead, while the deployed refresh is 91 ms and additionally includes network transfer, data processing, and control-side overhead.At 30 Hz, prefixes of two, four, six, and eight actions cover approximately 67, 133, 200, and 267 ms; only the first is below the reported 91 ms refresh. With P = 6, the committed prefix provides a nominal margin of 109 ms, without establishing a bound on every query or an observed queue-underflow rate.Source: Table 5 · p. 10, §4.5 · p. 10
• 80% success with three views exceeds the one-view and two-view point estimates of 40% and 55%, while shuffled three-view input also reaches 80%.The same checkpoint is used across settings; the aggregate sweep does not specify task selection, trial denominators, or seed variation. Action-prefix success is 80% without RTC and 65%, 70%, 80%, and 75% for P = 2, 4, 6, and 8, respectively.Source: Figure 6 · p. 10, §4.5 · p. 10

5 DISCUSSION AND CONCLUSION

• X-World transfers video representations through residual multi-view adaptation and a fixed-length token interface; removing video adaptation or cross-view fusion causes the largest tested success drops.Action pretraining helps after video adaptation but reduces performance when adaptation is skipped; gains on robot and language perturbations accompany losses on camera and noise shifts.Source: §5 · p. 10
• Evaluation does not establish transfer to unseen embodiments or a scaling law, and several ablations and comparisons do not isolate the effects they are intended to assess.Training-stage ablations also change supervision; the Q-Former comparison confounds compression with access to multiple layers, and its 0.6-point average difference needs repeated runs and a matched feature baseline. Baseline pretraining data and compute are uncontrolled, seed variability is unquantified, LIBERO-Plus denominators and weighting are unspecified, real-world estimates use 20 trials per task, and latency tail behavior and action continuity remain unquantified.Source: §5 · p. 11

ETHICS STATEMENT

• Teleoperation recordings should be treated as potentially identifying, and physical robot experiments require supervision, workspace safeguards, and an emergency stop.Public data are used according to their licenses, and any redistribution of teleoperation recordings should preserve applicable consent and privacy requirements; simulation success or a qualitative rollout does not establish general hardware safety.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

• The paper reports key training, model-input, inference, and evaluation details, but omits information that limits independent reproduction and uncertainty analysis.Full model dimensions, optimization configurations, post-training budgets, and per-run evaluation logs are not supplied.Source: REPRODUCIBILITY STATEMENT · p. 11

• The section summarizes data sources across X-World’s three training stages and the sampling strategy for heterogeneous action pretraining.Source: A DATA AND TRAINING DETAILS · p. 14

A.1 STAGE 1: VIDEO PRE-TRAINING

• Robot and human videos are combined to provide diverse visual interaction data for video pre-training.Source: §A.1 · p. 15
• Each sample is converted to a unified variable-view format that retains up to three camera views.Source: §A.1 · p. 15
• Robot and human videos supply future-frame supervision, and Table 6 summarizes the Stage 1 datasets and available views.Source: Table 6 · p. 15, §A.1 · p. 15

A.2 STAGE 2: ACTION PRE-TRAINING

• Stage 2 uses robot trajectories for action supervision across diverse manipulation settings; Table 7 summarizes data sources, camera counts, and dataset-specific projections for native state and action formats.Source: §A.2 · p. 15, Table 7 · p. 15

A.3 TEMPERATURE-SCALED MULTI-DATASET SAMPLING

• For all datasets, α_d = 1.0 and T = 2.0 make effective reweighting scale with the square root of dataset step count before normalization.Dataset steps are the sampleable state and action transitions in that dataset.Source: §A.3 · p. 16

A.4 STAGE 3: REAL-WORLD POST-TRAINING DATA

• Post-training uses self-collected teleoperation data for six real-world tasks.The task definitions and evaluation protocol are described in Sec. 4.2.Source: §A.4 · p. 16
• Each task has 30 minutes of demonstrations.Source: §A.4 · p. 16
• Demonstrations use synchronized three-view observations.Source: §A.4 · p. 16

B IMPLEMENTATION DETAILS

• Implementation details cover variable-view processing, the token-sequence interface, heterogeneous state/action alignment, and asynchronous real-time chunking.Source: B IMPLEMENTATION DETAILS · p. 16

B.1 VARIABLE-VIEW VIDEO PROCESSING

• View identity embeddings, randomized view order, independently noised replica slots, and cross-view masks support one-, two-, and three-view inputs without masked padding.Each sample uses three slots; when views are missing, an existing view is replicated, while masks prevent replica-original interactions.Source: §B.1 · p. 16

B.2 FPS-AWARE TEMPORAL ROPE SCALING

• Per-sample frame rates scale temporal RoPE coordinates in the LTX-Video backbone, aligning clips with different sampling rates within the same architecture.The same sample-level fps is repeated across views; the backbone builds on LTX-Video (HaCohen et al., 2024), and RoPE interpolation follows Su et al. (2024).Source: §B.2 · p. 16

B.3 Q-FORMER AND TOKEN SEQUENCE INTERFACE

• A layer-aligned Q-Former distills backbone hidden states into a fixed-length token sequence that the action expert attends to when generating action chunks.At each layer, query tokens cross-attend to the corresponding backbone hidden states; token-sequence size is independent of input view and video-token counts.Source: §B.3 · p. 16

B.4 ACTION REPRESENTATION AND DATASET-SPECIFIC MODULES

• Dataset-specific state projectors, action-input projections, and output heads align heterogeneous native interfaces with a shared action-generation backbone.For the real-world platform, actions are relative joint-position targets and proprioceptive state uses absolute joint positions; the design follows X-VLA (Zheng et al., 2026).Source: §B.4 · p. 17

B.5 ASYNCHRONOUS REAL-TIME CHUNKING

• Asynchronous chunking executes queued actions while refreshing future actions in the background, reusing the remaining queue as a committed prefix to avoid waiting gaps.When the remaining queue length R is at most threshold P, the policy discards overlapping predictions and appends E new suffix actions.Source: §B.5 · p. 17

C ADDITIONAL VIDEO PREDICTIONS

• Stage 1 examples show one-, two-, and three-view predictions, while real-world post-training examples show table clearing and block stacking; these videos are qualitative, not control-success evidence.The examples display observed frames and predictions at frames 8, 16, and 24 of a 24-frame clip, with synchronized camera views.Source: C ADDITIONAL VIDEO PREDICTIONS · p. 17, Figure 7 · p. 18, Figure 8 · p. 19

D SUPPLEMENTARY VIDEOS

• The supplementary package contains six real-world task videos; the policy uses synchronized three-view observations, while the videos display the third-person overview and do not compute success rates.Source: D SUPPLEMENTARY VIDEOS · p. 17