PaperlayerView exact revision on arXiv

Source-linked AI summary

One from Infinity: Actualizing Futures from Pretrained World Models into Robot Actions

Bang Du, Yichen Xie, Shuqi Zhao, Yuxin Chen, Menglin Wu, Masayoshi Tomizuka

arXiv:2609.36413v3cs.RO

TL;DR

Video world models represent many plausible futures, but robot policies must select the one matching a task and produce its actions; existing approaches require costly rollouts or backbone fine-tuning. RoboActualizer freezes a world model and trains a small actualizer to predict task-conditioned future latents and actions. It reports stronger performance than large baselines across simulation and real-world tasks, with real-time inference.

  • Problem

    World models represent multiple possible continuations, while robot control requires precise actions for one task; existing planning or fine-tuning approaches incur substantial inference or training costs.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    RoboActualizer freezes a video world model and jointly trains compact experts to predict task-conditioned future latents and the actions that realize them.

    Source: §1 · p. 2

  • Results

    RoboActualizer outperforms VLA and WAM baselines with billions of trainable parameters across LIBERO, LIBERO-Plus, RoboTwin 2.0, and two real-world setups, using 60M trainable parameters.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The task-conditioned selection and action readout can be learned on cached features without updating the pretrained world model, and deployment runs above 25 Hz on an NVIDIA RTX A6000 GPU.

    Source: §1 · pp. 2–3

    Experiments do not cover the complete RoboTwin 2.0 benchmark, and the method is tested only with the V-JEPA 2.1 world model.

    Source: §5 · p. 10

Abstract

from arXiv · show

A pretrained video world model admits many plausible futures for a scene, but a robot must realize the exact task-conditioned one. To turn world models into executable robot policies, existing methods fine-tune the heavy world model backbone using large-scale robot data and computational resources. Challenging this status quo, we argue that the expensive part has already been paid in the world model pretraining since the representation space of a video world model lays out the diverse potential futures. In this case, what remains is to select the future that accomplishes the task and to read out the actions that realize it. We formalize this task as actualization, which learns a task-conditioned selection and realization on top of a prior supplied by a frozen world model. This can be solved by a tiny actualizer model. We implement RoboActualizer with as few as 60M parameters on top of a frozen world model encoder. The actualizer is composed of two lightweight DiT experts that jointly predict future latents and actions by flow matching. The model can be trained entirely on a single GPU with 32 GB peak memory. With up to 100x fewer trainable parameters than existing WAMs and VLAs, RoboActualizer reaches great performance on simulation benchmarks including LIBERO, LIBERO-Plus, RoboTwin 2.0 and five tasks on two real-world platforms, with a low latency of 39 ms that allows real-time control.

1 INTRODUCTION

Pretrained video world models represent many plausible futures, but robot control requires selecting the task-achieving future and producing its actions. RoboActualizer freezes the world model and uses a lightweight module to make that selection and read out actions, with reported performance across simulation and real-world tasks.

Motivation

  • World models represent diverse scene evolutions, whereas robots must complete one specific task with precise actions.Existing approaches either search candidate actions using repeated imagined rollouts or fine-tune a heavy backbone with robot data and substantial computation.Source: §1 · p. 2

Approach

  • RoboActualizer selects the task-achieving future from the world model’s possibilities and reads out the actions that realize it, without updating the world model.Its task-conditioned future-latent and action experts are trained jointly from demonstrations.Source: §1 · p. 2

Strong Performance

  • RoboActualizer outperforms strong baselines across simulation benchmarks and five tasks on two real-world robots with only 60M trainable parameters.The paper reports up to 100× fewer trainable parameters than the baselines.Source: §1 · p. 2

Generalization

  • 63.1% on LIBERO-Plus versus 51.5% for Fast-WAM, which has 6.02B trainable parameters.The authors say the pretrained world model representations appear to preserve robustness that fine-tuning tends to erode.Source: §1 · p. 2

Cost

  • Training on cached latents allows the approximately 60M-parameter model to fit on a single GPU with 32 GB peak memory, without embodied pretraining.Source: §1 · p. 2

Efficiency

  • More than 25 Hz on an NVIDIA RTX A6000 GPU supports real-time robot control.Source: §1 · p. 3

2 RELATED WORK

Prior work extracts robot actions from world models through planning or by fine-tuning them, while another line applies pretrained visual representations to control. RoboActualizer instead relies on temporal features from a frozen world model and learns a lightweight task-conditioned readout.

Video World Models

  • Video world models either generate future frames in pixel space or predict future representations, with both approaches spanning plausible futures rather than committing to one continuation.The paper uses a pretrained latent world model as a fixed representation of that future space.Source: §2 · p. 3

From World Models to Robot Actions

  • Planning methods preserve the world model but use repeated rollouts at inference, while world-action models fine-tune pretrained models with future-frame supervision.Other approaches use future latent representations as supervision, but the paper says frame-by-frame encoding leaves temporal reasoning to a heavy trainable model.Source: §2 · p. 3

Pretrained Visual Representations for Robot Control

  • Frozen pretrained visual features can be sensitive to domain mismatch, but the authors argue that the gap may reflect how features are read out rather than what they contain.RoboActualizer uses V-JEPA 2.1 features as policy inputs and future-prediction targets.Source: §2 · p. 3

3 METHODOLOGY

The methodology formalizes actualization as selecting a task-conditioned future from a frozen world model’s prior and realizing it as actions. It instantiates this approach with frozen V-JEPA 2.1 and a narrow, two-expert actualizer.

3.1 FORMULATION: ACTUALIZING ONE FUTURE FROM MANY

  • The frozen representation supplies a mixture of plausible futures, while actualization learns task-conditioned selection and action realization without changing the prior.The actualizer therefore selects a component of the futures already represented rather than learning scene dynamics from scratch.Source: §3.1 · p. 4
  • Given demonstrations, the actualizer is fit by maximum likelihood to learn which future matches an instruction and which actions realize it.The method describes the prior as already encoded in the frozen latent space, making selection rather than dynamics modeling the learned future component.Source: §3.1 · p. 4

3.2 FUTURES FROM A FROZEN VIDEO WORLD MODEL

  • Frozen V-JEPA 2.1 encodes present and future clips into the same latent space, with future targets sampled at a few times across the action chunk.Because the encoder is frozen, latents can be cached offline for actualizer training.Source: §3.2 · p. 5

3.3 WHY A SMALL ACTUALIZER CAN WORK

  • At width 384, V-JEPA 2.1 latents retain 93.8% of variance, motivating a narrow actualizer; clean-sample prediction avoids the full-rank noise of v-prediction.The reported retained variance is stable across benchmark suites.Source: Figure 3 · p. 5, §3.3 · p. 5

3.4 LIGHTWEIGHT ACTUALIZER MODEL

  • Two lightweight DiT experts implement future-latent selection and action realization, using separate weights, shared self-attention, and instruction and proprioceptive conditioning.Observation and noisy future-latent tokens enter the latent expert, while noisy action tokens enter the action expert.Source: §3.4 · pp. 5–6
  • Each expert predicts clean tokens, while training computes velocity-space losses for the latent and action predictions.The loss balance is controlled by λ_z.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments evaluate RoboActualizer on three simulation benchmarks and two real-world platforms against VLA and WAM baselines. They report benchmark performance, real-world deployment, training and inference costs, and ablations of the lightweight actualization design.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers three simulation suites and two real-world platforms, comparing against VLA and WAM baselines; the actualizer is randomly initialized for each benchmark.The default setup uses a pretrained 300M-parameter V-JEPA 2.1 and a 60M-parameter DiT-S actualizer.Source: §4 · p. 6, §4.1 · p. 6

4.2 GENERALIZATION IN SIMULATION

  • RoboActualizer achieves 98.0% average success on LIBERO across four standard evaluation suites with 60M trainable parameters, and greatly outperforms baselines on RoboTwin 2.0 using 50 demonstrations per task.For LIBERO-Plus, it is trained only on LIBERO demonstrations, without augmentations fitted to the shifted distribution.Source: §4.2 · p. 7

4.3 REAL-WORLD DEPLOYMENT

  • Across five real-world tasks on a FANUC arm and Unitree G1, RoboActualizer consistently outperforms the corresponding baselines.The evaluations cover single-arm and bimanual manipulation.Source: §4.3 · pp. 7–8, Table 4 · p. 8

4.4 COST AND EFFICIENCY

  • 60M trainable parameters train in 15 hours on one 32-GB GPU, versus Fast-WAM’s estimated 112 GPU-hours; inference latency is 39 ms.Inference is measured on one NVIDIA RTX A6000 GPU.Source: §4.4 · p. 8, Figure 6 · p. 9

4.5.1 HOW LIGHTWEIGHT ACTUALIZER HELPS?

  • Removing future-latent prediction drops LIBERO-Plus success from 60% to 17.9%, and predicted latents track teacher features across time.The ablation compares joint latent-action prediction with action-only prediction.Source: §4.5.1 · p. 8, Table 5 · p. 8, Figure 5 · p. 9

4.5.2 HOW RoboActualizer DIFFERENT FROM LATENT WORLD ACTION MODELS?

  • Unlike latent-action models that learn futures with billion-parameter models, RoboActualizer selects futures already represented; DINO and single-frame features perform worse.The ablations indicate that future-prediction ability in the latent space matters more than representation ability alone.Source: §4.5.2 · pp. 8–9, Table 6 · p. 8

4.5.3 HOW TO ALLOCATE THE COMPUTATION

  • Scaling the frozen world model is reported as more beneficial than scaling the trainable actualizer, which only selects among futures the model admits.The computation-allocation comparison is reported in Table 7.Source: §4.5.3 · p. 9, Table 7 · p. 9

5 CONCLUSION

RoboActualizer pairs a frozen video world model with a compact actualizer that selects task-conditioned futures and reads out actions. The paper reports performance across simulation and real-world evaluations, while noting that its benchmark and world-model coverage remain limited.

5 CONCLUSION

  • RoboActualizer inherits future-dynamics priors from a frozen representation space, leaving robot-data learning to task-conditioned selection and action readout.The actualizer is a 60M-parameter DiT with a Mixture-of-Transformers architecture that jointly predicts latent features and actions.Source: §5 · pp. 9–10
  • The 60M-parameter model outperforms VLA and WAM baselines with billions of trainable parameters on LIBERO, LIBERO-Plus, RoboTwin 2.0, and two real-world setups.The experiments span simulation benchmarks and physical robot platforms.Source: §5 · p. 10
  • The study omits the complete RoboTwin 2.0 benchmark and tests only V-JEPA 2.1; combinations with other world models and large-scale benchmarks are future work.The paper says code will be released when accepted.Source: §5 · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL ARCHITECTURE ABLATIONS

The ablations compare the two-expert MoT design, prediction target, and head width. They support retaining separate experts and x-prediction, while a wide head does not improve performance.

Prediction Target

  • The x-prediction target is important for strong performance and allows the latent space to be represented with a narrow network.The narrow-network explanation is consistent with the paper's analysis in Sec. 3.3.Source: A ADDITIONAL ARCHITECTURE ABLATIONS · p. 14

Wide Head

Mixture-of-Transformers Architecture

B.1 MODEL ARCHITECTURE DETAILS

The architecture keeps the V-JEPA encoder frozen and uses a two-branch denoiser with separate conditioning and output paths. Its implementation also specifies positional encoding, latent normalization, and branch-specific components.

B.1 MODEL ARCHITECTURE DETAILS

  • The frozen V-JEPA 2.1 ViT-L/16-384 video encoder has about 300M parameters, remains in float32 and eval mode, and is excluded from optimization and released checkpoints.Inputs are mapped from the pipeline's [−1, 1] convention to ImageNet normalization; frames use the video branch.Source: §B.1 · p. 14
  • The two-branch denoiser gives image and action streams separate q, k, v, cross-attention, and FFNs, while mixing concatenated tokens through joint self-attention.Each branch has 12 pre-norm blocks; remaining parameters include per-branch context encoders and timestep embedders, a shared latent projection, and two output heads.Source: §B.1 · p. 14
  • RoPE is the sole positional encoding: image tokens use 3D frame-height-width positions, while action tokens use 1D positions aligned to the same step axis.Rotations apply to q and k, not v.Source: §B.1 · p. 14
  • Each branch receives 128 frozen umT5 text tokens and one embedded 8-D proprioception token, with separate text and proprio encoders.The resulting 129-token conditioning stream is shared across that branch's 12 layers.Source: §B.1 · p. 15
  • The image and action output heads produce 1024-D latent and 7-D action predictions, respectively, and both are zero-initialized.The image head includes non-affine LayerNorm and AdaLN modulation; the action head is a linear layer.Source: §B.1 · p. 15
  • Latents are normalized per channel using mean and standard deviation computed in float64 over the cache, and the same transform is applied to z0 at inference.The statistics are registered as checkpoint buffers so training and evaluation share the same constants.Source: §B.1 · p. 15

B.2 OBSERVATION AND TARGET CONSTRUCTION

The observation and target pipeline builds training windows from LIBERO demonstrations, encodes short motion context, and pairs future latent targets with action chunks over a shared horizon.

B.2 OBSERVATION AND TARGET CONSTRUCTION

  • The four LIBERO suites provide 1,712 demonstrations and 277,713 frames, with one stride-1 training window anchored at every frame.The data use the no-noops LeRobot conversion at 20 Hz.Source: §B.2 · p. 15
  • Each raw 512 × 512 frame is resized per camera, horizontally concatenated, then resized and center-cropped to 224 × 448 before normalization to [−1, 1].The same image pipeline is used for cache generation, online training, and closed-loop evaluation.Source: §B.2 · p. 15
  • The context latent z0 comes from the video clip at t−6, t−4, t−2, and t; its final temporal group yields 392 × 1024 tokens carrying short-horizon motion.Indices before the episode start repeat the earliest frame.Source: §B.2 · p. 15
  • Four future latents at t+4, t+8, t+12, and t+16 supervise the image branch alongside the next 16 normalized actions, covering the same horizon.Actions are 7-D: a 6-D delta end-effector pose and an absolute gripper command; anchor-frame proprioception is 8-D.Source: §B.2 · p. 15

B.3 TRAINING DETAILS

Training uses shifted rectified flow for both experts, with clean-token prediction and a shared optimization protocol across configurations. The action and latent noise levels are sampled independently.

B.3 TRAINING DETAILS

  • Both experts use a shifted rectified-flow schedule with 1,000 discrete noise levels, and predict clean action or future-latent tokens.Noise level τ is sampled through the stated shifted schedule with s = 5 for both branches.Source: §B.3 · p. 15
  • The velocity-space losses are equivalent to 1/(1 − τ)^2-weighted x-prediction losses, with the denominator floored at 0.05 and independent noise levels for the experts.The two loss terms are combined as L = La + λzLz with λz = 1.Source: §B.3 · p. 16
  • All configurations use AdamW, a peak learning rate of 2 × 10^-4, batch size 32, bf16 mixed precision, and one GPU without gradient accumulation.Training uses 277,713 windows, a 5% linear warm-up followed by cosine decay, and reports the final checkpoint without evaluation-based selection.Source: §B.3 · p. 16

C EVALUATION PROTOCOLS AND BASELINE PROVENANCE

Evaluations use closed-loop policies across LIBERO, LIBERO-Plus, and RoboTwin 2.0, with defined datasets, settings, and execution protocols. On RoboTwin 2.0, average success is 58.8%.

C EVALUATION PROTOCOLS AND BASELINE PROVENANCE

C.1 LIBERO AND LIBERO-PLUS

  • LIBERO training combines four standard suites, while the resulting policy transfers to LIBERO-Plus perturbations without additional fine-tuning.LIBERO uses 1,712 demonstrations and 2,000 evaluation episodes; LIBERO-Plus evaluates 10,030 perturbed instances across seven dimensions and reports their unweighted mean.Source: §C.1 · p. 16
  • LIBERO policies receive two camera views, language, and proprioception, then execute denoised action chunks receding-horizon in closed loop.The views are resized and concatenated; each chunk has 16 actions, and the first 10 are executed before replanning.Source: §C.1 · pp. 16–17

C.2 ROBOTWIN 2.0

  • RoboTwin 2.0 evaluates one policy trained on 50 demonstrations per task across Clean and Random settings, using 25 episodes per task and setting.The Random setting varies backgrounds, table clutter, lighting, and table height; evaluation instructions come from an unseen template pool and scenes are disjoint from training.Source: §C.2 · p. 17
  • For RoboTwin 2.0, the policy predicts 14-dimensional bimanual action chunks, executes 24 of 32 steps before replanning, and denoises with 10 Euler steps.Four-frame clips use stride four, with future latents at +8 through +32 steps to match the 50 Hz control rate.Source: §C.2 · p. 17
  • The overall success rate is 58.8% on average, with 57.8% on Clean and 59.9% on Random; complete task-wise results cover all 50 tasks.Source: §C.2 · p. 17

D REAL-ROBOT EXPERIMENTS

Real-robot experiments cover bimanual manipulation on a Unitree G1 and tabletop manipulation on a Fanuc CRX-10iA. The paper describes task-specific demonstrations, observation and action configurations, and deployment procedures for both platforms.

D REAL-ROBOT EXPERIMENTS

  • The real-robot setup uses a 300M V-JEPA encoder and DiT-S with 63.6M trainable parameters, conditioning on a single frame and four future latent targets.The targets are at +4, +8, +12, and +16 steps.Source: D REAL-ROBOT EXPERIMENTS · p. 17

D.1 UNITREE G1

  • Unitree G1 experiments train separate policies for plate handover, box packing, and cup insertion from successful 30 Hz policy rollouts.The policy uses synchronized third-person and head-mounted views; it predicts 16-dimensional absolute joint-position actions in 16-action chunks, executed receding-horizon at 30 Hz.Source: §D.1 · pp. 17–18

D.2 FANUC CRX-10IA

  • Fanuc CRX-10iA experiments train separate policies for mug hanging and multi-stage cooking from human teleoperation demonstrations collected at 10 Hz.The tasks use two RealSense views; the action vector comprises six Cartesian pose commands and two buttons, and each prediction contains 16 actions corresponding to 1.6 seconds.Source: §D.2 · p. 18
  • Fanuc deployment runs at 10 Hz with 10 Euler denoising steps, executes the first eight actions of each 16-step prediction, then re-observes and replans.Episodes are limited to 1,000 control steps, and checkpoints are validated by open-loop replay on held-out demonstrations before deployment.Source: §D.2 · p. 19

Related papers

Loading 2609.36413v3…