PaperlayerView exact revision on arXiv

Source-linked AI summary

CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight

Chensheng Peng, Wenhao Ding, Ran Tian, Zewei Zhou, Jef Packer, Maximilian Igl, Peter Karkus, Yan Wang, Masayoshi Tomizuka, Boris Ivanovic, Marco Pavone, Yuxiao Chen

arXiv:2610.00859v1cs.CV

TL;DR

World action models need their visual futures to match the actions they represent, but independently noising recorded actions and video does not preserve that correspondence. CTRLWAM pairs perturbed actions with simulated visual consequences and adds warped noise schedules and multi-agent streams; the reported experiments show gains across driving and robotics tasks.

  • Problem

    Independent corruption can leave a noised video tied to the recorded future even when perturbed actions imply a different trajectory.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    CTRLWAM pairs simulator-executed perturbed actions with noised visual consequences, uses warped video–action noise schedules, and adds agent streams for predicted or commanded futures.

    Source: §1 · pp. 2–3, §3.1 · p. 4, §3.2 · p. 4

  • Results

    Driving experiments show gains in command following, video–action consistency, and action forecasting, while robotics controls support improved motion fidelity.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The findings support physically aligned denoising as a practical route toward world action models whose imagined outcomes better reflect their actions with improved controllability.

    Source: §5 · p. 9

    Generation latency is a current bottleneck because the model inherits Cosmos 3’s bidirectional architecture.

    Source: §5 · p. 9

Abstract

from arXiv · show

World action models (WAMs) jointly predict actions (intent) and visual future (foresight). Standard training adds noise to recorded actions and video simultaneously, but such training paradigms introduce a mismatch: perturbed actions imply counterfactual future visual, while the noised video remains tied to the GT recording. In low-noise regime, the scene geometry and even the dynamic behavior remain clearly visible from the noisy future frames despite the added noise. We present CtrlWAM, which executes perturbed actions in a simulator and pairs them with their noised visual consequences for joint WAM learning. To accommodate the different denoising requirements of video and actions, we introduce warped video--action noise schedules that aim to keep visual layout responsive as action predictions evolve. We further extend the action interface from ego-only control to a variable number of agent streams, allowing a unified model to represent predicted or commanded futures for multiple agents. Driving experiments show more accurate action forecasts, closer agreement between generated video and actions, and better following of supplied commands; robotics experiments show stronger motion fidelity and controllability. Matched controls support the benefit of off-path renders for command following and manipulation fidelity. Together, these findings contribute to a more controllable world action model. Project page: https://ctrl-wam.github.io/

1 INTRODUCTION

CTRLWAM targets the mismatch between perturbed action predictions and videos that still depict the recorded future. It pairs perturbed actions with simulated visual consequences and adds warped noise schedules and multi-agent conditioning.

Motivation

  • Standard training corrupts recorded actions and video independently, leaving the video tied to the recorded future even when the perturbed action implies a different trajectory.The model receives no explicit supervision on physical correspondence between intermediate action and video states.Source: §1 · p. 2

Physically aligned noising

  • CTRLWAM executes perturbed actions in a simulator, renders their visual consequences, and trains on those paired inputs against the recorded actions and video.The paper uses RoboTwin for manipulation and AlpaSim for driving.Source: §1 · p. 2

Warped video–action noise schedules

  • Warped noise schedules assign video higher noise than actions to keep visual layout responsive as action predictions evolve.The intended progression is to preserve layout flexibility, then refine visual detail as actions approach convergence.Source: §1 · p. 2

Structured multi-agent conditioning

  • Structured multi-agent conditioning adds a variable number of agent streams, with masks specifying which futures are predicted and which are supplied as commands.It supports joint multi-agent prediction and action-conditioned generation, and reduces to ego-only control when no other agents are present.Source: §1 · p. 3

Evaluation

  • The evaluation covers driving action forecasting, video–action consistency, and command adherence, plus manipulation motion fidelity and video quality.Source: §1 · p. 3

2 RELATED WORK

Related work covers unified models that jointly represent video and action, as well as controllable world models that condition visual futures on commands, actions, or scene structure.

Unified World Action Models

  • Unified World Models and Unified Video Action couple video and action within shared generative representations, supporting policy learning and forward or inverse dynamics through conditioning.VideoVLA and DreamZero adapt pretrained video generators for joint prediction of actions and visual futures in robotic control.Source: §2 · p. 3

Controllable World Models

  • Controllable world-model work spans action-conditioned video for robotics planning and latent actions for interactive generation from unlabeled videos.Source: §2 · p. 3
  • Driving-focused controllable models use video, text, and action inputs, structured traffic constraints, or controls ranging from high-level commands to trajectories and vehicle parameters.Source: §2 · p. 3

3 CTRLWAM

CTRLWAM pairs perturbed actions with rendered consequences while retaining the recorded future as the target. It also adapts video noise separately from action noise and represents surrounding agents as prediction targets or commanded futures.

3.1 PHYSICALLY ALIGNED NOISING

  • Aligned noising pairs a perturbed action with a corrupted view of its simulated consequences rather than a noised view of the original future.The model is trained to recover the recorded future, correcting both behavioral and visual discrepancies between input and target.Source: §3.1 · pp. 3–4

3.2 WARPED VIDEO–ACTION NOISE SCHEDULES

  • A shared noise schedule can let video commit to a trajectory before action predictions settle, creating disagreement between video-implied and predicted actions.Source: §3.2 · p. 4
  • The warped schedule assigns video higher noise than actions to leave layout open to revision, then lets it refine detail as actions approach convergence.The video and action noise levels remain deterministically coupled; setting ς = 1 recovers the shared schedule.Source: §3.2 · pp. 4–5

3.3 STRUCTURED MULTI-AGENT CONTROL

  • Additional agent streams support joint prediction of ego and other-agent futures, or video generation conditioned on clean commanded agent tokens.Predicted agent futures provide estimates of intent that can inform ego-action prediction.Source: §3.3 · p. 5, Figure 4 · p. 5

4 EXPERIMENTS

CTRLWAM is evaluated on driving and bimanual-manipulation tasks, with measures of action forecasting, video–action consistency, command following, and visual motion fidelity. Results cover the main comparisons and ablations of render source and noise scheduling.

4.1 DATASETS AND EVALUATION METRICS

  • Driving uses 33,988 aligned training windows and evaluates on 1,156 clips; robotics evaluates 1,000 held-out episodes across six WorldArena dimensions.Source: §4.1 · p. 6

4.2 IMPLEMENTATION DETAILS

  • Driving experiments train the video and action transformer on Cosmos 3-Nano with its language pathway frozen; robotics generation proceeds autoregressively in 32-frame chunks.Source: §4.2 · p. 6

4.3.1 MAIN DRIVING COMPARISON

4.3.2 MULTI-AGENT CONTROLLABILITY

4.4 ROBOTICS RESULTS

  • On 1,000 RoboTwin episodes, CTRLWAM leads all six WorldArena categories, scoring EWMScore 61.75 versus 58.70 for SFT and 58.65 for GT-Src.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 ABLATIONS

  • Rendering and Cleaned inputs favor warped schedules for action errors, while shared schedules favor visual reconstruction; Cleaned reaches 21.23 dB PSNR.Source: §4.5 · pp. 8–9, Table 4 · p. 9
  • On Rendering inputs, increasing ς from 1 to 5 lowers Self ADE from 0.916 to 0.798 m, though ς = 2 slightly worsens consistency.Source: §4.5 · p. 9, Table 5 · p. 9

5 CONCLUSION

CTRLWAM pairs perturbed actions with simulated visual consequences, and its conclusion reports gains in driving and robotics that support physically aligned denoising as a route to more controllable world action models. The appendix describes how paired training states are constructed and supervised.

5 CONCLUSION

  • Driving experiments show gains in command following, video–action consistency, and action forecasting, while robotics experiments show stronger motion fidelity.The conclusion presents these findings as support for physically aligned denoising and improved controllability.Source: §5 · p. 9

A.1 PAIRED STATES AND CORRECTIVE TARGETS

  • Video and actions receive independent Gaussian noise draws, while history tokens remain clean and are excluded from the loss.With σ = 0 denoting clean data, standard flow matching constructs the noised state from the clean data and Gaussian noise.Source: §A.1 · p. 13
  • AlpaSim executes noised commands at predefined noise levels, pairing the executed commands and rendered observations with recorded actions and video as clean targets.Real history remains the conditioning prefix; rejected renders mean not every training window has every perturbation level.Source: §A.1 · p. 13
  • The executed noised action is used without additional action noise, while fresh noise is added to rendered video using a shared or warped schedule.Shared scheduling sets σv = σa; warped scheduling uses Equation 1.Source: §A.1 · p. 13

A.2 LOSS WEIGHTING AND RECORDED-DATA ANCHORS

This appendix describes endpoint-based loss weighting, recorded-data anchors, and sampling procedures, including how supplied commands remain fixed during generation.

A.2 LOSS WEIGHTING AND RECORDED-DATA ANCHORS

  • Multiplying velocity MSE by the modality’s squared noise level yields endpoint regression against the clean data.Predictions and targets are scaled in fp32, removing explicit low-noise amplification of endpoint error without implying equal empirical gradients across noise levels.Source: §A.2 · p. 14
  • Each training window contributes one recorded video–action anchor trained with the original unweighted flow-matching objective and fresh Gaussian draws.Size-proportional sampling gives approximately 20% anchors on four levels and 11% on eight levels.Source: §A.2 · p. 14

A.3 SAMPLING

  • Generation uses UniPC in data-prediction form, converting network velocity to a clean estimate for multistep predictor–corrector updates.Warped checkpoints use distinct, deterministically paired video and action noise grids, with independent solvers and multistep histories for each modality.Source: §A.3 · p. 14
  • Supplied future action tokens remain clean fixed points of the solver, so only video tokens are integrated during command-conditioned generation.Commands enter at σ = 0 with zero velocity and condition the network at every step.Source: §A.3 · p. 14

B.1 DATA AND ALIGNED EXPORTS

The appendix documents driving data exports and checks, the model’s packed video and action inputs, the unicycle action representation, and supervised initialization.

B.1 DATA AND ALIGNED EXPORTS

  • Export sweeps cover 8,924 scenes and 33,988 windows, and rejected renders mean not every training window has every perturbation level.Validation and test exports contain clean rollouts and paired recordings; Table 6 inventories training data.Source: §B.1 · p. 14, Table 6 · p. 14
  • Exports require complete camera streams, a uniform clock, timestamp pairing, finite poses and actions, and successful simulator decoding.Rollout-versus-dump disagreement must be below 5 cm, while action round-trip error must be below 0.5 m and 0.05 rad; Rendering uses simulator frames and Cleaned applies reference-conditioned Difix3D+.Source: §B.1 · pp. 14–15

B.2.1 PACKED INPUTS.

  • Each input packs nine history and 32 future frames with 14 history action rows and 64 future action rows; forward dynamics additionally conditions future actions.The video spans a 6.4 s future horizon at 5 Hz, and history is conditioned throughout.Source: §B.2.1 · p. 15

B.2.2 UNICYCLE ACTIONS.

  • Each action row stores acceleration and curvature, the two channels used in noise, loss, and sampling; decoding integrates from the recorded initial state.Counterfactual paths use the same encoding and decoding operators.Source: §B.2.2 · p. 15

B.2.3 SUPERVISED INITIALIZATION.

  • Video and action experts are initialized from Cosmos 3-Nano and trained with a fixed Wan 2.2 VAE; the step-20,000 checkpoint initializes driving post-training.Pretraining mixes joint, forward, and inverse tasks with probabilities 0.4/0.4/0.2.Source: §B.2.3 · p. 15

B.3.1 EVALUATION.

The appendix defines the driving evaluation metrics and counterfactual sample, then specifies optimization and schedule-comparison settings.

B.3.1 EVALUATION.

  • The standard benchmark contains 1,156 clips and 1,434 segments, with Self ADE, Rollout ADE, and command-following metrics defined against video-inferred, recorded, or supplied trajectories.The 100-clip counterfactual evaluation is randomly selected from the held-out evaluation set.Source: §B.3.1 · p. 15

B.3.2 OPTIMIZATION.

  • Main driving ablations use the same pretrained checkpoint, a fresh optimizer, 10,000 steps, 16 H100 GPUs, global batch 16, and peak learning rate 10−5.Full ego-only and multi-agent training instead uses the matched eight-node, batch-64 recipe.Source: §B.3.2 · p. 15

B.3.3 SCHEDULE-COMPARISON SETTINGS.

  • Table 4 compares render sources under different schedules with ς = 5 for warped rows, while Table 5 varies ς ∈{1, 2, 3, 5} on Rendering sources.Table 5 holds initialization, step count, batch size, and learning rate fixed.Source: §B.3.3 · p. 16

C COUNTERFACTUAL EGO-CONTROL PROTOCOL AND EXAMPLES

The counterfactual ego-control protocol compares generated futures under edited commands with the recording, while the accompanying training and generation setup specifies the manipulation stages and rollout configuration.

C COUNTERFACTUAL EGO-CONTROL PROTOCOL AND EXAMPLES

  • Figure 6 holds history frames, text, and sampling noise fixed across command variants and uses the training-time unicycle encoding to compare Cmd ADE with supplied commands.The evaluation samples 100 held-out clips.Source: C COUNTERFACTUAL EGO-CONTROL PROTOCOL… · p. 16
  • Figure 9 compares recorded scenes with factual, stop, accelerate, nudge-left, and nudge-right generations at matching times over 0.0–6.4 s, alongside ego trajectories.The examples cover a highway and an urban intersection.Source: Figure 9 · p. 16, C COUNTERFACTUAL EGO-CONTROL PROTOCOL… · p. 16
  • Counterfactual commands replace the recorded ego trajectory with braking, acceleration, or lateral nudges, with low, mid, and high settings defining paired severity values.Braking strengths are 1/2/3 m/s2, acceleration strengths 0.75/1.5/3 m/s2, and lateral offsets 2/3.5/5 m.Source: C COUNTERFACTUAL EGO-CONTROL PROTOCOL… · p. 16

D.1 TRAINING AND GENERATION

  • The manipulation comparison uses 14-dimensional joint positions in 32-step chunks and continues a shared SFT initialization through SFT, GT-Src, and aligned post-training stages.Each aligned stage uses 59,696 simulator-derived samples and a 6:2 aligned-to-anchor mixture.Source: §D.1 · p. 17
  • Aligned samples pair executed noised commands with simulator frames, while fresh Gaussian noise is added to video latents and endpoint supervision is applied to both modalities.Generation uses 30 UniPC steps; the last frame conditions the next chunk until the recorded episode length is reached.Source: §D.1 · p. 17

D.2 WORLDARENA EVALUATION

The WorldArena evaluation consolidates normalized visual, consistency, and motion metrics, while the accompanying agent interface packs selected surrounding actors with the ego stream.

D.2 WORLDARENA EVALUATION

  • EWMScore averages 15 normalized WorldArena metrics, and Table 7 consolidates the individual metric scores for the compared models.Evaluation uses 1,000 held-out RoboTwin episodes and the WorldArena harness.Source: §D.2 · p. 17, Table 7 · p. 17
  • Individual metrics do not uniformly favor aligned training: continued SFT scores higher on image and aesthetic quality, and lower-resolution oracle videos make oracle scores calibration references rather than strict upper bounds.The table caption identifies Oracle JEPA and depth as identity-comparison checks and notes that photometric consistency can favor smoother-than-real motion.Source: Table 7 · p. 17, §D.2 · p. 18

E.1 REPRESENTATION AND SELECTION

  • Each scene’s interface contains an ego block and up to eight selected surrounding-agent blocks, with histories as context and futures as prediction targets or supplied commands.With K = 0, the layout reduces to the ego-only model.Source: §E.1 · p. 18
  • Agent selection excludes static actors and retains up to eight nearest eligible actors that project into the front image within 60 m and meet visibility and occlusion criteria.Eligible actors must be at least 12 px tall at 832 × 480; tracks must also pass state, control-feasibility, and encode–decode checks.Source: §E.1 · p. 18

E.2 MATCHED TRAINING AND EVALUATION

The matched comparison controls initialization and training settings while testing whether the agent interface supports commanded actor futures; command compliance is assessed with a mean judge score.

E.2 MATCHED TRAINING AND EVALUATION

  • CTRLWAM-S and CTRLWAM-M share initialization, data, aligned-noising recipe, duration, optimizer, schedule, batch size, and evaluation protocol, differing only in the additional agent interface.The matched SFT baseline continues finetuning pretrained checkpoints with standard noising.Source: §E.2 · p. 18
  • Agent ADE/FDE compares generated trajectories with recordings for selected actors, but agent prediction errors are not evaluated in forward dynamics because no IDM estimates their trajectories from video.Under selection, 4,084 actors in 1,197 of the 1,434 benchmark segments enter the trajectory metrics.Source: §E.2 · p. 18
  • For counterfactual actor control, scene history, ego command, and sampling noise stay fixed while the selected actor’s future changes; only CTRLWAM-M receives the edited agent command.Other models’ scores measure agreement with the requested behavior without conditioning on that command.Source: §E.2 · p. 18
  • The reported compliance percentage is the mean Gemini score across examples, with 100 clips and four counterfactual command types per clip, not a thresholded success rate.Gemini scores generated video and commanded trajectories on a 0-to-1 scale.Source: §E.2 · p. 18

F DISCUSSION AND LIMITATIONS

The discussion identifies dependence on evaluation models, while the stated limitations bound the simulator-based pipeline and the multi-agent robotics setting.

F DISCUSSION AND LIMITATIONS

  • The experiments use offline open-loop metrics, and most robotics metrics rely on external models, leaving evaluation dependent on those models and exposed to model bias.Self ADE, Rollout ADE, and Cmd ADE/FDE use video–action comparisons or an inverse dynamics model.Source: F DISCUSSION AND LIMITATIONS · p. 19
  • Aligned noising requires a simulator to execute perturbed actions and render their consequences; the multi-agent extension is limited to driving for lack of suitable robotics action–observation datasets.The paper presents real–sim–real workflows as a future direction for robotics multi-agent prediction and control.Source: F DISCUSSION AND LIMITATIONS · p. 19

Related papers

Loading 2610.00859v1…