PaperlayerView exact revision on arXiv

Source-linked AI summary

OneLA: Learning Many Worlds from Latent Actions

Ziyu Zhu, Zhaoxu Sun, Fan Jiang, Chiyu Wang, Junfeng Ni, Mu Xu

arXiv:2610.20160v1cs.CV

TL;DR

World models across domains need a shared control space, but action interfaces differ and many videos lack command annotations. OneLA learns continuous latent actions from video and maps native controls into the same space for context-conditioned prediction. It improves action readout across domains and reduces action-cycle rotation error by 24.8% over the strongest evaluated baseline.

  • Problem

    World models across diverse domains need a shared action space, but control interfaces vary and much video data lacks command annotations.

    Source: §1 · p. 1

  • Method

    OneLA learns continuous latent actions from unlabeled videos with a shared orthogonal dictionary and predictive objective, then maps native controls into that space for world-model conditioning.

    Source: §1 · p. 2

  • Results

    OneLA improves action readout across domains and reduces action-cycle rotation error by 24.8% over the strongest evaluated baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results suggest a common action representation can connect diverse control interfaces without requiring separate world-model backbones.

    Source: §5 · p. 10

Abstract

from arXiv · show

Learning world models across diverse domains calls for a unified control space, yet ac- tion interfaces vary across domains and action annotations are often scarce. Latent ac- tions learned directly from videos offer a promising way to establish such a space without explicit action supervision. However, learning shared latent actions across domains re- mains challenging, as variations in appearance, embodiment, and dynamics can entangle domain-specific visual cues with transferable transition structure. To this end, we intro- duce OneLA, a simple framework that learns continuous latent actions through a shared orthogonal dictionary and a common predictive objective, without task-specific designs or cross-domain alignment losses during latent action pretraining. These latent actions pro- vide a shared conditioning interface for world models, with lightweight adapters mapping native control commands into the same space. Conditioned on latent actions and the cur- rent visual state, the world model predicts context-dependent transitions for controllable rollout. OneLA improves action readout across domains and reduces action cycle-transfer rotation error by 24.8% over the strongest evaluated baseline. A joint-training comparison further shows a 20.5% improvement in Human Preference Score v3 (HPSv3) for real- scene generation. These results suggest a path toward general-purpose world models that learn from diverse videos and support flexible control through latent action space.

1 Introduction

OneLA targets a shared control space for world models trained across domains, including when video lacks command labels. It learns latent actions from video, maps native controls into that space, and reports improved action readout and action-cycle transfer.

1 Introduction

  • World models need a shared action space to connect heterogeneous controls, but much video data lacks command annotations.Video-derived latents describe realized transitions, while command-derived latents specify controls whose effects depend on visual context.Source: §1 · pp. 1–2
  • OneLA learns continuous latent actions from unlabeled videos with an orthonormal motion dictionary, KL regularization, sparsity, and next-frame reconstruction.The frozen latent action model is followed by MLPs mapping native controls into its space and a shared diffusion transformer predicting video and realized-transition latent actions.Source: §1 · p. 2
  • OneLA reduces action-cycle translation and rotation errors by 15.5% and 24.8% relative to OLaF, respectively, while leading evaluated methods on nuScenes PSNR and SSIM.The evaluation spans six datasets across four domains.Source: §1 · p. 2
  • OneLA’s contributions include unified training on labeled and unlabeled videos, combining native-control alignment with joint video and latent-action prediction.The framework is evaluated across diverse domains for action readout, cross-view transfer, action-cycle consistency, and visual quality.Source: §1 · p. 2

2 Related Work

Prior latent-action work uses varied signals and regularizers to structure visual transitions, while world models must also reconcile different control interfaces. OneLA combines a geometrically regularized action space with joint multi-domain video and action prediction.

2.1 Latent Actions from Video

  • OneLA uses an orthonormal dictionary to give KL and sparsity regularization a consistent geometric scale, learning continuous actions with a common reconstruction objective and no additional supervision or alignment targets during pretraining.Prior approaches use signals such as agent masks, optical-flow targets, action supervision, or content–structure separation.Source: §2.1 · p. 3

2.2 World Models

  • OneLA couples a geometrically regularized action space with joint multi-domain video and action prediction, supporting native-command control and reference-action transfer through one shared backbone.Latent actions connect unlabeled-video pretraining to downstream commands and video-derived motion to imitation and feature-space dynamics.Source: §2.2 · p. 3

3 OneLA

OneLA learns continuous latent actions from frame-pair transitions across diverse video sources, then maps native controls into the same space. A shared world model uses either inferred actions or mapped controls to predict video and realized transitions.

3.1 Formulation

  • OneLA represents frame-pair transitions with continuous latent actions, aligns native controls to that space, and conditions a shared world model on either input.Source: §3.1 · p. 4

3.2 Latent Action Pretraining Data

  • The pretraining collection contains 1.79M clips spanning games, navigation, driving, egocentric interactions, and robot manipulation, with frame pairs sampled without action labels.Source: §3.2 · p. 4

3.3 Orthogonal Latent Action Learning

  • A shared orthonormal dictionary embeds transition coefficients for reconstruction; QR fixes their geometric scale, while KL and sparsity regularization constrain the coefficients.Source: §3.3 · pp. 4–5

3.4 World Model Training

  • World-model training conditions video generation on an initial frame and latent actions, while jointly predicting video and realized-transition actions from frozen-LAM targets.Source: §3.4 · pp. 5–6

4 Experiments

OneLA is evaluated across four domains using frozen action representations, video prediction, action-cycle transfer, ablations, and qualitative control examples. Results include stronger action readout and transfer, while ablations test prototype alignment, latent-action conditioning, model and data scaling, and training-data composition.

4.1 Experimental Setting

  • Evaluations span six datasets across four domains; representation tests include AgiBot and held-out Game data, while video prediction covers MIND, nuScenes, and RT-1.Prediction uses the initial frame and MLP-mapped native commands without target-frame latent conditions; cycle transfer instead extracts latent actions from the source video.Source: §4.1 · pp. 6–7

4.2 Main Results

  • OneLA achieves the best F1 on most datasets and second-best on the rest; it also lowers action-cycle translation and rotation RPE by 15.5% and 24.8% versus OLaF.MIND cross-view F1 is 78.31 and 72.88, exceeding the best baselines by 10.66 and 5.53 points without target-view labels; OneLA leads nuScenes PSNR/SSIM and ranks second on all three RT-1 fidelity metrics.Source: Table 2 · p. 7, §4.2 · p. 7, Table 3 · p. 8

4.3 Ablation Study

  • With QR, same-action prototype similarity rises across domains, and the same-versus-different action similarity gap increases from 0.715 to 0.836 across MIND views and 0.140 to 0.181 across robots.Figure 4 compares action-conditioned posterior means across MIND viewpoints and AgiBot/RT-1 robots; rows and columns denote action classes.Source: §4.3 · p. 7, Figure 4 · p. 8
  • Using LAM-inferred rather than zero latent actions for unlabeled videos raises real-scene HPSv3 from 4.94 to 5.95, improves MIND rotation RPE from 0.665 to 0.553, and raises nuScenes PSNR from 13.65 to 16.13.Adding latent-action prediction loss also reduces human-verified collisions from 36% (18/50) to 22% (11/50).Source: §4.3 · p. 8, Figure 5 · p. 9
  • Increasing encoder and decoder stacks from 6 to 12 layers improves Game, nuScenes, and RT-1 F1 by 6.44, 10.30, and 1.35 points; the 24-layer model scores highest among reported models.Data scaling improves all three datasets, peaking at 2× for Game and 3× for nuScenes and RT-1.Source: §4.3 · p. 8, Table 4 · p. 9
  • Expanding game-only training to the full mixture improves F1 on all six datasets, including Game (58.41 to 66.34) and AgiBot (39.18 to 54.42), while Sekai falls from 55.60 to 53.87 in the robot-inclusive setting.The Sekai decrease is described as consistent with a possible trade-off in domain-specific readout.Source: §4.3 · pp. 8–9, Table 5 · p. 9

4.4 Qualitative Results

  • Transferred latent actions preserve camera or character motion across scene changes, and keyboard commands drive navigation in real scenes.Figures 6 and 7 illustrate these transfer and control examples.Source: §4.4 · p. 9

5 Conclusion

OneLA connects control interfaces across games, navigation, driving, and robotics through a shared latent action space. The conclusion reports strong action readout, cross-view transfer, and action-cycle consistency, while identifying longer-horizon planning and adaptation to new environments as future work.

5 Conclusion

  • OneLA structures motion learned from unlabeled videos with an orthonormal dictionary and variational and sparsity regularization.Lightweight command mappings connect native controls to the shared space for unified video and action prediction from labeled and unlabeled data.Source: §5 · pp. 9–10
  • Figure 6 matches relative frame indices between source and transfer rollouts, while Figure 7 shows input frames followed by five generated frames across four domains.Source: Figure 6 · p. 10, Figure 7 · p. 10
  • Experiments show strong action readout, cross-view transfer, and action-cycle consistency.The conclusion also says ablations support using latent actions to learn from unlabeled videos.Source: §5 · p. 10
  • The conclusion suggests a common action representation can connect diverse control interfaces without separate world-model backbones.Source: §5 · p. 10
  • Future work will extend the interface to longer-horizon planning and closed-loop interaction, and test adaptation to new environments and embodiments.Source: §5 · p. 10

A Implementation and Reproducibility Details

The appendix documents the video sources, latent-action model, and training configurations used for reproducibility. It specifies source-specific subsampling, model and data comparisons, and frame-pair sampling.

A Implementation and Reproducibility Details

A.1 Latent Action Pretraining Video Composition

  • The pretraining collection contains 1,793,552 clips grouped into eight video categories spanning egocentric, driving, object-centric, scene-exploration, robot-manipulation, and handheld sources.Examples include Ego4D and EgoDex, NVIDIA driving and nuScenes, SSv2 and WildRGB-D, Sekai and RealEstate10K, RT-1 and AgiBot, and FastUMI.Source: §A.1 · p. 18
  • Source-specific subsampling retains 20% of Spatial-Vid, 50% of cinematic scenes, and 5% of daily activities; other configured manifests use all records.Retained counts are max(1, floor(rN)); reported counts are clip records after subsampling, and shares use the total retained count as denominator.Source: §A.1 · p. 18, Table 6 · p. 18

A.2 Latent Action Model

  • The 32-dimensional LAM uses 24 encoder and decoder blocks, width 1024, 16 attention heads, and 16-pixel patches.Its loss uses KL coefficient β = 10^-6 and sparsity coefficient λ = 0.01; extraction uses the posterior mean, while training samples from the posterior.Source: §A.2 · p. 19
  • The default LAM schedule uses up to 50,000 updates over a 6.4M-pair sampling budget, with 16 GPUs and global batch 640.Training uses AdamW, a cosine learning-rate schedule from 5 × 10^-5 to 5 × 10^-6, and gradient clipping at 0.3.Source: §A.2 · p. 19
  • Frame pairs are usually sampled at 16 FPS with offsets {0, 1, 2, 4, 8, 16}, with source-specific rates and offsets for RT-1, SSv2, nuScenes, NVIDIA driving, and daily activities.Zero-offset pairs provide static transitions; most sources advance pair starts by two sampled frames, while RT-1 uses stride one.Source: §A.2 · p. 19
  • The appendix compares three LAM sizes and three data amounts, while composition comparisons evaluate four mixtures with the same six-dataset linear-probe protocol.The size comparison varies encoder/decoder depth from 6/6 to 24/24; data amounts use 10,000, 20,000, or 30,000 updates with a fixed source mixture.Source: §A.2 · p. 19

A.3 World-Model Training

The world-model appendix describes training data and conditioning, video-control alignment, and frozen-representation evaluation. It specifies how native commands and video-derived actions are used, and how representation probes are fit and reported.

A.3 World-Model Training

  • World-model training draws from 11 sources, with native annotations and command MLPs for ABot-World, AgiBot, nuScenes, and RT-1; other sources use frame-pair-derived latent actions.Across 6,000 updates with batch size 16, approximately 12,873 samples (13.41%) use native controls; Table 7 reports expected sample counts before action dropout.Source: §A.3 · p. 19, Table 7 · p. 20
  • The video model shares pretrained Wan2.2-TI2V-5B backbone weights across domains, freezes the LAM, and keeps the command MLPs trainable.Its joint objective combines video flow matching with latent-action prediction, weighted 0.05, and cosine loss, weighted 0.1.Source: §A.3 · p. 20
  • Before joint training, command MLPs align native controls to frozen-LAM posterior means; an auxiliary head predicts four consecutive 32-dimensional actions from adjacent temporal token groups.Prealignment uses SmoothL1 loss plus cosine distance; the auxiliary loss combines SmoothL1 regression and cosine distance, with total action-loss weight λ_act = 0.05.Source: §A.3 · p. 20

A.4 Video and Control Alignment

  • Pack1 concatenates four consecutive 32-dimensional frame-pair actions into a 128-dimensional condition for each stride-four video-latent step, preserving transition order.It encodes consecutive pairs rather than only the endpoints four frames apart; the initial frame is excluded from fidelity scores.Source: §A.4 · p. 21

B Probe Labels and Statistical Reporting

  • Table 8 summarizes frame-pair counts and action labels; binary labels are predicted independently, and Sekai macro-F1 averages labels with positive test examples.Sekai I/K have no positive examples; reporting retains simultaneous binary controls as multilabel and uses each robot dataset’s native action convention.Source: B Probe Labels and Statistical Reporting · p. 21, Table 8 · p. 21, §B.1 · p. 22

B.1 Reported Frozen-Representation Protocol

  • Frozen-representation comparisons retain each method’s native representation dimension and use the same evaluation pairs and dataset-specific readout protocol.OneLA uses a 32-dimensional representation; comparison methods include OLaF, AdaWorld, DiLA, and RLA-WM, with DiLA at 256 dimensions and RLA-WM residual features at 2,048 dimensions.Source: §B.1 · p. 21
  • The primary readout is macro-F1 from a linear layer over the full representation, with shared within-dataset probe-fitting protocols and held-out test videos.The restricted readout selects one dimension per action by absolute correlation on fit labels, then fits a sign and threshold; it still uses labeled fit data.Source: §B.1 · pp. 21–22

C.1 Reported World-Model Comparison

The comparison uses shared checkpoints, fixed test cases, and specified video and trajectory metrics. It also compares latent-action training settings on real-scene preference and collision tests, with the preference metric measuring visual quality rather than control correctness.

C.1 Reported World-Model Comparison

  • All methods are evaluated after 6,000 fine-tuning updates, using the same fixed test cases, 50 denoising steps, guidance scale 3, and seed 42.OneLA uses a shared checkpoint with the corresponding keyboard, pose, or RT-1 command MLP; future target RGB frames are not used to extract action conditions.Source: §C.1 · p. 22
  • Video prediction is scored with PSNR, SSIM, and LPIPS, while visual quality uses IQ; IQ measures visual quality rather than action correctness.Visual metrics exclude the conditioning frame except IQ, which includes it; IQ uses VBench long-side-512 preprocessing.Source: §C.1 · p. 22
  • Trajectory evaluation uses ViPE pose estimates and Sim(3) alignment, reporting adjacent-frame translation and rotation errors separately for each benchmark.Cycle evaluation compares the returned video with its source at 16 FPS and uses the source video’s actual length.Source: §C.1 · p. 22
  • Collision rates are 36% (18/50) without the latent-action prediction loss and 22% (11/50) with it.The separate 50-case collision comparison uses human inspection; the real-scene HPSv3 comparison measures visual preference, not control correctness.Source: §C.1 · pp. 22–23
  • Qualitative control examples show an initial frame and five generated frames, with each label marking the action arriving at its displayed frame.Transfer examples use the same three frame indices for source and transfer videos, and the first transfer frame is the generation input.Source: §C.1 · p. 23

D Orthogonal Parameterization: Scope of the Constraint

The orthogonal parameterization preserves distances between latent coefficients, but this constraint applies to the embedding map rather than the generated transition. Rotational ambiguity remains, so the learned coordinates are not identified as a unique physical basis.

D Orthogonal Parameterization: Scope of the Constraint

E Additional Qualitative Results

The appendix presents additional qualitative OneLA rollouts across scene control and several domains, alongside reference-action transfer examples and enlarged cross-domain similarity matrices.

E.1 Scene Control

E.2 Rollouts across Domains

  • Additional rollouts cover first- and third-person games, driving, navigation, and robot interactions.The examples include vehicle following, open roads, outdoor and indoor camera motion, and interactions with a drawer, can, snack bag, and sponge.Source: Figure 10 · p. 25, Figure 11 · p. 25, Figure 12 · p. 26

E.3 Reference-Action Transfer

F Enlarged Cross-Domain Similarity Matrices

Related papers

Loading 2610.20160v1…