PaperlayerView exact revision on arXiv

Source-linked AI summary

Olaf-World: Orienting Latent Actions for Video World Modeling

Yuxin Jiang, Yuchao Gu, Ivor W. Tsang, Mike Zheng Shou

arXiv:2602.10104v2cs.CVcs.AIcs.LG

TL;DR

Action-controllable world models remain difficult to scale from unlabeled video because latent actions can entangle visual context and lack a shared coordinate system. Olaf-World introduces Seq∆-REPA to align latent actions with observable semantic video changes, improving zero-shot action transfer and data-efficient adaptation to new control spaces.

  • Problem

    Unsupervised latent action objectives lack a global action basis, allowing context-entangled representations that transfer poorly across videos.

  • Method

    Seq∆-REPA anchors latent action trajectories to semantic feature changes from a frozen self-supervised video encoder, then uses them to train an action-conditioned world model.

  • Results

    Olaf-World learns more linearly decodable and context-invariant latents, outperforming AdaWorld under cross-domain evaluation while improving zero-shot transfer and data-efficient adaptation.

  • Takeaways & Limitations

    Effect-aligned latent actions provide a consistent control interface for transferring actions across contexts and adapting to new control spaces with limited data.

  • Takeaways & Limitations

    Inverse-dynamics objectives alone do not identify a global action basis, so context-dependent cues can produce non-transferable latent actions.

Abstract

from arXiv · show

Scaling action-controllable world models is limited by the scarcity of action labels. While latent action learning promises to extract control interfaces from unlabeled video, learned latents often fail to transfer across contexts: they entangle scene-specific cues and lack a shared coordinate system. This occurs because standard objectives operate only within each clip, providing no mechanism to align action semantics across contexts. Our key insight is that although actions are unobserved, their semantic effects are observable and can serve as a shared reference. We introduce Seq$Δ$-REPA, a sequence-level control-effect alignment objective that anchors integrated latent action to temporal feature differences from a frozen, self-supervised video encoder. Building on this, we present Olaf-World, a pipeline that pretrains action-conditioned video world models from large-scale passive video. Extensive experiments demonstrate that our method learns a more structured latent action space, leading to stronger zero-shot action transfer and more data-efficient adaptation to new control interfaces than state-of-the-art baselines.

1. Introduction

The introduction identifies cross-context non-identifiability and shortcut learning as barriers to transferable latent actions. It proposes Seq∆-REPA and Olaf-World to align latent actions with observable semantic effects and pretrain action-conditioned world models from passive video.

  • Motivation: Latent action learning discovers action spaces from unlabeled video, but transfer requires the same underlying action to produce similar latent actions across changing visual contexts.Inverse-dynamics encoders infer latent actions from observed transitions, while forward models predict future frames conditioned on those actions.
  • Challenges: Two failure modes undermine transfer: shortcut learning entangles latent actions with scene-specific visual cues, while local reconstruction objectives are non-identifiable across contexts.The resulting latent coordinates need not share a common basis across clips, even when transitions reconstruct successfully.
  • Seq∆-REPA: Seq∆-REPA aligns integrated latent actions with semantic control-effect directions from a frozen, self-supervised video encoder.Observable semantic effects provide a shared reference despite unavailable explicit action labels, encouraging consistent action meanings across contexts.
  • Olaf-World: Olaf-World pretrains action-conditioned video world models on large-scale passive video using latent actions learned with Seq∆-REPA as a consistent control interface.The introduction highlights context-invariant zero-shot action transfer and efficient adaptation with minimal labeled data as downstream capabilities.

2. Related Work

Related work spans latent action models for world-model control, robotics policy learning, and observation-only offline RL, alongside action-controllable world models and encoder-alignment methods. The paper addresses the consistency of latent action spaces across contexts, extending temporal alignment ideas from video generation.

  • Latent action models: Latent action models infer controls from unlabeled video for interactive world-model interfaces, cross-embodiment robotics policies, and observation-only offline RL.These applications include unified control interfaces, robotics transfer across embodiments, and offline reinforcement learning without action labels.
  • Video world models: Action-controllable video world models commonly use explicit frame-level controls logged from interactive game engines, whereas latent-action models infer control interfaces directly from video.Examples of explicit signals include keyboard and mouse inputs and other interaction annotations from environments such as Unreal Engine and Minecraft.
  • Video world models: Latent-action controllability and transfer depend on whether the learned action space remains consistent across contexts.This cross-context consistency is identified as the bottleneck addressed by the paper.
  • Feature alignment: Alignment methods match generative-model features to large self-supervised encoders, progressing from spatial image features to temporal structure in video generators.Recent video methods align internal states of video generators with pretrained video encoders.

3. Method

Olaf-World learns a transferable latent action space before using it as a unified control interface for action-conditioned video world-model pretraining. SeqΔ-REPA aligns integrated latent actions with context-comparable temporal effects, enabling consistent action semantics across videos.

  • Overview: Olaf-World has two stages: learning a transferable latent action space that disentangles dynamics from visual context, then training a latent-action-conditioned video world model.The pipeline applies the frozen latent action model to unlabeled videos to extract latent-action sequences for world-model pretraining.
  • Latent Action Model: The standard step-wise β-VAE can achieve low one-step prediction error without ensuring semantic consistency across contexts.Context leakage can encode cues correlated with the next frame, while unconstrained latent coordinates may drift across trajectories and break transfer.
  • SeqΔ-REPA: SeqΔ-REPA anchors integrated latent actions to a comparable effect direction computed from temporal differences in a frozen self-supervised video encoder.Spatially pooled per-frame descriptors define the clip effect direction, making the signal emphasize coherent temporal change and reduce sensitivity to static appearance.
  • SeqΔ-REPA: Cosine alignment between integrated control and effect directions imposes a control-to-effect constraint that encourages consistent action meaning across contexts.The reference encoder remains frozen, and the projection from latent-action space to feature space is trainable.
  • World-Model Pretraining and Adaptation: Action-aware pretraining projects per-frame latent actions into diffusion timestep embeddings, while target-environment adaptation learns an action adapter and small backbone LoRA.For discrete actions, the adapter can use an embedding table initialized with class-wise prototypes inferred by the frozen latent action model.

4. Experiments

Experiments evaluate whether Seq∆-REPA improves latent-action structure, zero-shot transfer, and data-efficient adaptation. Across probing, transfer, adaptation, and ablation studies, Olaf-World shows stronger cross-context action alignment, controllability, and robustness.

  • Experimental setup: Experiments assess latent-action structure, zero-shot control transfer, and data-efficient adaptation through three research questions.The evaluation uses MiraData for pretraining and MIND’s disjoint 1ST-P and 3RD-P subsets, which share an 8-action label space.
  • RQ1: Structure: Seq∆-REPA improves in-domain linear decodability and consistently outperforms AdaWorld in cross-domain probing across both 1ST-P↔3RD-P directions.The largest gains occur on the more challenging 3RD-P subset, where AdaWorld saturates at low Macro-F1.
  • RQ1: Structure: Seq∆-REPA yields more diagonal-dominant cross-context prototype similarity, preserving high similarity for matching actions while separating non-matching pairs.AdaWorld instead shows high similarity across many action pairs, indicating weak cross-context identifiability.
  • RQ2: Transfer: Olaf-World transfers action sequences more reliably than AdaWorld while preserving scene and subject persistence under viewpoint and appearance shifts.AdaWorld exhibits temporal wash-out, scale drift, loss of controlled subjects, and trajectory drift toward generic motions.
  • RQ3: Adaptation: Olaf-World achieves the lowest RPE-trans and RPE-rot across all adaptation budgets on both 1ST-P and 3RD-P, with comparable video quality to AdaWorld.After 50-video adaptation, it follows multi-key controls more reliably and synthesizes newly visible regions with stable details matching the initial frame.
  • OOD robustness: Olaf-World retains the lowest RPE under unseen visual contexts, while qualitative results show stronger appearance consistency and action-consistent changes for unseen styles and objects.These results indicate improved out-of-distribution robustness of action-conditioned dynamics.

5. Conclusion · Appendix

The conclusion identifies cross-context non-identifiability as a central limitation of unsupervised latent action learning and presents Seq∆-REPA and Olaf-World as responses. It also outlines embodiment-bridging robotics applications and the supplementary materials’ supporting analyses, protocols, results, and future-work discussion.

  • 5. Conclusion: Unsupervised latent action learning suffers from cross-context non-identifiability, causing inverse-dynamics objectives to produce context-entangled latents.These latents transfer poorly because the objectives do not identify a global action basis.
  • 5. Conclusion: Seq∆-REPA anchors latent actions to action effects measured as feature differences from a self-supervised video encoder.Its sequence-level objective is designed to encourage context-invariant semantics.
  • 5. Conclusion: Olaf-World builds on these effect-aligned latents as a scalable latent-action world-modeling pipeline.The supplied conclusion passage introduces Olaf-World after presenting Seq∆-REPA.
  • 5. Conclusion: Effect-aligned latent actions could become transferable robotics skills through embodiment-specific action-to-skill adapters.The conclusion gives human→robot transfer as an example.
  • Appendix: The appendix provides a formal analysis of cross-context non-identifiability and implementation details.These materials appear in Sections A and B of the supplementary information.
  • Appendix: The supplementary materials also contain evaluation protocols, additional results, and limitations and future-work discussion.These appear in Sections C, D, and E, respectively.
  • Appendix: A project page supplies video visualizations for evaluating the temporal quality of the method’s outputs.The supplied appendix passage identifies the project page as https://showlab.github.io/Olaf-World.

A. Formal Analysis of Cross-Context Non-Identifiability

Standard local inverse-dynamics objectives cannot identify a shared latent-action coordinate system across contexts because context-dependent reparameterizations preserve transition predictions and training loss. Consequently, latent codes may fail to transfer as the same semantic action, while a β-VAE diagonal-Gaussian constraint only reduces the symmetry to context-varying signed permutations.

  • A.1. Setup: Local encoder-decoder training predicts transitions within each context but provides no shared latent-action coordinate system.The objective applies to transitions (x_t, x_t+1) indexed by context c and uses a reconstruction or prediction loss.
  • A.2. Proposition (context-dependent latent-coordinate symmetry): Any family of context-specific bijections can redefine the encoder and decoder while preserving the local prediction objective.The proposition allows one bijection G_c per context.
  • A.2. Proposition (context-dependent latent-coordinate symmetry): The transformed model produces identical predictions for every sample, so expectations over transitions and contexts leave the prediction loss unchanged.The proof establishes equality sample by sample before taking expectations.
  • A.3. Implication for cross-context transfer: Different contexts can therefore use different latent coordinate systems despite achieving the same training objective.This is the direct implication of Proposition A.1 for cross-context anchoring.
  • A.3. Implication for cross-context transfer: A code representing a desired semantic effect in one context need not transfer as the same action in another context.If z_A := G_cA(z⋆), applying z_A in context c_B is not generally equivalent because G_cB G_cA is approximately not the identity.
  • Remark (what changes with a β-VAE KL term?): A β-VAE isotropic prior does not fully resolve the issue when the posterior is restricted to factorized diagonal Gaussians.The diagonal-Gaussian family is not closed under arbitrary orthogonal rotations.
  • Remark (what changes with a β-VAE KL term?): Within that variational family, exact rotational symmetries reduce to signed permutations, which can still vary across contexts without an explicit cross-context constraint.The reduction holds up to degenerate isotropic cases.

B. Implementation Details · B.1. Latent Action Model · B.2. Olaf-World

The implementation uses a causal VAE-based latent action model with Transformer components and trains Olaf-World by injecting 32-dimensional latent actions into a SkyReels I2V DiT backbone. Training and adaptation use specified AdamW and LoRA configurations across H200 GPUs.

  • B.1. Latent Action Model: The latent action model uses a VAE-based video prediction framework with a causal spatio-temporal encoder and spatial-only decoder.Both components use Transformer architectures with 16 blocks, 1024-dimensional embeddings, and 16 attention heads.
  • B.1. Latent Action Model: Latent actions have dimension dz = 32, while training clips contain T=16 frames at 272×480 resolution.The encoder applies causal temporal masking to prevent leakage from future frames.
  • B.1. Latent Action Model: The latent action model is trained with AdamW at learning rate 2.5×10−5, weight decay 10−2, and total batch size 32 on 8×H200 GPUs.Training lasts 100 epochs (∼146k steps) and takes ∼4.5 days.
  • B.2. Olaf-World: Olaf-World uses the SkyReels I2V 1.3B DiT backbone and injects latent actions through a 32→1536 projection into the timestep embedding stream.A learned gain γ is initialized to 2.0; adaptation uses rank-16 LoRA on attention and feed-forward linear layers.
  • B.2. Olaf-World: The latent-action-conditioned video generator is pretrained for 10k steps with AdamW at learning rate 5×10−5.The provided training configuration specifies the optimizer and learning rate for pretraining.
  • B.2. Olaf-World: Pretraining uses weight decay 10−3 across 4×NVIDIA H200 GPUs with batch size 4 per device, while downstream adaptation fine-tunes only rank-r=16 LoRA parameters.Adaptation uses learning rate 1×10−4 and zero weight decay.

C. Evaluation Details … C.2. World model

The evaluation probes whether latent actions transfer across domains, remain action-consistent between viewpoints, and support visually coherent, controllable world-model generation under in-distribution and novel-scene conditions.

  • C.1.1. CROSS-CONTEXT LINEAR PROBING: A linear classifier is trained on frozen latent actions with focal loss and selected by highest in-domain validation Macro-F1.Training uses SGD with momentum 0.9, weight decay 10^-6, 12 epochs, and a StepLR schedule; focal-loss γ=2 addresses class imbalance.
  • C.1.1. CROSS-CONTEXT LINEAR PROBING: The selected probe is evaluated zero-shot on the other domain using Macro-F1.This cross-domain evaluation tests whether latent-action semantics transfer between domains without retraining the probe.
  • C.1.2. CROSS-CONTEXT ACTION CONSISTENCY: The 1ST-P and 3RD-P prototype matrices are ℓ2-normalized and compared with a cosine-similarity heatmap.The matrices have shape C×d_z with C=8 actions, enabling cross-view action correspondence analysis.
  • C.1.2. CROSS-CONTEXT ACTION CONSISTENCY: Latent-action interpolation and extrapolation vary α across {0, 0.5, 1, 1.5, 2} between no-operation and action prototypes.The same interpolated latent is repeated across action steps from one initial image, with generated frames shown at timesteps 20 and 40 for left, backward, and right actions.
  • C.2. World model: World-model visual quality is evaluated with selected VBench dimensions, including Imaging Quality and Temporal Consistency.These dimensions assess generated-video appearance and temporal coherence.
  • C.2. World model: Controllability is measured by relative pose error after reconstructing and Sim(3)-aligning camera trajectories from ground-truth and generated videos.For a fixed action sequence, ViPE estimates per-frame camera poses for both videos before trajectory comparison.
  • C.2. World model: An OOD novel-scene set contains 50 initial frames spanning photorealistic and stylized visual domains to test robustness under large appearance shifts.The distribution shift is motivated by MIND’s primarily near-photorealistic 3D game-rendering training data.

D. Additional Results

Additional results qualitatively support Olaf-World’s robustness, while latent-action interpolation indicates coherent local geometry. The reported failure cases expose limitations involving collision avoidance and large scene reveals.

  • Additional Qualitative Results: Qualitative examples cover zero-shot action-sequence transfer, data-efficient adaptation, and generalization to novel scenes, further supporting robustness against baselines.Dynamic video quality is difficult to convey through sparsely sampled frames, so corresponding videos are provided on the supplementary project page.
  • Latent Action Interpolation: Increasing α produces progressively stronger motion in the intended direction when the same interpolated latent action is repeated across steps.Values 0 < α < 1 interpolate between the NO-OP and target-action prototypes, while α > 1 extrapolates.
  • Failure Cases: Transferred actions that would cause collisions may trigger hallucinated scene changes that remove or alter obstacles while preserving the intended motion.The example drive-forward-then-turn-left illustrates this control–physics mismatch.
  • Failure Cases: Actions requiring large newly visible regions, such as zooming out, can produce degraded completion.These failures arise when the model must synthesize a large amount of newly visible content.

E. Limitations and Future Work

The section identifies promising future directions for strengthening transferable latent actions and action-conditioned world modeling.

  • Future Work: Future work targets stronger transferable latent actions.The paper frames these directions as promising ways to further improve transferability.
  • Future Work: Future work also targets action-conditioned world modeling.The stated research directions are intended to advance this modeling paradigm.
  • Future Work: The paper presents several promising directions rather than a single next step.These directions are described as avenues for further strengthening both capabilities.

E.1. Effect-aligned latent actions · E.2. Latent actions for planning and reasoning

The paper identifies extensions for making effect-aligned latent actions more robust, structured, physically faithful, and interpretable across entities. It also proposes using latent-action sequences for direct planning and reasoning, including latent-action traces as a possible successor to frame-level visual chain-of-thought.

  • E.1. Effect-aligned latent actions: SeqΔ-REPA currently uses cosine alignment between latent actions and feature-difference effect directions from a frozen video encoder.Alternative effect targets and alignment formulations may improve robustness across contexts and the structure of the learned latent action space.
  • E.1. Effect-aligned latent actions: Current latent actions operate at the step level, with one latent per frame at 16 FPS.Hierarchical latents could compose short-horizon controls into longer-horizon skills for long rollouts, multi-rate control, and cleaner downstream decision-making.
  • E.1. Effect-aligned latent actions: Physics-grounded constraints could augment effect-aligned latent actions so transferred trajectories remain visually faithful and physically plausible.Suggested constraints include verifiable kinematic or collision-consistency rewards, such as Newtonian acceleration for falling objects and collision rules.
  • E.1. Effect-aligned latent actions: SeqΔ-REPA summarizes observed change with a single effect signal that can mix ego motion, other-agent behavior, controllable-agent motion, and environment-driven events.Factorizing effects and learning entity-specific latent control could improve interpretability and support richer multi-entity controllable world modeling.
  • E.2. Latent actions for planning and reasoning: Latent actions currently support transfer and adapter-based control, while a key next step is planning directly over latent-action sequences.The world model could enable imagination-based search or trajectory optimization in latent-action space.
  • E.2. Latent actions for planning and reasoning: Large video models show emergent zero-shot capabilities, and video generation increasingly uses visual chain-of-thought guidance for long-horizon coherence and controllability.Examples include sparse keyframes, intermediate thought prompts, and storyboard plans.
  • E.2. Latent actions for planning and reasoning: A proposed direction is to move from frame-level visual chain-of-thought toward latent-action traces for video-model reasoning.This direction follows work using visual plans and intermediate guidance to improve long-horizon generation and controllability.
Loading 2602.10104v2…