PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Latent Action World Models In The Wild

Quentin Garrido, Tushar Nagarajan, Basile Terver, Nicolas Ballas, Yann LeCun, Michael Rabbat

arXiv:2601.05230v2cs.AIcs.CV

TL;DR

World models need action information to predict consequences, but action labels are difficult to obtain at scale, motivating latent actions learned from videos alone. This paper studies latent-action world models on diverse in-the-wild videos and finds that constrained continuous actions capture complex behavior, transfer across videos, and support planning through a learned control interface.

  • Problem

    World models commonly require action labels, while learning a general action space from videos alone remains challenging on diverse in-the-wild videos.

  • Method

    The paper jointly learns inverse-dynamics latent actions and a forward world model from in-the-wild videos, regulating latent information and training a controller for known actions.

  • Results

    Constrained continuous latent actions capture complex natural-video actions, transfer changes such as agents entering scenes across videos, and enable planning performance close to action-conditioned baselines.

  • Takeaways & Limitations

    Latent actions can serve as a universal interface for world models trained on natural videos, despite differing embodiments across videos.

  • Takeaways & Limitations

    The information constraint is static although videos contain actions of varying complexity, and direct sampling or planning in latent space remains less developed for high-dimensional structured latents.

Abstract

from arXiv · show

Agents capable of reasoning and planning in the real world require the ability of predicting the consequences of their actions. While world models possess this capability, they most often require action labels, that can be complex to obtain at scale. This motivates the learning of latent action models, that can learn an action space from videos alone. Our work addresses the problem of learning latent actions world models on in-the-wild videos, expanding the scope of existing works that focus on simple robotics simulations, video games, or manipulation data. While this allows us to capture richer actions, it also introduces challenges stemming from the video diversity, such as environmental noise, or the lack of a common embodiment across videos. To address some of the challenges, we discuss properties that actions should follow as well as relevant architectural choices and evaluations. We find that continuous, but constrained, latent actions are able to capture the complexity of actions from in-the-wild videos, something that the common vector quantization does not. We for example find that changes in the environment coming from agents, such as humans entering the room, can be transferred across videos. This highlights the capability of learning actions that are specific to in-the-wild videos. In the absence of a common embodiment across videos, we are mainly able to learn latent actions that become localized in space, relative to the camera. Nonetheless, we are able to train a controller that maps known actions to latent ones, allowing us to use latent actions as a universal interface and solve planning tasks with our world model with similar performance as action-conditioned baselines. Our analyses and experiments provide a step towards scaling latent action models to the real world.

1 Introduction

This work studies latent-action world models learned from large-scale in-the-wild videos, where actions are diverse, noisy, and lack a common embodiment. It finds that constrained continuous latent actions support complex action modeling, transfer across videos, and control for planning tasks.

  • Approach: Latent action models discover an action space from videos alone without action annotations or a known embodiment.They jointly use an inverse dynamics model to infer latent actions and a forward model to predict future observations.
  • Motivation: Large-scale in-the-wild videos provide a richer and more general action distribution than classical navigation or manipulation datasets.They include actions such as objects entering scenes and people dancing, beyond camera or hand movements.
  • Findings: Continuous, sparse, or noisy latent actions model complex in-the-wild actions more effectively than discrete latent actions.The study focuses on regulating latent-action information to balance complex-action modeling against environmental noise and future-state encoding.
  • Findings: Without a common embodiment, learned latent actions encode more spatially localized transformations relative to the camera.The paper reports that this absence of shared embodiment does not prevent learning latent actions.
  • Findings: Complex motions and agent actions, including someone entering a frame, can be transferred between videos.This demonstrates generality of the learned action space across distinct video contexts.
  • Downstream use: A controller mapping known actions to latent actions enables the natural-video world model to solve robotic manipulation and navigation planning tasks.Its planning performance is reported as close to models trained on domain-specific, action-labeled data.

2 Related works

Prior latent-action and world-model research has concentrated on games, robotics, manipulation, and other constrained settings. Existing methods infer latent actions from unlabeled videos with inverse and forward models, using discrete or continuous representations while addressing information leakage and distractors.

  • World Models: World-model research spans game data, simulated robotics, and real-world environments with varied embodiments and action spaces.Examples include locomotion, whole-body control, and textual control across embodiments.
  • Latent Action Models: Latent action models infer actions from unlabeled videos using either latent policies or inverse dynamics models paired with forward models.The inverse dynamics model predicts a latent action from past and future frames, while the forward model predicts the future frame.
  • Latent Action Models: Discrete latent actions are common, while continuous latent spaces offer greater flexibility.The related methods cited include both discretized approaches and continuous alternatives.
  • Challenges: Inverse dynamics can leak excessive future information, so latent actions must be constrained to avoid encoding the entire next frame.Latent actions may also capture background noise rather than the desired actions, motivating supervision or careful latent-space design.

3 Problem setting

The model represents stochastic future evolution with latent actions inferred by an inverse dynamics model and used by a forward world model. The section compares sparsity, noise, and discretization as mechanisms for limiting latent-action information while preserving complex behavior.

  • Latent action formulation: Latent actions z_t augment the world model because agents and environmental stochasticity make future prediction non-deterministic.The forward model predicts s_t+1 from the history and latent action.
  • Latent action formulation: An inverse dynamics model infers z_t from past and future states, while a forward model predicts the future state from history and z_t.The inverse model introduces future information during training, and the trained model can support world-model or VLA pipelines.
  • Information regularization: Information regularization seeks minimal latent actions that explain future predictions without encoding exogenous noise or the entire next state.In-the-wild videos increase the risk of capturing environmental changes such as oscillating leaves.
  • Information regularization: Sparsity constrains latent actions toward low L1 norm while adding regularizers to avoid trivial concentration or mode-focused solutions.The additional regularization addresses degenerate solutions that reduce L2 norm or concentrate information in a few dimensions.
  • Information regularization: Noise addition limits latent information through perturbations whose norm is controlled, using a VAE-like prior-matching mechanism.The target standard deviation adds noise, while the target mean reduces latent-action norm.
  • Information regularization: Vector quantization provides a baseline discretization scheme using classical quantization and codebook reset for unused codes.The scheme follows the approach used in UniVLA.

4 Experimental details

Experiments use frozen frame representations and jointly train inverse and forward models with latent-action regularization. Prediction quality is evaluated on in-the-wild videos, with a decoder used for visualization and perceptual metrics.

  • Training setup: Videos are encoded by a frozen frame-causal V-JEPA 2-L encoder, producing representations for model training.The encoder outputs s_0:T−1 and remains frozen during training.
  • Training setup: The world model and inverse dynamics model are trained jointly with prediction loss and latent-action regularization using teacher forcing.Teacher forcing is used to improve training efficiency.
  • Evaluation: Figure 4 evaluates one-step prediction error on in-the-wild videos while varying the capacity of sparsity- and noise-based latent actions.Quantized latent actions are compared against these continuous regularization strategies.
  • Evaluation: A frame-causal ViT-L decoder trained with L1 and perceptual losses supports visualization and perceptual metrics, although video generation is not central to the work.The decoder is used to inspect model predictions.

5 Performance of information regularizations

Continuous sparse and noisy latent actions adapt their information capacity to complex in-the-wild actions, whereas vector quantization struggles to scale capacity. Qualitative rollouts show that continuous methods capture actions such as a person entering a scene more accurately.

  • Evaluation interpretation: Inverse-dynamics rollouts provide an upper bound on attainable prediction performance because the inferred latent actions use the future frame.This evaluation can therefore involve shortcut solutions unavailable in ordinary deployment.
  • Main findings: Sparse and noisy latent actions achieve a range of prediction performance between unconstrained latent actions and a deterministic world model.Their capacity can be adjusted through the regularization strength.
  • Main findings: Vector quantization remains close to the deterministic baseline and struggles to scale capacity, unlike sparsity and noise-based approaches.The comparison is made with the other training components held constant.
  • Qualitative behavior: Sparse and noisy latent actions accurately capture a person entering and moving through a scene, whereas quantization produces mainly a blob-like entrance.The exact shirt color is not encoded, indicating abstraction beyond the precise changing pixels.
  • Main findings: Sparse and noisy latent actions capture more complex actions when given sufficient capacity, while vector quantization struggles with such actions.This is the central comparison across the tested information-regularization methods.

6 What kind of actions do we learn ?

The evaluations test whether latent actions leak future frames and whether they transfer across videos. Results indicate limited future leakage, reliable transfer and re-inference, and camera-relative spatial localization of learned transformations.

  • Future leakage: Scene-change tests measure future leakage by comparing prediction-error increases after swapping video endings with baseline predictions.A missing error spike would indicate that the latent action copied the next frame, so an increase is necessary but not sufficient evidence against cheating.
  • Future leakage: Prediction error more than doubles under scene changes for every tested latent-action capacity, indicating that no studied model simply encodes the next frame.The authors hypothesize that dataset complexity makes this shortcut difficult to learn.
  • Future leakage: Only minor next-frame information is captured in latent actions, attributed to objects appearing or disappearing across frames rather than problematic leakage.The transfer evaluations suggest this residual information is not harmful in practice.
  • Transferability: Cycle consistency evaluates transfer by inferring actions on video A, applying them to video B, re-inferring them, and applying them back to A.A small prediction-error increase after the cycle indicates reliable transfer and re-inference.
  • Transferability: Latent actions transfer a man’s leftward movement to a flying ball, which stops and then moves left, before the motion is successfully re-inferred on the original video.This demonstrates transfer across visually different objects.
  • Transferability: Higher-capacity latent actions transfer less well, but remain better after transfer than more constrained actions on Kinetics and RECON.The reported transfer gap is a minor prediction-error increase on both datasets.
  • Action locality: Latent actions are spatially localized and camera-relative, affecting the person at the relevant position rather than requiring a shared embodiment.This abstraction allows motion transfer between entirely different objects.

7 Leveraginglatentactionworldmodels for planning

The authors train controllers that map known actions to latent actions, enabling latent-action world models trained on natural videos to support planning. Controller-based planning reaches performance similar to action-conditioned baselines, although rollout quality and planning performance depend differently on latent-action capacity and regularization.

  • Planning performance: Latent actions can serve as a generic interface for different embodiments and enable goal-based planning with existing protocols.Planning searches for action sequences that minimize the distance between predicted and goal states, using CEM in the described evaluations.
  • Controller training: A lightweight controller maps known actions, optionally using past representations, to latent actions for controlling the world model.The controller is trained with an L2 loss; actions alone use an MLP, while actions plus representations use a cross-attention adapter.
  • Rollout quality: On DROID and RECON, controller predictions approximate inverse-dynamics latents and apply movements correctly, although physical appearance degrades over time.The unrollings duplicate frames to associate one action with one latent, unlike training.
  • Rollout quality: Balanced latent-action regularization produces the best controller predictions, while the most and least constrained settings are suboptimal.Over-constrained latents may lack information, whereas under-constrained latents may encode too much future information.
  • Planning performance: Latent-action models achieve similar planning performance to V-JEPA 2-AC, while higher-capacity latents can plan better despite producing worse rollouts.Noisy latent actions achieve the best planning performance even when their unrollings are comparatively poor.
  • Planning performance: On navigation, controller-based models beat policy baselines such as NoMaD but remain below NWM, partly because egocentric views continually introduce new frame content.The authors report similar conclusions for navigation and use Relative Pose Error as the main metric.
  • Scaling trends: Scaling model size, training time, and data improves inverse-dynamics predictions, while planning benefits most clearly from longer training.The figure reports similar overall trends for planning, with more nuanced effects from model size and data quantity.

8 Scaling models and data.

The scaling study varies model size, training time, and training data for sparse and noisy latent actions. Larger models, longer training, and more data improve inverse-dynamics predictions, but planning responds most consistently to longer training.

  • Scaling results: Increasing model size, training time, or training data improves inverse-dynamics predictions on natural videos.This trend holds across the scaling axes studied for both regularization settings.
  • Scaling results: Planning performance improves most clearly with longer training, while model size mainly helps noisy latent actions and data scaling shows no significant trend.The planning effects are therefore more nuanced than the inverse-dynamics scaling trends.

9 Limitations and future work

The authors identify future work on adapting latent-information constraints, planning directly in latent-action space, and jointly shaping representations with the world model. These directions address calibration, high-dimensional latent planning, and limitations from frozen encoders.

  • Variable latent information content: A static latent-information constraint cannot adapt to videos with differing action complexity or deterministic dynamics.The authors propose variable constraints, which may increase latent-action-space complexity while enabling better calibration.
  • Sampling and planning in latent action space: Direct sampling and planning in latent-action space remain underexplored, especially for high-dimensional structured latent actions.The paper provides initial supplementary analysis of these possibilities.
  • Shaping representations with single stage training: Training the world model on frozen representations may hinder inverse-dynamics learning and prediction quality because the representation space was not designed for prediction.The authors suggest joint encoder/world-model pretraining as a future direction.

10 Conclusion

The paper demonstrates latent-action world models trained on large-scale in-the-wild videos despite action complexity, environmental noise, and inconsistent embodiment. Continuous constrained latents adapt better than vector quantization, become spatially localized, and can support transfer and planning through a learned controller.

  • Conclusion: The work demonstrates effective latent-action world models trained directly on large-scale in-the-wild natural videos.The study addresses high action complexity, environmental noise, and the lack of a common embodiment.
  • Conclusion: Continuous latent actions adapt more effectively to natural-video action complexity than vector quantization.The authors report that vector quantization struggles at this scale.
  • Conclusion: Without a common embodiment, learned latent actions become spatially localized relative to the camera while remaining inferable and reusable.The paper also reports qualitative transfer of complex motions between different objects.
  • Conclusion: A controller maps known actions into the learned latent space, enabling planning with performance close to action-conditioned baselines.This provides practical utility for applying latent actions learned from natural videos.

A Training and evaluation protocols

The paper trains continuous latent-action world models from video, evaluates sampling and planning, and uses controllers to connect known actions to latent actions. Sampling quality depends on regularization: lower-capacity latents align better with inferred latents, while higher capacity improves expressiveness but complicates sampling.

  • Latent action training: The world model uses 128-dimensional continuous latent actions and AdaLN-zero frame-wise conditioning to predict the next frame.The training objective combines next-state prediction with latent-action regularization, where the inverse dynamics model infers z_t from consecutive representations.
  • Latent action training: Regularization coefficients are selected by increasing or decreasing information constraints until capacity no longer reduces prediction error or vector-quantized codebooks become underused.The sweep also considers when latent actions behave like noise.
  • Planning and control: The controller maps real actions and latent visual states to latent actions, enabling known-action control of the learned world model.Planning recursively unrolls predicted states using the controller and predictor, then optimizes action sequences with the Cross-Entropy Method.
  • Limitations: Direct planning in latent action space remains an open problem because the geometry of the latent space can make optimization more difficult.The paper instead uses latent actions as an interface for transfer experiments or through controllers that map interpretable actions into latent actions.
  • Sampling latent actions: Lower-capacity latent actions are easier to sample because sampled and inferred latents overlap more closely, whereas higher-capacity latents become separable and sampling worsens.The sampling analysis compares inferred latents with equally numerous random samples under different capacity constraints.

D Robot manipulation vs in-the-wild videos

The experiments compare domain-specific DROID pretraining with in-the-wild video training and analyze how latent capacity affects action transfer. Diverse pretraining improves coverage of human-centric actions, while continuous latent actions balance fidelity, transfer, and generality better than vector quantization.

  • Cross-domain transfer: A DROID-only model struggles with human-centric in-the-wild actions such as entering a room and hand motion, while both models handle simple object translation.For a person entering the room, the DROID-trained model produces a robotic arm, reflecting its training distribution.
  • Cross-domain transfer: Pretraining on more diverse data benefits capture of diverse actions, although constrained-domain training can still learn actions that generalize.The result supports using broader video sources for learning action spaces beyond a single embodiment.
  • Planning performance: Adding domain-specific DROID data can strongly improve rollouts and planning, with some settings receiving a substantial boost from only 10% domain data.The latent-action model reaches planning performance similar to an action-labeled baseline: 0.06 versus 0.05 for the best reported model.
  • Capacity and transfer: Weaker latent constraints improve the appearance and motion of a person entering the scene, especially the fidelity of leg movements.Overly constrained latents cannot make the person appear, while further capacity gains eventually plateau.
  • Capacity and transfer: Increasing latent capacity improves cross-object motion transfer, changing a ball from continuing its trajectory to following the transferred motion.Strong regularization can leave the ball unaffected, whereas adequate capacity enables motion identification and transfer.
  • Capacity and transfer: Higher capacity initially improves cycle consistency but can localize motion to part of a human, trading generality for more precise actions.The paper identifies a balance between capacity and generality as important for planning performance.
  • Latent representation comparison: Continuous noisy or sparse latents capture actions in unseen SSv2 videos more effectively than vector-quantized latents, especially for precise object rotation.The qualitative rollouts use inverse dynamics models and therefore represent an upper bound that may include shortcut solutions.

G Additional human action transfer results.

Across additional transfer scenarios, latent actions transfer human movements and scene-entry behaviors between videos, with capacity affecting the naturalness and extent of the generated motion. The results also show that controllers can map real actions to latent actions, while missing visual context limits control.

  • Action transfer: Human entry actions transfer across videos, while the behavior of existing people varies with latent-action capacity.At higher capacities, an existing person may move with the newcomer or disappear; when originally still, the target person remains still.
  • Action transfer: Transferring two people raising an arm to one person raises both target arms and preserves the original movement despite task ambiguity.The target arms expand less horizontally, consistent with spatially localized actions, and this behavior is consistent across capacities.
  • Action transfer: Higher-capacity latents produce more natural turning and walking when transferring motion to a still person.The person begins moving only when the transferred motion reaches their current location, reinforcing action locality.
  • Action transfer: No background leakage appears in these videos, suggesting the model learns transferable latent actions rather than copying future frames.The overall result is adequate cross-video transfer, including ambiguous transfers between different numbers of people.
  • Controller rollouts: Controller rollouts accurately map real actions to latent actions on RECON, but DROID control fails when the robotic arm is absent from view.Without past representations, the DROID arm remains static because camera and arm-position ambiguity prevents successful contextualized control.
Loading 2601.05230v2…