PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Act without Actions

Dominik Schmidt, Minqi Jiang

arXiv:2312.10812v2cs.LGcs.AI

TL;DR

Videos are abundant behavioral data, but they lack the action labels typically required to learn policies from demonstrations. LAPO infers latent actions from observed dynamics and uses them to train latent-action policies and dynamics models. Across Procgen games, these policies can be adapted to expert-level behavior with a small labeled dataset or online fine-tuning.

  • Problem

    Web videos lack the action labels required by common imitation-learning and offline-RL methods, limiting their direct use as demonstrations.

    Source: §1 · p. 1

  • Method

    LAPO learns compressed latent actions from video transitions using predictive consistency between inverse and forward dynamics models, then trains policies over those actions.

    Source: §1 · pp. 1–2

  • Results

    Across 16 Procgen games, LAPO learns latent-action spaces that often align with the true action space, and about 200 labeled transitions let a decoder exceed the performance of 4M steps of regular RL via PPO.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The results show that action information can be recovered from pure video and that latent policies can be adapted to expert-level behavior with limited labels or online fine-tuning.

    Source: §1 · p. 2

    Delayed-effect actions are represented with their visible delay, while significant stochasticity can make useful transition information harder for the IDM to compress.

    Source: §6.4 · p. 9

Abstract

from arXiv · show

Pre-training large models on vast amounts of web data has proven to be an effective approach for obtaining powerful, general models in domains such as language and vision. However, this paradigm has not yet taken hold in reinforcement learning. This is because videos, the most abundant form of embodied behavioral data on the web, lack the action labels required by existing methods for imitating behavior from demonstrations. We introduce Latent Action Policies (LAPO), a method for recovering latent action information, and thereby latent-action policies, world models, and inverse dynamics models, purely from videos. LAPO is the first method able to recover the structure of the true action space just from observed dynamics, even in challenging procedurally-generated environments. LAPO enables training latent-action policies that can be rapidly fine-tuned into expert-level policies, either offline using a small action-labeled dataset, or online with rewards. LAPO takes a first step towards pre-training powerful, generalist policies and world models on the vast amounts of videos readily available on the web.

1 INTRODUCTION

LAPO infers latent actions from videos without action labels, then uses them to train policies and dynamics models. The introduction reports that these policies can be adapted to expert-level behavior in Procgen using either a small labeled dataset or online fine-tuning.

1 INTRODUCTION

  • The motivation is that web videos lack the action or reward labels typically required by imitation learning and offline RL.Source: §1 · p. 1
  • Unlike prior IDM-based approaches that use ground-truth action labels, LAPO infers latent actions purely from observed environment dynamics.The introduction names Torabi et al. (2018), Schmeckpeper et al. (2020), Baker et al. (2022), and Zheng et al. (2023) as prior approaches.Source: §1 · p. 1
  • LAPO trains an inverse dynamics model to infer compressed latent actions from observed transitions and a forward dynamics model to predict the next observation from each latent action.Predictive consistency between the models makes the latent action an information bottleneck, preventing the IDM from simply forwarding the future observation.Source: §1 · p. 2
  • Across 16 Procgen games, learned latent-action clusters often align with the true action space; about 200 labeled transitions let a decoder exceed the performance of 4M steps of regular RL via PPO.Online fine-tuning of the latent policy can also recover or sometimes exceed expert performance.Source: §1 · p. 2, Figure 1 · p. 2
  • The authors frame recovering comprehensive action information from pure video as a first step toward training generalist policies and world models on large video datasets.Source: §1 · p. 2

2 RELATED WORK

LAPO differs from related approaches by inferring latent actions without action labels and directly producing policies that can be adapted online. Related work includes dynamics-based latent-action methods, world-model pretraining, semi-supervised labeling pipelines, and imitation from observation.

2 RELATED WORK

  • Unlike ILPO, LAPO learns an IDM jointly with an FDM using continuous latent actions, avoiding enumeration of hypothetical transitions; ILPO’s discrete-action objective can suffer mode collapse and break down across Procgen levels.Source: §2 · p. 3
  • FICC pretrains world models to distill dynamics, whereas LAPO directly learns latent policies that imitate behavior and can be rapidly adapted online.FICC’s adaptation maps true actions to latents and requires continued world-model training and application.Source: §2 · p. 3
  • Semi-supervised methods such as VPT and ACO train IDMs on labeled data before pseudo-labeling web footage, while LAPO infers latent actions without that labeled IDM-training data.Other examples include RLV, BCO, and SS-ORL, which use action labels, interaction, or reward labels in their pipelines.Source: §2 · p. 3
  • Unlike imitation-from-observation methods that typically require access to the true action space during training, LAPO learns from action-free demonstrations without that access (Torabi et al., 2019a;b; Yang et al., 2019).Source: §2 · p. 3

3 BACKGROUND

The background sets up offline-to-online reinforcement learning under partial observability, explains imitation when actions are missing, and introduces the dynamics models and vector quantization used by LAPO.

3.1 REINFORCEMENT LEARNING

  • The paper frames reinforcement learning as a partially observable process in which agents receive observations and may first learn from offline demonstrations before online fine-tuning.Source: §3.1 · p. 4

3.2 LEARNING FROM OBSERVATIONS

  • When demonstrations lack action labels, LAPO infers latent actions for observed transitions and trains a latent-action policy, rather than relying on an IDM trained with ground-truth actions.Behavior cloning otherwise imitates labeled expert actions by minimizing cross-entropy between the policy distribution and expert actions.Source: §3.2 · p. 4

3.3 DYNAMICS MODELS

  • LAPO’s inverse dynamics model predicts continuous latent actions from consecutive observations, while its forward dynamics model predicts the next observation from the previous observation and action.Both models are deterministic deep neural networks in this work.Source: §3.3 · p. 4

3.4 VECTOR-QUANTIZATION

  • Vector quantization maps a continuous representation to the nearest codebook vector, producing discrete features while allowing gradients to pass through the quantization step.Source: §3.4 · p. 4

4 LATENT ACTION POLICIES

LAPO learns latent actions from observation transitions, trains a policy to imitate those latents, and adapts it to true actions using labeled data or online rewards.

4.1 LEARNING A LATENT ACTION REPRESENTATION

  • The IDM infers z_t from transition observations, and the FDM predicts o_t+1 from past observations and z_t; both models minimize next-state prediction error.The IDM uses o_t-k through o_t+1, while the FDM receives only o_t-k through o_t and z_t.Source: §4.1 · p. 5
  • An information bottleneck discourages copying o_t+1 into z_t, while vector quantization encourages reuse of discrete latents across states.The paper hypothesizes that compressing transition information may yield representations structured like the true action space.Source: §4.1 · p. 5

4.2 BEHAVIOR CLONING A LATENT ACTION POLICY

  • Behavior cloning trains a latent policy on the IDM-generated labels, minimizing squared error between the policy output and inferred latent action.The policy is initialized as a mapping from observations to latent actions.Source: §4.2 · p. 5

4.3 DECODING LATENT ACTIONS

  • With a small true-action-labeled dataset, a decoder learns to map IDM-predicted latents to true actions and is composed with the frozen latent policy.The resulting decoded policy can be deployed online.Source: §4.3 · p. 5
  • Online interaction supplies labeled transitions for decoder training and rewards for RL, enabling joint decoder bootstrapping and fine-tuning of the composed policy.The paper says RL can potentially improve performance beyond the data-generating policy, including when data mixes skill levels or goals.Source: §4.3 · p. 6

5 EXPERIMENTAL SETTING

The experiments use Procgen and an expert-generated observation-only dataset, with model architectures and training choices selected for the evaluation.

5 EXPERIMENTAL SETTING

  • Procgen tests generalization and pixel-level dynamics modeling under procedural variation, visual complexity, partial observability, and stochasticity.The authors note that partial observability and stochasticity can make transitions ambiguous when inferring actions from observations.Source: §5 · p. 6
  • The observation-only dataset contains approximately 8M frames from an expert policy trained with PPO for 50M frames.The synthetic data permits access to true actions and episodic-return metadata for evaluation.Source: §5 · p. 6
  • The IDM and policy use IMPALA-CNN networks with a 4x channel multiplier, while the FDM uses a ResNet-based U-Net with approximately 8M parameters.The latent-action decoder is a fully connected network with hidden sizes (128, 128).Source: §5 · p. 6

6 RESULTS AND DISCUSSION

LAPO’s latent policy reaches expert-level performance with limited online interaction, while offline decoding can match a from-scratch policy with fewer than 256 labeled transitions. Learned latent actions often align with true actions, though their structure varies across environments and can be fragmented under partial observability.

6.1 DECODING THE LATENT POLICY ONLINE

  • Within 4M frames, LAPO recovers expert performance and exceeds it in 9 of 16 tasks, while PPO from scratch reaches 44% of expert performance.The comparison uses 4M frames for PPO rather than the 25M suggested by Cobbe et al. (2020).Source: §6.1 · p. 6, Figure 3 · p. 7

6.2 DECODING THE LATENT POLICY OFFLINE

  • Fewer than 256 labeled transitions let offline decoding match a policy trained from scratch for 4M steps, though its performance plateaus below online decoder training.The authors suggest this gap may reflect latent actions that are not always state-invariant.Source: Figure 4 · p. 7, §6.2 · p. 7

6.3 INSPECTING THE LATENT ACTION SPACE

  • Across most Procgen games, latent-action clusters closely correspond to true discrete actions, while some environments show more fragmented structure.The UMAP points are colored by true action labels for visualization; those labels were not used for training.Source: §6.3 · p. 8, Figure 5 · p. 8
  • Vector quantization simplifies latent-action structure, and BigFish shows four latent clusters for each true action, consistent with state-dependent effects.The authors report duplicated clusters for distinct true actions in several environments.Source: §6.3 · p. 9

6.4 LIMITATIONS

  • Delayed visible effects can shift inferred action timing, stochasticity can degrade latent representations, and scaling to complex domains requires larger models.The authors suggest multi-timestep architectures and larger datasets as possible mitigations for some of these constraints.Source: §6.4 · p. 9

7 CONCLUSION

LAPO learns policies over a latent action space from observational data, without access to the true action space or predefined discrete latents. Across 16 Procgen games, its latent policies can be rapidly adapted to recover or exceed the source expert’s performance.

7 CONCLUSION

  • LAPO learns latent actions end-to-end through predictive consistency between inverse- and forward-dynamics models, with vector quantization encouraging state-invariant transition information.The learned latent action space is inferred from purely observational data.Source: §7 · p. 9
  • Across all 16 Procgen games, learned latent action spaces reflect the structure of true action spaces despite never accessing them.Latent-action policies can be rapidly adapted to recover or exceed the performance of the expert that generated the action-free data.Source: §7 · p. 9
  • The authors propose scaling world models and policies to richer observations and multiple timesteps as a direction toward generalist policies for complex, multi-task settings.They frame LAPO as a potential route to pretraining from web-scale action-free demonstrations such as videos.Source: §7 · p. 9

A.1 OFFLINE DECODING RESULTS

Offline decoding can be effective with a small action-labeled dataset, but its performance generally plateaus below online reinforcement-learning decoding.

A.1 OFFLINE DECODING RESULTS

  • An effective offline decoder can be trained with only a few hundred action-labeled samples.The decoder is combined with the latent policy.Source: Figure 6 · p. 14
  • Offline decoding performance generally plateaus before reaching the level of online reinforcement-learning decoding.The figure reports mean and standard deviation across three seeds.Source: Figure 6 · p. 14
  • The offline decoder is trained on action-labeled (z_t, a_t) tuples, with performance evaluated against dataset size.The figure presents test performance versus the number of labeled transitions.Source: Figure 6 · p. 14

A.2 LATENT SPACE ANALYSIS FOR NO-VQ ABLATION

Without vector quantization, the latent space is visualized against true action labels, while the ablation has lower FDM loss but worse online performance.

A.2 LATENT SPACE ANALYSIS FOR NO-VQ ABLATION

  • No-VQ has significantly lower FDM loss yet worse online performance, consistent with vector quantization constraining information passed to the FDM.The caption attributes the better downstream policy representation to compressing transition information.Source: Figure 7 · p. 15
  • Figure 7 projects continuous IDM-generated latent actions for transitions from the observation-only dataset across all 16 Procgen games.Each point is colored by the true action, which is used only for visualization; NOOP actions are omitted.Source: Figure 7 · p. 15

A.3 CONTINUOUS CONTROL EXPERIMENTS

The appendix reports continuous-control online-decoding curves, method hyperparameters, and a qualified ILPO reproduction.

A.3 CONTINUOUS CONTROL EXPERIMENTS

  • Figure 8 plots mean test returns during online decoding of a LAPO latent policy versus PPO from scratch on continuous-control tasks, averaged across 2 seeds.The experiments use the same latent-IDM and behavior-cloning hyperparameters as the Procgen experiments.Source: Figure 8 · p. 16

A.4 HYPERPARAMETERS

  • Latent actions are 128-dimensional, split and quantized into 8 discrete latents with 16-dimensional embeddings; Procgen uses easy mode and online fine-tuning totals 4,000,000 interactions.PPO hyperparameters follow Cobbe et al. (2020).Source: Table 1 · p. 17, §A.4 · p. 17

A.5 ILPO REPRODUCTION

  • Figure 9 reproduces one CoinRun level; on the full level distribution ILPO's minimizing discrete latent and policy collapse, and comparison is uncertain because Edwards et al.'s level is unknown.The reported reproduction results are not exactly comparable to Edwards et al. (2019).Source: Figure 9 · p. 17

Related papers

Loading 2312.10812v2…