PaperlayerView exact revision on arXiv

Source-linked AI summary

GenRL: Multimodal-foundation world models for generalization in embodied agents

Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Aaron Courville, Sai Rajeswar

arXiv:2406.18043v2cs.AIcs.CVcs.LGcs.RO

TL;DR

Embodied RL is difficult to scale because reward design is complex, while VLMs and embodied domains face adaptation and multimodal-data challenges. GenRL aligns VLM representations with generative world-model latents using vision-only data, then learns prompted tasks in imagination; it reports multi-task performance across locomotion and manipulation and data-free policy learning close to its offline performance.

  • Problem

    Scaling embodied RL across tasks is challenging because reward design is complex, VLMs often require adaptation, and embodied domains lack multimodal data.

    Source: §1 · p. 2

  • Method

    GenRL connects VLM representations to generative world-model latents using vision data, grounds visual or language prompts in dynamics, and learns behaviors in imagination.

    Source: §1 · p. 2

  • Results

    GenRL enables multi-task generalization from language and visual prompts across locomotion and manipulation, while data-free policy learning remains close to offline GenRL and outperforms other approaches.

    Source: §4.1 · p. 7, §4.2 · p. 9, §5 · p. 10

  • Takeaways & Limitations

    Data-free behavior learning lays groundwork for embodied RL foundation models that generalize to new tasks without additional data.

    Source: §1 · p. 2, §5 · p. 10

    GenRL inherits VLM multimodality-gap and prompt-tuning issues, and its reliance on reconstructions can fail with complex observations.

    Source: §5 · p. 10

Abstract

from arXiv · show

Learning generalist embodied agents, able to solve multitudes of tasks in different domains is a long-standing problem. Reinforcement learning (RL) is hard to scale up as it requires a complex reward design for each task. In contrast, language can specify tasks in a more natural way. Current foundation vision-language models (VLMs) generally require fine-tuning or other adaptations to be adopted in embodied contexts, due to the significant domain gap. However, the lack of multimodal data in such domains represents an obstacle to developing foundation models for embodied applications. In this work, we overcome these problems by presenting multimodal-foundation world models, able to connect and align the representation of foundation VLMs with the latent space of generative world models for RL, without any language annotations. The resulting agent learning framework, GenRL, allows one to specify tasks through vision and/or language prompts, ground them in the embodied domain's dynamics, and learn the corresponding behaviors in imagination. As assessed through large-scale multi-task benchmarking in locomotion and manipulation domains, GenRL enables multi-task generalization from language and visual prompts. Furthermore, by introducing a data-free policy learning strategy, our approach lays the groundwork for foundational policy learning using generative world models. Website, code and data: https://mazpie.github.io/genrl/

1 Introduction

GenRL addresses the difficulty of scaling RL across embodied tasks, where reward design is complex and VLM-based approaches often require fine-tuning or work reliably in only a few visual settings. It connects VLM representations to generative world-model latents using vision data, so agents can learn prompted tasks in imagination and generalize to new tasks without additional data.

1 Introduction

  • Reward design makes multi-task embodied RL difficult to scale, while VLM-based reward approaches often require fine-tuning or work reliably in only a few visual settings.The paper identifies expert knowledge and errors in reward design as further challenges.Source: §1 · p. 2
  • Embodied domains often lack multimodal training data, and translating language tasks into hardware-level actions such as motor currents or joint torques is non-trivial.Source: §1 · p. 2
  • GenRL connects a foundation video-language model’s joint embedding space to generative world-model representations using vision data, without language annotations.It grounds visual or language prompts in embodied dynamics and uses RL in imagination to learn the specified behaviors.Source: §1 · p. 2
  • After MFWM training, GenRL can generalize to new tasks without additional data, combining environment-dynamics priors with multimodal knowledge.The paper presents this as groundwork for foundation models in embodied RL.Source: §1 · p. 2

2 Preliminaries and background

The background frames embodied task learning around visual or language specifications, model-based RL, and efforts to apply foundation models to RL. It highlights the potential of generative world models alongside limitations of current VLM-based approaches.

Problem setting

  • The task setting allows visual or language prompts and does not assume a reward signal for solving the task; rewards, when available, evaluate performance.Source: §2 · p. 2

Generative world models for RL

  • Generative world models support model-based RL by learning latent environment dynamics and enabling agents to roll out and score imagined trajectories.The overview figure depicts GenRL connecting VLM representations with world-model latent states and translating prompts into targets learned through RL in imagination.Source: §2 · p. 2, Figure 2 · p. 3

Foundations models for RL

  • VLMs let agents evaluate behaviors in visual space, but prior VLM approaches to RL have generally required fine-tuning, prompt hacking, or visual environment modifications.Video generation models, unlike action-conditioned world models, are typically conditioned on language or are unconditional.Source: §2 · p. 3

3 GenRL

GenRL builds a task-agnostic world model, connects its latent space to a VLM, and learns prompted behaviors by matching imagined trajectories. Its aligner addresses the multimodality gap using vision-only data, while temporal matching accommodates differences between agent and target starting states.

3.1 World models for RL

  • GenRL models sequential environment dynamics in a compact discrete latent space, with an encoder and decoder that represent individual observations while temporal information resides in the sequence model’s hidden state.The latent states are sampled from independent categorical distributions.Source: §3.1 · pp. 3–4

3.2 Multimodal-foundation world models

  • The VLM input is a sequence of visual observations and a text prompt; the implementation uses InternVideo2 with k = 8 frames.Source: §3.2 · p. 4
  • A multimodal-foundation world model connects VLM representations to world-model latents; its connector predicts latent states from VLM embeddings, while the aligner maps points near visual embeddings toward them.The connector is trained against the world model’s encoder distribution, and the aligner uses noise-sampled points around visual embeddings, requiring no language annotations.Source: §3.2 · p. 4
  • Contrastive VLMs exhibit a multimodality gap, so GenRL assumes nearby visual and language embeddings and learns alignment without the embodied-domain language annotations that are typically unavailable.The method uses an aligner rather than changing the embedding geometry through the prior approach of injecting noise into vision embeddings.Source: §3.2 · p. 4
  • The aligner enables a noise-free connector with higher prediction accuracy for visual inputs, similar language-input alignment, and flexibility to adapt to different noise levels.The aligner can be retrained independently or omitted when unnecessary.Source: §3.2 · p. 5

3.3 Specifying and learning tasks in imagination

  • The policy learns prompted tasks by matching its visited-state distribution to prompt-generated latent trajectories, using cosine distance between linear latent projections for faster, more stable learning.The objective is converted into an RL reward maximized by an actor-critic; the experiments use b = 8 frames as a compromise between initial-state comparison and no alignment.Source: §3.3 · p. 5
  • Best-matching trajectory alignment slides the target’s first b states across imagined agent trajectories to find the timestep with the highest reward, then aligns subsequent states relative to it.This addresses the mismatch that can arise when the agent’s initial state differs from the connector-generated target, including when reaching the target behavior takes longer than the VLM’s temporal span.Source: §3.3 · p. 5

4 Experiments

Across locomotion and manipulation, GenRL is evaluated on 35 reward-free tasks using visual or language prompts. The experiments compare offline approaches, test video prompts and data-free policy learning, and analyze which training-data subsets support generalization.

4 Experiments

  • Across 35 tasks in four locomotion environments and Kitchen, agents train without rewards using visual or language prompts.The evaluation compares offline RL methods, including IQL, TD3+BC, and TD3, with model-based GenRL and WM-CLIP.Source: §4 · p. 5, §4.1 · p. 6

4.1 Offline RL

  • GenRL performs strongly overall, particularly on dynamic quadruped and cheetah tasks, while other methods sometimes outperform it on static Kitchen tasks.Slight motion in inferred target sequences may explain the static-task limitation; the authors retain the general method rather than setting target sequence length to 1 for static prompts.Source: §4.1 · p. 7
  • GenRL significantly outperforms model-free approaches on tasks excluded from training, especially in quadruped and cheetah, approaching specialized-agent performance there.The comparison averages performance across tasks by domain; detailed per-task results are in Appendix K.Source: §4.1 · p. 7, Figure 4 · p. 7

4.2 Data-free policy learning

  • Data-free learning samples latent states, rolls out policies in imagination, and computes rewards from prompt-derived targets, without requiring action, reward, or observation data during adaptation.The MFWM is pretrained on a varied dataset and supports task specification directly in latent space.Source: Figure 6 · p. 8, §4.2 · pp. 8–9

4.1 Offline RL

  • Video prompts produce performance aligned with language prompts on the same tasks, and generalize across visual styles, camera viewpoints, and morphologies.The decoder can visualize the model’s interpretation of prompts, while prompts can also be converted into behaviors.Source: Figure 5 · p. 7, §4.1 · p. 8
  • The experiments identify three ingredients associated with stronger performance: video-language models for dynamic tasks, model-based algorithms, and the proposed connection-alignment system over reversed connections.The comparison also finds video-language rewards tend to outperform image-language rewards on dynamic tasks; TD3 is the strongest model-free baseline in most tasks, and WM-CLIP-V is closest to GenRL.Source: §4.1 · pp. 7–8

4.2 Data-free policy learning

  • Data-free policy learning is slightly below offline GenRL overall but remains close and still outperforms other approaches; it can even perform better in Kitchen.The authors report policy training can often converge within 30 minutes, and does not require CPU-GPU memory transfers of data.Source: §4.2 · p. 9

4.3 Analysis of the training data distribution

  • The complete training dataset performs best, followed by varied exploration data; the usefulness of task-specific data depends on task complexity.Run data generalizes better than walk or stand data across tasks, while low-variation stand data limits general-agent learning but can support simpler tasks such as lying down and sitting on knees.Source: §4.3 · p. 9, Figure 7 · p. 9

5 Discussion

GenRL grounds vision-language prompts in embodied dynamics and learns behaviors in imagination, including from unimodal training data. The discussion highlights prompt interpretability, inherited limitations, and research directions, while extended related work situates its representation and approach among prior world and action models.

5 Discussion

  • GenRL trains multimodal-foundation world models on unimodal data and learns prompt-conditioned behaviors in imagination, addressing the lack of multimodal data in embodied RL.Its data-free behavior-learning capacity lays groundwork for RL foundation models that generalize to new tasks without data.Source: §5 · p. 10
  • Decoded latent targets make prompt interpretations visible before behavior training, enabling faster prompt-tuning iteration than model-free approaches that require training to reveal rewarded behaviors.GenRL still inherits the foundation VLM’s prompt-tuning issue.Source: §5 · p. 10
  • GenRL inherits the VLM multimodality gap and prompt-tuning reliance, plus world-model dependence on reconstructions that can fail with complex observations.Connection alignment mitigates the multimodality gap, while target decoding facilitates prompt tuning.Source: §5 · p. 10
  • Future work includes composing learned behaviors with an LLM for long-horizon tasks and investigating temporal flexibility, which may improve static-task performance.The authors also suggest greater temporal awareness for actions extending beyond a VLM’s time comprehension.Source: §5 · p. 10

A Extended Related Work

  • Unlike DynaLang’s shared vision-language world-model representation, GenRL trains its world-model representation on vision-only data and connects it to multimodal foundation representations.Other cited world-model work uses latent actions for 2D games or simple robotics, or frame-by-frame video prediction for RL rewards [14].Source: A Extended Related Work · p. 14
  • Prior embodied action foundations include behavior cloning across 604 tasks, Minecraft-trained VPT, STEVE-1’s VPT–MineCLIP connection, and RT-X’s shared end-effector action space.Source: A Extended Related Work · p. 14

B Implementation details

The implementation uses actor-critic learning to maximize rewards in imagination, follows selected DreamerV3 design choices, and configures the world-model and actor-critic stages independently.

B Implementation details

  • Rewards are maximized over time in imagination using actor-critic models.Source: B Implementation details · p. 14
  • The actor-critic follows DreamerV3 advances including a two-hot critic distribution and return scaling in the actor loss.When computing the GenRL reward, the implementation uses the mode of the target-state distribution.Source: B Implementation details · p. 14
  • MFWM and actor-critic training use different batch sizes and lengths because the stages are independent; the connector matches world-model sequential-dynamics hyperparameters.The aligner is a small U-Net with a bottleneck half the 768-dimensional embedding size; Table 2 presents world-model and actor-critic hyperparameters.Source: B Implementation details · p. 14, Table 2 · p. 14

C Tasks

The task suite includes a modified Walker-based Stickman environment and language prompts for tasks, with prompt choices informed by VLM behavior and task-specific movement styles.

C Tasks

  • Stickman adds four arm joints to the Walker environment, for ten total, to study upper-body tasks such as boxing and handstands without humanoid-training complexity.Its action space is normalized to [-1,1], and it includes a head resembling a humanoid.Source: C Tasks · p. 15
  • The task list pairs tasks with language prompts; flipping tasks include both forward and backward directions because the VLM struggles to distinguish them.Table 3 presents the task and prompt used for each task.Source: C Tasks · p. 15, Table 3 · p. 15
  • Prompts were tuned for InternVideo2 and mostly also improved SigLIP; specifying embodiment or style words can help clarify behaviors amid human-action bias and varied movement styles.Examples include “spider running fast,” “running like a quadruped,” and specifying “fast” or “clean.”Source: C Tasks · p. 15

D Experiments settings

The experiments compare model-free offline RL with model-based agents and include a reversed-connector baseline. GenRL’s data-free policy-learning mode reduces training time, while its world-model reuse becomes advantageous at higher run counts.

D Experiments settings

  • Model-free baselines use 64×64 RGB images with a three-frame stack and a DrQ-v2-adapted encoder; image augmentations such as random shifts hurt performance.The baselines use larger, deeper networks and layer normalization.Source: D Experiments settings · p. 16
  • WM-CLIP predicts VLM embeddings from world-model latent states with a reversed connector, then computes imagined-state rewards by cosine similarity to prompt embeddings.WM-CLIP and GenRL use the same world model; the former pre-trains a reversed connector, while the latter trains a connector and aligner.Source: D Experiments settings · p. 16
  • Model-free IQL, TD3, and TD3+BC retrain the full agent per task, whereas GenRL and WM-CLIP train a model per domain and an actor-critic per task.GenRL actor-critic training uses 50k gradient steps, with convergence significantly faster than the other methods.Source: D Experiments settings · p. 16
  • MFWM pretraining takes about five days for 500k gradient steps; actor-critic training takes under five hours with data and under three hours in data-free mode.Model-free baselines take about seven hours for 500k gradient steps; these experiments use V100 GPUs with 16GB VRAM.Source: D Experiments settings · p. 16
  • Beyond 60 runs, GenRL starts becoming advantageous on a single GPU; data-free policy learning halves training time by eliminating dataset transfers.Source: D Experiments settings · p. 16

E Temporal alignment ablation

The temporal-alignment ablation analyzes how temporal alignment affects the proposed RL objective for matching sequential targets.

E Temporal alignment ablation

  • The ablation measures the impact of temporal alignment in the proposed RL objective for matching sequential targets.Results are averaged over 10 seeds.Source: Figure 8 · p. 17

F Aligner model ablation study

The aligner is crucial to GenRL, while adding it to baseline reward computation provides no overall advantage. The discussion also describes LIV’s Kitchen results and a data-free policy-learning strategy.

F Aligner model ablation study

  • The aligner is crucial to GenRL, but adding it to baseline reward computation changes performance by domain without providing an overall advantage.GenRL’s no-aligner ablation feeds language embeddings directly to the connector; the baselines instead use the aligner before computing cosine similarity.Source: F Aligner model ablation study · p. 17

G Comparison with LIV

  • LIV’s general pre-trained representation does not work well for Kitchen vision-language rewards without fine-tuning on domain-specific vision-language pairs.Such pairs are unavailable in the authors’ no-language-annotations setting.Source: G Comparison with LIV · p. 18, Table 6 · p. 18

H Extended Discussion on Data-free Policy Learning

  • Data-free GenRL samples initial latent states internally, combining random latent-space samples with embeddings mapped through the aligner and connector.The method also uses five-step rollouts mixing trained-policy and random actions to obtain varied initial states with dynamic information.Source: H Extended Discussion on Data-free… · pp. 18–19

I Scaling to complex observations

In Minecraft, GenRL was tested with a small DreamerV3-collected dataset, but reconstructing complex observations remained difficult. Blurrier reconstructions leave precise prompt targets less clear.

I Scaling to complex observations

  • GenRL was tested in Minecraft using a small dataset collected by a DreamerV3 agent, and complex-observation reconstruction was the primary challenge.The setup was similar to DreamerV3’s experimental settings.Source: I Scaling to complex observations · p. 19
  • The agent identified different Minecraft biomes from language prompts even with the smaller model, but its reconstructions were significantly blurrier than in other environments.Larger models added some detail, yet still struggled to provide accurate prompt-derived targets.Source: I Scaling to complex observations · p. 19, Figure 10 · p. 19
  • Unclear targets may be less problematic for high-level tasks such as navigating to a beach, but may hinder precise actions such as attacking a zombie.The authors suggest improving the GRU-based architecture or using transformers or diffusion models to improve representation quality.Source: I Scaling to complex observations · p. 19

J Additional limitations

GenRL’s task generalization is bounded by the available behaviors and exploration data: complex behaviors require demonstrations, and out-of-distribution Kitchen tasks remain difficult. Detailed results are reported across three figures.

J Additional limitations

  • Solving more complex tasks requires expert data or demonstrations of the complex behavior during training.The experiments used exploration data from Plan2Explore and task-specific data collected with DreamerV3.Source: J Additional limitations · p. 19
  • Adding Kitchen tasks is harder because meaningful behaviors are difficult to explore in manipulation environments.GenRL retrieves the four dataset tasks and can achieve similar tasks such as reaching the microwave, but more interesting out-of-distribution tasks generally pose difficulty.Source: J Additional limitations · pp. 19–20
  • For locomotion, varied exploration data enables tasks beyond those included in the training dataset.Source: J Additional limitations · p. 20

K Detailed results

Related papers

Loading 2406.18043v2…