PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language Agents Can Learn in Their Own Latent Worlds

Haoxi Li, Sikai Bai, Jianfei Ma, Qinglin Hou, Xiaocheng Lu, Tao Han, Jie Zhang, Song Guo

arXiv:2610.36925v1cs.AI

TL;DR

Agents often need substantial environmental interaction to improve through reinforcement learning, while policy shifts and expensive updates complicate world-model adaptation. DWELL uses a recurrent VLM in its own latent space, learning from real interactions and imagined trajectories. Across five interactive visual environments, it achieves strong overall performance among trainable methods while reducing imagination compute relative to generative world models.

  • Problem

    Model-free reinforcement learning can require substantial environment interaction, and policy shifts expose states poorly covered by the world model, making improvement from limited experience a challenge.

    Source: §1 · pp. 1–2

  • Method

    DWELL recasts a pretrained VLM as a recurrent latent transition operator, using a shared representation for prediction and policy learning, with real interactions refining models and imagined trajectories guiding policy improvement.

    Source: §1 · p. 2

  • Results

    Across five interactive visual environments, DWELL achieves superior performance among trainable methods while reducing imagination compute relative to a world model that reconstructs observations.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    A useful world model for VLM agents need not reconstruct future observations; it needs to predict futures the agent can act and learn upon.

    Source: §6 · p. 10

    Evidence covers one VLM backbone and five simulated interactive visual environments, leaving scaling to other capacities, richer action spaces, and real-world interaction unestablished.

    Source: §5 · p. 9

Abstract

from arXiv · show

Learning from experience is essential for building the next generation of agents that continually expand their capabilities. Pretrained vision-language models (VLMs) already internalize rich world knowledge, yet this knowledge is rarely leveraged to drive policy improvement through online interaction. Realizing this potential efficiently remains challenging, as policy updates shift the agent's interaction distribution, requiring the world model to adapt accordingly, while continually updating large pretrained models is computationally expensive. To address this challenge, we introduce Dwell, a model-based reinforcement learning (RL) framework that reframes a VLM as a recurrent latent transition operator, endowing it with the ability to predict, adjust, and act in latent space. Leveraging the VLM's own representations and its dual role as both world model and policy, the agent continually learns from real interactions to anticipate and evaluate action outcomes, and from latent imagination to improve its behavior. Across five interactive visual environments, Dwell achieves the strongest overall performance among trainable methods while substantially reducing the compute required for imagination, showing that effective world models for VLM agents need not reconstruct the future; they need to predict futures that are useful for action. Our code is publicly available at https://github.com/Tom111-lhx/Dwell

1 INTRODUCTION

DWELL addresses the challenge of improving policies from limited real experience while keeping world-model predictions compatible with a changing VLM policy. It uses a recurrent VLM in a shared latent space for world modeling and action, with real interactions refining models and imagined trajectories guiding policy improvement.

1 INTRODUCTION

  • Model-free RL can require substantial environment interaction, while policy shifts can expose poorly covered states where transition and reward errors bias later updates.Repeated updates to a large pretrained VLM can also make adapting the world model computationally expensive.Source: §1 · pp. 1–2
  • Latent-space imagination avoids reconstructing observations at every predicted step, and pretrained VLM representations may support this policy learning.Predictive representations can emphasize predictable structure over incidental visual detail.Source: §1 · p. 1
  • DWELL recasts a pretrained VLM as a recurrent latent transition operator, sharing a representation interface for world modeling and policy learning.Separate lightweight LoRA adapters enable independent, parameter-efficient adaptation on the same VLM backbone.Source: §1 · p. 2
  • Across five interactive visual environments, DWELL outperforms trainable methods and reduces imagination compute relative to a world model that reconstructs observations.The comparison is reported under the training protocol described in the introduction.Source: §1 · p. 2
  • Real interactions continually refine the world and reward models, while imagined trajectories guide online policy improvement.The VLM serves as both world model and policy within the learning loop.Source: §1 · p. 2

2 METHOD

DWELL turns a pretrained VLM into a recurrent latent transition model that uses interaction history to predict states and improve behavior through latent imagination. Its world model and policy share a latent interface, and online interaction alternates with imagined planning to refine both prediction and action.

2.1 RECURRENCE AND DUALITY: A COMMON INTERFACE

  • The recurrent latent state uses visual tokens, prior state, and action to represent history for subsequent prediction and decision-making.The VLM's posterior incorporates the realized observation; its prior predicts the outcome without access to that observation.Source: §2.1 · p. 3
  • Prior and posterior outputs are deterministic latent representations, not probability distributions, and are distinguished by the information available to each update.The posterior uses the realized outcome; the action-conditioned prior anticipates it without the observation.Source: §2.1 · p. 3
  • The shared latent interface lets the policy act on posterior states during interaction and predicted states during imagination without decoding observations.Ordered latent slots serve transition prediction, reward evaluation, and action selection.Source: §2.1 · p. 3

2.2 DYNAMICS, REPRESENTATION, AND REWARD LEARNING

  • The action-conditioned prior learns to predict posterior states, while posterior representations are grounded through a projection of frozen pretrained visual features.This keeps learning in a compact representation space rather than reconstructing raw observations.Source: §2.2 · p. 4
  • Reward learning uses prior lookahead from a posterior state under future actions, bridging observed history and imagined rewards.The lookahead horizon ranges from per-step prediction at L = 0 to prior prediction decoupled from history at L ≥ T.Source: §2.2 · p. 4
  • The critic is warmed up on real experience before guiding policy improvement, with the actor and world model held fixed during warmup.This addresses the dependence of early finite-imagination updates on value estimates not yet fitted to task returns.Source: §2.2 · p. 5
  • Planning rolls out imagined actions, predicted states, and rewards from a posterior state, then uses PPO to update the policy and value function.The rollout batch is generated by sampling actions from the policy and advancing states with the action-conditioned prior.Source: §2.2 · p. 5

2 METHOD

  • DWELL alternates real interaction, which refines its world and reward models, with latent imagination that guides policy improvement.The learned policy returns to the environment to collect new experience and revise its beliefs.Source: §2 · p. 2, §2.2 · p. 5

3 EXPERIMENTS

Across five interactive visual environments, DWELL achieves the best overall performance among trainable methods and reduces computation relative to a visual-generation world model. Its benefits vary by task, and its objectives and continual model updates each contribute to learning from latent imagination.

3.1 EXPERIMENTAL SETUP

  • The evaluation spans five environments, from spatial planning and navigation to manipulation and long-horizon household tasks, using task success rate as the primary metric.Trainable methods share a Qwen2.5-VL-3B backbone and environmental feedback and interaction budget; off-the-shelf VLMs are evaluated without online adaptation.Source: §3.1 · pp. 5–6, Table 1 · p. 6

3.2 OVERALL PERFORMANCE

  • DWELL achieves the best overall performance among trainable methods, with gains most evident on SOKOBAN, FROZENLAKE, and EB-ALFRED.Model-free GRPO remains stronger on NAVIGATION and slightly better on PRIMITIVESKILL; DWELL consistently improves over adapted LEWORLDMODEL.Source: §3.2 · p. 6, Table 1 · p. 6

3.3 LEARNING AND COMPUTATIONAL EFFICIENCY

  • Across SOKOBAN and FROZENLAKE, DWELL reaches higher success rates than alternative model-based approaches, especially on FROZENLAKE.The reported results suggest recurrent latent imagination supplies learning signal from experience already collected, rather than relying solely on new environment rollouts.Source: §3.3 · p. 7, Figure 2 · p. 7
  • DWELL reduces GPU-hours by approximately 38–58% versus ADAWORLD across four environments, while LEWORLDMODEL remains least expensive and DWELL uses less compute than PPO and GRPO.The end-to-end measure includes world-model updates, imagined rollouts, and policy/value optimization under the same environment-specific schedule and hardware.Source: §3.3 · p. 7, Figure 3 · p. 7

3.4 ABLATION STUDIES

  • Removing either predictive dynamics learning or representation grounding degrades performance on SOKOBAN and NAVIGATION, with the larger drop occurring without Ldyn.Ldyn supervises action-conditioned state evolution; Lrep anchors latent states to perceptually meaningful visual representations.Source: §3.4 · p. 8, Figure 4 · p. 8
  • Freezing the world model yields competitive early performance but quickly saturates on SOKOBAN and FROZENLAKE, while full DWELL continues to improve with interaction.The authors attribute this to policy-induced distribution shift and say that refreshing the model keeps imagined trajectories aligned with the evolving policy distribution.Source: §3.4 · p. 8, Figure 4 · p. 8

4 RELATED WORK

Prior work shows that control can use compact latent world models and that generative world models can provide agent training experience. DWELL instead makes a VLM’s predicted beliefs the states on which its policy acts and learns.

Latent World Models for Behavior Learning

  • Compact latent world models and predictive-representation methods establish that control need not operate on reconstructed observations; DWELL studies this interface within a VLM agent.Its focus is on using the representations for real decisions as the state space for imagined policy learning.Source: §4 · p. 8

World Models for Agent Learning

  • Generative world models provide training environments for embodied and web agents, but a simulator held fixed during policy optimization cannot incorporate evidence from later policy updates.World-VLA-Loop adds new policy rollouts to world-model training, while DWELL uses predicted beliefs as interaction states and refreshes its latent world from real experience.Source: §4 · pp. 8–9

World Modeling within VLM Agents

  • Unlike approaches that use world modeling mainly to shape representations or action prediction, DWELL makes predicted states the states on which the policy continues to act and learn.A frozen VLM backbone supports separate trainable adapters for action-conditioned latent dynamics and policy learning, with real interaction supervising both.Source: §4 · p. 9

5 LIMITATION AND DISCUSSION

Latent imagination helps unevenly across tasks, and compact deterministic worlds face trade-offs in decision-relevant detail and rollout reliability. Evidence is limited to one VLM backbone and five simulated environments, with further constraints from frozen visual grounding and compute costs.

When Does Latent Imagination Help?

  • DWELL gains clearly on SOKOBAN, FROZENLAKE, and EB-ALFRED, but its advantage is smaller on NAVIGATION and PRIMITIVESKILL.The discussion suggests selective imagination rather than a fixed rollout horizon across states and tasks.Source: §5 · p. 9

What Must a Latent World Preserve?

  • Compact abstraction may discard fine-grained dynamics needed for control, while latent transition and reward errors can accumulate in long-horizon planning.The current deterministic dynamics do not represent uncertainty, so the policy cannot distinguish confident predictions from ambiguous ones.Source: §5 · p. 9

Scope and Scalability

  • Evidence covers one VLM backbone and five simulated environments, and uses a frozen pretrained visual encoder to ground the latent world.DWELL reduces end-to-end GPU-hours versus a generative world model, but simpler latent dynamics remain cheaper.Source: §5 · p. 9

6 CONCLUSION

DWELL recasts a VLM as a recurrent latent transition operator, sharing representations for prediction, value estimation, and action. Across interactive visual environments, it achieves strong performance under controlled interaction budgets while reducing imagination cost relative to generative world models.

6 CONCLUSION

  • DWELL enables VLM agents to learn from experience in their own latent worlds.Source: §6 · p. 9
  • DWELL uses a shared representational interface for prediction, value estimation, and action instead of reconstructing future observations.The framework interleaves real interaction with latent imagination to improve the policy without explicitly generating future observations.Source: §6 · p. 10
  • Across diverse interactive visual environments, DWELL achieves strong overall performance under controlled interaction budgets while substantially reducing imagination cost relative to generative world models.The conclusion argues that a useful world model need not reproduce the world, but should predict futures the agent can act and learn upon.Source: §6 · p. 10

A.1 ALGORITHM

The algorithm alternates online interaction, world-model learning, latent planning, and actor-critic updates. It initializes the replay buffer and critic before repeating these stages until convergence.

A.1 ALGORITHM

  • The loop gathers environment transitions, stores them in replay, and repeatedly updates the world and reward models.It begins by prefilling replay with a random policy, pretraining the world model, and warming up the critic on n-step returns.Source: §A.1 · p. 13
  • For latent planning, the algorithm samples posterior states from replay trajectories and unrolls the prior model for H steps to form imagined batches.Source: §A.1 · p. 13
  • The actor and critic learn from imagined batches using advantages and λ-returns, with joint updates over PPO epochs and minibatches.Source: §A.1 · p. 13

A.2 BASELINE

The baseline protocol matches initialization, interaction and evaluation budgets, and key optimization settings across trainable methods. It also describes the shared-latent-interface comparison and the environments' decision horizons and PrimitiveSkill setup.

A.2 BASELINE

  • Trainable methods share Qwen2.5-VL-3B initialization, real-interaction budgets, evaluation seeds, and matched environment-specific optimization settings.Methods differ in world-model representation and how they produce imagined transitions; evaluation levels are excluded from training and model selection.Source: §A.2 · pp. 13–14
  • Separating prediction and decision spaces degrades performance across the three evaluated environments, most markedly on SOKOBAN.Table 2 compares direct policy use of DWELL's predicted latent states with an adapter-bridged policy space; results are success rates averaged over three seeds.Source: Table 2 · p. 14
  • Environment horizons are maximum training or evaluation decisions; PrimitiveSkill includes task-state text and evaluates 64 episodes across four tasks.Each PrimitiveSkill episode has three language turns, with at most two primitive commands per turn.Source: Table 3 · p. 14

A.3 ADDITIONAL ABLATION STUDY RESULTS

DWELL uses a shared latent-state interface so the policy can consume predicted states directly. Separating prediction and decision spaces degrades performance across all three evaluated environments.

A.3 ADDITIONAL ABLATION STUDY RESULTS

  • DWELL’s shared latent interface lets predicted states pass directly to the policy without representation conversion.The bridged variant instead maps world-model states into a policy-specific latent space with a learned adapter.Source: §A.3 · p. 14
  • Separating prediction and decision spaces degrades performance across SOKOBAN, FROZENLAKE, and NAVIGATION, with the largest drop on SOKOBAN.The comparison uses a bridged variant with a learned adapter while keeping other components unchanged.Source: §A.3 · p. 14

A.4 ENVIRONMENT SETTINGS

The appendix specifies environment and training protocols, including reward assignment, latent regularization, reward and termination prediction, critic warm-up, and PPO-based latent planning.

A.4 ENVIRONMENT SETTINGS

  • PrimitiveSkill uses one environment decision per step, with each language turn allowing at most two sequential primitive commands.The reported setting covers four tasks and 16 held-out episodes per task.Source: §A.4 · p. 14

A.5 REWARD ASSIGNMENT

  • DWELL learns its feedback head from real transitions and uses its predicted rewards, rather than real environment rewards, in imagined trajectories.Offline episodes are sampled uniformly without reward-, success-, or difficulty-based prioritization.Source: §A.5 · pp. 14–15

A.6 KEY HYPERPARAMETERS

  • SOKOBAN and FROZENLAKE roll back candidate actor updates when held-out action-ranking accuracy falls below a threshold; EB-ALFRED uses scripted demonstrations for behavior-cloning initialization.EB-ALFRED also retains a behavior-cloning term during PPO.Source: §A.6 · p. 15

A.7.1 REGULARIZATION

  • The latent regularizer combines projected-distribution matching to a standard Gaussian with a variance floor on latent dimensions.The stated weights are λep = 0.05 and λvar = 0.005.Source: §A.7.1 · pp. 16–17

A.8 REWARD AND TERMINATION LEARNING

  • Reward and termination predictors are lightweight MLPs that mean-pool latent slots; reward outputs are Gaussian for continuous tasks or categorical for sparse or discrete rewards.The terminal predictor outputs a Bernoulli distribution.Source: §A.8 · p. 17

A.9.1 CRITIC WARM-UP

  • The critic is warmed up using n-step returns computed from inferred posterior states to provide a starting point for policy improvement.The passages identify the n-step returns as critic warm-up targets.Source: §A.9.1 · p. 17

A.9.2 MODEL-BASED LATENT PLANNING

  • PPO trains the actor and critic on imagined trajectories, reusing each batch multiple times to improve sample efficiency.Generalized advantage estimation supplies advantage and λ-return estimates for critic fitting and policy-gradient learning.Source: §A.9.2 · p. 17

Related papers

Loading 2610.36925v1…