PaperlayerView exact revision on arXiv

Source-linked AI summary

From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignment

Yilin Wu, Ran Tian, Gokul Swamy, Andrea Bajcsy

arXiv:2502.01828v3cs.ROcs.LG

TL;DR

Generative robot policies can fail at deployment, while VLMs struggle to evaluate low-level actions because their representations differ from robot sensorimotor data. FOREWARN separates outcome prediction from evaluation using a latent world model and an aligned VLM, improving policy steering across manipulation tasks while remaining constrained by base-policy competence and world-model accuracy.

  • Problem

    VLMs struggle to verify low-level robot action outcomes because robot actions and VLM training representations differ fundamentally.

  • Method

    FOREWARN uses a latent world model for future-state prediction and aligns a VLM to evaluate those predicted states through natural-language reasoning.

  • Results

    FOREWARN improves the base generative imitation policy by over 30% and outperforms alternative VLM approaches by ∼40%.

  • Takeaways & Limitations

    Latent alignment enables VLM-in-the-loop policy steering that provides interpretable, reliable, and flexible steering across diverse manipulation tasks.

  • Takeaways & Limitations

    FOREWARN assumes the base policy is sufficiently competent, and its primary failures stem from imprecise world-model imagination worsened by limited training data.

Abstract

from arXiv · show

While generative robot policies have demonstrated significant potential in learning complex, multimodal behaviors from demonstrations, they still exhibit diverse failures at deployment-time. Policy steering offers an elegant solution to reducing the chance of failure by using an external verifier to select from low-level actions proposed by an imperfect generative policy. Here, one might hope to use a Vision Language Model (VLM) as a verifier, leveraging its open-world reasoning capabilities. However, off-the-shelf VLMs struggle to understand the consequences of low-level robot actions as they are represented fundamentally differently than the text and images the VLM was trained on. In response, we propose FOREWARN, a novel framework to unlock the potential of VLMs as open-vocabulary verifiers for runtime policy steering. Our key idea is to decouple the VLM's burden of predicting action outcomes (foresight) from evaluation (forethought). For foresight, we leverage a latent world model to imagine future latent states given diverse low-level action plans. For forethought, we align the VLM with these predicted latent states to reason about the consequences of actions in its native representation--natural language--and effectively filter proposed plans. We validate our framework across diverse robotic manipulation tasks, demonstrating its ability to bridge representational gaps and provide robust, generalizable policy steering. Videos can be found on the project website: https://yilin-wu98.github.io/forewarn/.

I. INTRODUCTION

Generative robot policies can fail or behave in ways misaligned with deployment context, motivating runtime steering without retraining. FOREWARN combines world-model outcome prediction with VLM-based verification through latent-space alignment.

  • Motivation: Runtime policies can knock over objects or produce behaviors misaligned with user preferences and task context.Examples include knocking down a cup during grasping and placing a gripper inside a cup of water when serving a guest.
  • Motivation: Runtime policy steering selects among candidate plans from an imperfect generative policy using an external verifier, avoiding retraining.The remaining challenge is autonomous steering under changing environments, task contexts, and base-policy performance.
  • Problem: Open-world steering separates action-outcome prediction from verification, but conventional formulations require difficult dynamics models and well-defined rewards.These requirements are challenging when dynamics are complex and steering criteria are hard to specify.
  • FOREWARN: FOREWARN uses a world model for latent future-state prediction and a VLM as an open-world verifier for nuanced task specifications.The framework addresses the representational mismatch between world models and VLMs by aligning the VLM with predicted latent states.
  • Results: FOREWARN improves a base generative imitation policy by over 30%, including novel task descriptions.The introduction reports extensive hardware experiments and robust filtering of proposed action plans.
  • Results: FOREWARN outperforms alternative VLM approaches by ∼40% when outcome prediction and verification are decoupled.The comparison is reported from hardware experiments.

II. RELATED WORK

Prior work addresses generative-policy failures through detection, monitoring, prediction, steering, and learning-to-search methods. FOREWARN instead performs local search inside a learned world model with a learned verifier.

  • Failure Detection, Monitoring & Prediction: Failure-handling methods include posthoc detection, runtime monitoring, and prediction before failures occur.Runtime monitors can identify nuanced failures, but monitoring activates only after the robot begins failing.
  • Policy Steering: Runtime policy steering improves generalist imitation policies without additional demonstration data by re-ranking candidate generations toward good outcomes.Previous verifiers include humans and Q-functions learned from large offline datasets; FOREWARN uses a VLM for autonomous, interpretable guidance.
  • Learning to Search: Learning to search plans action sequences at test time rather than merely imitating training demonstrations.This paradigm can reason about action consequences and potentially recover from mistakes.
  • Learning to Search: FOREWARN performs local search against a learned verifier inside a learned world model instead of extensive global search in the real world.This avoids the computational expense of global search and potential safety violations from real-world interaction.

III. PROBLEM FORMULATION

The paper formulates policy steering as stochastic model-predictive control over plans sampled from a base policy. Solving it requires predicting future outcomes and verifying their alignment with a language-specified task.

  • Problem Formulation: Policy steering is formalized as stochastic model-predictive control over action plans proposed by a base action-generation model.The objective selects the best plan from candidate generations.
  • Setup & Notation: The base policy maps observation sequences to a distribution over T-step end-effector action plans.Observations combine RGB images and proprioceptive states, while real-world execution produces future observations conditioned on the plan.
  • Problem: The steering objective samples K independent candidate plans from the base policy and returns the plan that optimizes the task objective.The candidate plans are evaluated under predicted future outcomes.
  • Prediction and Verification: Solving the objective requires forward-simulating each candidate plan to predict future observations and evaluating them with a task-conditioned reward.The language description specifies the desired behavior, and the reward measures alignment with it.
  • Challenges: Directly solving the coupled objective with a VLM is difficult because low-level action data lies outside current VLM training distributions and direct action-plan fine-tuning is sample-inefficient.The proposed strategy instead leverages VLM strengths while addressing the prediction-verification coupling.

IV. OUR APPROACH: FOREWARN

FOREWARN decouples future prediction from outcome evaluation by rolling candidate actions through a latent dynamics model and conditioning a fine-tuned VLM on the resulting latent states. The VLM evaluates these predicted outcomes against task descriptions.

  • FOREWARN: FOREWARN adopts a divide-and-conquer strategy that separates the VLM’s prediction of action outcomes from their evaluation.The framework uses latent dynamics models to predict lower-dimensional future states from high-dimensional robot data.
  • Training FOREWARN: The world model is pretrained as a Recurrent State Space Model on observations and actions, then kept frozen during VLM alignment.A linear layer projects learned latent embeddings into the text embedding space, while the projection layer and Llama model are fine-tuned with LoRA.
  • FOREWARN: A world model efficiently predicts diverse future outcomes that are otherwise difficult to model directly.The predicted latent states provide the basis for subsequent VLM evaluation.
  • VLM Latent-State Narration: FOREWARN maps latent world-model states into textual representations so the VLM can narrate and evaluate robot behavior.The alignment is framed as a visual question-answering task over sequences of predicted latent states.
  • VLM-in-the-Loop Steering: The steering objective uses a probabilistic latent dynamics model, an observation encoder, and a VLM-embedded reward to rank candidate plans.World-model rollouts approximate future-outcome expectations, while the fine-tuned VLM evaluates latent outcomes conditioned on the task description.

A. Foresight: Predicting Outcomes via Latent World Models

FOREWARN uses a latent world model to predict future outcomes of action plans in a lower-dimensional representation. Training on successful and failed rollouts enables prediction of both favorable and unfavorable outcomes, while the frozen model supports later policy steering.

  • DreamerV3 predicts future latent outcomes from robot observations and action plans.
  • The world-model dataset contains robot observation-action trajectories with outcomes induced by action plans.
  • Training includes successful and failed base-policy rollouts plus demonstrations, allowing the model to predict outcomes of both good and bad action plans.
  • The training loss encourages latent states that support image decoding and predict the next latent state conditioned on an action.
  • After training, the encoder and latent dynamics model are frozen for use in the policy-steering algorithm.

B. Forethought: Latent-Text Alignment for Outcome Reasoning and Policy Steering

FOREWARN aligns a VLM with latent world-model predictions so it can narrate and evaluate candidate robot outcomes in natural language. The same aligned VLM is then queried as a verifier to select action plans under nuanced task specifications.

  • The latent dynamics model predicts future latent states, allowing the VLM to evaluate imagined outcomes of candidate low-level action plans.
  • Because VLMs are not naturally equipped to reason about predicted latent states, FOREWARN treats the problem as latent-text alignment.
  • FOREWARN frames alignment as VQA, prompting the VLM to narrate robot behavior conditioned on world-model latent-state sequences.
  • The adapted VLM replaces image tokenization with the world-model encoder and latent dynamics model to connect latent representations with text embeddings.
  • Fine-tuning targets fine-grained behavior narrations, such as grasping a cup by its handle, to expose nuanced predicted behaviors and potential failures.
  • The fine-tuned VLM is queried again as a verifier to select the best action sequence using open-world reasoning and natural-language understanding.

T VLM

Policy steering translates candidate action plans into behavior narrations and asks the VLM to choose among them for a specified task. This uses the VLM’s implicit reward knowledge rather than requiring explicit reward assignment.

  • The constraint z_t = E_ϕ(o_t) encodes the current observation into the latent state used by the steering pipeline.
  • The mapping ψ translates a candidate action plan into its associated robot-behavior narration.
  • Rather than explicitly scoring predicted outcomes, the VLM selects the best plan among K candidates conditioned on the task description.
  • FOREWARN combines world-model predictions, behavior narration, and commonsense reasoning to guide plans toward task goals and deployment-context alignment.

V. EXPERIMENTS

The experiments evaluate FOREWARN on real-world manipulation tasks with multimodal behaviors, difficult interaction outcomes, and nuanced failures. The setup combines a teleoperation-trained diffusion policy, a learned world model, latent-to-text fine-tuning, and runtime aggregation of candidate plans.

  • Experiments study latent-to-language behavior translation, closed-loop policy steering, and robustness to novel task descriptions.
  • The real-robot setup uses a Franka manipulator with a 3D-printed gripper, wrist and third-person RGB cameras, and 15Hz end-effector and gripper control.
  • The Cup and Bag tasks contain diverse valid behaviors and hard-to-model interaction outcomes, while Fork-to-Bowl Transfer adds longer-horizon object interactions.
  • The base diffusion policy is trained on 100 teleoperated demonstrations per task and predicts 64-step action plans from visual and proprioceptive inputs.
  • The world model uses 250 trajectories per task, including successful and failed rollouts, and predicts 64 future latent states from the current observation and action plan.
  • The VQA fine-tuning data is constructed from trajectory snippets in the same offline dataset used for world-model training.
  • Latent states are downsampled to reduce semantic repetition, and LoRA fine-tuning keeps the encoder and latent dynamics model frozen.
  • For deployment, 100 sampled action plans are aggregated into 6 modes, whose most likely future latent predictions are passed to the VLM.

A. From Action Rollouts to Behavior Narration

FOREWARN studies how to translate predicted low-level action outcomes into behavior narrations that a VLM can evaluate. Explicit world-model prediction and VLM fine-tuning improve narration alignment over direct action-to-text or image-based baselines.

  • Baselines: FOREWARN compares latent-aligned narration against four baselines, including direct action narration, predicted-image narration, and oracle variants.FOREWARN-Oracle uses ground-truth future observations as an upper bound, while VLM-Act predicts narrations directly from observations and action plans.
  • Evaluation: The evaluation measures narration alignment with LLM Score and human-assessed GT Accuracy across 30 test rollouts per task.LLM Score ranges from 0 to 1, while GT Accuracy is binary.
  • Explicit outcome prediction: FOREWARN outperforms VLM-Act by over 50% on every task despite using the same fine-tuning dataset.VLM-Act achieves less than 50% GT Accuracy across all tasks, whereas explicit world-model prediction provides grounding for narration.
  • VLM fine-tuning: FOREWARN exceeds VLM-Img and VLM-Img-Oracle by at least 30% in GT Accuracy and 16% in LLM Score.The comparison indicates that even ground-truth visual observations do not match the performance of the fine-tuned latent-aligned approach.
  • Qualitative behavior narration: Qualitative examples show FOREWARN capturing nuanced motion and contact details that baselines often hallucinate or omit.In the Bag task, several baselines incorrectly describe grasping the bag’s edge instead of its middle.

B. Policy Steering for Open-World Alignment

FOREWARN uses world-model rollouts and aligned VLM reasoning to select action plans under task and environmental variation. It improves policy steering across training and novel contexts while retaining faster narration generation than VLM-Act.

  • System comparison: The system compares FOREWARN with baselines that remove explicit world-model narration, latent-to-text alignment, or VLM-based evaluation.The baselines include direct action narration, latent-state index prediction, and binary success classification.
  • Novel task descriptions: FOREWARN outperforms all baselines in both training-distribution and novel task contexts.Novel descriptions change task preferences, such as avoiding an oily cup handle or maximizing fork contact during grasping.
  • Policy steering: FOREWARN steers the base policy toward safe and preference-aligned behavior by using the VLM to interpret and evaluate predicted latent outcomes.Success is measured over 20 trials per method, with a trial requiring task completion and alignment with end-user preferences.
  • Environmental generalization: FOREWARN generalizes across six environmental variations in object color, object size, and table background with a small performance drop.These tests assess robustness to appearance and background changes beyond the training conditions.
  • Inference speed: FOREWARN’s complete inference time is 3.7 seconds, including 1.3 seconds to narrate six candidate plans, versus 22.0 seconds for VLM-Act.The system queries the VLM twice, and latent communication avoids image decoding and re-encoding between the world model and VLM.

VI. LIMITATIONS

FOREWARN’s main limitations concern assumptions about the base policy, world-model accuracy, inference cost, and visual reasoning capacity.

  • Base policy: FOREWARN assumes the base policy already contains the correct behavior, so it cannot identify when all generated plans are unsuitable.The paper calls for detecting this case and improving the base policy with targeted fine-tuning data.
  • World model: World-model failures primarily arise from imprecise imagination of real-world interaction dynamics, worsened by limited training data.The authors suggest stronger visual features and larger generalizable manipulation world models as possible improvements.
  • Inference: Inference overhead is a bottleneck because FOREWARN combines large autoregressive language and world models.Suggested directions include hierarchical execution, quantization and caching, distillation, and selective allocation of reasoning time.
  • VLM backbone: The Llama-3.2-11B-Vision-Instruct backbone has weaker visual reasoning than language-based commonsense reasoning.A stronger backbone could improve behavior descriptions, particularly for video captioning.

VII. CONCLUSION

The paper presents FOREWARN as a framework for using VLMs as open-vocabulary verifiers by separating future prediction from outcome assessment. Experiments across manipulation tasks report interpretable failure detection, higher policy success rates, and flexible steering.

  • Framework: FOREWARN decouples policy steering into foresight and forethought using an explicit world model and latent-space alignment.This lets VLMs reason about sensorimotor data through natural language.
  • Results: Experiments across diverse manipulation tasks report interpretable, reliable failure detection and significantly improved policy success rates.The reported steering is flexible and generalizable across the evaluated settings.
  • Implication: Combining learned dynamics models with language-based reasoning is presented as a way to improve test-time robot-policy performance.

A. Algorithm & Implementation Details

The implementation uses a diffusion policy, a latent world model, and a modified VLM to narrate and evaluate imagined action outcomes. Supplementary monitoring experiments assess generalization across task descriptions and behaviors.

  • Policy and action plans: Diffusion Policy generates multimodal action behaviors from high-dimensional observations, while sampled trajectories are aggregated into 6 modes using time-series K-Means with DTW.Each trajectory is represented as a T × 7 sequence of end-effector position and quaternion orientation actions.
  • World model: FOREWARN uses DreamerV3 with an encoder, recurrent latent dynamics model, and decoder to predict future latent states from observations and actions.The world model is pretrained on demonstrations and policy rollouts, using dynamics, representation, and prediction losses.
  • Latent-text alignment: The modified VLM is fine-tuned to narrate predicted latent states, then evaluates those behavior narrations under task descriptions to select or monitor action plans.During steering, each of the 6 action-plan modes is translated into a narration for VLM evaluation.
  • Monitoring results: FOREWARN maintains over 75% failure-monitoring accuracy with balanced TPR and TNR across tasks and varying task descriptions.End-to-end baselines perform well on familiar descriptions but drop sharply on novel descriptions.
  • Analysis: The reported generalization is attributed to decoupling behavior narration from outcome evaluation.This separation preserves VLM reasoning while avoiding direct evaluation of low-level action sequences.
  • Qualitative comparison: FOREWARN consistently produces accurate descriptions and monitoring results, whereas VLM-Act is failure-biased and other baselines ignore differences among action plans.
Loading 2502.01828v3…