PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning

Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, David Lindner

arXiv:2310.12921v2cs.LGcs.AI

TL;DR

RL reward specification is either difficult to hand-design or expensive to learn from human feedback. The paper proposes zero-shot VLM reward models using natural-language prompts, showing CLIP-based agents solve complex humanoid tasks and that larger VLMs provide better rewards, subject to generalization and visual-domain limitations.

  • Problem

    Manually specifying RL rewards is often infeasible, while learning reward models from human feedback is typically expensive.

  • Method

    VLM-RMs use pretrained VLMs, implemented with CLIP, to derive RL rewards from natural-language task prompts and optionally a baseline prompt.

  • Results

    VLM-RMs solve classic RL benchmarks and train a MuJoCo humanoid on complex tasks, while reward-model quality strongly improves with VLM scale and humanoid learning requires the largest publicly available CLIP model.

  • Takeaways & Limitations

    VLMs can serve as zero-shot reward models and may support increasingly sophisticated RL tasks specified by human-written descriptions.

  • Takeaways & Limitations

    The reward can be misspecified when prompts underspecify human intent or VLM generalization is poor, with risk increasing for complex tasks difficult to express in one prompt.

Abstract

from arXiv · show

Reinforcement learning (RL) requires either manually specifying a reward function, which is often infeasible, or learning a reward model from a large amount of human feedback, which is often very expensive. We study a more sample-efficient alternative: using pretrained vision-language models (VLMs) as zero-shot reward models (RMs) to specify tasks via natural language. We propose a natural and general approach to using VLMs as reward models, which we call VLM-RMs. We use VLM-RMs based on CLIP to train a MuJoCo humanoid to learn complex tasks without a manually specified reward function, such as kneeling, doing the splits, and sitting in a lotus position. For each of these tasks, we only provide a single sentence text prompt describing the desired task with minimal prompt engineering. We provide videos of the trained agents at: https://sites.google.com/view/vlm-rm. We can improve performance by providing a second "baseline" prompt and projecting out parts of the CLIP embedding space irrelevant to distinguish between goal and baseline. Further, we find a strong scaling effect for VLM-RMs: larger VLMs trained with more compute and data are better reward models. The failure modes of VLM-RMs we encountered are all related to known capability limitations of current VLMs, such as limited spatial reasoning ability or visually unrealistic environments that are far off-distribution for the VLM. We find that VLM-RMs are remarkably robust as long as the VLM is large enough. This suggests that future VLMs will become more and more useful reward models for a wide range of RL applications.

1 INTRODUCTION

The paper proposes using pretrained VLMs as zero-shot reward models to specify vision-based RL tasks through natural-language prompts. CLIP-based rewards solve benchmarks and complex humanoid tasks, with quality improving strongly as VLM scale increases.

  • Motivation: VLM-RMs provide a more sample-efficient and natural alternative to manually specified rewards or expensive human-feedback reward modeling.The approach uses pretrained vision-language models to ground reward signals in natural-language task descriptions.
  • Method: The proposed VLM-RM method uses pretrained VLMs as reward models, with CLIP cosine similarity between environment-state and task-prompt embeddings as one implementation.A baseline prompt can optionally regularize the reward by projecting representations along the baseline-to-target direction.
  • Evaluation: CLIP-based VLM-RMs correlate highly with ground-truth rewards and successfully train agents on CartPole and MountainCar.Reward quality also improves when environments are rendered with more realistic textures.
  • Evaluation: Single-sentence CLIP prompts train a MuJoCo humanoid on complex tasks including raising its arms, sitting in a lotus position, doing the splits, and kneeling.The tasks require no manually specified reward function beyond the natural-language descriptions.
  • Scaling: VLM scale strongly correlates with VLM-RM quality, and the humanoid tasks are learnable only with the largest publicly available CLIP model.This identifies underlying-model capability as a practical determinant of reward-model success.
  • Conclusion: The results indicate that VLMs can serve as powerful zero-shot reward models for increasingly sophisticated tasks specified by human-written descriptions.The authors relate remaining limitations to current VLM capabilities and anticipate broader usefulness as models become more capable.

2 BACKGROUND

The paper formulates vision-based RL as a partially observable Markov decision process and defines VLMs and CLIP encoders as the relevant modeling background.

  • Partially observable Markov decision processes: A POMDP is specified by state and action spaces, transitions, rewards, observations, an observation distribution, discounting, and an initial-state distribution.The tuple is (S, A, θ, R, O, ϕ, γ, d0).
  • Partially observable Markov decision processes: At each timestep, an action causes a probabilistic state transition, after which the agent receives an observation and reward; trajectories collect these states and actions.Returns are defined as discounted sums of rewards over finite-horizon trajectories.
  • Partially observable Markov decision processes: The agent seeks a policy maximizing expected trajectory returns under the reward function.The formulation allows stochastic policies and considers finite-horizon trajectories.
  • Vision-language models: Vision-language models process language and vision inputs, including bounded-length language strings and sequences of RGB images.The paper denotes language inputs by l ∈ L≤n and image inputs by i ∈ I≤m.
  • CLIP models: CLIP uses jointly trained language and image encoders that map captions and images into a shared latent space using contrastive learning.Matching image-caption pairs are encouraged to have similar embeddings, while non-matching pairs are separated.

3 VISION-LANGUAGE MODELS AS REWARD MODELS (VLM-RMS)

VLM-RMs use a VLM to convert natural-language task descriptions and visual observations into rewards for RL without manually engineering a reward function. The paper implements this with CLIP similarity and improves it through goal-baseline regularization.

  • Using VLMs as rewards: VLM-RMs offer a learning-free way to specify rewards from natural-language descriptions, avoiding manual reward engineering and expensive reward-model data collection.The formulation is intended as a drop-in reward signal for RL.
  • Using VLMs as rewards: The task is defined by a language description l, while a vision-based environment maps each state s to an image observation ψ(s).For example, “a humanoid robot kneeling” describes the target task.
  • Using VLMs as rewards: The method uses a VLM to provide the reward R(s) needed to train the RL agent.This replaces the need to design a task-specific reward function directly.
  • Using VLMs as rewards: The general formulation accepts an optional language context c and can encompass image- or video-encoding VLMs as reward models.Context can support interactive reward definition with a VLM.
  • CLIP as a reward model: The basic CLIP reward is the cosine similarity between the language embedding of the task description and the image embedding of the current environment state.The language and image encoders act as task and state encoders, respectively.
  • Goal-Baseline Regularization: Goal-baseline regularization projects state representations onto the line defined by baseline and goal embeddings to remove observation information irrelevant to distinguishing them.The baseline describes the environment’s default state, while the goal describes the target state.
  • Goal-Baseline Regularization: The regularized reward uses normalized goal, baseline, and state encodings, with α controlling regularization strength.At α = 0 it recovers the initial CLIP reward; at α = 1 it removes state components orthogonal to g − b.
  • Goal-Baseline Regularization: The baseline-to-goal direction is intended to capture the change from the environment baseline to the target, but α remains a hyperparameter because that direction may omit relevant information.Intermediate α values are reported as generally robust and often better than the unregularized setting.

4 EXPERIMENTS

Experiments show that CLIP-based VLM reward models can solve benchmark and complex humanoid tasks from language, but performance depends strongly on visual realism, regularization, and model scale. The method succeeds on most tested tasks, while remaining failures reflect limitations of the environment or current CLIP models.

  • Classic control benchmarks: CLIP reward models correlate highly with ground-truth rewards in CartPole and MountainCar, with the strongest correlation under goal-baseline regularization.More realistic textures further improve reward-model outputs.
  • Humanoid tasks: Zero-shot CLIP rewards enable reinforcement learning to learn most attempted humanoid tasks with little to no prompt engineering or hyperparameter tuning.The experiments use short language prompts for tasks that are challenging to specify manually.
  • Classic control benchmarks: CLIP reward landscapes are well-shaped around CartPole’s goal, but MountainCar becomes well-shaped only with realistic textures and goal-baseline regularization.For larger α, the MountainCar reward shape resembles the environment’s hill slope.
  • Classic control benchmarks: 100% task success rate is achieved in CartPole and textured MountainCar for most α regularization strengths, whereas untextured MountainCar cannot be successfully trained.The result supports the hypothesis that abstract visualizations are difficult for CLIP to interpret.
  • Humanoid tasks: 5 out of 8 tested humanoid tasks are solved with minimal prompt engineering and tuning; the remaining failures are hypothesized to reflect CLIP capability limitations.The imperfect tasks include hands on hips, standing on one leg, and arms crossed.
  • Scaling: VLM model scale strongly improves reward-model quality: EPIC distance improves close to log-linearly, but kneeling succeeds only with the largest evaluated CLIP model.The kneeling task has 0% success with smaller models and a sharp transition to successful learning with ViT-bigG-14.

5 RELATED WORK

The paper positions VLM-RMs as a zero-shot alternative to prior reward-modeling approaches that require fine-tuning or environment-specific data, extending language-grounded rewards to visual RL tasks.

  • VLM-RMs address visual RL tasks, which cannot always use language models operating on structured environment representations.
  • Unlike related methods requiring labeled, environment-specific data or carefully crafted robotics datasets, VLM-RMs use pretrained models directly.
  • The paper focuses on using CLIP’s zero-shot ability to specify rewards from natural-language task descriptions without fine-tuning.

6 CONCLUSION

The paper introduces VLM-RMs for training RL agents with pretrained vision-language models and reports successful benchmark and humanoid-task learning alongside a strong model-size scaling trend. It also identifies specification, generalization, robustness, and safety boundaries for practical use.

  • VLM-RMs use pretrained vision-language models as reward models with standard RL algorithms for classic benchmarks and simulated humanoid tasks.
  • A strong scaling trend with model size suggests larger VLMs may support reward modeling across a broader range of tasks.
  • Reward misspecification remains possible when prompts omit intended information or VLMs generalize poorly, especially for complex tasks described in one prompt.
  • Practical deployments require independent monitoring, with multi-step or dialogue-based reward specification recommended for complex tasks.
  • Future work includes environment-specific VLM fine-tuning, video-based reward models, and dialogue-enabled models for more complex task specification.
  • Robustness and safety against optimization pressure from RL agents remain important open questions for practical applications.

A COMPUTING AND INTERPRETING EPIC DISTANCE

The section defines EPIC distance and explains how, for goal-based tasks, it can evaluate CLIP reward models through their separation of goal and non-goal states.

  • EPIC distance compares reward functions without requiring policy training and is invariant among reward functions inducing the same optimal policy.
  • For goal-based tasks, the ground-truth reward assigns 1 to goal states and 0 to non-goal states, making EPIC distance easier to compute.
  • The CLIP reward model’s quality depends on the Pearson correlation between CLIP rewards and the ground-truth reward.
  • Positive Pearson correlation means CLIP assigns higher task-embedding cosine similarity to goal states than to non-goal states.
  • The experiments estimate EPIC distance under either a uniform state distribution or one induced by a pretrained expert policy.

B HUMAN EVALUATION

The paper evaluates humanoid policies through manual trajectory labeling and examines whether goal-baseline regularization improves unsuccessful tasks.

  • Manual evaluation labels how long each rollout satisfies task-specific criteria because humanoid tasks lack ground-truth reward functions.
  • Goal-baseline regularization did not improve humanoid tasks that failed without regularization at α = 0.
  • The authors suggest regularization may be more useful for smaller CLIP models, but human-evaluation costs prevented a more thorough study.
  • A second evaluation constructs a human-labeled state dataset from rollouts across checkpoints selected from the largest VLM training run.

C IMPLEMENTATION DETAILS & HYPERPARAMETER CHOICES

Implementation details cover the experimental setup and the integration of CLIP reward computation with batched reinforcement-learning training.

  • The implementation section describes experiment design, simulated-environment modifications, and reinforcement-learning hyperparameter choices.
  • Algorithm 1 integrates CLIP reward computation with a batched SAC algorithm.

C.1 CLASSIC CONTROL ENVIRONMENTS

The classic-control experiments use modified CartPole and MountainCar environments, CLIP-based reward computation, and evaluation procedures designed to test visual realism and reward quality.

  • Evaluation: Human rollout evaluation counts a trajectory as successful when task criteria hold for at least 50% of its timesteps.
  • Evaluation: Scaling evaluations compare differently sized CLIP models on humanoid kneeling, with a stark difference between ViT-H-14 and ViT-bigG-14.
  • Prompt setup: The listed goal and baseline prompts were the first prompts tried, without prompt engineering.
  • Reward computation: The SAC-with-CLIP algorithm precomputes the task embedding, encodes states with CLIP, computes cosine-similarity rewards, and trains SAC from a replay buffer.
  • Environment setup: The environments remove early termination conditions to prevent task-completion information from leaking through termination behavior.MountainCar’s goal state becomes absorbing after this modification.
  • RL algorithms: CartPole uses DQN, while MountainCar uses SAC because the former has discrete actions and the latter continuous actions.

C.2 HUMANOID ENVIRONMENT

Humanoid experiments use a fixed SAC training setup, with training over 10 million steps and evaluation based on saved checkpoints.

  • All humanoid experiments use SAC with the same hyperparameters tuned on preliminary kneeling-task experiments.The setup uses τ = 0.005, γ = 0.95, and learning rate 6 · 10−4.
  • 10 million training steps use episodes of 100 steps, with learning beginning after 50000 initial steps.The procedure performs 100 SAC updates every 100 environment steps.
  • Final evaluation uses the checkpoint with the highest training performance among saved checkpoints.Checkpoints are saved every 128000 steps.
Loading 2310.12921v2…