PaperlayerView exact revision on arXiv

Source-linked AI summary

RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback

Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, Zackory Erickson

arXiv:2402.03681v4cs.ROcs.AIcs.LG

TL;DR

Reward engineering typically requires substantial human effort and trial-and-error. RL-VLM-F learns rewards from VLM preferences over image observations using only a task description, and the paper reports effective policies across diverse manipulation settings without human supervision, outperforming prior methods.

  • Problem

    Reward engineering requires extensive human effort and trial-and-error to design effective reward functions.

  • Method

    RL-VLM-F queries VLMs for preferences over pairs of visual observations conditioned on a task description, then learns a reward function from those labels.

  • Results

    RL-VLM-F generates effective rewards and policies across rigid, articulated, and deformable manipulation tasks and greatly outperforms prior methods.

  • Takeaways & Limitations

    The method removes extensive manual reward crafting and preference labeling while supporting policy learning across a wide range of manipulation tasks.

  • Takeaways & Limitations

    Biases in pretrained VLMs may be inherited by the learned reward and policy, so learned policies should be examined before safety-critical deployment.

Abstract

from arXiv · show

Reward engineering has long been a challenge in Reinforcement Learning (RL) research, as it often requires extensive human effort and iterative processes of trial-and-error to design effective reward functions. In this paper, we propose RL-VLM-F, a method that automatically generates reward functions for agents to learn new tasks, using only a text description of the task goal and the agent's visual observations, by leveraging feedbacks from vision language foundation models (VLMs). The key to our approach is to query these models to give preferences over pairs of the agent's image observations based on the text description of the task goal, and then learn a reward function from the preference labels, rather than directly prompting these models to output a raw reward score, which can be noisy and inconsistent. We demonstrate that RL-VLM-F successfully produces effective rewards and policies across various domains - including classic control, as well as manipulation of rigid, articulated, and deformable objects - without the need for human supervision, outperforming prior methods that use large pretrained models for reward generation under the same assumptions. Videos can be found on our project website: https://rlvlmf2024.github.io/

1. Introduction

RL-VLM-F addresses the human-intensive, trial-and-error process of reward engineering by automatically learning rewards from VLM preferences over visual observations. It requires only a task description and images, and is reported to outperform prior large-model reward-generation methods across diverse manipulation tasks.

  • Reward engineering requires substantial human effort and repeated trial-and-error to design functions that produce desired behavior.
  • Prior methods often require environment code or low-level state information, limiting their applicability to high-dimensional observations and complex deformable-object manipulation.
  • RL-VLM-F uses only a text description of the task goal and the agent’s visual observations to automatically generate reward functions.
  • Instead of requesting noisy raw reward scores, RL-VLM-F queries VLMs for preferences between image observations and learns a reward function from those labels.
  • RL-VLM-F generates rewards and policies for rigid, articulated, and deformable manipulation tasks and greatly outperforms prior methods.

2. Related Works

Related work replaces manually specified rewards with demonstrations, human preferences, large pretrained models, or contrastive vision-language alignment. RL-VLM-F instead uses VLM-generated comparisons to remove extensive human preference-labeling labor.

  • Inverse reinforcement learning typically learns rewards from expert demonstrations, whereas RL-VLM-F requires a task description instead.
  • Human-feedback methods learn rewards from pairwise trajectory preferences, but many comparisons may be needed because each conveys limited information.
  • RL-VLM-F has a VLM provide preference comparisons, removing the need for extensive human labor in labeling preferences.
  • Large pretrained models have been used to generate rewards or write reward code, but many tasks are difficult to specify through code.
  • Contrastive vision-language methods align images or trajectories with task descriptions, but their reward signals can be noisy and highly dependent on the task specification.

3. Background

The paper uses standard reinforcement learning and preference-based reward learning: agents maximize discounted returns while a parameterized reward model is trained from labeled comparisons and alternated with policy updates.

  • In the standard Markov decision process, an agent observes a state, selects an action, receives a reward, and transitions to the next state.
  • Preference-based RL learns a reward function from labels indicating which of two behavior segments is preferred, incomparable, or equally preferable.
  • The reward model assigns preference probabilities to segment pairs using the Bradley-Terry model.
  • The reward function is optimized by minimizing a preference-label loss over a dataset of comparisons.
  • Policy and reward updates alternate, with the policy trained using the learned reward; the implementation uses PEBBLE.

4. Assumptions

RL-VLM-F assumes VLMs can generalize across environments, compare multiple images, and assess task success from visual observations. Its procedure repeatedly collects images, queries preferences, trains the reward, and updates the policy.

  • Assumptions: The method assumes VLMs are trained on diverse text and image corpora and can generalize and reason across environments and tasks.
  • VLM querying: Preference generation uses analysis followed by labeling: the VLM first compares images in free form, then extracts a preference label from those responses.
  • Training procedure: The training loop collects image observations, samples image pairs, queries the VLM for labels, updates the reward, and relabels the replay buffer.
  • Assumptions: RL-VLM-F requires VLMs to process multiple images simultaneously and compare image pairs to generate preference labels.
  • Assumptions: The method is designed for tasks whose state quality or success can be judged from a single image or an image sequence.

5. Method

RL-VLM-F automates preference-based reward learning by querying a VLM about image pairs and using those preferences to train a reward model and policy.

  • Preference-Based Reward Learning: RL-VLM-F initializes a policy and reward function, then samples image pairs from the policy’s observation buffer for VLM labeling.The VLM judges which image better achieves the text-described task goal.
  • Policy Training: The resulting labels train the reward model, whose updated rewards are used to relabel SAC replay-buffer transitions for policy training.The policy-gradient update step is set to 1, and other algorithm parameters are specified in the appendix.
  • VLM Querying: A two-stage query first elicits comparative image analyses, then extracts a preference label from those textual responses.The stages are analysis followed by labeling.
  • Automation: A unified prompt template means adapting RL-VLM-F to a new environment requires only a task-goal description.Labels and the reward function are then trained automatically through the same process.

6. Experiments

RL-VLM-F is evaluated across seven tasks spanning classic control and rigid, articulated, and deformable manipulation, using image-and-text reward-generation baselines. It outperforms all baselines across tasks and closely tracks ground-truth preference and task progress, while deformable-task results show stronger final states.

  • 6.1. Setup: The benchmark contains CartPole, three MetaWorld manipulation tasks, and three SoftGym deformable-object tasks.The tasks cover classic control, rigid and articulated objects, and deformable objects.
  • 6.1. Setup: Compared methods include VLM Score, CLIP Score, BLIP-2 Score, RoboCLIP-Text, GT Preference, and RL-VLM-F under text-and-image assumptions.VLM Score directly predicts image scores, whereas GT Preference uses benchmark reward functions for preference labels.
  • 6.2. Does RL-VLM-F Learn Effective Rewards and Policies?: RL-VLM-F outperforms every baseline on all seven tasks and matches or surpasses GT Preference on six of seven tasks.Results are averaged over five seeds; RoboCLIP is evaluated only on MetaWorld tasks.
  • 6.2. Does RL-VLM-F Learn Effective Rewards and Policies?: On Sweep Into, RL-VLM-F surpasses GT Preference, which the authors associate with the ground-truth reward including a nonessential grasping term.RL-VLM-F instead uses the goal of minimizing cube-to-hole distance.
  • 6.2. Does RL-VLM-F Learn Effective Rewards and Policies?: RL-VLM-F achieves better final states than baselines on Fold Cloth, Straighten Rope, and Pass Water.It reaches a diagonal cloth fold, fully straightens the rope, and transports water without spilling.
  • 6.3. What is the Accuracy of VLM Preference Labeling?: VLM preference labels are more often correct than incorrect, and their accuracy is sufficient to learn effective rewards and policies.Accuracy is analyzed across ten bins of ground-truth task-progress differences between image pairs.
  • 6.4. How Does the Learned Reward Align With the Task Progress?: RL-VLM-F rewards align better with ground-truth task progress than VLM Score rewards on three MetaWorld expert trajectories.Rewards and task progress are normalized to [0, 1] for comparison, with averages over three reward-model seeds.

7. Conclusion and Future Work

RL-VLM-F automatically generates rewards from VLM preferences over task-related image observations, and is demonstrated across rigid, articulated, and deformable manipulation tasks. Future work considers more efficient querying, stronger VLMs, longer-horizon tasks, and real-world applications.

  • RL-VLM-F generates reward functions by querying VLMs with task descriptions and image observations.
  • The method is demonstrated on rigid, articulated, and deformable object manipulation tasks.
  • Future Work: Future work includes active learning for more efficient VLM queries and integration of more advanced VLMs.
  • Future Work: The authors also propose testing longer-horizon tasks through subtask decomposition and applying the approach in real-world settings.

Impact Statement

Because pretrained VLMs generate the rewards, their biases may pass into the reward function and learned policy. The authors recommend examining learned-policy behavior before safety-critical deployment.

  • Biases in pretrained VLMs may be inherited by the generated reward function and learned policy.
  • The authors recommend examining learned-policy behavior before deploying it in safety-critical applications.

A. Details on Tasks and Environments

The experiments combine state observations for policy learning with rendered RGB images for reward learning across classic control and object-manipulation environments. Tasks use environment-specific observation and action spaces, with SAC policy learning and image-based reward models.

  • Tasks and Environments: The benchmark includes CartPole, three MetaWorld manipulation tasks, and three SoftGym deformable-object tasks.
  • Observation Space: Policy learning uses state observations, while reward learning uses high-dimensional RGB images rendered by the simulator.
  • Observation Space: MetaWorld state observations have 39 dimensions, including robot, object, and goal information.
  • Observation Space: CartPole observations contain cart position and velocity plus pole angle and angular velocity in four dimensions.
  • Observation Space: Cloth Fold observations use the picker position and uniformly subsampled cloth-particle positions from a 40 x 40 mesh.
  • Observation Space: Straighten Rope uses all rope-particle positions, while Pass Water uses seven variables describing containers, target position, and water.
  • Action Space: Actions are environment-specific: MetaWorld uses four dimensions, CartPole uses continuous force, and manipulation tasks use pick-and-place or picker/container displacements.
  • Training Details: Reward models use CNNs for MetaWorld and CartPole, ResNet-18 for deformable tasks, three-model ensembles, and tanh reward outputs; SAC trains the policy.

B.3. Training details

Training details specify feedback scheduling, VLM prompting, baselines, and task-dependent model choices. GPT-4V performs better than Gemini-Pro on Fold Cloth, while direct VLM scoring and text-only RoboCLIP are evaluated as comparisons.

  • Feedback Schedule: Feedback learning schedules summarize queries per session, collection frequency, and maximum query budgets for each task.
  • VLM Comparison: On Fold Cloth, GPT-4V performs better than Gemini-Pro, possibly because cloth has complex visual appearance.
  • VLM Score: The VLM Score baseline asks the VLM for a 0-to-1 score and trains the reward model with regression loss instead of preference classification.
  • RoboCLIP: RoboCLIP uses an S3D video-language backbone but is evaluated here in a text-only variant on the three MetaWorld tasks.
  • Prompting: RL-VLM-F and VLM Score use unified query templates whose only task-specific component is the task-goal description.
  • Baseline Prompts: CLIP and BLIP-2 baselines use semantically similar task descriptions with different structure, while RoboCLIP follows its original format.
  • Ablation: An ablation compares the proposed two-stage prompting strategy with a closely matched single-stage prompt.

E.1. GT Task Reward (Oracle) and GT Sparse Reward (Oracle)

The study compares RL-VLM-F with oracle ground-truth task and sparse rewards using SAC. For most tasks, RL-VLM-F reaches final performance comparable to the ground-truth reward.

  • GT Task Reward (Oracle) uses the original human-written ground-truth reward function with SAC to train the policy.It provides a reference using the task’s original reward design.
  • Figure 10 reports learning curves for GT Task Reward (Oracle), GT Sparse Reward (Oracle), RL-VLM-F, and all baselines.The comparison includes both oracle reward settings alongside the proposed method and prior methods.
  • For most tasks, RL-VLM-F’s final performance can match that of using ground-truth reward.The authors identify this comparison as evidence of the method’s effectiveness.

E.2. Ablation Study: Influence of Using Different VLMs

The VLM ablation compares Gemini-Pro and GPT-4V on Fold Cloth and examines reward alignment across MetaWorld tasks. GPT-4V performs better on Fold Cloth, while RL-VLM-F’s learned reward aligns best with ground-truth task progress across all three visualized tasks.

  • Influence of Different VLMs: On Fold Cloth, GPT-4V achieves much better performance than Gemini-Pro.The authors use GPT-4V instead of Gemini-Pro for Fold Cloth in both RL-VLM-F and the VLM Score baseline.
  • Influence of Different VLMs: The poorer Gemini-Pro performance on Fold Cloth could be due to the more complex visual reasoning required for deformable cloth.This is presented as a possible explanation rather than a confirmed cause.
  • Reward Alignment: Across Open Drawer, Soccer, and Sweep Into, RL-VLM-F’s learned reward aligns best with ground-truth task progress among the compared methods.The comparison includes the VLM Score baseline, CLIP, and BLIP-2 scores along expert trajectories.
Loading 2402.03681v4…