PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning

Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, Sergey Levine

arXiv:2405.10292v3cs.AIcs.CLcs.CVcs.LG

TL;DR

Visual instruction tuning may not efficiently train VLMs for multi-step, goal-directed decision-making in interactive environments, motivating a reinforcement-learning approach. The paper prompts chain-of-thought reasoning, parses text outputs into executable actions, and fine-tunes the entire VLM with task rewards; across tasks, 7b models outperform commercial models, while removing CoT substantially reduces performance.

  • Problem

    Visual instruction tuning may be limited for multi-step interactive decision-making because it trains on pre-collected datasets without environment interaction.

  • Method

    The framework prompts task-specific chain-of-thought reasoning, parses open-ended text into executable actions, and fine-tunes the entire VLM with environment task rewards.

  • Results

    Across diverse tasks, the method enhances VLM decision-making and enables 7b models to outperform commercial models such as GPT-4V and Gemini on most tasks.

  • Takeaways & Limitations

    CoT reasoning is a crucial component for enabling effective RL training and improving VLM decision-making across domains requiring visual recognition or visual semantic understanding.

  • Takeaways & Limitations

    The method’s environment-step cost makes scaling to Atari workloads impractical, requiring roughly half a year for 2 million steps.

Abstract

from arXiv · show

Large vision-language models (VLMs) fine-tuned on specialized visual instruction-following data have exhibited impressive language reasoning capabilities across various scenarios. However, this fine-tuning paradigm may not be able to efficiently learn optimal decision-making agents in multi-step goal-directed tasks from interactive environments. To address this challenge, we propose an algorithmic framework that fine-tunes VLMs with reinforcement learning (RL). Specifically, our framework provides a task description and then prompts the VLM to generate chain-of-thought (CoT) reasoning, enabling the VLM to efficiently explore intermediate reasoning steps that lead to the final text-based action. Next, the open-ended text output is parsed into an executable action to interact with the environment to obtain goal-directed task rewards. Finally, our framework uses these task rewards to fine-tune the entire VLM with RL. Empirically, we demonstrate that our proposed framework enhances the decision-making capabilities of VLM agents across various tasks, enabling 7b models to outperform commercial models such as GPT4-V or Gemini. Furthermore, we find that CoT reasoning is a crucial component for performance improvement, as removing the CoT reasoning results in a significant decrease in the overall performance of our method.

1 Introduction

The paper identifies a gap in using supervised visual instruction tuning to train VLMs for multi-step interactive decision-making and proposes end-to-end RL fine-tuning with chain-of-thought reasoning. Across five tasks and two domains, the method improves 7b VLM decision-making and surpasses commercial models such as GPT4-V and Gemini.

  • Motivation: Visual instruction tuning may be insufficient for multi-step interactive decision-making because it relies on supervised learning from pre-collected datasets without environment interaction.The paper notes that limited dataset diversity may leave decision-making scenarios underrepresented.
  • Motivation: End-to-end RL fine-tuning for goal-directed, multi-step VLM tasks had not been studied, despite RL’s use in interactive agents and text-based language-model tasks.
  • Framework: The framework prompts task-specific chain-of-thought reasoning before a text action, parses that action into an executable environment action, and uses resulting rewards for RL training.The environment supplies goal-directed rewards and the next state for training.
  • Evaluation: Across five tasks spanning fine-grained visual recognition, language reasoning, and visual semantic reasoning, the method enhances the decision-making capabilities of a 7b VLM.The evaluation covers an original domain and an embodied AI domain.
  • Evaluation: CoT reasoning is crucial for performance improvement, because removing it significantly decreases the method’s overall performance.

2 Related Work

The related work situates the paper among RL training of language and vision-language models, prompting-based decision-making agents, and evaluations of VLMs in interactive environments. The paper distinguishes its approach by directly fine-tuning the entire VLM for decision-making rather than relying only on prompting or frozen models.

  • RL training: Prior RL research trains LLMs and VLMs either with human-feedback reward models or with task-specific reward functions from environments.
  • RL training: The paper aligns with task-specific RL approaches but applies them to directly train VLMs for interactive decision-making.
  • Decision-making agents: Prompting-based studies aim to enhance large foundation-model decision-making, whereas this work directly uses RL to fine-tune the entire VLM.
  • Evaluation: Prior evaluations examine VLMs in non-interactive tasks, while this paper focuses on interactive environments requiring visual recognition and language reasoning.
  • CoT prompting: Chain-of-thought prompting has been shown to improve complex reasoning capabilities in language models across reasoning tasks.

3 Preliminaries

The paper formulates standard RL with an MDP and adapts it to VLMs by treating image-prompt pairs as states and generated text sequences as actions. VLM outputs are later parsed into executable environment actions.

  • Standard RL terminologies: An MDP is represented as M = {S, A, P, r, γ}, with state and action spaces, transition dynamics, rewards, and a discount factor.
  • Standard RL terminologies: The policy-learning objective is to learn a policy π : S → A that selects actions from states.
  • Standard RL terminologies: The maximum number of steps per episode is denoted by T.
  • Standard RL terminologies: The probability π(a|s) ∈ [0, 1] denotes the likelihood that policy π chooses action a in state s.
  • Adapting RL to VLMs: For VLMs, the state space combines RGB observations with input text, S = O × V_m, while the output text space V_n serves as the action space.
  • Adapting RL to VLMs: A VLM policy π_θ maps an image and prompt to an output sequence, whose probability is evaluated before the sequence is parsed into an environment action.

4 Training VLMs with RL

The framework fine-tunes VLMs with RL by prompting chain-of-thought reasoning, parsing generated text into legal environment actions, and using task rewards for policy optimization. It estimates action probabilities from output-token likelihoods while scaling the contribution of reasoning tokens.

  • 4.4 Formal Implementation: The parsed action interacts with the environment to produce a reward and next observation, which are stored for RL training.The training procedure collects on-policy replay data and applies PPO to update the VLM parameters.
  • 4.1 Prompt Design for Domain-Specific Outputs: VLMs generate chain-of-thought reasoning followed by a formatted text action from a task-specific prompt.The prompt includes task descriptions, observation-dependent information, legal actions, and a desired output format.
  • 4.2 Post-Processing Open-Ended Text for Legal Actions: A post-processing function extracts the requested action from open-ended output and maps it to a legal symbolic or text action.For example, it maps “stand” to a symbolic Blackjack operator and “look” to an Alfworld text action.
  • 4.2 Post-Processing Open-Ended Text for Legal Actions: If generated output lacks a legal action, the framework performs uniform random exploration over legal actions so RL training can continue.This fallback handles outputs that do not contain the required action keywords.
  • 4.3 Estimating Action Probabilities of VLM Policies: Summing log-likelihoods over all output tokens is undesirable because the reasoning tokens can dominate the action-token likelihood.The framework therefore scales the reasoning-token contribution with λ before estimating the action probability.
  • 4.3 Estimating Action Probabilities of VLM Policies: Extreme λ values near 1 or 0 degrade overall performance, so experiments use λ between 0.2 and 0.5.The scaling factor substantially affects final performance.

5 Evaluation Tasks

The evaluation spans the gym_cards and ALFWorld domains, testing fine-grained arithmetic and visual-language reasoning alongside embodied visual-semantic decision-making. Tasks range from deterministic number manipulation and formula construction to stochastic blackjack and state-dependent embodied actions, with image-based Atari benchmarks excluded for computational reasons.

  • Evaluation domains: The evaluation uses gym_cards for fine-grained visual recognition and arithmetic reasoning, and ALFWorld for embodied visual-semantic understanding.The two domains target complementary decision-making capabilities.
  • Evaluation scope: The study omits standard image-based Atari benchmarks because of limited computation resources.Such tasks generally require at least 2 million environment steps, while this method needs roughly 30 hours for 15k steps.
  • gym_cards: gym_cards includes deterministic NumberLine, EZPoints, and Points24 tasks with increasing complexity, plus stochastic Blackjack.The first three tasks assess processing numbers or mathematical operators; Points24 uses four cards and targets 24.
  • gym_cards: NumberLine requires moving a current number to a target using “+” and “−” actions, each changing the number by 1.The observation provides the target x and current number y_t.
  • gym_cards: EZPoints requires using each card number once to construct a formula evaluating to 12, with legal outputs appended to the evolving formula.Its action space includes natural numbers in [1] and the operators “+”, “∗”, and “=”.
  • ALFWorld: ALFWorld contains six goal-conditioned task types and state-dependent admissible actions executed through text-based navigation and interaction.Actions such as going to a shelf or examining a sidetable depend on the current state; some actions require earlier actions.

6 Experimental Results

The experiments evaluate the method across arithmetic and visual-semantic decision-making tasks, compare it with alternative methods, and examine the role and scaling of CoT reasoning. Results show consistent gains across evaluated tasks, while removing CoT substantially reduces performance and moderate λ values train best on NumberLine.

  • Evaluation scope: The evaluation covers deterministic and stochastic gym_cards arithmetic tasks plus visual-semantic reasoning in alfworld.The gym_cards tasks include NumberLine, EZPoints, Points24, and Blackjack; alfworld includes overall and task-specific evaluation.
  • Cross-task performance: 27.1% average improvement over LLaVA-sft on arithmetic tasks raises performance from 18.4% to 45.5%.
  • Cross-task performance: 4.0% average improvement on visual-semantic decision-making raises performance from 17.7% to 21.7%.
  • Comparisons: The method surpasses the second-best method by 14.0% on gym_cards and 2.3% on alfworld.The cited comparisons identify CNN+RL as second-best on gym_cards and GPT4-V on alfworld.
  • CoT ablation: Removing CoT significantly decreases performance and prevents improvement on the deterministic NumberLine and EZPoints tasks.
  • CoT scaling: Moderate λ values support effective NumberLine training, with the observed effective range at 0.3–0.5 and the typical optimum at 0.2–0.5.The method fails when λ is at least 0.7 or at most 0.1.

7 Conclusions, Limitations, and Future Directions

The paper concludes that direct RL fine-tuning with CoT improves VLM decision-making across visual-recognition and visual-semantic tasks. It also identifies limited prompting exploration and reduced gains in larger-action-space, more difficult environments as boundaries of the current results.

  • Conclusion: The framework directly fine-tunes VLMs with RL using CoT reasoning to improve decision-making across diverse domains.
  • Conclusion: 7b VLMs outperform GPT-4V and Gemini on most evaluated tasks.
  • Future directions: The study does not extensively explore different prompting techniques, leaving them as a future direction.
  • Limitations: Performance gains are limited by action-space size and task difficulty, with alfworld gaining less than gym_cards because it is multitask and has a larger action space.

B Additional Details of the Evaluation Tasks

The evaluation tasks define visual or multimodal states, discrete action spaces, task-specific transitions, and reward functions for NumberLine, EZPoints, Points24, Blackjack, and alfworld.

  • B.1.1 NumberLine: NumberLine requires moving a current integer to a target using sequential “+” or “−” actions.The current value changes by one per action and remains at the boundary when an action crosses it.
  • B.1.1 NumberLine: NumberLine ends at the target or after T = 2nmax steps, rewarding success with 1 and penalizing actions that fail to move closer with −1.
  • B.1.2 EZPoints: EZPoints asks the agent to use two observed cards to construct a formula equaling 12, with each card usable only once.Its action space contains card numbers, arithmetic symbols, and “=”.
  • B.1.3 Points24: Points24 extends the card task to four cards and target 24, using a larger action space that includes subtraction, division, and parentheses.
  • B.1.4 Blackjack: Blackjack presents dealer and player cards and requires choosing “stand” or “hit” to win the game.Rewards are 1, 0, or −1 for win, draw, or loss, with 1.5 for a blackjack.
  • B.1.5 alfworld: alfworld combines RGB images with text descriptions and admissible language actions for goal-conditioned embodied tasks.Rewards include task completion, subgoal achievement, and penalties for inadmissible actions.

C.2 Experimental Setup for Comparative Methods

Comparative experiments use GPT4-V, Gemini, LLaVA-sft, and a CNN-based RL method under specified evaluation and representation protocols.

  • Commercial baselines: GPT4-V and Gemini results were tested on March 15, 2024 using the same prompt as RL training.gym_cards averages use 200 deterministic-task episodes and 1000 Blackjack episodes; alfworld uses 1000 collected episodes for GPT4-V.
  • LLaVA-sft: LLaVA-sft fine-tunes LLaVA-1.6-7b for one epoch on task-specific instruction-following data and initializes downstream RL training.
  • CNN-based baseline: CNN-based RL uses the CLIP ViT-L/14 feature and adds RoBERTa-base text features for tasks requiring text inputs.The visual and text features are concatenated for downstream RL training.

C.3 General Setup for End-to-End RL Training

The experiments train VLMs end-to-end with PPO while varying hardware, seeds, transition collection, and evaluation averaging across tasks.

  • General setup: All experiments run on 8 A100 GPUs with a maximum VRAM requirement below 40G.Each Figure 5 and 6 curve takes at most 36 hours; training uses DeepSpeed zero2 for multi-GPU execution.
  • General setup: The VLM training updates all trainable components: the vision encoder, language model, and MLP projector.The implementation uses an open-source PPO implementation and a 3-layer MLP value component.
  • NumberLine and Blackjack: NumberLine and Blackjack use four GPUs, four random seeds, 512 transitions per PPO update per GPU, and total batch size 512.Returns and success rates are averaged over 200 NumberLine episodes and 1,000 Blackjack episodes.
  • NumberLine and Blackjack: Blackjack is stochastic whereas NumberLine is deterministic, with CNN-based baselines also averaged across four random seeds.Both CNN-based tasks use the same transition count and batch size for on-policy training.
  • EZPoints and Points24: EZPoints and Points24 use four GPUs, four seeds, 1,024 transitions per PPO update per GPU, and total batch size 512.The larger transition count accommodates their typically longer episode horizons.
  • ALFWorld: ALFWorld uses one GPU and four seeds, collecting 1,024 transitions per PPO update with batch size 256.Multi-GPU training is avoided because on-policy sampling time varies substantially across GPUs and would increase synchronization time.

C.4 Additional Experimental Results

The appendix provides additional return and task-specific success-rate results, plus qualitative Points24 failures involving perception and reasoning.

  • Additional results: Additional results include episode returns for gym_cards and task-specific training curves for alfworld.These results extend the main experimental visualizations with return and task-level performance views.
  • Additional results: Table 4 reports average episode success rates and returns for different gym_cards methods using each RL method’s best Figure 18 result.The table includes both CNN and proposed-method RL baselines.
  • Additional results: The appendix includes episode success-rate curves for the proposed method across alfworld task types.The curves use the same run as the alfworld experiments associated with Figure 5.
  • Failure cases: Points24 can fail through incorrect perception even when the generated reasoning produces a formula equal to 24.One example recognizes only two of four card numbers correctly while the reasoning remains correct.
  • Failure cases: Points24 can also fail when both card perception and reasoning are incorrect.One example recognizes three numbers correctly and generates 10+10+5+7, which does not equal 24.

D Details on Supervised Fine-tuning Data

The supervised fine-tuning data section documents task-specific demonstrations, with and without chain-of-thought reasoning, across gym_cards and alfworld environments.

  • Overview: The section provides detailed descriptions of supervised fine-tuning data for all evaluated tasks.Examples are organized around task-specific inputs, response formats, and CoT variants.
  • NumberLine: NumberLine uses 20k instruction-following expert samples collected with a task solver.The appendix presents examples both with and without CoT reasoning.
  • NumberLine: NumberLine prompts describe a target and current number in an image, then require adding or subtracting one through valid JSON actions.The task objective is to move the current number closer to the target.
  • Blackjack: Blackjack demonstrations use visual game states, allow stand or hit, and provide examples with and without CoT.The response is specified as a valid JSON file containing the selected action.
  • EZPoints: EZPoints uses 50k instruction-following expert samples collected with a task solver.The appendix includes supervised examples with and without CoT reasoning.
  • Points24: Points24 uses 50k solver-generated samples, but its supervised data distribution differs because every expert state has a viable solution whereas environment states may not.The examples include CoT and non-CoT formats, with card-game prompts requiring formula construction toward 24.
  • ALFWorld: ALFWorld data comes from 1k GPT4-V-generated episodes, totaling approximately 45k instruction-following samples for CoT fine-tuning.The environment lacks solvers, so the authors use the task prompt and scene descriptions; a non-CoT version is also collected.
Loading 2405.10292v3…