Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning

Yingdong Hu; Fanqi Lin; Tong Zhang; Li Yi; Yang Gao

Source-linked AI summary

Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning

Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, Yang Gao

arXiv:2311.17842v2Nov 29, 2023cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

Existing LLM-based planners lack world grounding and rely on external affordance models that cannot jointly reason with them. VILA uses visual observations and language goals to generate actionable plans with closed-loop execution, and outperforms LLM-based baselines across real-world and simulated manipulation tasks.

Problem
LLM-based planners lack world grounding and depend on external affordance models to perceive environmental information, limiting unified reasoning about scenes.
Method
VILA prompts vision-language models with current visual observations and high-level language goals to generate step-by-step plans, selecting and executing actions in a closed loop.
Results
VILA consistently outperforms LLM-based planners across 16 real-world and 16 simulated manipulation tasks; across 8 evaluated tasks, it achieves 80% average success versus 13% for SayCan and 20% for GD.
Takeaways & Limitations
VILA supports visual-world commonsense reasoning, multimodal goal specification, and visual feedback for complex long-horizon robotic manipulation.
Takeaways & Limitations
VILA assumes all necessary single-step primitive skills already exist, while its black-box VLM limits steerability and error explanation.

Abstract

from arXiv · show

In this study, we are interested in imbuing robots with the capability of physically-grounded task planning. Recent advancements have shown that large language models (LLMs) possess extensive knowledge useful in robotic tasks, especially in reasoning and planning. However, LLMs are constrained by their lack of world grounding and dependence on external affordance models to perceive environmental information, which cannot jointly reason with LLMs. We argue that a task planner should be an inherently grounded, unified multimodal system. To this end, we introduce Robotic Vision-Language Planning (ViLa), a novel approach for long-horizon robotic planning that leverages vision-language models (VLMs) to generate a sequence of actionable steps. ViLa directly integrates perceptual data into its reasoning and planning process, enabling a profound understanding of commonsense knowledge in the visual world, including spatial layouts and object attributes. It also supports flexible multimodal goal specification and naturally incorporates visual feedback. Our extensive evaluation, conducted in both real-robot and simulated environments, demonstrates ViLa's superiority over existing LLM-based planners, highlighting its effectiveness in a wide array of open-world manipulation tasks.

I. INTRODUCTION

The paper targets long-horizon robotic planning that adapts to changing visual scenes, addressing the grounding limits of language-only planners and one-way affordance modules. It introduces VILA, which uses a unified VLM to plan from visual observations and supports commonsense reasoning, multimodal goals, and visual feedback.

Motivation: Humans adapt task behavior to scene context, motivating robots that can perform long-horizon planning across varying environments.For example, obtaining a coke can may involve immediate pickup or searching storage locations depending on visibility.
Limitations of Existing Planners: LLMs provide world knowledge for step-by-step planning but cannot directly perceive physical states, object properties, or real-world constraints.
Limitations of Existing Planners: External affordance models ground LLMs through one-directional perceptual channels but may omit task-dependent information in complex environments.
VILA: VILA directly prompts VLMs to generate actionable plans from visual observations and high-level language instructions, avoiding independent affordance models.
VILA: VILA supports commonsense reasoning about spatial layouts and object attributes, flexible language-and-image goals, and visual feedback for closed-loop planning.
Evaluation: VILA was evaluated on 16 real-world and 16 simulated manipulation tasks and consistently outperformed LLM-based planners such as SayCan and Grounded Decoding.

III. METHOD

The planning problem is to generate short-horizon text actions for a long-horizon manipulation instruction, assuming the required primitive skills already exist. The method takes visual state and language input, repeatedly selects and executes the first VLM-generated step until completion.

Problem Formulation: The planner receives an initial visual observation, a high-level instruction, and a set of available skills.
Problem Formulation: The formulation assumes the visual observation accurately represents the world state and allows instructions to be long-horizon or under-specified.
Planning Procedure: VILA repeatedly prompts a VLM for plan steps, selects the first step, executes its corresponding skill, updates the observation, and stops when the plan reaches “done”.
Problem Formulation: The objective is to generate a sequence of short-horizon text actions, each specifying a subtask associated with a primitive skill.

B. Vision-Language Models as Robot Planners

VILA addresses the grounding problem by using a VLM to combine visual and language information when decomposing high-level instructions into executable skills. It performs closed-loop replanning after each executed step and uses GPT-4V to support challenging zero-shot planning.

Grounded Planning: External affordance models are separate grounding channels, whereas VLMs can jointly reason over images and language for physical-world planning.
VILA Pipeline: VILA prompts a VLM with the current visual observation and language goal to produce a step-by-step plan, then executes its first step as a text action.
VILA Pipeline: After execution, VILA adds the selected action to the query and repeats planning until a termination token such as “done” is reached.
Implementation: The approach uses GPT-4V and can solve varied challenging planning problems in zero-shot mode without in-context examples.
Visual Commonsense: VILA emphasizes visual information because images provide scene details that language-centered planners often overlook.

1) Spatial Layout Understanding:

VILA uses visual reasoning to interpret spatial relationships, object attributes, and multimodal goals that language-only or externally grounded planners may miss. It also incorporates direct visual feedback for correction and replanning.

1) Spatial Layout Understanding:: VILA reasons over spatial relationships and environmental constraints, enabling it to identify when an occluding object must be moved first.This supports plans such as opening a container before retrieving an object hidden inside.
2) Object Attribute Understanding:: VILA jointly reasons about visual and linguistic information to assess task-dependent object attributes, such as whether scissors are hazardous or necessary.Standalone affordance models provide only a limited, unidirectional subset of object attributes.
2) Object Attribute Understanding:: VILA accepts current observations, language instructions, and goal images, including images from different domains such as photographs, drawings, or pointing targets.This enables image-only or combined image-and-language goal specification.
2) Object Attribute Understanding:: VILA uses visual feedback to recognize object states, detect task success, and correct or replan when the environment changes or a skill fails.The VLM serves both as a scene descriptor and a success detector within the planning loop.

IV. EXPERIMENTS AND ANALYSIS

The evaluation combines real-world and simulated tabletop experiments to assess VILA across diverse long-horizon manipulation tasks. The real-world study uses strict zero-shot GPT-4V planning, while quantitative simulation results compare VILA with baselines on commonsense-heavy tasks.

IV. EXPERIMENTS AND ANALYSIS: The study evaluates VILA through extensive real-world experiments followed by quantitative comparisons against baseline methods in a simulated tabletop environment.The experiments are organized into real-world everyday manipulation evaluation and simulated baseline comparison.
IV. EXPERIMENTS AND ANALYSIS: The real-world setup uses a 7-DoF Franka Emika Panda arm, a 1DoF parallel-jaw gripper, and a fixed-view Logitech Brio camera.The fixed camera view is maintained for consistency across tasks.
IV. EXPERIMENTS AND ANALYSIS: Table I reports VILA’s superior performance on tasks requiring rich commonsense knowledge about spatial layouts and object attributes.The table separates spatial-layout tasks in its top half from object-attribute tasks in its bottom half.
IV. EXPERIMENTS AND ANALYSIS: 16 real-world long-horizon tasks cover visual commonsense, flexible goal specification, and visual feedback across 10 environmental variations per task.Variations include changes in scene configuration and lighting conditions.
IV. EXPERIMENTS AND ANALYSIS: GPT-4V generates plans in a strict zero-shot setting using high-level language instructions and simple robot constraints without in-context examples.The full prompt is provided in the appendix.

4) Primitive Skills:

VILA uses composable primitive skills for long-horizon manipulation, supporting multimodal goals and visual-feedback-driven replanning. Across real-world tasks, it outperforms LLM-based baselines by integrating visual commonsense reasoning into planning.

Primitive Skills: VILA composes five primitive skills—picking, placing, opening, closing, and pouring—for high-level temporally extended planning.Both VILA and the baselines use script policies for these primitive skills.
Real-World Evaluation: 80% average success across 8 tasks outperforms SayCan at 13% and Grounded Decoding at 20%.The tasks require visual reasoning about spatial layouts and object attributes.
Visual Commonsense Reasoning: VILA resolves spatial configurations and contextual object attributes that cause SayCan to choose unsuitable actions.Examples include relocating objects before picking up a plate and distinguishing hazardous tools from scissors needed for class.
Error Analysis: VILA significantly reduces baseline understanding errors by integrating vision and language reasoning.Understanding errors include failures involving occlusions, complex spatial layouts, and context-specific object attributes.
Multimodal Goals: VILA interprets diverse image and image-plus-language goals, including reference arrangements, drawings, pointing cues, and object swaps.It demonstrates these capabilities across four multimodal-goal tasks.
Evaluation Scope: VILA evaluates 16 simulated tasks grouped into Blocks & Bowls and Letters, alongside its real-world manipulation evaluation.The simulated task suite is based on the RAVENS environment.
Visual Feedback: Closed-loop VILA substantially outperforms open-loop planning on dynamic tasks by replanning from visual feedback.In Find Stapler, it searches the bottom drawer after the stapler is absent from the top drawer.

B. Simulated Tabletop Rearrangement

The simulated evaluation tests robotic rearrangement across seen and unseen RAVENS tasks against imitation-learning, ungrounded LLM, and grounded-decoding baselines. VILA consistently outperforms these alternatives, supporting synergistic vision-language reasoning for high-level planning.

Experimental Setup: The RAVENS evaluation contains 16 simulated tabletop rearrangement tasks, including 6 seen and 10 unseen tasks.A UR5 robot performs object rearrangement in these tasks.
Baselines: Baselines comprise CLIPort imitation learning, ungrounded Llama 2 and GPT-4 planners, and Grounded Decoding with an affordance model.Grounded Decoding uses Llama 2 and task-specific affordance values or CLIPort logits.
Results: VILA consistently outperforms baselines across seen and unseen simulated tasks.Table IV averages success rates over 20 episodes per task within each category.
Results: Grounded Decoding exceeds other LLM-based planners but significantly trails VILA, highlighting the benefit of synergistic vision-language reasoning.Llama 2 and GPT-4 show comparable performance across the tasks, supporting the comparison between Grounded Decoding and VILA.

V. CONCLUSION, LIMITATIONS, & FUTURE WORKS

VILA uses vision-language models to plan complex, long-horizon robotic tasks by integrating perception into reasoning. The authors identify limitations involving primitive-skill availability, black-box VLMs, and prompt design.

VILA decomposes high-level language instructions into actionable steps while integrating perceptual information into planning.
The approach supports visual-world commonsense reasoning, multimodal goals, and visual feedback across real-world and simulated tasks.
VILA’s limitations include assuming all single-step primitive skills exist, which leaves robust low-level control outside the study’s scope.The authors identify transferable web knowledge as a possible route toward more generalizable robotic skills.
Dependence on a black-box VLM limits steerability and makes some errors difficult to explain.The authors suggest parameter-efficient fine-tuning as a possible future direction.
Excluding in-context examples makes output more versatile but can leave output consistency for future prompting methods to improve.

APPENDIX

The appendix describes the real-robot setup, evaluation design, zero-shot prompting, and the prompt’s scene-dependent planning instructions. The study evaluates varied environments using a tabletop camera and repeated trials.

The real-robot platform uses a 7-DoF Franka Emika Panda arm, a 1DoF parallel-jaw gripper, and a tripod-mounted Logitech Brio camera.The camera captures 1920 × 1080 color images of the tabletop.
The real-world evaluation contains 16 long-horizon tasks spanning visual commonsense, multimodal goals, and visual feedback.Each task is evaluated 10 times under scene, lighting, and other environmental variations.
The prompt uses only high-level language instructions and simple constraints, without in-context examples, constituting strict zero-shot evaluation.
The prompt instructs the robot to manipulate visible objects directly, search for unseen objects, remove blockers, and ignore irrelevant distractors.
Figure A.1 presents the prompt used for the real-world environment.

4) Primitive Skills:

The study composes scripted primitive skills to evaluate high-level robotic planning across real and simulated task suites. These tasks test visual commonsense, multimodal goals, and visual feedback, with simulated prompts adapted to object-recognition constraints.

The planner composes five primitive-skill categories: picking up, placing, opening, closing, and pouring objects.The study focuses on high-level temporally extended planning rather than learning low-level skills.
The task suite tests spatial-layout and object-attribute understanding, alongside multimodal goals and visual feedback.Tables A.1–A.3 organize these tasks by their required capability.
The simulated RAVENS suite contains 16 rearrangement tasks in which a UR5 robot configures tabletop objects from language instructions.A top-down camera supplies images for task planning.
The simulated tasks use Blocks & Bowls and Letters-style object configurations, with spatial and word-based descriptors such as corners, colors, and object names.
Blocks & Bowls prompts include three in-context examples because GPT-4V cannot recognize those objects reliably in a zero-shot setting.Letters prompts omit examples because GPT-4V recognizes them proficiently.

2) Prompts:

The evaluation compares planner and policy baselines using simulator primitives designed to isolate high-level planning quality. Script policies replace CLIPort primitives because they provide cleaner execution measurements.

3) Primitive Skills:: The study uses script policies instead of CLIPort primitives because CLIPort can execute objects correctly even when the planner generates an incorrect step.
3) Primitive Skills:: The baseline design separates high-level instruction planning from low-level policy execution when assessing planner capabilities.
3) Primitive Skills:: The comparison includes direct CLIPort policies, ungrounded LLM planners, and Grounded Decoding as three baseline categories.
3) Primitive Skills:: CLIPort baselines include Short and Long variants trained on single-step or high-level instructions, respectively.Both receive only high-level instructions during evaluation.
3) Primitive Skills:: Llama 2 70B and GPT-4 represent LLM-based planners without grounding or affordance models.Llama 2 is included because Grounded Decoding requires token probabilities unavailable through the OpenAI API.

4) Baselines:

The simulation baseline evaluation covers seen and unseen tasks, with success rates averaged over 20 episodes. Prompts specify object-identification and primitive-action planning requirements for Blocks & Bowls and Letters tasks.

Seen and unseen tasks: Simulation results distinguish ‘seen’ tasks from ‘unseen’ tasks using blue and orange background colors, respectively.Seen tasks support supervised-baseline training or high-level-planner prompts, with an exception for VILA’s Letters prompt.
Evaluation protocol: Each table entry reports success rate averaged across 20 episodes.
Blocks & Bowls: Blocks & Bowls prompts require counting, color and object identification, followed by one-block-at-a-time placement actions.The specified command is to pick up a colored block and place it on a designated place, without picking up multiple stacked blocks.
Letters: Letters prompts require identifying letters in images and converting tasks into primitive placement skills.The listed skills place a letter at a corner or side, or to the right of a previous letter.