PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning

Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, Yang Gao

arXiv:2311.17842v2cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

Robots need to adapt long-horizon plans to changing scenes, as humans do. VILA uses visual observations and language goals to generate actionable steps, and evaluations report that it outperforms LLM-based planners across real-world and simulated tasks.

  • Problem

    The paper investigates how robots can emulate human-like adaptability by performing long-horizon task planning across varying scenes.

    Source: §I · p. 2

  • Method

    VILA prompts a vision-language model with visual observations and a high-level language goal to generate a step-by-step plan, executing its first step for closed-loop planning.

    Source: §I · p. 2, B. Vision-Language Models as Robot… · p. 3

  • Results

    Across 16 real-world and 16 simulated tasks, VILA consistently outperformed LLM-based planners; on 8 tasks, average success rate was 80%, versus 13% for SayCan and 20% for GD.

    Source: §I · p. 2, 5) Baselines · p. 6

  • Takeaways & Limitations

    Evaluations support VILA's effectiveness for varied, complex, long-horizon robotic tasks in real-world and simulated settings.

    Source: §V · p. 9

    VILA presupposes that all single-step primitive skills are available, leaving robust low-level control policies outside its scope.

    Source: §V · p. 9

Abstract

from arXiv · show

In this study, we are interested in imbuing robots with the capability of physically-grounded task planning. Recent advancements have shown that large language models (LLMs) possess extensive knowledge useful in robotic tasks, especially in reasoning and planning. However, LLMs are constrained by their lack of world grounding and dependence on external affordance models to perceive environmental information, which cannot jointly reason with LLMs. We argue that a task planner should be an inherently grounded, unified multimodal system. To this end, we introduce Robotic Vision-Language Planning (ViLa), a novel approach for long-horizon robotic planning that leverages vision-language models (VLMs) to generate a sequence of actionable steps. ViLa directly integrates perceptual data into its reasoning and planning process, enabling a profound understanding of commonsense knowledge in the visual world, including spatial layouts and object attributes. It also supports flexible multimodal goal specification and naturally incorporates visual feedback. Our extensive evaluation, conducted in both real-robot and simulated environments, demonstrates ViLa's superiority over existing LLM-based planners, highlighting its effectiveness in a wide array of open-world manipulation tasks.

I. INTRODUCTION · II. RELATED WORK

VILA aims to give robots human-like, scene-aware long-horizon planning by reasoning directly over visual observations and language instructions. The related work contrasts this approach with planners that rely on separate grounding modules, low-level control methods, or training for new environments.

I. INTRODUCTION

  • VILA targets scene-aware adaptability because LLMs lack physical-world grounding and separate affordance models can miss task-relevant visual information, risking infeasible or unsafe plans.The paper motivates joint vision-language reasoning with examples involving obstacles and task-dependent interpretations of scissors.Source: §I · pp. 1–2
  • VILA prompts vision-language models to turn environmental visual observations and high-level instructions into actionable steps, without independent affordance models.It is designed for long-horizon robotic planning.Source: §I · p. 2
  • VILA supports language, image, or combined language-and-image goals, and uses visual feedback for closed-loop planning in dynamic environments.Its visual commonsense is intended to capture spatial layouts and object attributes.Source: §I · p. 2
  • VILA consistently outperforms SayCan [2] and Grounded Decoding [40] across 16 real-world manipulation tasks and shows marked enhancements on 16 simulated RAVENS tasks.Source: §I · p. 2

II. RELATED WORK

  • Unlike VQA, image captioning [1] [34], and OCR, this work applies VLM world knowledge to long-horizon robotic planning; high-level visual semantics alone may not ensure control generalizability [35] [91] [30].Prior VLMs align pretrained visual encoders with LLM input embeddings using crossmodal connectors [3] [53] [52] [10] [47] [36] [90] [97] [5] [81].Source: §II · p. 2
  • Unlike LLM-based robotic planners using affordance models [40], perception APIs [49], or textual scene descriptions [38] [94], VILA integrates perception directly into planning without those auxiliary grounding modules.Source: §II · p. 3
  • RT-2 [8] applies VLMs to low-level control, whereas VILA focuses on high-level planning; unlike PaLM-E [18], VILA is open-world and zero-shot without added training data or in-context examples.PaLM-E requires training on robotics and general visual-language data [11] [54].Source: §II · p. 3
  • Where traditional TAMP combines continuous motion planning with discrete task planning, VILA uses visually grounded commonsense to handle complex tasks beyond previous LLM-based planners.TAMP [26] has also drawn on symbolic planning, optimization, machine learning [20] [25] [41], and LLMs [16] [12] [72] [86].Source: §II · p. 3

III. METHOD · A. Problem Statement

ViLa plans manipulation tasks by turning visual observations and high-level language instructions into short-horizon actions that invoke available skills. The formulation permits long-horizon or under-specified instructions and assumes the observation accurately represents the world state.

III. METHOD

  • The method section proceeds from the planning-problem formulation to VILA’s use of VLMs as planners, then to properties contributing to its advantages.Source: §III · p. 3

A. Problem Statement

  • The robot receives a visual observation assumed to accurately represent the world state and a high-level language instruction describing a manipulation task.The instruction may be arbitrarily long-horizon or under-specified, requiring contextual understanding.Source: A. Problem Statement · p. 3
  • The procedure starts from an initial visual observation and high-level instruction, with an available set of skills Π.It continues until the selected action is “done.”Source: A. Problem Statement · p. 3
  • At each iteration, the VLM receives the current observation, task instruction, and prior actions, then selects the first step of its proposed plan.The selected skill executes and updates the observation before the next iteration.Source: A. Problem Statement · p. 3
  • The planning problem is to generate a sequence of short-horizon text actions, each specifying a subtask or primitive skill.The required skills are assumed to be available; skill acquisition is outside the paper’s contribution.Source: A. Problem Statement · p. 3
  • Each text action ℓt maps to a primitive skill πℓt drawn from the available skill set Π.Skills may be predefined scripts or acquired through reinforcement learning or behavior cloning.Source: A. Problem Statement · p. 3

B. Vision-Language Models as Robot Planners · C. Intriguing Properties of VILA

VILA uses visual and language input together to plan robot actions, then executes and revises its plan in a closed loop. The paper motivates this approach by arguing that images ground commonsense reasoning in scene information that language alone cannot fully capture.

B. Vision-Language Models as Robot Planners

  • VILA addresses the limits of LLM planning with separate affordance models, which transmit visual information but cannot jointly reason with the LLM.The paper says this separation can prevent task-relevant visual information from informing accurate plans.Source: B. Vision-Language Models as Robot… · p. 3
  • The paper advocates directly employing VLMs, whose extensive world knowledge is grounded in processed visual data and whose image-language reasoning can decompose high-level instructions into low-level skills [53] [5] [97].Source: B. Vision-Language Models as Robot… · p. 3
  • VILA prompts a VLM with the current visual observation and language goal, executes the first step’s policy, then replans until termination.The executed step is added to the finished plan before the VLM is queried again.Source: B. Vision-Language Models as Robot… · pp. 3–4, Figure 2 · p. 4
  • VILA uses GPT-4V(ision) [88], whose internet-scale training and generalization capabilities make it suited to the paper’s open-world scenarios.The paper reports that VILA can solve challenging planning problems in zero-shot mode, without in-context examples.Source: B. Vision-Language Models as Robot… · p. 4
  • VILA’s zero-shot planning reduces prompt-engineering effort compared with previous approaches [2] [37] [40].The paper describes zero-shot operation as requiring no in-context examples.Source: B. Vision-Language Models as Robot… · p. 4

C. Intriguing Properties of VILA

  • VILA’s visual reasoning draws on images’ fine-grained scene information to support commonsense understanding of the physical world, which language alone may not fully represent.The paper contrasts semantically rich but limited language with images that can capture an entire scene.Source: C. Intriguing Properties of VILA · p. 4

1) Spatial Layout Understanding: · 2) Object Attribute Understanding:

ViLa uses joint visual-language reasoning to understand spatial relationships and task-dependent object attributes, while supporting image-based goal specification and direct visual feedback for replanning.

1) Spatial Layout Understanding:

  • ViLa can infer that a hidden target is likely inside a container and that the container must be opened, while recognizing when an obstructing object must be moved first.The paper argues that language alone poorly conveys complex spatial layouts and that an external affordance model may assign zero retrieval probability to an unseen target.Source: 1) Spatial Layout Understanding · p. 4

2) Object Attribute Understanding:

  • Because object attributes are task-dependent and natural language conveys them incompletely, ViLa jointly reasons over images and language rather than relying on limited standalone affordance models.For example, scissors may be hazardous for children but useful in a paper-cutting class.Source: 2) Object Attribute Understanding · p. 4
  • In the illustrated tasks, SayCan picks up a plate before clearing its contents and puts scissors in a box when they should remain on the table.These examples illustrate errors involving spatial relationships and task-specific object attributes.Source: Figure 3 · p. 5
  • ViLa accepts a goal image alongside the current observation and language instruction, allowing goals from different domains, such as internet photos, drawings, or images indicating a target location.The goal image need only convey essential task elements, and images can be combined with language to specify goals.Source: 2) Object Attribute Understanding · pp. 4–5
  • Converting visual environment feedback into natural language adds complexity and risks losing information, making it cumbersome for LLM-only planning methods.The paper notes that much feedback is initially observed visually.Source: 2) Object Attribute Understanding · p. 5
  • In ViLa, the VLM recognizes object states and detects task success from visual feedback, enabling correction or replanning when the environment changes or a skill fails.This provides closed-loop feedback directly in visual form.Source: 2) Object Attribute Understanding · p. 5

IV. EXPERIMENTS AND ANALYSIS … 3) VLM and Prompting:

ViLa is evaluated through real-world everyday-manipulation experiments and simulated tabletop comparisons, with a 16-task suite testing visual commonsense, flexible goal specification, and visual feedback. The real-world experiments use GPT-4V in a strict zero-shot prompt setup.

IV. EXPERIMENTS AND ANALYSIS

  • The evaluation first tests ViLa on real-world everyday manipulation, then quantitatively compares it with baseline methods in a simulated tabletop environment.Source: §IV · p. 5

1) Hardware:

  • The real-world setup uses a Franka Emika Panda 7-DoF arm with a 1DoF parallel-jaw gripper.Source: 1) Hardware · p. 5
  • A tripod-mounted Logitech Brio color camera provides perception from a fixed angled view of the tabletop across tasks.Video demonstrations were recorded from different views for visual aesthetics.Source: 1) Hardware · p. 5

2) Tasks and Evaluation:

  • The evaluation comprises 16 long-horizon tasks: eight test visual-world commonsense, four flexible goal specification, and four visual feedback.Source: 2) Tasks and Evaluation · p. 6
  • Each task is evaluated across 10 environment variations, including changes in scene configuration and lighting.Source: 2) Tasks and Evaluation · p. 6
  • Figure 1 illustrates 12 selected tasks from the visual-world commonsense and flexible-goal domains; the visual-feedback domain is not among those selections.Source: 2) Tasks and Evaluation · p. 6

3) VLM and Prompting:

  • The planner uses OpenAI GPT-4V with high-level language instructions and simple robot constraints, without in-context examples, unlike previous approaches [2] [40].Source: 3) VLM and Prompting · p. 6

4) Primitive Skills: · 5) Baselines:

VILA outperforms SayCan [2] and Grounded Decoding [40] on tasks requiring spatial and object-attribute understanding; the section describes the primitive skills and multimodal, feedback-driven evaluations behind this planning approach.

4) Primitive Skills:

  • The study composes five script-policy primitive skills: picking up, placing in or on, opening, closing, and pouring objects.The focus is high-level, temporally extended planning rather than acquiring low-level primitive skills; the same script policies are used for VILA and the baselines.Source: 4) Primitive Skills · p. 6

5) Baselines:

  • The baselines are SayCan [2] and Grounded Decoding [40], implemented with Llama 2 70B [79] for token probabilities and OWL-ViT [57] [56] as the affordance detector.The OpenAI API did not return output token probabilities, motivating the use of Llama 2 70B as an alternative.Source: 5) Baselines · p. 6
  • VILA averages 80% success across 8 spatial-layout and object-attribute tasks, versus 13% for SayCan and 20% for Grounded Decoding.Both baselines approach zero success on particularly challenging tasks, including Take Out Marvel Model and Righteous Characters 1.Source: 5) Baselines · p. 6
  • In rollout examples, VILA accounts for spatial constraints when moving items before picking up a plate and uses context to distinguish hazards from needed tools.SayCan attempts to pick up a plate before moving the items on it and misclassifies scissors as dangerous in the art-class setup.Source: 5) Baselines · p. 6
  • VILA significantly reduces understanding errors and has the lowest overall error, while baseline errors often reflect failures to understand spatial layouts and context-specific object attributes.Baseline perception errors refer to failures of the open-vocabulary detector; VLM outputs can also fail to recognize objects, and prompt examples may reduce response-structure errors.Source: Figure 4 · p. 6, 5) Baselines · p. 7
  • VILA supports multimodal goals across 4 tasks, including image-based puzzle, sushi, and vegetable arrangements, and combined image-and-language instructions for desk tidying.Examples include interpreting a child’s drawing, using a reference photograph, and inferring from a pointing finger that vegetables belong on the pink plate.Source: 5) Baselines · p. 7, Figure 5 · p. 7
  • Closed-loop VILA significantly outperforms an open-loop variant on dynamic tasks, recovering from disturbances and adapting its strategy using real-time visual observations.Tests include noisy block stacking, experimenter-reversed packing progress, variable stapler locations, and pausing for human-robot interaction; VILA also searches the bottom drawer after the stapler is absent from the top drawer.Source: 5) Baselines · pp. 7–8, Table III · p. 8, Figure 6 · p. 8
  • In simulation, VILA consistently outperforms baselines across seen and unseen tasks; the RAVENS environment includes 16 Blocks & Bowls and Letters tasks.The supplied table caption reports average success rates but gives no specific values in these passages.Source: Table IV · p. 8, Figure 7 · p. 8

B. Simulated Tabletop Rearrangement

In simulated tabletop rearrangement, VILA outperforms the compared LLM-based planning methods, including Grounded Decoding, across tasks. The evaluation covers seen and unseen rearrangement tasks using several planner and imitation-learning baselines.

Experimental Setup

  • The evaluation uses 16 RAVENS-based tasks: 6 seen and 10 unseen, divided evenly between Blocks & Bowls and Letters.A UR5 rearranges objects to match high-level language instructions; the seen tasks support few-shot prompting or supervised-baseline training.Source: B. Simulated Tabletop Rearrangement · p. 8
  • Baselines include CLIPort’s Short and Long variants, ungrounded Llama 2 and GPT-4 planners, and Grounded Decoding with Llama 2.Grounded Decoding uses CLIPort-predicted affordances for Blocks & Bowls and simulation ground-truth affordances for Letters; LLM-based planners use script policies as primitive skills.Source: B. Simulated Tabletop Rearrangement · pp. 8–9

Analysis

  • VILA significantly outperforms Grounded Decoding and other LLM-based planners, while Llama 2 and GPT-4 perform comparably across tasks.Each method was evaluated over 20 episodes per task within each category.Source: B. Simulated Tabletop Rearrangement · p. 9
  • CLIPort-based methods generalize poorly to unseen tasks, while Grounded Decoding benefits from an external affordance model but still lags behind VILA.Grounded Decoding uses Llama 2 because it requires access to LLM output-token probabilities.Source: B. Simulated Tabletop Rearrangement · p. 9

V. CONCLUSION, LIMITATIONS, & FUTURE WORKS

VILA uses VLMs to turn high-level instructions into actionable steps by integrating perception into planning, supporting multimodal goals and visual feedback. The paper identifies assumptions about primitive skills, limited VLM steerability, and output consistency as areas for future work.

Conclusion

  • VILA integrates visual perception into VLM-based planning to decompose high-level instructions into actionable steps for complex, long-horizon tasks.It supports multimodal goal specification and incorporates visual feedback; evaluation in real-world and simulated settings demonstrates its effectiveness.Source: §V · p. 9

Limitations & Future Works

  • The approach presupposes single-step primitive skills, while robust low-level control policies remain challenging to obtain.The authors suggest that transferring web knowledge to robotic control [7] [8] may help build a repertoire of generalizable skills.Source: §V · p. 9
  • Dependence on a black-box VLM limits steerability and makes some errors difficult to explain.The authors propose parameter-efficient finetuning [32] [33] to customize VLMs [24].Source: §V · p. 9
  • Omitting in-context examples from prompts enables a more versatile output format but leaves output consistency open to improvement.The authors suggest applying prompting methods [45] [89] to refine consistency.Source: §V · p. 9

APPENDIX … 4) Primitive Skills:

The real-world appendix describes the evaluation setup, strict zero-shot prompting, and five scripted primitive skills composed for high-level planning. The skills are tailored to the study’s tasks, while developing a generalizable, robust skill set is left for future work.

1) Hardware Setup:

  • The real-world robot is a 7-DoF Franka Emika Panda arm with a 1DoF parallel-jaw gripper, operated using Deoxys’s joint controller [98].Source: 1) Hardware Setup · p. 14
  • Perception uses an angled tripod-mounted Logitech Brio color camera capturing 1920 × 1080 tabletop images.Source: 1) Hardware Setup · p. 14

2) Tasks and Evaluation:

3) Prompts:

  • The prompt uses high-level language instructions and simple constraints without in-context examples, making the setup strict zero-shot.Figure A.1 shows the full prompt.Source: 3) Prompts · p. 14

4) Primitive Skills:

  • Five scripted primitive skills—picking up, placing, opening, closing, and pouring—support complex behaviors through composition and planning.The study focuses on high-level, temporally extended planning rather than acquiring low-level primitive skills.Source: 4) Primitive Skills · p. 14
  • Primitive policies are taught by teleoperation with a 3D SpaceMouse for simple pickup and kinematic teaching for intricate, contact-rich tasks such as opening a drawer.Developing generalizable and robust primitive skills is identified as important future work.Source: 4) Primitive Skills · p. 14

B. Simulated Environment · 1) Tasks:

The simulated benchmark uses 16 RAVENS tasks for a UR5 to rearrange objects from high-level instructions, with randomized object layouts and binary ground-truth evaluation. Blocks & Bowls and Letters use different prompting strategies because GPT-4V recognizes letters but struggles with simulated blocks and bowls zero-shot.

1) Tasks:

  • 16 RAVENS tasks require a UR5 to rearrange objects into configurations specified by high-level language instructions.The tasks are split evenly between Blocks & Bowls and Letters.Source: 1) Tasks · p. 17
  • A top-down camera view supports planning, and a binary reward based on ground-truth object states enables automatic evaluation.Source: 1) Tasks · p. 17
  • At each reset, task-relevant objects and distractors are randomly distributed across the workspace.Source: 1) Tasks · p. 17
  • Eight spatial targets, including corners and sides, are listed for attribute variation in some task instructions.The listed targets span the top, bottom, left, and right sides and corners.Source: 1) Tasks · p. 17
  • Twenty word options, including “cat,” “bread,” and “table,” are listed for attribute variation in some task instructions.Source: 1) Tasks · p. 17
  • Blocks & Bowls prompts include three in-context examples because GPT-4V cannot recognize these simulated objects zero-shot amid the domain gap from real-world counterparts.Letters prompts omit examples because GPT-4V recognizes and understands all letters proficiently.Source: 2) Prompts · p. 17

2) Prompts: … 5) Full Results on Simulated Environments:

The evaluation uses script policies as primitive skills to isolate planner performance, compares ViLa with language-conditioned and LLM-based baselines, and presents simulated results for seen and unseen tasks alongside the prompts used by high-level planners.

3) Primitive Skills:

  • Script policies replace CLIPort primitive skills because they access simulator ground-truth object states, avoiding noisy outcomes when measuring planner success.The authors observed CLIPort policies sometimes interacted correctly even when the planner generated an incorrect step.Source: 3) Primitive Skills · p. 17
  • The CLIPort [71] baselines use high-level instructions at evaluation and include Short policies trained on single-step tasks and Long policies trained on high-level instructions.Source: 4) Baselines · p. 17
  • Ungrounded LLM baselines include Llama 2 70B and GPT-4; Llama 2 supports Grounded Decoding reimplementation because the OpenAI API lacks output-token probabilities.Source: 4) Baselines · p. 17
  • Grounded Decoding [40] uses Llama 2 with beam search, deriving Blocks & Bowls affordances from CLIPort logits and Letters affordances from simulator ground truth.Ground-truth affordances are used for Letters because CLIPort logits generalize poorly to unseen letters.Source: 4) Baselines · p. 17

5) Full Results on Simulated Environments:

2) Prompts:

  • Figure B.1 presents the prompt for Blocks & Bowls in the simulated environment.Source: Figure B.1 · p. 19
  • Figure B.2 presents the prompt for Letters in the simulated environment.Source: Figure B.2 · p. 20

Related papers

Loading 2311.17842v2…