Source-linked AI summary
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingnan Zhang, Tsang-Wei Edward Lee, Kuang-Huei Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn, Sergey Levine, Brian Ichter
TL;DR
VLMs’ textual outputs do not directly provide the continuous spatial values needed for robotic control and related tasks. PIVOT addresses this by iteratively visualizing candidate proposals and asking a VLM to select and refine them, demonstrating zero-shot spatial control while exposing limitations in current models.
Problem
VLMs mainly output text, but robotic control and other spatial tasks require continuous coordinates, actions, or trajectories without task-specific fine-tuning.
Method
PIVOT maps candidate spatial outputs into labeled image annotations, queries a VLM to select promising candidates, and iteratively refines their proposal distribution.
Results
PIVOT is evaluated across robotic navigation, manipulation, simulated instruction following, and localization, with zero-shot VLM inference and improved performance from larger, more capable models.
Takeaways & Limitations
Visual prompting and iterative optimization provide a route to using VLMs for spatial reasoning and low-level robotic control without robot training data.
Takeaways & Limitations
Current VLMs cannot reliably choose actions based on depth, and occlusion in closed-loop interaction impairs decision-making during navigation and manipulation.
Abstract
from arXiv · showhide
Vision language models (VLMs) have shown impressive capabilities across a variety of tasks, from logical reasoning to visual understanding. This opens the door to richer interaction with the world, for example robotic control. However, VLMs produce only textual outputs, while robotic control and other spatial tasks require outputting continuous coordinates, actions, or trajectories. How can we enable VLMs to handle such settings without fine-tuning on task-specific data? In this paper, we propose a novel visual prompting approach for VLMs that we call Prompting with Iterative Visual Optimization (PIVOT), which casts tasks as iterative visual question answering. In each iteration, the image is annotated with a visual representation of proposals that the VLM can refer to (e.g., candidate robot actions, localizations, or trajectories). The VLM then selects the best ones for the task. These proposals are iteratively refined, allowing the VLM to eventually zero in on the best available answer. We investigate PIVOT on real-world robotic navigation, real-world manipulation from images, instruction following in simulation, and additional spatial inference tasks such as localization. We find, perhaps surprisingly, that our approach enables zero-shot control of robotic systems without any robot training data, navigation in a variety of environments, and other capabilities. Although current performance is far from perfect, our work highlights potentials and limitations of this new regime and shows a promising approach for Internet-Scale VLMs in robotic and spatial reasoning domains. Website: pivot-prompt.github.io and HuggingFace: https://huggingface.co/spaces/pivot-prompt/pivot-prompt-demo.
1. Introduction
PIVOT addresses the mismatch between VLMs’ textual outputs and spatial tasks requiring continuous coordinates or actions by iteratively refining visually annotated proposals. The paper evaluates this zero-shot approach across robotic and spatial inference settings, finding both useful capabilities and model-dependent limitations.
- VLMs handle rich visual modalities but generally produce textual answers, whereas robotic control and other spatial tasks require grounded coordinates, trajectories, or actions.
- PIVOT annotates images with candidate actions or spatial proposals, asks the VLM to select promising options, and refines the proposal distribution iteratively.The procedure frames spatial control as visual question answering and uses numbered visual representations that the VLM can reference.
- The method supports zero-shot spatial inference without domain-specific training by converting continuous outputs into selections among visually represented candidates.
- PIVOT is studied on robotic navigation, manipulation, simulated instruction following, and keypoint localization using GPT-4 and Gemini without modification or finetuning.
- The paper analyzes where zero-shot visual prompting succeeds and fails, reporting that performance improves with larger, more capable VLMs.The authors frame the work as an empirical study of current VLM potentials and limitations rather than a best-possible task-specific controller.
2. Related Work
Related work situates PIVOT among studies of visual annotations, automatic prompt optimization, and foundation-model reasoning and control for robotics. Its distinctive focus is using visual prompting and iterative optimization to obtain spatial or control outputs from VLMs.
- Visual annotations with VLMs: Prior work studies whether VLMs recognize visual annotations and how annotations can support perception or decision-making.
- Prompt optimization: Prompt optimization has been explored through gradients, human engineering, and automatic language-space methods using model, answer, or environment feedback.
- PIVOT contributes a visual, iterative route for grounding continuous spatial outputs while retaining zero-shot VLM inference.
- Foundation models for robot reasoning and control: Foundation-model robotics research spans high-level reasoning and planning as well as approaches for applying such models to control tasks.
3. Prompting with Iterative Visual Optimization
PIVOT turns continuous spatial-output problems into iterative VQA: candidate actions are visualized and labeled, the VLM selects promising options, and the distribution is refined. Parallel calls improve robustness, while the algorithm supports varied action representations and zero-shot control settings.
- 3. Prompting with Iterative Visual Optimization: PIVOT treats tasks as producing an action or spatial value from a task description and image, including continuous coordinates, 3D locations, robot actions, or trajectories.
- 3.1. Grounding VLMs to Robot Actions through Image Annotations: The mapping lifts low-level actions into the VLM’s image-and-text input space, enabling the model to judge which represented action is promising.
- 3.1. Grounding VLMs to Robot Actions through Image Annotations: The visual prompt mapping transforms an image and candidate actions into an annotated image with textual labels that refer to the candidates in image space.Projected 3D locations can be shown with visual markers, allowing the VLM to select spatial concepts through label references.
- 3.2. Prompting with Iterative Visual Optimization: At each iteration, PIVOT samples candidate actions, projects and labels them visually, queries the VLM to select the best candidates, and fits the next proposal distribution.The process repeats until convergence or a maximum number of iterations is reached.
- 3.3. Robust PIVOT with Parallel Calls: Parallel PIVOT instances aggregate independently selected actions to improve robustness and mitigate local minima in the optimization process.
- 3.3. Robust PIVOT with Parallel Calls: Algorithm 1 initializes the action distribution, samples and projects candidates, and returns an action selected from the VLM’s best actions.
- 3.4. PIVOT Implementation: Actions can be visualized as numbered arrows, with colors and label sizes encoding forward and backward movement for 3D problems; distributions are approximated as isotropic Gaussians.
4. Experiments
The experiments test PIVOT across diverse robotic embodiments, control settings, and spatial tasks, including zero-shot real-world control, simulated manipulation, and visual grounding. Across evaluations, iterative optimization and parallel calls improve performance, while current VLMs retain limitations in scaling, 3D understanding, interaction, and multi-step reasoning.
- Experimental setup: PIVOT is evaluated on mobile-manipulator navigation and manipulation, Franka-arm manipulation, simulated tabletop pick-and-place, and visually grounded spatial tasks.The robot setups vary in camera views, action-space dimensionality, and control settings.
- Zero-shot robotic control: Real-world experiments assess zero-shot control across mobile-manipulator navigation and manipulation, as well as Franka manipulation.These experiments use varied robot embodiments, camera views, and action-space dimensionalities.
- Zero-shot robotic control: Iterations and parallel calls improve navigation success, manipulation performance, and policy efficiency in the reported evaluations.Table 1 reports improved navigation with iterations and parallel calls, while Table 2 reports improved manipulation performance and efficiency; increasing iterations also improves performance.
- Offline performance and ablations: PIVOT outperforms a language-only baseline by a large margin in offline evaluations, while visual prompting and iterative optimization together perform best.The offline evaluation compares PIVOT components against language actions mapped to robotic actions.
- Scaling: Performance increases monotonically across four progressively larger Gemini models, suggesting PIVOT can leverage stronger foundation models despite remaining capability gaps.The scaling evaluation measures cosine similarity and L2 error against demonstration data.
- Limitations: Current limitations include unreliable depth-based action selection, challenges from occlusion and fine-grained interaction, greedy multi-step behavior, and incorrect links between reasoning and visual actions.The authors relate these limitations to missing 3D, embodied, video, or in-domain training data and to autoregressive decoding errors.
5. Conclusion
PIVOT enables zero-shot spatial reasoning by casting challenging tasks such as low-level robotic control as vision problems and iteratively refining image-space representations. The approach exposes both opportunities and limitations of current VLMs.
- PIVOT enables VLMs to control robot arms zero shot by iteratively refining spatially grounded continuous outputs.The paper frames robotic control and other spatial tasks through visual representations that VLMs can select and refine.
- PIVOT suggests casting traditionally challenging problems, including low-level robotic control, as vision tasks.
- The approach can benefit from alternative sampling initialization procedures, optimization algorithms, and search-based strategies.
- Current state-of-the-art VLM limitations include 3D understanding and interaction.The paper identifies datasets representing these areas and task-specific fine-tuning as future directions.
- The paper reports that PIVOT's zero-shot performance is likely to improve as VLM capabilities improve.This expectation is connected to the paper's scaling experiments.
A. Robotic Embodiments
The paper applies PIVOT across mobile-manipulator navigation, mobile-manipulator manipulation, Franka manipulation, and simulated RAVENS pick-and-place tasks. Each embodiment uses image annotations tailored to its action or spatial representation.
- Mobile Manipulator Navigation: Mobile-manipulator navigation uses arrows from the image bottom center to represent 2D actions, then maps selected pixel actions to 3D targets using depth.Evaluation includes real-robot scenarios and an offline dataset.
- Mobile Manipulator Manipulation: Mobile-manipulator manipulation represents 3D relative Cartesian end-effector positions with arrows originating from the end-effector.The z-dimension is studied using either color grading or a setting where the arm only uses part of the representation.
- Franka: Franka manipulation uses wrist-camera arrows to represent 3D relative Cartesian end-effector positions, with height encoded by a red-to-blue color spectrum.The evaluation examines both pick and place tasks with five objects per task.
- RAVENS: RAVENS simulation annotates pick and place locations from an overhead camera to evaluate higher-level action representations.
B.1. Dataset
The offline navigation dataset contains 60 indoor examples pairing instructions with target locations in image space. Tasks are categorized as in-view finding, semantic understanding, or out-of-view finding.
- Dataset: 60 examples were collected from an on-robot camera while walking the robot through an indoor environment.Each example includes an instruction and an associated image-space target location.
- Task Categories: In-view finding asks the robot to approach an object visible within its line of sight.
- Task Categories: Semantic understanding uses instructions that implicitly refer to an object visible in the current view.
- Task Categories: Out-of-view finding targets objects absent from the current view but visible in past images from different locations.The dataset examples are illustrated in Figure 10, where a red dot marks the ground-truth target.
B.2. Evaluation Results
Offline navigation evaluates image-space target deviation across three task categories using repeated runs. Parallel VLM calls and additional PIVOT iterations improve accuracy, while image annotations outperform text-based region reasoning for GPT-4V.
- Evaluation Setup: Navigation accuracy is measured by image-space deviation from the target, normalized by image width, across three task categories.The evaluation reports mean and standard deviation over three runs on the entire dataset.
- Ablations: Parallel calls significantly improve performance over a single VLM call, and multiple PIVOT iterations further improve accuracy.Increasing either setting beyond the tested improvements did not produce notably better performance.
- Image Annotations vs. Text: For GPT-4V, the text baseline has higher mean and standard deviation of errors across all navigation tasks than PIVOT with one iteration and zero parallel calls.The baseline asks the VLM to select among nine imagined image regions rather than annotated image proposals.
C. Mobile Manipulator Manipulation Online Evaluation
PIVOT can recover from inaccurate action predictions during real-world mobile-manipulator manipulation, including errors from imperfect depth perception or action precision.
- PIVOT can recover from action-prediction inaccuracies during real-world mobile-manipulator manipulation.The reported inaccuracies may result from imperfect depth perception or challenges in action precision.
D. Mobile Manipulator Manipulation Offline Evaluation
Offline ablations show that zero-shot chain-of-thought performs best overall, while few-shot direct prompting is similarly effective with greater token efficiency. Prompt order has only a small effect, and qualitative episodes reveal difficulty during interaction-heavy moments.
- Prompt ablations: Zero-shot chain-of-thought performs best overall, while few-shot direct prompting performs similarly with significantly greater token efficiency.Neither prompting choice is a panacea.
- Prompt ablations: Preamble-image-task ordering yields the highest performance, although differences between prompt orders are small.The authors hypothesize that this order most closely mirrors the training mixture.
- Qualitative evaluation: PIVOT is more accurate when the next movement is clear but struggles to generate correct actions during interaction-heavy middle-of-episode states.
- Qualitative evaluation: PIVOT generates reasonable actions following arrow annotations in offline mobile-manipulator episodes.
E. RAVENS Online Simulation Evaluation
In RAVENS pick-and-place tasks, visual prompting error generally decreases during the first few iterations and eventually converges, but precise one-action execution remains difficult.
- Each task requires picking a specified fruit and placing it in a specified colored bowl.
- RAVENS pick-and-place localization error generally decreases over the first few visual-prompting iterations and eventually converges.Evaluation covers five random task instances and measures error against ground-truth pick and place locations.
- Although selected pick and place locations are usually close to the desired objects, VLMs often cannot choose points precisely enough for successful one-action execution.
F. Franka Online Evaluation
Franka experiments show non-zero success across PIVOT variants, with more optimization iterations and parallel processes improving performance and stability. The broader evaluation also examines manipulation rollouts and VLM robustness to arrow annotations.
- Franka online evaluation: All Franka PIVOT variants achieve non-zero success on seven real-world tabletop manipulation tasks.The setup uses a wrist-mounted camera and a 4D relative Cartesian delta action space.
- Franka online evaluation: Increasing optimization iterations and parallel PIVOT processes improves manipulation performance and stability.Table 5 reports XY and YZ reaching success rates and successful-trial step counts.
- Arrow-annotation robustness: The evaluation also studies VLM robustness to absolute arrow direction and object-referential arrow selection across synthetic datasets.The toy dataset uses arrows on blank backgrounds, while the realistic dataset overlays arrows on robotics scenes.
- Manipulation rollouts: Additional rollouts include moving an orange to complete a fruit smiley face, tracing a blue road with a marker, and sorting a held object.These promising successes use three optimization iterations on the real-world mobile manipulator.