PaperlayerView exact revision on arXiv

Source-linked AI summary

VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models

Zixuan Wang, Yuxin Chen, Yuqi Liu, Jinhui Ye, Pengguang Chen, Changsheng Lu, Shu Liu, Bei Yu, Jiaya Jia

arXiv:2603.22003v3cs.RO

TL;DR

Existing VLA models often combine instruction interpretation, spatial reasoning, and execution in a monolithic process that can limit grounding and robustness. VP-VLA separates these functions with a planner-controlled visual prompting interface and an auxiliary grounding objective. Across simulation and real-world evaluations, it reports consistent gains over strong baselines, while its current 2D prompts leave richer representations as future work.

  • Problem

    Existing VLA models can struggle with instruction grounding, spatial precision, and robustness to novel objects and spatial positions.

  • Method

    VP-VLA uses a System 2 Planner to decompose tasks and generate structured visual prompts for a System 1 Controller, with auxiliary visual grounding during training.

  • Results

    VP-VLA consistently outperforms strong baselines across Robocasa-GR1-Tabletop, SimplerEnv, and real-world cluttered scenarios.

  • Takeaways & Limitations

    Explicit visual prompts and grounding supervision provide a structured interface for precise manipulation across simulated and real-world evaluations.

  • Takeaways & Limitations

    The current interface uses 2D bounding boxes and crosshair coordinates, while more complex manipulations may benefit from richer visual prompts.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models typically map visual observations and linguistic instructions directly to control signals. This "black-box" mapping forces a single forward pass to simultaneously handle instruction interpretation, spatial grounding, and low-level control, often leading to poor spatial precision and limited robustness in out-of-distribution scenarios. To address these limitations, we propose VP-VLA, a dual-system framework that decouples high-level reasoning and low-level execution via a structured visual prompting interface. Specifically, a "System 2 Planner" decomposes complex instructions into sub-tasks and identifies relevant target objects and goal locations. These spatial anchors are rendered directly within the native RGB observation space as modality-consistent visual prompts, such as crosshairs and bounding boxes. This avoids the modality mismatch introduced by dense masks, affordance maps, or additional control-specific representations. Guided by these prompts and enhanced by a novel auxiliary visual grounding objective during training, a "System 1 Controller" reliably generates precise low-level execution motions. Extensive experiments in simulation and real world demonstrate that VP-VLA surpasses state-of-the-art end-to-end baselines including QwenOFT and GR00T-N1.6. Project page: https://visualprompt-vla.github.io/

1 Introduction

Existing VLA models can struggle with instruction grounding, novel spatial configurations, and dynamic multi-stage tasks. VP-VLA addresses these challenges by separating high-level reasoning from low-level control through structured visual prompts and achieves consistent gains across simulation and real-world evaluations.

  • 1 Introduction: Existing VLA frameworks may overfit training scene distributions, with gibberish language substitutions sometimes barely affecting performance.This pattern indicates limited reliance on meaningful linguistic grounding.
  • 1 Introduction: Novel object categories and unseen spatial positions remain challenging for existing policies, while prior interfaces often target static, single-task settings.Earlier approaches use goal images or dense geometric supervision but rely on rigid intermediate representations.
  • 1 Introduction: VP-VLA decouples high-level reasoning from low-level control through a structured visual prompting interface.The framework uses a System 2 Planner and a System 1 Controller rather than requiring one model to solve all stages simultaneously.
  • 1 Introduction: 5% average success-rate improvement over the baseline on Robocasa-GR1-Tabletop accompanies gains on SimplerEnv and real-world scenarios.VP-VLA surpasses GR00T-N1.6 on Robocasa-GR1-Tabletop and reports +8.3% absolute improvement over baseline on SimplerEnv.
  • 1 Introduction: The visual grounding objective enhances the spatial precision and robustness of VLA models.It provides an auxiliary training signal aligned with the visual prompting interface.
  • 1 Introduction: Experiments on Robocasa-GR1-Tabletop, SimplerEnv, and real-world scenarios demonstrate consistent gains over strong baselines.The evaluations cover both simulated benchmarks and real-world settings.

2 Related Work

Related work uses visual intermediates to improve spatial reasoning and action grounding in robotic manipulation. However, many such representations are not natively understood by standard VLA models and require task-specific supervision or complex training.

  • 2 Related Work: Many existing VLA models use monolithic architectures that tightly couple semantic instructions, multimodal inputs, and visuomotor policy learning.This design places several reasoning and control functions within one architectural pathway.
  • 2 Related Work: Prior visual intermediates include future image frames, key poses, trajectories, and hierarchical spatiotemporal traces for downstream robotic policies.These approaches provide intermediate signals for spatial reasoning and action grounding.
  • 2 Related Work: Intermediate modalities often require task-specific supervision and complex end-to-end training because standard VLA models do not natively understand them.The modality mismatch can complicate the training of downstream policies.

3 Method

VP-VLA separates deliberative task reasoning from visuomotor execution through an explicit visual-prompt interface. Its event-driven planner grounds subtasks in crosshairs and bounding boxes, while the controller uses these prompts and auxiliary grounding training to produce precise actions.

  • 3 Method: The System 2 planner decomposes complex instructions into subtasks and identifies target objects and goal locations.It produces a visual interface image after semantic reasoning and spatial grounding.
  • 3 Method: VP-VLA uses a dual-system architecture that decouples high-level reasoning from low-level control through an explicit visual interface.The System 2 planner and System 1 controller divide instruction interpretation, spatial grounding, and execution.
  • 3.2 System 2 Planner: The planner is invoked at transition events, using changes in the robot’s physical interaction state to trigger re-evaluation of the next sub-goal.In tabletop manipulation, gripper-state changes serve as the proxy for semantic phase shifts.
  • 3.2 System 2 Planner: Visual prompts encode interaction anchors as crosshairs and placement constraints as bounding boxes overlaid on the observation.The crosshair localizes object interaction, while the bounding box specifies the spatial region for placement.
  • 3.3 System 1 Controller: The System 1 controller combines the original observation with the visual prompt image to generate continuous control signals.Its policy uses a VLM backbone for multimodal embeddings and an action decoder for control outputs.
  • Training Objective: An auxiliary grounding objective aligns policy representations with visual prompts during training to improve spatial precision and robustness.The total loss combines action prediction with grounding loss, and the grounding loss is applied to the VLM parameters.

4 Experiment

VP-VLA is evaluated across simulation and real-world manipulation settings, including in-domain, out-of-distribution, cluttered, and ablation scenarios. Results show improved success rates, generalization, and the value of selective grounding and structured prompt design.

  • Experimental scope: The experiments cover simulation benchmarks and real-world tasks assessing cluttered-scene grounding, OOD robustness, and visual prompting for spatial reasoning.The evaluation uses multiple datasets, robot settings, and task suites.
  • Simulation benchmarks: VP-VLA achieves 53.8% average success on RoboCasa Tabletop, exceeding QwenOFT by 5.0 percentage points.It also improves performance on novel placement configurations involving varied backgrounds, object attributes, and positions.
  • Simulation benchmarks: 58.3% average success on SimplerEnv exceeds QwenOFT’s 50.0% by +8.3%.VP-VLA also surpasses π0.5 and Isaac-GR00T-N1.6-Bridge, each at 57.1%.
  • Real-world evaluation: 87.5% in-domain and 85% out-of-distribution success in cluttered categorization exceed QwenOFT’s 80% and 63.3%.The generalization gap is 2.5% for VP-VLA versus 16.7% for QwenOFT.
  • Real-world evaluation: VP-VLA reaches 77.1% in-domain accuracy and 75.0% OOD color success, while QwenOFT reaches 29.2% on OOD colors.The reported ID-to-OOD color gap is 2.1% for VP-VLA.
  • Real-world evaluation: 91.25% in-domain and 68.75% OOD accuracy outperform QwenOFT’s 70.63% and 55% in the reported spatial-combination setting.The result is reported for novel row–column combinations.
  • Ablation study: Removing grounding reduces success from 53.8% to 49.4%, while grounding every frame reaches 49.5%, below selective key-frame grounding.The ablation supports grounding supervision at selected decision frames rather than densely across all frames.
  • Ablation study: Changing crosshairs to points lowers average performance to 47.3%, and direct RGB overlay reaches 50.8%.These results indicate that prompt geometry and separation from raw visual features affect prompt interpretation.

5 Conclusion

VP-VLA decouples high-level reasoning from low-level execution through structured visual prompts and auxiliary grounding. Across simulation and real-world cluttered scenarios, the framework demonstrates improved performance and generalization.

  • Conclusion: VP-VLA uses a System 2 Planner to convert complex instructions into structured visual prompts for a System 1 Controller.The prompts encode spatial anchors, while auxiliary visual grounding guides precise manipulation.
  • Conclusion: Evaluations on RoboCasa-GR1-Tabletop, SimplerEnv, and real-world cluttered scenarios demonstrate superior performance and generalization.The conclusion reports gains across both simulated benchmarks and real-world deployment.

A Limitations

The current interface represents spatial anchors with 2D bounding boxes and crosshair coordinates. More complex manipulation may require richer visual prompts.

  • Limitations: VP-VLA currently uses 2D bounding boxes and crosshair coordinates, which may be insufficient for threading needles or handling deformable objects.The paper suggests exploring richer prompts such as 3D Gaussian splats or contact-point heatmaps.

B Compute Resources

The experiments use substantial GPU resources for training, while real-world inference runs on a single NVIDIA 4090. The reported pipeline includes measurable planner, controller, and segmentation latency.

  • Compute resources: Training uses eight NVIDIA H200 GPUs with 140 GB memory each, while real-world inference uses one NVIDIA 4090 with 24 GB.Training on the full dataset takes approximately 46 hours.
  • Inference latency: Mean latency is 0.78 seconds per VLM reasoning call, 0.108 seconds per action chunk, and 0.36 seconds for SAM3 segmentation.SAM3 remains faster than semantic VLM planning and is compatible with asynchronous prompt updates.

D Additional Demonstration to Tool-Use Manipulation Tasks

VP-VLA is extended to long-horizon tool-use manipulation through staged subtask reasoning and visual grounding. The scoop-bean task requires sequential spoon pickup, scooping, and pouring actions.

  • Tool-Use Task: The scoop-bean task tests whether VP-VLA extends beyond pick-and-place to sequential tool-use manipulation.The robot must pick up a spoon, scoop from one bowl, and pour into another.
  • Subtask Detection: Subtask boundaries are detected from end-effector motion using z-axis rise events as transition cues.This replaces gripper-state heuristics for identifying stage changes.
  • Visual Planning: At the initial frame and detected boundaries, a VLM predicts whether to continue or proceed, plus the target object and target location when needed.The VLM receives a reference frame and the current frame for each decision.
  • Visual Grounding: The predicted semantic targets are grounded with SAM, producing frame-level masks and boxes that are saved per episode and injected into training via visual prompting.This connects subtask-level predictions to frame-level visual supervision.

E Extended Analyze on Experiments

The extended experiments examine VP-VLA across task-specific categorization, attribute-based reference, and spatial placement settings. Results emphasize robustness to novel objects, colors, and spatial configurations compared with QwenOFT.

  • Robotic Waste-Sorting Categorization: VP-VLA remained robust on a red-shoe categorization case, achieving 9/10 versus QwenOFT’s 7/10 in the recyclable box.The comparison suggests category grounding independent of surface appearance in this case.
  • Robotic Waste-Sorting Categorization: On novel OOD objects in the kitchen-waste box, VP-VLA achieved 85% accuracy versus 75% for the baseline.The evaluated novel objects were a pear and a carrot.
  • Object Reference by Attribute: VP-VLA consistently surpassed the baseline across in-domain training colors in object-reference tasks.Baseline failures included imprecise grasp positioning and failure to move above the target-colored egg.
  • Location Reference with Spatial Grounding: For in-domain positions, VP-VLA achieved near-perfect scores on most coordinates, while QwenOFT dropped to 1/5 at L3C3 versus 4.5/5 for VP-VLA.The comparison concerns row–column spatial parsing.
  • Location Reference with Spatial Grounding: For OOD positions, VP-VLA scored 5/5 at L3C2 and 4/5 at L4C3, while QwenOFT ranged from 1.75/5 to 3.5/5.The tasks require following target masks at coordinate combinations not jointly observed during training.

F Extended Ablation Results

The extended ablations examine VP-VLA’s experimental design, showing that temporal reasoning decomposition supports performance and that the planner re-evaluates task stages through transition events.

  • Impact of Reasoning Decomposition: Removing temporal decomposition degraded success, especially on the “Put Eggplant in Yellow Basket” task.Without decomposition, interaction anchors and spatial constraints are rendered simultaneously.
  • Impact of Reasoning Decomposition: Concurrent crosshair and bounding-box prompts can introduce visual noise that confuses the policy’s attention.This is the authors’ hypothesis for the observed degradation.
  • Impact of Reasoning Decomposition: Without decomposition, prompting fails to generalize to complex multi-step sequences such as the “PnP * to * Close” task suite.These sequences require distinguishing between sequential objectives.
  • Planner Design: The System 2 planner first converts a high-level instruction into an ordered list of atomic robotic manipulation subtasks.The planner is selectively invoked by transition events during execution.
  • Planner Design: When triggered, the planner determines whether to proceed to the next subtask and identifies the corresponding target object and destination names.This provides stage-specific semantic targets during execution.

H VLA Experimental Demonstrations

VP-VLA demonstrations span simulation and real-world manipulation, including OOD localization, cluttered multi-step interactions, and physical deployment. The accompanying tables and prompt templates document task-level results and planner design.

  • Demonstration Coverage: VP-VLA demonstrations cover real-world tasks and the SimplerEnv and RoboCasa simulation environments.The figures visualize inference on these deployment and benchmark settings.
  • SimplerEnv Demonstrations: In SimplerEnv’s OOD setting, VP-VLA decomposes manipulation into discrete stages while completing each phase.Success requires high-precision localization and robust control.
  • RoboCasa Demonstrations: In cluttered RoboCasa kitchen environments, VP-VLA handles complex multi-step interactions and generalizes to humanoid embodiments using egocentric observations.The demonstrations concern long-horizon activities.
  • Real-World Demonstrations: Real-world demonstrations show object manipulation and placement in both in-distribution and OOD scenarios with novel objects and unseen placement settings.The rollout requires precise reasoning and grounding in a physical environment.
  • Supporting Experimental Materials: The extended materials include per-task RoboCasa ablations, task-decomposition ablations, and planner prompts for decomposition and subtask completion detection.Tables 5–8 organize these results and prompt structures.
  • Supporting Experimental Materials: Additional tables report categorization, colored-egg reference, egg-carton placement, and GR1 RoboCasa results across in-domain and OOD conditions.These materials provide task-specific evaluations and model comparisons.
Loading 2603.22003v3…