PaperlayerView exact revision on arXiv

Source-linked AI summary

Show-Harness: Just a VLM Agent Can Play Robots

Yanzhe Chen, Zechen Bai, Zhijun Cao, Wenzheng Zeng, Kevin Qinghong Lin, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Qiming Huang, Mike Zheng Shou

arXiv:2609.10522v1cs.ROcs.AIcs.CVcs.MM

TL;DR

Foundation VLMs contain broad semantic knowledge, but translating it into fine-grained robot control remains difficult. Show-Harness addresses this with a compact semantic action interface and deterministic embodiment-specific grounding, enabling zero-shot frontier-VLM control, low-cost small-model adaptation, and shared GUI demonstrations. Experiments report robust generalization across tasks, embodiments, and environments, while the current evaluation is concentrated on limited manipulation embodiments.

  • Problem

    Foundation VLM knowledge does not readily translate into robot behavior, while existing VLA approaches often require embodiment-specific adaptation across tasks and environments.

  • Method

    Show-Harness exposes discrete semantic action units for VLM reasoning and deterministically grounds them into executable embodiment-specific robot motions within a closed perception–reasoning–action–feedback loop.

  • Results

    Show-Harness-equipped VLM agents generalize across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms while supporting zero-shot frontier-VLM control and few-GPU-hour small-model adaptation.

  • Takeaways & Limitations

    A suitable semantic interface can unlock substantial embodied capability from foundation VLMs and support human–agent demonstration collection without specialized teleoperation hardware.

  • Takeaways & Limitations

    Show-Harness is evaluated primarily on single- and dual-arm manipulation with parallel-jaw grippers, leaving more complex embodiments and richer contact modalities for future work.

Abstract

from arXiv · show

Foundation vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging. We present Show-Harness, an Embodied Harness that enables VLMs to "play" robots through a compact semantic interface linking intent to action. Show-Harness exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions, keeping the VLM directly responsible for fine-grained physical decisions. Through the same interface, Show-Harness demonstrates the feasibility of (1) directly unlocking closed-source frontier VLMs for zero-shot robot control, and (2) adapting small-scale open-source VLMs for low-cost deployment with just a few GPU-hours of fine-tuning. We further develop GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to "play" robots across embodiments without specialized teleoperation hardware. Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms. These results suggest that the right interface can unlock substantial embodied capability from foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.

1 Introduction

Show-Harness addresses the gap between VLM semantic intelligence and robot control with a compact interface of fine-grained semantic action units. It supports both zero-shot frontier-VLM control and efficient adaptation of smaller models, while GUMI extends the interface to human and agent demonstration collection.

  • Show-Harness: Show-Harness converts robot control into discrete semantic action units that VLMs can interpret while remaining fine-grained enough for physical control.An embodiment-specific interpreter deterministically grounds each unit into bounded robot motion, and the harness returns execution feedback after every unit.
  • Robot control modes: Closed-source frontier VLMs achieve zero-shot robot control through Show-Harness, while representative agentic harnesses are outperformed across tasks, embodiments, and environments.The approach requires no fine-tuning for the frontier-VLM mode.
  • Robot control modes: 2B-scale VLMs can play robots after just a few GPU-hours of fine-tuning, with stronger generalization and sim-to-real transfer than representative VLA paradigms.This provides a lower-cost deployment mode using the same semantic action space.
  • GUMI: GUMI lets humans and agents collect robot demonstrations through a GUI in the same semantic action space without specialized teleoperation hardware.The interface supports cross-embodiment reuse and human–agent collaborative data collection.
  • Contributions: The paper presents Show-Harness as a model-agnostic embodied harness for directly operating robots with foundation VLMs and reports strong generalization across tasks, embodiments, and environments.The contribution is framed as unlocking embodied capability without requiring additional model capacity or costly embodiment-specific pretraining.

2 Related Work

Related work spans low-level VLA policies, hierarchical intermediate representations, and agentic systems that translate VLM decisions into robot behavior. Show-Harness positions interface design as the central distinction by exposing fine-grained semantic actions whose physical realization is deterministic and transparent.

  • Foundation models as low-level policies: VLA models attach learned low-level action generation to VLM backbones, using continuous, discretized, spatial, or latent action representations.These approaches directly predict or encode robot controls rather than relying only on intermediate semantic decisions.
  • Foundation models as intermediate decision makers: Hierarchical systems preserve VLM semantics by emitting subgoals, keypoints, affordance targets, value maps, or spatial constraints for downstream controllers.Their physical realization depends on engineered, system-specific grounding pipelines and can require long-horizon planning, feedback, and recovery.
  • Agentic architectures: Agentic architectures keep the foundation VLM inside a harness that translates decisions into robot behavior and feeds back outcomes through a closed loop.They differ mainly in whether the model composes programs, selects skills, or steers learned policies.
  • Interfaces for embodied execution: Existing interfaces often expose skills, controllers, or VLAs as callable primitives, leaving an opaque executor to determine how an intended action is physically realized.Show-Harness instead exposes fine-grained semantic action units and their deterministic physical grounding.
  • From digital interfaces to physical manipulation: Show-Harness extends compact, interpretable interfaces used by digital and simulated agents to real-world manipulation through deterministic grounding and shared human–VLM actions.The paper reports strong sim-to-real transfer while using the same action space for VLM agents and humans.

3 Show-Harness

Show-Harness places a foundation VLM in a modular perceive–reason–act loop, where it selects fine-grained semantic actions and an embodiment-specific interpreter deterministically grounds them into robot control. The same interface supports frontier-model zero-shot control and lightweight fine-tuning of small VLMs.

  • 3.1 Overview: Show-Harness translates foundation VLM intelligence into robot behavior through an iterative perception–reasoning–action loop.The model receives perceptual inputs and interaction history, selects an action, observes its effects, and continues iteratively.
  • 3.2 Physically Grounded Semantic Action Interface: An embodiment-specific interpreter deterministically converts each semantic decision into executable control while keeping embodiment-specific variables outside the VLM interface.The interpreter updates pose setpoints, applies calibrated motion increments and safety projections, and maps gripper units to open or close commands.
  • 3.2 Physically Grounded Semantic Action Interface: The semantic action interface uses compact, fine-grained units for end-effector movement, rotation, and gripper intent.Units are designed to be interpretable to the VLM while remaining sufficiently fine-grained for direct physical control.
  • 3.3 Embodied Harness Architecture: The harness organizes configurable plugins across perception, reasoning, and action stages.These stages include multi-view guidance, proprioception, subtask planning, situated planning, action history, and failure recovery.
  • 3.4 Two Modes on One Interface: The shared interface supports both zero-shot frontier VLM agents and fine-tuned small VLMs.Frontier VLMs operate without fine-tuning, while small VLMs learn semantic action units from demonstrations using lightweight adaptation and native vocabulary prediction.

4 GUMI: A GUI-based Manipulation Interface

GUMI extends Show-Harness’s semantic action space into a lightweight GUI through which humans and agents can operate robots and collect demonstrations. Its shared controls support flexible, cross-embodiment data collection without specialized teleoperation hardware.

  • 4 GUMI: A GUI-based Manipulation Interface: GUMI exposes the shared semantic action units through labeled GUI controls and keystrokes for humans, computer-use agents, and general VLM agents.Humans can operate the robot from a keyboard, while agents use the same GUI or predict the units directly.
  • 4 GUMI: A GUI-based Manipulation Interface: GUMI supports step-wise control, queued action chunks, single- and dual-arm operation, and mixed human–agent demonstration collection.Humans can intervene to correct agent rollouts during collection.
  • 4 GUMI: A GUI-based Manipulation Interface: Demonstrations recorded in the shared semantic action space can be reused across embodiments without specialized teleoperation hardware.The lightweight digital interface also supports remote collection without physical colocation with the robot.

5.1 Experimental Setup

The experiments use ten real-robot manipulation tasks, standardized trials and metrics, multiple VLM control modes, and demonstrations collected through GUMI across two robot embodiments.

  • Tasks and metrics: Ten real-robot tasks pair five physically diverse objects with plate or bowl receptacles, requiring locating, approaching, grasping, transporting, and placing.The objects include a block, banana, tennis ball, teddy bear, and chess piece.
  • Tasks and metrics: Each task reports success rate and average steps over 10 randomized trials, with 50-step timeouts counted as failures.
  • VLM agents and harness: The study evaluates zero-shot frontier VLM agents and fine-tuned small VLMs, using Gemini-3.1 Pro by default for ZS and Qwen3.5-2B with LoRA for FT.The fine-tuned setup updates about 3% of parameters while freezing the vision encoder and multimodal projector.
  • Baselines: Baselines include low-level VLA systems such as π0.5 and GR00T, fine-tuned on continuous end-effector trajectories from the same demonstrations.
  • Demonstration data collection: GUMI collects shared demonstrations through human keyboard control and frontier-VLM browser rollouts across Franka and AgileX arms.The collection contains 164 real-robot episodes and 7.8K decision steps.

5.2 Main Results: Generalization across Task, Environment, and Embodiments

Show-Harness agents consistently outperform representative baselines across task, environment, and embodiment shifts, with advantages extending to held-out combinations and sim-to-real transfer.

  • Evaluation axes: The evaluation covers cross-task, cross-environment, and cross-embodiment generalization on real robots.Cross-environment includes background, lighting, viewpoint, distractor, and sim-to-real shifts.
  • Overall generalization: Show-Harness ZS and FT consistently outperform representative baselines across task, environment, and embodiment shifts.
  • Cross-embodiment: The same semantic interface transfers effectively between Franka and AgileX embodiments.
  • Sim-to-real: FT succeeds in sim-to-real transfer using only simulated demonstrations, while trainable VLA baselines fail.

5.3 Capability Analysis: Physical and Semantic Adaptability

Show-Harness supports physical and semantic adaptation without changing the VLM–action interface or retraining in several settings, including finer control, composition, workspace shifts, coordination, and novel reasoning tasks.

  • Adaptability framework: Physical adaptability covers changes in motion precision, composition, workspace, and embodiment without policy retraining, while semantic adaptability covers reasoning and in-context learning from novel demonstrations.
  • Physical adaptability: Reducing interpreter steps from 2 cm to 1 cm improves ZS from 60% to 82% and FT from 40% to 65% on fine-grained tasks without retraining.π0.5 achieves 18% with the same demonstrations and reaches 62% only after additional fine-grained training.
  • Physical adaptability: Composing two orthogonal translation units into a diagonal displacement reduces execution steps on five Plate tasks without a notable success drop.
  • Physical adaptability: FT reaches 70% at an unseen 90° carrot orientation after training on 0° and 45° demonstrations, while π0.5 reaches 20%.Each rotation unit changes orientation by 15°.
  • Physical adaptability: Show-Harness degrades only mildly as the workspace expands, whereas π0.5 drops sharply.The evaluation spans nested workspace regions from S@1 at 25% to S@3 at 90%.
  • Physical adaptability: Joint action prediction substantially improves multi-arm success and eliminates collisions on table tidying and banana handover.
  • Semantic adaptability: ZS with Situated Planning achieves 85% on reasoning-intensive tasks, compared with 10% for FT and 0% for π0.5 alone.With shared Gemini-generated subtask instructions, FT rises to 70%.
  • Semantic adaptability: With a video demonstration, ZS follows the demonstrated object order in 20/20 trials from either human or robot sources.Without a demonstration, ZS succeeds in 20% when the instruction only says tidy up.

5.4 Ablation Studies

Ablations show that performance depends on model choice, perception and state inputs, planning and recovery plugins, selective action compression, adaptive stepping, and explicit action conventions.

  • Model and reasoning ablations: Zero-shot performance generally improves with stronger frontier VLMs, while higher thinking effort mainly reduces redundant interaction steps and can increase wall-clock cost.More than 98% of responses produce valid action units, and errors concentrate on fine-grained grasping and placement.
  • Model and reasoning ablations: Performance is already strong at 2B, while larger backbones mainly help stacking and peg insertion; 1B models make excessive local adjustments.The 2B backbone offers a balance between precision and responsiveness.
  • Perception and state: Multi-view guidance and proprioception improve performance, especially when small-object alignment or ambiguous visual cues make manipulation difficult.Wrist views provide close-up evidence for fine-grained discrimination, while compact state signals include gripper height and contact.
  • Harness plugins: Removing subtask planning drops success to 60% because the model often drags objects without lifting.Planning makes manipulation stages and completion criteria explicit.
  • Harness plugins: Selective action chunking preserves fine-grained feedback while compressing redundant transport steps; forcing chunking throughout drops success to 74%.Disabling chunking preserves 96% success but increases model calls.
  • Harness plugins: Adaptive switching between fine and coarse steps achieves 96% success with 30 steps per episode on average.Fine-only control is inefficient and timeout-prone, while coarse-only control can overshoot.
  • Harness plugins: Visual Prompt improves handle-aware grasping from 40% to 85% by aligning language instructions with the marked interaction point.It has little effect on regular tasks.
  • Harness plugins: Situated Planning improves hidden-object search from 35% to 85% by deferring unresolved decisions until sufficient visual evidence is available.It has no effect on regular tasks.

5.5 Qualitative Analysis

Qualitative demonstrations show Show-Harness handling broad visual and physical variation beyond standard pick-and-place. The interface also supports semantic rearrangement and coordinated bimanual manipulation.

  • Show-Harness handles novel objects, background and lighting changes, cluttered scenes, spatial-reasoning tasks, and bimanual control.
  • The same interface supports rearranging letter blocks to satisfy semantic goals and coordinating two arms to open a drawer and place an object inside.

6 Conclusion and Limitations

The paper concludes that Show-Harness exposes a compact semantic action space that keeps VLMs involved in stepwise physical decisions, while GUMI enables GUI-based demonstration collection. Experiments indicate strong generalization with minimal embodiment-specific adaptation, but evaluation remains concentrated on parallel-jaw manipulation.

  • Conclusion: Show-Harness uses a semantic action space that is interpretable to VLMs yet fine-grained enough for direct physical control within a perception–reasoning–action–feedback loop.
  • Conclusion: GUMI enables humans and agents to collect robot demonstrations through a GUI without specialized teleoperation hardware.
  • Conclusion: Experiments demonstrate strong generalization and adaptability with minimal embodiment-specific adaptation.
  • Limitations: Show-Harness is evaluated primarily on single- and dual-arm manipulation with parallel-jaw grippers.The paper identifies humanoids, dexterous hands, tactile feedback, and force feedback as directions for extension.

7 Appendix

The appendix documents lightweight fine-tuning, demonstration data, controlled comparisons, prompt conventions, and the planner-controller harness used to execute semantic actions. It also specifies recovery, proprioception, multi-view, and action-selection rules.

  • Implementation: Fine-tuning uses 7.9K single-arm samples for 40 epochs and can run on 24 GB-class GPUs; Qwen3.5-2B takes less than 2 hours on one H200.
  • Demonstration Data: The real-robot corpus covers 19 tasks across Franka and AgileX platforms, including pick-and-place, stacking, shelf rearrangement, and grasp-recovery episodes.
  • Demonstration Data: Simulation preserves the real-Franka control and observation conventions while varying rendering, scene composition, and embodiment appearance.
  • Comparison: Under matched demonstrations, Show-Harness completes the chess-piece, tennis-ball, and teddy-bear examples, whereas π0.5 exhibits grasping or object-interaction failures.
  • Agent Architecture: Show-Harness uses one VLM in planner and controller roles: the planner decomposes tasks, while the controller selects the next semantic action from current context.
  • Controller: The controller selects one movement, gripper, or termination action from a compact JSON action contract.
  • Controller: The harness augments prompts with proprioception, multi-view guidance, action chunking, adaptive step sizes, action history, and failure recovery.
  • Action Selection: When the target is not in the wrist view, the agent plans multiple movement tokens; when it is visible, the agent chooses a single move.
Loading 2609.10522v1…