PaperlayerView exact revision on arXiv

Source-linked AI summary

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

Yehang Zhang, Haojian Huang, Yifan Chang, Jianchong Su, Bohan Zhou, Yingjie Xu, Wosong Chen, Tianhao Zhou, Chenxu Wang, Tianyi Zhang, Yangkai Wei, Wenqian Li, Shiyuan Deng, Yinchuan Li, Ying-Cong Chen, Zexi Li

arXiv:2609.29964v1cs.ROcs.AI

TL;DR

VLMs can provide broad knowledge for manipulation, but existing systems either use them indirectly or show them a scene without a world in which to act. WAA lets VLMs pilot robots through a visual workspace for observing, rehearsing, and correcting actions, while building skills and training smaller pilots. WAA reaches 75.6% average success on LIBERO-Pro with LIBERO-90-only skills, and harness traces improve a smaller model’s out-of-domain success.

  • Problem

    Existing systems use VLMs indirectly for manipulation or provide a scene interface without a world in which the model can act.

    Source: §1 · p. 2

  • Method

    WAA is a multi-agent harness where VLMs use basic tools to observe, rehearse, and correct robot actions within one visual action workspace.

    Source: §1 · pp. 2–3, §3.2 · p. 5

  • Results

    75.6% average success on LIBERO-Pro is state of the art for WAA with skills evolved only from LIBERO-90, and Qwen3.5-9B harness-trace fine-tuning raises out-of-domain success from 1.7% to 43.3%.

    Source: §1 · p. 3, §5 · p. 12

  • Takeaways & Limitations

    The same skills evolved only from LIBERO-90 remain effective on robosuite without further learning.

    Source: §1 · p. 3

    Performance remains bounded by the backbone’s multi-view perception, and an episode still requires tens of model calls.

    Source: §5 · p. 12

Abstract

from arXiv · show

General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act. We present World Action Agent (WAA), a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace. The workspace has three properties. Contact views, selected automatically from the scene geometry, present the scene around the current interaction. Action rehearsal turns each action into an editable proposal that the agent, alone or through an Imagination Agent, previews and revises against planning feedback before execution. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the view where it observes them. Through the same workspace, WAA acquires embodied procedural knowledge in two ways: it evolves multimodal skills from expert videos and human teaching under evidence-based review and consults them through a Skill Agent, and its interaction traces train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone; the same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.

1 Introduction

WAA addresses the gap between VLMs that guide manipulation indirectly and visual interfaces that show scenes without giving the model a world in which to act. It combines interaction-centered views, rehearsable actions, and in-view correction, while using the same workspace to develop skills and train smaller VLM pilots.

Motivation

  • Existing approaches either use VLMs to provide constraints or write programs, or expose a scene interface without giving them a world in which to act.The motivating gap is the lack of interaction-centered observation, pre-execution action trials, and correction in the observed view.Source: §1 · p. 2

Learning through the harness

  • WAA evolves multimodal skills from expert videos and human teaching and distills interaction traces into smaller VLM pilots through the same workspace.These provide nonparametric skill evolution and parametric training, respectively.Source: §1 · p. 3

Results

  • 75.6% average success on LIBERO-Pro is state of the art for WAA with skills evolved only from LIBERO-90; the same skills transfer to robosuite without further learning.Fine-tuning Qwen3.5-9B on harness traces raises out-of-domain success from 1.7% to 43.3%.Source: §1 · p. 3

World Action Agent

  • WAA makes general-purpose VLMs robot pilots by letting them observe, rehearse, and correct actions through basic tools in one visual action workspace.The design changes what the VLM sees and how its decisions take effect rather than changing the VLM itself.Source: §1 · pp. 2–3

2 Related Work

Prior manipulation systems use foundation models to predict actions, guide control with spatial representations, or generate programs and interact through visual interfaces. WAA is distinguished by giving the VLM direct control over robot primitives in a visual workspace, without a VLA executor or generated policy code.

Foundation models for robot manipulation

  • VLA and world action models learn or ground robot actions, while VLM-guided spatial representations supply paths, constraints, primitives, or affordances for downstream control.The passage notes that executable control for world action models typically requires robot data for action alignment, while cross-embodiment transfer is sensitive to morphology and action-space differences.Source: §2 · p. 3

Agentic robot control and visual interfaces

  • Code-generation systems connect language-model reasoning to execution through programs, while hierarchical and agentic systems coordinate reasoning with VLA execution.The effectiveness of code-based systems depends on perceptual grounding and control abstractions, and VLA calls remain bounded by learned policy capabilities.Source: §2 · pp. 3–4
  • Visual interfaces expose targets or semantic actions, whereas WAA gives the VLM direct control over robot primitives without a VLA executor or generated policy code.Numerical planners and controllers realize the VLM’s decisions.Source: §2 · p. 4

3 World Action Agent

WAA formulates manipulation as tool-mediated decisions over a multi-view canvas, action proposals, and execution feedback. Its harness centers views on interactions, previews and revises feasible actions before execution, and supports both evolving multimodal skills and training smaller VLM pilots.

3.1 Problem Formulation

  • At each step, the agent receives a multi-view Canvas and control context, then selects a tool call to query information, revise a proposal, or execute motion.Query and proposal tools leave the physical world unchanged, allowing repeated inspection and revision before commitment; execution ends with task success or exhaustion of the interaction budget.Source: §3.1 · p. 4

3.2 Designing the World Action Agent Harness

  • The Canvas combines global context with automatically selected Contact views that expose local interaction relations from complementary directions.View selection accounts for interaction and gripper visibility, compact framing, redundant directions, and viewpoint stability; orthogonal Contact views help read alignment errors along independent directions.Source: §3.2 · pp. 5–6, Figure 2 · p. 5
  • Action rehearsal turns a target pose into an editable proposal, previews its planned configuration and feasibility, and allows an Imagination Agent to revise it before execution.Only proposals with executable plans become physical motion, coupling the VLM’s geometric judgment with planner feasibility checks.Source: §3.2 · p. 6
  • In-view correction converts a drag in a calibrated Contact view into bounded end-effector motion, letting the agent address residual errors through fresh observations.The paired views support complementary correction directions, and all three workspace designs share calibrated views for observing, rehearsing, and correcting the same targets.Source: §3.2 · pp. 5–7, Figure 3 · p. 6

3.3 Learning and Evolution through the Harness

  • WAA skills encode applicability conditions, procedures, reference images, and observable outcome checks as relational guidance rather than fixed coordinates or displacements.The Skill Agent compares relevant skill states and images with the current Canvas, reports applicability and unconfirmed relations, and lets scene-specific advice expire when the observation or control state changes.Source: §3.3 · p. 7
  • Skill evolution starts from text-only seeds and adds evidence from expert trajectories and human corrections through editing and independent review.The Learner cites source frames and extracts recovery guidance by comparing failed attempts with human corrections; the Editor–Reviewer loop admits only supported changes that preserve applicability conditions.Source: §3.3 · pp. 7–8
  • Successful harness episodes and separately reviewed recovery segments train smaller VLMs to operate the same interface from observations, instructions, context, and available skill guidance.Human corrections can also supervise recovery behavior, and fine-tuning updates only the main policy while leaving the workspace and sub-agents unchanged.Source: §3.3 · pp. 7–8

4 Experiments

Experiments evaluate WAA on LIBERO-Pro and robosuite, finding state-of-the-art LIBERO-Pro success with skills evolved only from LIBERO-90. They also test zero-shot use, transfer, alternate point-cloud sources, and fine-tuning a smaller VLM to pilot the harness.

4.1 Experimental Setup

  • Evaluation covers six LIBERO-Pro splits and three robosuite tasks; skills are learned only from LIBERO-90 and frozen without evaluation-time updates.WAA uses Gemini 3.7 Flash in zero-shot, text-only seed-skill, and evolved-skill settings.Source: §4.1 · p. 8

4.2 Main Results

  • 75.6% average success is state of the art on LIBERO-Pro; WAA leads both Spatial splits and Goal Task, while ASPIRE leads both Object splits and Goal Pos.The skill library was learned only from LIBERO-90 and frozen before evaluation.Source: Table 1 · p. 9, §4.2 · p. 9
  • 28.9% average success without skills exceeds π0.5 at 12.8% and both CaP-Agent0 evaluations; WAA also reaches 53.3% and 46.7% on the two Object splits.The passage reports that end-to-end VLAs almost completely fail under task perturbations.Source: §4.2 · p. 9
  • Against same-backbone Show-Harness, zero-shot WAA reaches 28.9% versus 6.7% and uses 31 model calls and 150 s versus 120 calls and 874 s per episode.The comparison uses the same backbone and no skills for both systems.Source: §4.2 · pp. 9–10, Table 2 · p. 11
  • On the two Spatial splits, WAA reaches 80.0% and 73.3%, versus 12.0%–31.0% for CaP-Agent0 and RATs and 51.0% and 60.0% for ASPIRE.Figure 4 illustrates proposal repair before execution and in-view bowl alignment.Source: §4.2 · p. 10, Figure 4 · p. 10

4.3 Skill Learning, Transfer, and Generality

  • 75.6% average success with evolved skills exceeds 43.3% with seed skills and 28.9% without skills; evolved skills beat zero-shot in all six splits.Text-only seed skills help inconsistently and slightly lower Goal Pos. success.Source: §4.3 · p. 10
  • Frozen LIBERO skills raise robosuite restacking from 60.0% to 100.0%, versus 63.3% average for CaP-Agent0 with RATs skills.One LIBERO-90 stove demonstration also produced 10/10 LIBERO-Pro successes with an accepted skill requiring an explicit activation signal.Source: Table 3 · p. 11, §4.3 · p. 11
  • Fine-tuning Qwen3.5-9B raises out-of-domain success from 1.7% to 43.3%.Using fused RGB-D or VGGT point clouds yields 71.1% or 68.9% average success, with both settings surpassing every baseline on the Spatial splits.Source: Figure 5 · p. 11, §4.3 · p. 11

5 Conclusion

WAA lets a general-purpose VLM make and revise robot actions in a visual action workspace; its LIBERO-90 skills transfer to robosuite, and harness traces improve a smaller VLM’s out-of-domain success.

5 Conclusion

  • 75.6% average success on LIBERO-Pro is state of the art, and the same frozen skills transfer to robosuite.The skills were evolved only from LIBERO-90.Source: §5 · p. 11
  • WAA changes what the VLM sees and how its decisions take effect, rather than retraining the original model.The workspace supports action rehearsal and a closed loop between observation, rehearsal, and low-level execution.Source: §5 · p. 11
  • 43.3% out-of-domain success follows fine-tuning a 9B VLM on harness traces, up from 1.7%.The conclusion identifies this as teaching a smaller VLM to pilot the harness.Source: §5 · pp. 11–12
  • Imperfect multi-view integration in Gemini 3.7 Flash leaves WAA less stable on some tasks, while episodes still require tens of model calls.The limitation is attributed to backbone perception; training smaller VLMs to pilot the harness is proposed as a way to reduce cost and latency.Source: §5 · p. 12

A Multimodal Skill Examples

The appendix presents condensed English versions of two frozen-library skills used in evaluation, with stored visual references for the Skill Agent to interpret alongside the current Canvas.

A Multimodal Skill Examples

A.1 Grasping Cylindrical Objects

The cylindrical-object skill selects grasp guidance from observed shape, checks contact in calibrated views, verifies the lift, and changes the approach after unsuccessful contact.

A.1 Grasping Cylindrical Objects

  • Use the skill for a visible can, bottle, or similar cylinder based on its shape, and reconsult it after slipping, one-sided contact, or an unsuccessful lift.Source: Grasping cylindrical objects · p. 16
  • For an upright object, approach with the gripper open from a pre-grasp position 3–5 cm above its top, using small visually checked adjustments.Source: Grasping cylindrical objects · p. 16
  • Check both Contact views before closing: center the fingers across the diameter, keep the body within the closing region, and retain palm clearance.Source: Grasping cylindrical objects · p. 16
  • Verify a grasp with a short lift; object motion with the gripper confirms it leaves its support, while tilt alone does not establish failure.Source: Grasping cylindrical objects · p. 16
  • After unsuccessful contact, reopen, return above the object, and revise centering or approach depth rather than repeating the same contact.Source: Grasping cylindrical objects · p. 16
  • A stored unstable-contact reference shows a tilted object not centered between the fingers, guiding reassessment and a new approach.Source: Figure 6 · p. 17

A.2 Placing a Bowl on a Plate

The bowl-placement skill applies when a bowl is held above a plate or needs correction after an off-center release; its stored references include a negative example of apparent alignment.

A.2 Placing a Bowl on a Plate

  • Use the skill when a bowl is held above a plate or when a released bowl needs correction because it is off-center.Source: Placing a bowl on a plate · p. 18
  • The skill stores both positive and negative placement references.Source: Figure 7 · p. 18
  • The negative reference warns that apparent alignment in one view does not establish centered placement.Source: Figure 7 · p. 18

A.3 Learning Stove Activation from One Demonstration

A skill learned from one LIBERO-90 stove demonstration was reviewed and frozen before evaluation. The section also compares two ways to construct point-cloud views of the same LIBERO-Pro scene.

A.3 Learning Stove Activation from One Demonstration

  • One LIBERO-90 stove demonstration supplied aligned RGB frames and end-effector orientations, whose relative rotations revealed turning direction despite knob occlusion.Visual evidence was still needed to assess the knob response and burner state.Source: §A.3 · p. 19, Figure 8 · p. 20
  • The Reviewer rejected knob motion as sufficient success evidence, and the Editor revised the skill to require an explicit activation signal.The accepted package contained the procedure and five source-frame references; its text was not manually edited before evaluation.Source: §A.3 · p. 19, Figure 8 · p. 20
  • 10/10 stove-task executions succeeded with the frozen skill in the LIBERO-Pro goal-swap suite.The ten trials covered five initial states, with two trials per state.Source: §A.3 · p. 19

B Canvas Geometry from Visual Inputs

  • RGB-D + Hand combines four RGB-D views with FK gripper geometry, whereas VGGT reconstructs from eight RGB views without measured depth or a hand model.Both point-cloud renderings depict the same LIBERO-Pro scene; the inputs were captured in simulation.Source: Figure 9 · p. 21, B Canvas Geometry from Visual Inputs · p. 21

C Function List

The function list separates information gathering and proposal editing from physical execution. The prompts guide the Main Agent to use current visual evidence and delegate skill consultation without giving the Skill Agent robot-control access.

C Function List

  • Preview functions edit a pending action without moving the robot, while execute_action runs the plan and local drag, rotation, and gripper commands act on the current state.Skill consultation provides guidance but does not execute robot actions.Source: C Function List · pp. 22–23

D Agent System Prompts

  • The prompts are English translations of Chinese system prompts, with runtime instructions, Canvas images, execution feedback, and skill content supplied separately.The Main Agent prompt combines shared instructions, a skill-mode block, and coordinate instructions.Source: D Agent System Prompts · p. 25

D.1 Main Agent

  • The Main Agent is instructed to inspect the latest Canvas, cross-check global and local views, and use skill verification conditions against current observations.It should focus on the physical relation that still needs to change rather than follow a fixed tool order.Source: §D.1 · p. 25
  • Local translation uses a drag in a Contact Front or Side panel, which the harness converts through camera calibration into bounded displacement.Single-axis motion is generally limited to 3 cm except upward motion, which may reach 8 cm; diagonal displacement is limited to 8 cm.Source: §D.1 · p. 25
  • The Main Agent must treat previews as unexecuted and recheck physical outcomes, since reaching a target pose does not establish correct contact or task success.Local physical commands do not modify or execute a pending preview.Source: §D.1 · p. 25

D.2 Skill Agent

  • The Skill Agent first selects the minimum reference evidence for a question, then compares selected images with the current scene and reports relevant differences and uncertainty.It provides consultation only and does not execute robot actions.Source: §D.2 · p. 27

E Pilot Fine-Tuning Details

The pilot fine-tuning setup trains Qwen3.5-9B on harness interaction traces. Samples pair the pre-action context with the executed tool call, and evaluation uses the final-epoch adapter without a held-out validation split.

E Pilot Fine-Tuning Details

  • Each training sample pairs the pre-action Canvas, task instruction, control context, and available skill guidance with the executed tool call.Earlier turns are masked so the loss covers only the target tool call.Source: E Pilot Fine-Tuning Details · p. 28
  • No separate validation split is held out, and the adapter after the final epoch is used for evaluation.This is the stated evaluation setup for the pilot fine-tuning configuration.Source: E Pilot Fine-Tuning Details · p. 28
  • Table 7 reports the fine-tuning configuration used for the Qwen3.5-9B main-agent pilot.The supplied text does not enumerate configuration values.Source: Table 7 · p. 28, E Pilot Fine-Tuning Details · p. 28

F Failure Cases

Three unsuccessful Object/Spatial episodes illustrate difficulties with cross-view grasp verification, object identity tracking, and judging whether placement is complete.

F Failure Cases

  • The discussion inspects three unsuccessful episodes from the 200-episode Object/Spatial evaluation.The reported cases illustrate specific failure modes rather than estimating their prevalence.Source: F Failure Cases · p. 29
  • The agent judged a pudding box grasp from one view, but the Side view showed it offset from the fingers; empty closures and retries consumed episode budget.The agent recognized the empty grasp in the next observation.Source: F Failure Cases · p. 29
  • After losing track of the instructed patterned bowl, the agent redirected to a bowl on the stove and later attempted to grasp a ramekin.The case illustrates inconsistent preservation of object identity across scene changes and repeated localization.Source: F Failure Cases · p. 29
  • The agent called a released bowl centered and complete although it remained offset toward the plate edge, and the task was unsuccessful.The case illustrates insufficient geometric verification of the final object–target relation.Source: F Failure Cases · p. 29
Loading 2609.29964v1…