PaperlayerView exact revision on arXiv

Source-linked AI summary

MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation

Bingxuan Li, Siqi Song, Yizhuo Wu, Jiarui Yao, Tong Zhang, Huan Zhang

arXiv:2609.38078v1cs.RO

TL;DR

Zero-shot transfer remains limited for specialized VLA models, and agentic robotic systems often rely on external models and tools. MotorMind connects a general-purpose VLM to robot control through mid-level actions, feedback, and asynchronous execution. It reports strong simulation and real-robot success, with higher success using a stronger VLM backbone.

  • Problem

    Specialized VLA models have limited zero-shot transfer and cannot directly benefit from emerging general-purpose VLMs, while agentic robotic systems often depend on substantial external models and tools.

    Source: §1 · p. 2

  • Method

    MotorMind uses a general-purpose VLM to propose parameterized mid-level actions connected to robot control and feedback, with monitoring and memory updates alongside execution.

    Source: §1 · p. 2, §2 · p. 3, §3.2 · p. 5

  • Results

    66.7% success on LIBERO-PRO base suites and 53.8% under perturbations compare with prior zero-shot results of at most 13.3% and 19.2%, respectively; the system reaches 95% on real xArm6 tasks and 83.3% average success with GPT-6 Sol.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    The results support effective zero-shot manipulation by a general-purpose VLM when paired with an appropriate action representation and execution harness.

    Source: §7 · p. 12

    Remaining failures primarily involve visual grounding, embodied reasoning, and insufficient action knowledge.

    Source: §1 · p. 3

Abstract

from arXiv · show

Vision-language-action (VLA) models have advanced robotic manipulation, but their zero-shot generalization in new tasks and environments remains limited, and their reliance on specialized training keeps them from benefiting directly from rapidly advancing general-purpose vision-language models (VLMs). In parallel, recent agentic robotic systems leverage VLMs for high-level reasoning or coding agents for robot control, but often depend on extensive external models and tools, introducing additional complexity and cost. This motivates us to ask: Can a general-purpose VLM itself operate a robot more like the human teleoperator by reasoning directly from observations, issuing actions, and continuously adapting to execution feedback, without relying on external models such as learned action experts, coding agents or grounding tools like SAM3? In this work, we introduce MotorMind, a robot manipulation harness that connects VLM-proposed mid-level actions to deterministic robot control and feedback, with asynchronous monitoring and background memory updates. Without task-specific policy training, coding agents, or additional grounding tools such as SAM3, MotorMind achieves 66.7% success on the base LIBERO-PRO suites and 53.8% under perturbations, compared with at most 13.3% and 19.2%, respectively, for the prior zero-shot methods we evaluate. The same interface reaches 95% average success on a real xArm6 robot across direct manipulation and human-perturbation settings. Replacing the backbone with a stronger VLM further improves performance, while the remaining failures - primarily due to visual grounding, embodied reasoning, and action knowledge - decrease as VLM capability improves. These results show that a general-purpose VLM, when equipped with an appropriate mid-level action representation and asynchronous execution harness, can perform effective zero-shot robotic manipulation.

1. Introduction

Specialized VLA models have limited zero-shot transfer, while agentic robotic systems often depend on substantial external models and tools. MotorMind instead connects a general-purpose VLM to robot control and feedback, achieving strong results in simulation and on a real robot.

1. Introduction

  • Specialized VLA models have limited zero-shot transfer across changes in object appearance, spatial configuration, task semantics, and environment structure, and often rely on additional robot-action data or policy adaptation.Their specialized robotics training also prevents them from directly benefiting from emerging general-purpose VLM capabilities.Source: §1 · p. 2
  • Agentic robotics uses general-purpose VLMs for planning or high-level reasoning but often adds learned experts, segmentation models, skill libraries, motion planners, or execution modules.The paper motivates a more direct VLM-operated robot that adapts from observations and execution feedback without those external components.Source: §1 · p. 2
  • 66.7% success on LIBERO-PRO base suites and 53.8% under perturbations exceed the strongest evaluated prior zero-shot results of 13.3% and 19.2%, respectively.On a real xArm6, the same interface reaches 95% average success across direct manipulation and human disturbance settings.Source: §1 · p. 2
  • Replacing Qwen3.8-Flash-Next with GPT-6 Sol raises LIBERO-PRO base-suite success from 66.7% to 83.3%.The reported improvements in general-purpose multimodal models may translate into stronger robotic behavior without redesigning the control architecture or retraining with specialized data or model architecture.Source: §1 · pp. 2–3
  • MotorMind uses a VLM directly for semantic action generation without a coding agent, learned action model, SAM-based perception, or external inverse kinematics solver.Most existing works rely on learned action policies, generated robot code, or extra perception and control modules.Source: Table 1 · p. 3
  • Remaining failures primarily involve imperfect visual grounding, embodied reasoning errors, and insufficient action knowledge.Examples include selecting the wrong object or target, incorrectly judging task completion, and repeatedly proposing ineffective motions.Source: §1 · p. 3
  • The results frame zero-shot robotic control as aligning general-purpose model capabilities with a suitable control representation rather than relying solely on specialized action policies.The introduction contrasts this direction with learning robotic behavior fully from demonstrations.Source: §1 · p. 3

2. Diagnosing VLMs for Robotic Manipulation

The section probes three local VLM decisions needed for manipulation: choosing an action, assessing progress, and judging subgoal completion. Action selection is consistently the weakest diagnostic capability, while stronger judgments remain imperfect and model accuracy trades off against latency.

Diagnostic Benchmark

  • The 240-question diagnostic covers action selection, progress assessment, and subgoal completion as local visual decisions rather than closed-loop control of complete tasks.It contains 80 questions for each capability; action selection and progress assessment use past and current observations, while completion judges the current observation against a stated criterion.Source: §2 · p. 3, Figure 2 · p. 4, Table 2 · p. 4

Diagnostic Results and Analysis

  • Action selection is the least accurate capability for every evaluated model, ranging from 18.75% to 60.00%.These findings motivate short action proposals that can be revised after observing their effects.Source: §2 · pp. 3–4
  • 36.25% action accuracy for Qwen3.8-Flash-Next is below its 55.00% progress-assessment and 65.00% completion accuracy.The task answer spaces differ, so these percentages are diagnostic performance rather than equivalent measures of intrinsic difficulty.Source: §2 · p. 4
  • GPT-6 Astra achieves the highest accuracy on all three capabilities but takes 8.724 s per query, compared with 0.276 s for Qwen3.8-Flash-Next.The reported accuracy–latency trade-off motivates considering both decision quality and feedback frequency when choosing a backbone.Source: §2 · p. 4

3. MotorMind: VLM Harness for Zero-Shot Robotic Manipulation

MotorMind uses a frozen general-purpose VLM to plan and revise subgoals, propose parameterized actions, and assess outcomes from robot feedback. Its sequential motion loop is coordinated with asynchronous monitoring and memory updates.

3. MotorMind: VLM Harness for Zero-Shot Robotic Manipulation

  • MotorMind connects a frozen general-purpose VLM to measured robot feedback through mid-level actions, with motion decisions in a sequential loop and monitoring and memory updates running asynchronously.The VLM makes decisions from camera images and measured robot state without task-specific policy training.Source: §3 · p. 4, §3.1 · p. 5

3.1. Task Formulation

  • The Planner creates ordered subgoals with intended state changes, target descriptions, and success criteria, then can revise unfinished requirements when execution invalidates the initial plan.The active subgoal specifies both what actions should accomplish and what evidence is needed to accept the outcome.Source: §3.1 · p. 5

3.2. Mid-level Action Representation

  • The Executor proposes parameterized move, rotate, and gripper actions, while the Controller converts them into embodiment-specific motion and records measured execution.Fresh observations and feedback inform later proposals, so an intended movement is not treated as an achieved one.Source: §3.2 · p. 5

3.3. Architecture Overview

  • Five VLM roles divide planning, action proposals, monitoring, verification, and memory, while a deterministic Controller handles physical execution.The Monitor reports task-inconsistent events; the Verifier assesses the success criterion using observations and robot-state evidence, and outcomes determine whether to advance, retry, or replan.Source: §3.3 · pp. 5–6, Figure 3 · p. 6

3.4. Asynchronous Scheduling

  • The main motion loop remains sequential, while a background Monitor can request cancellation at an action boundary and discard pending commands.After an interruption or normal attempt, updated observations, measured state, and alerts inform outcome assessment and possible retry or replanning; memory summaries are generated in the background without blocking the next subgoal.Source: §3.4 · pp. 6–7

4. Experiments

The experiments evaluate whether MotorMind can turn a frozen general-purpose VLM into a manipulation system without task-specific policy training, comparing it with zero-shot and fine-tuned baselines and testing adaptation to changing tasks. MotorMind leads the evaluated zero-shot methods on LIBERO-PRO and succeeds across four adaptive task groups.

4. Experiments

  • The experiments compare task success, robustness to perturbations, and episode wall time on LIBERO-PRO against zero-shot and task-fine-tuned baselines.The base setting uses original episodes; perturbations change semantics, objects, positions, or tasks.Source: §4 · p. 7, §4.1 · p. 7

4.1. Main Experiment

  • 66.7% base and 53.8% perturbed average success made MotorMind the strongest evaluated zero-shot method; its 53.8% perturbed success compared with 51.2% for OpenVLA-OFT.These results are reported for the LIBERO-PRO evaluation.Source: Table 3 · p. 8, §4.1 · p. 8

4.2. Adaptive Tasks Experiment

  • 70% vs 30% on Dynamic Reasoning, 90% vs 70% on Scene Shift, 80% vs 40% on Dynamic Manipulation, and 60% vs 60% on Prompt Shift.The task groups test adaptation to semantic or temporal targets, scene changes, conveyor motion, and revised instructions.Source: Table 4 · p. 9, §4.2 · p. 9
  • Observation-driven execution and asynchronous inference support decisions while objects, scenes, or instructions change during execution.The adaptive evaluation includes moving objects, scene shifts, dynamic manipulation, and prompt shifts.Source: §4.2 · p. 9

5. Real Robot Deployment

MotorMind is deployed on a physical xArm6 robot without task-specific demonstrations or fine-tuning, using camera observations and the standard robot SDK. It achieves high success across direct placement, human perturbations, and semantic-understanding tasks, and a case study illustrates observation-driven correction of an execution error.

Deployment Setup

  • The xArm6 receives observations from the robot and RealSense D455 cameras and executes actions through the standard SDK, without task-specific demonstrations or fine-tuning.The three settings test direct perception, adaptation to human interventions, and semantic understanding.Source: §5 · pp. 9–10

Results and Case Study

  • 95% pooled success across Direct Perception (98%) and Human Perturbations (92%) on the real robot.Each object-destination pair in these settings was evaluated over 10 trials.Source: Table 5 · p. 10, §5 · pp. 10–11
  • 80%, 100%, 100%, and 60% success across the four Semantic Understanding tasks.Each task was evaluated over five trials, and object arrangements were randomized.Source: Table 5 · p. 10, §5 · pp. 10–11
  • After an initial approach toward the wrong region, the Monitor flags the mismatch and the Executor re-localizes the cube and completes placement.The case study depicts placing a blue cube in a box after the robot initially moves toward a white bowl.Source: Figure 5 · p. 10, §5 · p. 11

6. Discussion

Discussion results show that a stronger VLM backbone improves success, while removing replanning or planning reduces it. Failure analysis identifies grounding and premature completion as prominent error sources.

Backbone Sensitivity

  • 83.3% average success with GPT-6 Sol versus 66.7% with Qwen3.8-Flash-Next across the evaluated suites, with longer wall time in every suite.On Spatial, wall time rises from 199.0 s to 370.4 s.Source: §6 · p. 11

Ablation Study

  • Removing the planner reduces success to 0.0%, while removing replanning lowers it from 66.7% to 36.7%.Removing the verifier yields 60.0% success, a 6.7-point drop; shorter wall times in ablated variants do not mean more effective execution.Source: §6 · pp. 11–12

Failure Analysis

  • Grounding errors account for the largest share overall, while premature completion claims are also a major failure source.Grounding errors are especially common under Semantic, Object, and Position perturbations; premature completion is especially common under Task and Object perturbations.Source: §6 · p. 12

7. Conclusion

MotorMind uses a frozen VLM, mid-level actions, and an execution loop with monitoring, memory updates, and outcome verification. The paper situates this approach among robot foundation models, coding agents, and VLM-based control systems.

7. Conclusion

  • MotorMind achieves 66.7% success on base LIBERO-PRO suites, 53.8% under perturbations, and 95% on real-world xArm6 placement tasks.Using GPT-6 Sol raises average success to 83.3%.Source: §7 · p. 12

A.1. Robot Foundation Models

  • Robot foundation policies remain bounded by robot-action training coverage, while MotorMind uses a general-purpose VLM without task-specific robot data or policy fine-tuning.The paper also notes that adapting VLMs to low-level actions can weaken multimodal understanding, instruction following, and semantic reasoning.Source: §A.1 · p. 17

A.2. Coding Agents for Robot Manipulation

  • Coding-agent systems trade online revision latency against offline programs or skills that cannot adapt to unexpected execution changes.Prior approaches include executable robot programs, reusable skills, and execution-feedback optimization of control software.Source: §A.2 · p. 17

A.3. VLM-Based Robot Control

  • VLM-planned systems such as VoLo and Harness VLA rely on frozen VLA policies or analytic primitives, whose physical capabilities limit ready deployment in new settings.Other compact general-purpose VLM interfaces use semantic actions such as translation, rotation, and gripper control.Source: §A.3 · p. 17
  • MotorMind and Show-Harness share a teleoperation-like interface motivation, but differ in scheduling, decision autonomy, motion parameterization, and correction mechanisms.The comparison is methodological rather than experimental because Show-Harness appeared concurrently.Source: §A.3 · pp. 17–18

B. Diagnostic Evaluation Details

The diagnostic evaluates local manipulation decisions using image-conditioned questions, expert demonstrations, and human-reviewed labels. Its prompts distinguish image-space references from robot-frame actions and define separate progress and completion judgments.

B.1. Data Construction

  • The diagnostic separates local action, progress, and completion decisions, with one image-conditioned question about a local subgoal as its evaluation unit.The 240-question set was built from temporally selected demonstration segments and includes 138 distinct subgoal strings and 142 subgoal–criterion pairs.Source: §B.1 · p. 18

B.2. Question Generation and Labels

  • Question construction combines camera views, a local subgoal, its success criterion, and reference answers proposed by Codex and reviewed by a human expert.Codex also had access to construction-time state and rollout evidence beyond the images shown in each question.Source: §B.2 · p. 19
  • Action labels specify one prioritized move, rotation, or gripper primitive, with rotational evidence converted to a rotation vector in the robot base frame.The labels do not specify distance, angle, speed, or a complete executable command.Source: §B.2 · p. 19
  • Progress labels mark whether a transition advances the specified subgoal, while completion labels require the stated success criterion to be satisfied.Negative examples include wrong-object actions, incompatible states, incomplete progress, and simulated perturbations that fail to advance the subgoal.Source: §B.2 · p. 19

Action Selection Prompt

  • Action prompts interpret left and right in the fixed scene-camera image, but express output motion directions in robot-base coordinates.The prompt requests one primary next primitive and no distance or angle.Source: Action Selection Prompt · pp. 19–20

Progress Verification Prompt

  • Progress verification returns a binary label for whether the observed change advances the subgoal; completion is not required for progress.Preparation counts when it advances the stated subgoal.Source: Progress Verification Prompt · p. 20

Subgoal Completion Prompt

  • Subgoal completion returns a positive label only when the stated subgoal and success criterion are satisfied at the later observation.When release is required, holding the object at the target is insufficient.Source: Subgoal Completion Prompt · p. 21

B.3. Prompts and Scoring

  • Table 7 reports action-selection accuracy separately for translation, rotation, and gripper questions.The supplied caption identifies the primitive-type breakdown but gives no outcome values.Source: Subgoal Completion Prompt · p. 21, Table 7 · p. 21

C. MotorMind Implementation Details

MotorMind assigns separate VLM roles to planning, execution, monitoring, verification, memory, and target localization. Structured proposals, measured feedback, and target estimates connect those roles to robot control.

C.1. VLM Roles and Prompts

  • Role-specific VLM calls separate task reasoning, action generation, execution monitoring, outcome verification, and memory construction through distinct inputs and output schemas.The executor receives one subgoal, while the motion supervisor and scene monitor cannot replace an action already being executed.Source: §C.1 · p. 21

Planner Prompt / Replanner Prompt

  • The planner converts a one-sentence task into subgoals whose wording and success criteria guide the executor; the replanner supplies only the remaining steps after a failure.Completed steps are retained by the caller rather than repeated in the replanned list.Source: Planner Prompt · p. 22, Replanner Prompt · p. 22

Executor Prompt

  • The executor receives one subgoal and proposes the next few actions using current camera views and reported robot state.Its decisions concern the subgoal, not the whole task plan.Source: Executor Prompt · p. 22

Motion Supervisor Prompt / Scene Monitor Prompt / Verifier Prompt

Memory Manager Prompt / Target Locator Prompt

C.2. Visual Grounding and Measured Feedback

  • For target localization, the VLM uses labeled image-space boxes, triangulated fixed-camera views, and close wrist views to estimate target offsets and uncertainty.Those estimates are supplied to the executor in the same base-frame direction vocabulary as its actions.Source: §C.2 · p. 23

C.3. Structured Proposals and Auxiliary Actions

  • Executor proposals structure assessment, completion, optional commands, expectations, and confidence, with translation distances in millimeters and rotation angles in degrees.Schema and conversion errors are returned for correction before execution; proposals also support wait and home actions.Source: §C.3 · p. 23

D.1. LIBERO-Pro Tasks and Perturbations

The evaluation spans LIBERO-PRO task families under base and perturbed conditions, compares frozen direct VLA policies and agent frameworks, and includes adaptive-reasoning stress tests across 30 online tasks.

D.1. LIBERO-Pro Tasks and Perturbations

  • Each LIBERO-PRO task family is evaluated under a base condition and four perturbations that vary language, objects, positions, or the instructed task.The base condition is unperturbed; the task matrix uses benchmark-provided configurations.Source: §D.1 · p. 23

D.2. Baseline Configurations

  • The baselines include frozen direct VLA policies and agent frameworks that add planning, tools, or programmatic execution, without additional policy training or fine-tuning.The compared frameworks include Harness VLA, VoLo, and CAP-X; direct policies retain their model-specific input and action interfaces.Source: §D.2 · pp. 24, 26, Table 9 · p. 24

E. Adaptive Reasoning Tasks

  • Four adaptive-reasoning suites comprise 30 tasks testing responses to object-location changes, moving targets, semantic or temporal target selection, and revised instructions.Object identities, state variables, and intervention triggers are not supplied as privileged policy inputs; the study frames these as stress tests rather than precise quantitative evidence.Source: E. Adaptive Reasoning Tasks · p. 26

E.1. Details of the Four Task Suites

The four online suites test scene changes, moving targets, constrained target selection, and instruction updates. Their protocol preserves ongoing scene dynamics, with task success assessed by environment predicates.

E.1. Details of the Four Task Suites

  • Scene Shift displaces a target object or receiver while keeping the instruction fixed, with one to three displacements testing initial adaptation and repeated correction.Later displacements require physical-progress triggers to be reached.Source: §E.1 · p. 26, Table 11 · p. 27
  • Dynamic Manipulation requires locating and placing a named target on a single-pass conveyor moving at 1.5 mm/s.An ungrasped target can leave the available conveyor region; motion does not loop for another pass.Source: §E.1 · p. 27, Table 12 · p. 27
  • Dynamic Reasoning combines conveyor manipulation with target selection using semantic properties, exclusion, spatial relations, or observation history.Spatial references point to the initial arrangement.Source: §E.1 · p. 27, Table 13 · p. 28
  • Prompt Shift changes the instruction after a physical trigger while preserving the robot and scene state, requiring adaptation from the current state rather than a task restart.The suite covers destination redirection, intermediate operations, obstacle reminders, and target replacement.Source: §E.1 · p. 27, Table 14 · p. 28
  • The diagnostic characterizes VLM local decision-making, not reliable low-level control; a gripper-closing trigger alone does not certify a successful grasp.In one task, the trigger requires a positive closing command and three consecutive steps with less than 1 mm change in gripper width.Source: §E.1 · p. 27

E.2. Dynamic Environment Protocol

  • During planning or inference waits, elapsed wall-clock time advances the environment at 20 Hz while the robot receives a zero-motion command.This keeps conveyor and environment dynamics progressing rather than pausing the scene.Source: §E.2 · pp. 27–28
  • Agents receive task instructions, supported camera observations, and robot measurements, but not ground-truth target identities, trajectories, intervention triggers, or evaluator state.Methods share task definitions and intervention rules, although they may reach the trigger at different times.Source: §E.2 · p. 28
  • After an instruction update, MotorMind interrupts the active subgoal and replans from the current observation, prior plan, accumulated memory, and recent execution evidence without resetting the scene.Cancellation and switching time count toward the episode budget.Source: §E.2 · p. 28

F. Additional Ablation Study

The execution-budget ablation finds that additional time alone does not consistently improve task completion; the 300 s setting matches the 450 s setting with less wall time.

Effect of Execution Budget

  • 66.7% average success is achieved with both the 300 s and default 450 s execution budgets.The shorter budget reduces mean wall time from 210.0 s to 159.8 s, a 23.9% reduction.Source: F. Additional Ablation Study · p. 29
  • 25.03 time-normalized success score is reached at 300 s, compared with 19.05 at 450 s.The 300 s setting uses less mean wall time while matching the 450 s average success.Source: F. Additional Ablation Study · p. 29
  • 50.0% average success at 900 s and 60.0% at 3,600 s show that average success does not rise monotonically with the execution budget.The authors conclude that an effective harness must use observations to judge when further action is useful and when execution should stop.Source: F. Additional Ablation Study · p. 29

G. Error Analysis Details

Figure 8 traces MotorMind episodes through failure, recovery, and outcomes for unperturbed and perturbed LIBERO-PRO tasks.

G. Error Analysis Details

  • Figure 8 contrasts episode failure and recovery outcomes on unperturbed tasks at left and perturbed tasks at right.Band thickness represents each outcome’s share of episodes within its panel.Source: Figure 8 · p. 29
  • Each failed episode is assigned to its last unrecovered failure, while recovered episodes had failures but recovered from all of them.Source: Figure 8 · p. 29
  • The figure defines a false-done outcome as declaring task completion while the goal predicate remains false.Source: Figure 8 · p. 29
Loading 2609.38078v1…