PaperlayerView exact revision on arXiv

Source-linked AI summary

VLMs are Native Action Models: Unleashing the Manipulation Capabilities of VLMs with Explicit Action-Space Grounding

Jingkun Yi, Shujie Zhang, Zirui Zhou, Haoxiao Wang, Haiyang Sun, Yuhang Dong, Xudong Xu, Xing Shen, Tao Zhang, Yong Liu, Jiangning Zhang

arXiv:2610.12832v1cs.RO

TL;DR

Pretrained VLMs have manipulation-relevant visual and spatial reasoning, but existing action interfaces either rely on robot data to teach action semantics or separate zero-shot reasoning from a unified trainable policy. EASG grounds continuous action semantics in wrist-camera observations and uses the VLM’s native text generation interface for robot actions and fine-tuning. The paper reports comparable zero-shot performance to fine-tuned VLA models at 27B scale and competitive performance with several VLA baselines after fine-tuning a 0.8B-scale VLM.

  • Problem

    Pretrained VLMs have manipulation-relevant reasoning, but existing VLA interfaces learn action semantics from robot data, while modular zero-shot approaches rely on external execution modules rather than a unified trainable policy.

    Source: §1 · pp. 1–2

  • Method

    EASG projects a TCP-centered action frame into wrist-camera observations and uses the VLM’s native text-generation interface to ground continuous robot actions for direct generation and fine-tuning.

    Source: §1 · pp. 2–3, §3.1 · p. 4

  • Results

    A 27B-scale VLM with EASG achieves comparable zero-shot performance with fine-tuned VLA models, while a fine-tuned 0.8B-scale VLM achieves competitive performance with several VLA baselines.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The same grounded action interface supports zero-shot manipulation and robot-data fine-tuning, including improvements beyond the model’s initial zero-shot capabilities.

    Source: §1 · p. 3, Q2. Can VLMs be fine-tuned beyond… · p. 7

    The paper leaves open how capabilities vary across more VLM families and scales, and whether the interface extends to bimanual and mobile manipulation as action dimensionality increases.

    Source: §7 · p. 10

Abstract

from arXiv · show

Pretrained vision-language models (VLMs) already exhibit strong visual, spatial and reasoning capabilities relevant to robotic manipulation. The key challenge in turning pretrained VLMs into generalist manipulation policies is to expose these capabilities through an appropriate action interface. Vision-language-action (VLA) models provide a unified trainable policy, but typically introduce learned action tokens, action heads, or action experts whose semantics must be learned from robot data pretraining. In contrast, modular foundation-model-based manipulation methods can exploit pretrained VLM capabilities for zero-shot manipulation, but rely on external planning and control scaffolds that are not end-to-end trainable as a unified continuous-action policy. In this work, we introduce EASG, an Explicit Action-Space Grounding framework that bridges these two paradigms. By projecting a tool center point (TCP)-centered action frame into the wrist camera observation and expressing actions in the form of VLM's native textual generation interface, EASG makes the semantics of a continuous robot action space directly observable in the VLM's perceptual context. This allows pretrained VLMs to directly generate executable robot actions using their existing visual-spatial reasoning capabilities, without robot-action pretraining or additional action modules. Besides, the same action interface supports direct fine-tuning with robot trajectories, allowing the models to continuously improve beyond their manipulation capabilities unleashed by EASG. Overall, EASG enables zero-shot manipulation and generalization across object positions, categories, and robot embodiments, while supporting effective fine-tuning in the same continuous action space. These results demonstrate that, given an appropriate action interface, pretrained VLMs can themselves serve as native action models for robotic manipulation.

1 INTRODUCTION

Pretrained VLMs have manipulation-relevant reasoning, but existing action interfaces either require robot-data training to learn action semantics or separate zero-shot reasoning from trainable control. EASG grounds continuous action semantics in the VLM’s perceptual context and uses its native text generation interface for both zero-shot action generation and fine-tuning.

1 INTRODUCTION

  • VLA models offer unified, trainable policies, but their action representations or modules are learned from robot trajectories rather than being directly available to pretrained VLMs.The model must learn action coordinates, metric geometry, and embodiment-specific mappings, placing demands on robot data and model capacity.Source: §1 · pp. 1–2
  • Modular foundation-model methods can support zero-shot manipulation, but external planners and controllers prevent the same VLM policy from being trained end-to-end through that interface.The paper frames this and VLA models as complementary trade-offs: zero-shot use versus a unified trainable action policy.Source: §1 · p. 2
  • EASG projects a TCP-centered action frame into wrist-view observations and expresses actions through the VLM’s native text interface, grounding continuous action semantics in its perceptual context.This lets pretrained VLMs generate executable actions without robot-action pretraining or auxiliary learned planning modules.Source: §1 · pp. 2–3
  • The same action interface supports zero-shot manipulation and robot-data fine-tuning, allowing models to improve existing skills and acquire capabilities beyond their zero-shot capabilities.The paper presents fine-tuning as extending the capabilities initially unleashed by EASG.Source: Figure 1 · p. 2, §1 · p. 3
  • A 27B-scale VLM with EASG achieves comparable zero-shot performance with fine-tuned VLA models, while a fine-tuned 0.8B-scale VLM achieves competitive performance with several VLA baselines.These results demonstrate both zero-shot and fine-tuned manipulation through the same interface.Source: §1 · p. 3

2 RELATED WORK

Related work addresses robot action generation through learned VLA interfaces, modular foundation-model systems, and 3D visuomotor policies. EASG contrasts with these approaches by grounding action-centric geometry for pretrained VLM reasoning without reconstructing the full 3D scene.

VLA Models

  • VLA models use discrete action tokens or additional action modules, while video- or world-model-based policies use pretrained video generators for future visual dynamic priors.The paper characterizes action pretraining as data-intensive because models must jointly infer action coordinates, depth information, and inverse dynamics from robot data.Source: §2 · p. 3

Modular Foundation-Model-Based Manipulation

  • Modular foundation-model methods use intermediate representations and external planning or control, enabling zero-shot manipulation without robot-action pretraining but leaving the perception-to-action pipeline distributed across modules.That distribution prevents end-to-end optimization of the complete pipeline within a unified policy.Source: §2 · p. 3

3D Visuomotor Manipulation

  • 3D visuomotor methods use voxelized RGB-D scenes or virtual multi-view projections, whereas EASG dynamically projects the TCP-frame z-axis toward the target without reconstructing the full 3D scene.The paper says this action-centric grounding with depth information supports pretrained VLMs’ direct reasoning about 3D motion for zero-shot manipulation and subsequent fine-tuning.Source: §2 · p. 3

3 METHOD

EASG grounds continuous robot-action semantics in the VLM’s perceptual context and uses the model’s native text interface to generate actions. The same action space supports zero-shot execution and robot-data fine-tuning.

3.1 EXPLICIT ACTION-SPACE GROUNDING

  • EASG anchors a local action frame at the current TCP and grounds its origin, axes, parameterization, and metric scale in the observation.The frame axes define incremental translation and rotation directions; the prompt specifies their units and output semantics.Source: §3.1 · p. 4
  • The action-frame origin and axes are projected into the wrist-camera image, while aligned depth estimates distance to the nearest visible surface along the approach direction.This supplies metric 3D information rather than requiring precise physical distance to be inferred from RGB appearance alone.Source: §3.1 · pp. 4–5
  • The VLM predicts motion in the grounded frame, while deterministic transforms, inverse kinematics, and low-level control produce robot-specific commands.The predicted translation and rotation increments are first transformed into the robot base frame.Source: §3.1 · p. 5

3.2 VLM-NATIVE TEXTUAL ACTION LEARNING

  • Continuous actions use the pretrained VLM’s native text vocabulary, encoding translation as a unit direction and distance and rotation as an axis and angle.The directions are expressed in the grounded action frame, with distance in centimeters and angle in degrees.Source: §3.2 · p. 5
  • Demonstrated action sequences fine-tune the same autoregressive policy with causal language-model cross-entropy, while deterministic grammar-constrained decoding ensures valid action formats.Fine-tuning retains the existing action interface; unconstrained free-form decoding may produce malformed values, missing fields, or extraneous text.Source: §3.2 · p. 5

4 EXPERIMENTS

Experiments separate easier Task Group 1 tasks for zero-shot evaluation from harder Task Group 2 tasks for fine-tuning, then examine simulation performance, memory and thinking, and model scaling.

4 EXPERIMENTS

  • Task Group 1 probes zero-shot capability on relatively simple tasks, while Task Group 2 tests fine-tuning on harder tasks requiring greater precision, complex interactions, or longer horizons.Source: §4 · p. 6

Q1. Can pretrained VLMs perform zero-shot manipulation with explicit action-space grounding?

Q2. Can VLMs be fine-tuned beyond their zero-shot manipulation capabilities?

4.3 CROSS-EMBODIMENT TRANSFER

  • Thinking is key to extending VLM-native manipulation, while memory mainly adds gains on longer-horizon tasks through temporal context.Source: Table 7 · p. 8, §5.1 · p. 9

Q5. How do VLM family and scale shape the boundary of zero-shot manipulation?

6 DIAGNOSTIC ABLATIONS

The ablations identify action-frame grounding, metric-depth grounding, and grammar-constrained decoding as important to zero-shot action grounding and reliable inference.

6 DIAGNOSTIC ABLATIONS

  • Figure 5 presents a zero-shot capability frontier across VLM families and scales.Source: Figure 5 · p. 9

6.1 WHAT ENABLES EFFECTIVE ZERO-SHOT ACTION GROUNDING?

  • Removing either action-frame or metric-depth grounding reduces average success from 79.4% to 0% in real-world zero-shot manipulation.Table 8 reports performance on individual tasks under the same setup as Sec. 4.1.Source: §6.1 · p. 9, Table 8 · p. 9

6.2 DOES GCD IMPROVE RELIABLE INFERENCE?

  • Removing grammar-constrained decoding reduces average LIBERO success from 83.28% to 72.6%, with GCD improving performance across all suites.The comparison fine-tunes Qwen3.5-0.8B with and without GCD in the same setting as Sec. 4.2.Source: §6.2 · p. 9, Table 9 · p. 9

7 DISCUSSIONS AND FUTURE WORKS

The discussion identifies open questions about the scope and scaling of EASG. The related-work discussion describes VLA action-generation approaches and the grounding tasks their training must learn.

7 DISCUSSIONS AND FUTURE WORKS

  • Future work will characterize zero-shot capabilities and reasoning budgets across more VLM families and scales, and test EASG for bimanual and mobile manipulation.The authors also propose studying whether pretraining with the native textual action interface improves visual-spatial reasoning and zero-shot manipulation without task-specific demonstrations.Source: §7 · p. 10

A.1 VISION-LANGUAGE-ACTION MODELS

  • VLA models generate actions through discrete tokens or added continuous-action modules, while robot data supports their trainable manipulation capabilities.Examples include discretized scalar actions, adaptive spatial tokens, compressed action chunks, action experts, diffusion-transformer modules, and MLP action heads.Source: §A.1 · p. 17
  • VLA training must implicitly learn action-frame semantics, depth from RGB, and mappings among joint configurations, end-effector poses, observations, and executable poses.The authors say this joint learning consumes robot data and adds data and training costs; multimodal-pretraining/action-data distribution mismatch may further degrade pretrained reasoning.Source: §A.1 · p. 17

A.2 CODE AS POLICIES AND ROBOT-USE AGENTS

Code-as-policy and Robot-Use Agent methods use pretrained-model outputs or tools with external components to produce robot motion. EASG instead grounds continuous actions in the VLM interface, which is also used for robot-data fine-tuning.

A.2 CODE AS POLICIES AND ROBOT-USE AGENTS

  • Code-as-policy methods turn model outputs into robot motion through external perception, planning, or control, using programs, spatial constraints, keypoints, points, or image-space trajectories.Examples include generated programs, 3D value maps, relational keypoint constraints, affordance points lifted with depth, and 2D paths for a separate control policy.Source: §A.2 · pp. 17–18
  • EASG grounds executable continuous actions in the VLM interface for zero-shot generation and robot-data fine-tuning; Figure A1 compares methods across five interface and capability dimensions.Source: §A.2 · p. 18, Figure A1 · p. 18
  • Robot-Use Agents orchestrate callable perception, planning, execution, and control tools; cited systems also use execution feedback, skill synthesis, or coding-model post-training.CaP-X and VIA illustrate agent/tool interfaces, while ENPIRE and ASPIRE explore autonomous policy improvement.Source: §A.2 · pp. 18–19
  • Modular methods do not directly generate executable continuous actions, so performance can depend on scaffold abstractions and capabilities.Their pipelines can also add perception and planning components, latency, and engineering overhead, especially in closed-loop manipulation.Source: §A.2 · p. 19
  • When zero-shot reasoning is insufficient, modular methods lack a unified way to fine-tune the same foundation model as a continuous-action policy.Other adaptations may improve programs, skills, coding agents, or separate downstream policies instead.Source: §A.2 · p. 19

Linguistic Action-Space Grounding

  • The zero-shot prompt identifies a gripper-attached TCP frame and specifies incremental translation and rotation in that frame.It also gives the axes and positive directions, instructs use of metric TCP-distance information, and defines a single-line action format with task and gripper state.Source: Linguistic Action-Space Grounding · pp. 19, 21

B.2 IMPLEMENTATION OF GRAMMAR-CONSTRAINED DECODING

Grammar-constrained decoding restricts generated actions to valid syntactic forms while retaining the VLM's preferences among valid continuations. Zero-shot execution uses the resulting textual actions in a closed loop.

B.2 IMPLEMENTATION OF GRAMMAR-CONSTRAINED DECODING

  • The action grammar serializes task state, translation, rotation, and gripper state, with signed values and fixed decimal precision.Translation uses a 3D unit direction and distance in centimeters; rotation uses a 3D unit axis and signed angle in degrees.Source: §B.2 · pp. 21–22
  • The grammar enforces serialization syntax, not semantic constraints such as unit norms, nonnegative distance, valid gripper range, or task-completion meaning.Those constraints are specified by the policy prompt and handled separately from decoding grammar.Source: §B.2 · p. 22
  • Token-level prefix validation accepts a token only when its entire decoded text fits the grammar, including subword tokens that span grammar symbols.End-of-sequence is allowed only in an accepting state; malformed separators, missing signs, invalid decimal formats, whitespace, and natural-language tokens are rejected.Source: §B.2 · p. 22
  • Grammar-constrained decoding masks tokens that cannot continue a valid action syntax, while preserving the original decoder's preferences among valid continuations.The grammar is implemented as an inference-time logits processor without learned parameters, an extra action head, or robot-specific action tokens.Source: §B.2 · p. 22

C ZERO-SHOT EXPERIMENTS

  • In zero-shot execution, the pretrained VLM receives observations, an instruction, and action-space grounding, outputs a textual action, and repeats after execution and re-observation.Source: C ZERO-SHOT EXPERIMENTS · p. 23

C.1 EXPERIMENT SETTINGS

The experiments span real-world manipulation, data-intensive VLA comparisons, randomized object configurations, and LIBERO simulation tasks. On unseen objects and configurations, fine-tuned VLA baselines drop substantially while EASG largely preserves its zero-shot performance.

C.1 EXPERIMENT SETTINGS

  • Real-world experiments use a UR3 with wrist and external cameras and cover rigid, deformable, articulated, precise/contact-rich, and long-horizon manipulation tasks.Task definitions and success criteria are summarized in Table A2.Source: §C.1 · p. 23, Table A2 · p. 24

C.2 BASELINE MODELS

  • VLA baselines are fine-tuned with 10–50 task demonstrations, while EASG uses none, to assess the robot data needed to match EASG’s zero-shot performance.The comparison includes π0.5, GR00T N1.6, π0-FAST, and OpenVLA-OFT.Source: §C.2 · p. 25, Table A3 · p. 25

C.3 RANDOMIZATION SETTINGS

  • Generalization tests replace objects in Tasks 1.1–1.3 with five unseen objects and vary object positions and poses across three configurations per object.Positions and poses vary independently within the 60 × 60 cm workspace while task semantics and success criteria are preserved.Source: §C.3 · p. 25, Figure A4 · p. 26
  • Despite randomized-position and pose demonstrations, fine-tuned VLA baselines drop substantially on unseen objects and configurations, while EASG largely preserves zero-shot performance.The randomization evaluates five unseen objects under three configurations per object.Source: §C.3 · p. 25

D.1 TASKS AND METRICS OF SIMULATION BENCHMARKS

  • LIBERO evaluation covers its Spatial, Object, Goal, and Long suites and reports episode-level task success rate.Success is the fraction of rollouts satisfying the environment’s language-conditioned goal.Source: §D.1 · p. 25

D.2 TASKS AND METRICS OF REAL-WORLD SCENES

Real-world fine-tuning targets tasks beyond zero-shot capability and evaluates success over repeated trials. Representative trajectories show the fine-tuned VLM completing those tasks after training with EASG.

D.2 TASKS AND METRICS OF REAL-WORLD SCENES

  • Fine-tuning tasks are harder than their zero-shot counterparts, and each task-method pair is evaluated over 30 trials with mean success rate and standard deviation reported.The trials comprise three independent rounds of 10; Tasks 1.1–1.3 assess improvement over zero-shot ability, while Tasks 2.1–2.4 test harder tasks.Source: §D.2 · p. 26, Table A4 · p. 26

D.3 REAL-WORLD TRAINING PROTOCOL

  • The real-world training set contains 30 demonstrations per task by default, recorded at 20 Hz and converted into EASG’s camera-projected, TCP-centered action representation.RGB observations, proprioception, end-effector poses, and gripper commands are recorded during teleoperation.Source: §D.3 · p. 26
  • After EASG fine-tuning, Qwen3.8-27B reliably completes Task Group 2 tasks designed beyond its native zero-shot capability.These tasks require more precise spatial reasoning, object-orientation control, or longer-horizon action sequences; Figure A5 shows representative successful rollouts.Source: §D.3 · p. 27, Figure A5 · p. 27

E.1 EXPERIMENT DESIGN

Cross-embodiment evaluation tests whether one fine-tuned EASG checkpoint transfers from UR3 to Flexiv without further fine-tuning. The evaluation uses matched starting end-effector poses and repeated trials on both robots.

E.1 EXPERIMENT DESIGN

  • The same fine-tuned checkpoint and action-grounding interface are tested on UR3 and Flexiv from matched end-effector poses, with no additional Flexiv fine-tuning or embodiment-specific demonstrations.Both robots use the same gripper but differ in morphology, kinematics, and low-level control.Source: §E.1 · pp. 27–28, Figure A6 · p. 28
  • Cross-embodiment success rates are reported as mean ± standard deviation over three rounds of 10 trials per task.Table A6 reports results for the original UR3 and unseen Flexiv embodiments.Source: §E.1 · p. 28, Table A6 · p. 28

F FULL ANALYSIS OF ABLATION STUDY

The ablations isolate grounding choices under a shared setup and compare TCP-centered grounding with alternative coordinate frames. A separate synchronized-rollout visualization depicts execution across robot embodiments.

F.1 WHAT ENABLES EFFECTIVE ZERO-SHOT ACTION-SPACE GROUNDING?

  • All zero-shot ablations use the same robot setup, VLM backbone, prompting protocol, and inference configuration, changing only the grounding component.Source: §F.1 · p. 28
  • The action-frame ablation compares TCP-centered grounding with robot-base-centered and wrist-camera-centered frames to test the role of aligning actions with the controllable tool frame.Figure A8 visualizes representative rollouts for the full grounding and both alternatives on the same zero-shot task.Source: §F.1 · p. 28, Figure A8 · p. 30
  • A synchronized rollout figure shows UR3 and Flexiv executing the same task with one EASG checkpoint, while embodiment-specific inverse kinematics and low-level controllers carry out predicted task-space actions.Source: Figure A7 · p. 29
Loading 2610.12832v1…