GTA-2: A Multi-VLM Framework for Synthesizing Robot Manipulation Skills via Grounded Task Axes

M. Yunus Seker; Shobhit Aggarwal; Ruwan Wickramarachchi; Jonathan Francis; Oliver Kroemer

Source-linked AI summary

GTA-2: A Multi-VLM Framework for Synthesizing Robot Manipulation Skills via Grounded Task Axes

M. Yunus Seker, Shobhit Aggarwal, Ruwan Wickramarachchi, Jonathan Francis, Oliver Kroemer

arXiv:2609.09808v1Sep 9, 2026cs.RO

TL;DR

Robotic manipulation skills often lack an explicit representation for the geometric, control, and scene-dependent decisions needed during execution. GTA-2 uses specialized VLM agents to synthesize and ground modular task-axis skills, achieving stronger zero-shot performance than evaluated baselines while supporting targeted refinement.

Problem
Existing manipulation behaviors can be too coarse to expose the scene-specific geometric and control choices required for execution.
Method
GTA-2 separates task decomposition, abstract task-axis skill synthesis, parameter assignment, and visual grounding into modular VLM stages that produce executable robot programs.
Results
73.9% average zero-shot success exceeded CaP-Primitive at 42.5% by 31.4 percentage points, while one targeted refinement raised success to 90.7%.
Takeaways & Limitations
Explicit intermediate representations enable targeted human feedback that preserves correct skill components while refining incorrect stages.
Takeaways & Limitations
GTA-2 is constrained by the expressivity of its controller and feature libraries, single-observation grounding, and evaluation on one robot and sensing configuration.

Abstract

from arXiv · show

Robotic manipulation tasks are often decomposed into behaviors or skills. However, one often needs to predefine these behaviors for specific tasks or try to cover a wide range of tasks using generic skills. As a result, these behaviors can remain too coarse to expose the geometric, control, and scene-dependent decisions required for execution. We introduce Grounded Task Axes v2 (GTA-2), a modular multi-VLM framework that constructs executable, task-bespoke manipulation skills from reusable object-centric task-axis components. Rather than predicting actions end-to-end or composing fixed task-level primitives, GTA-2 represents each skill as semantic subtasks comprising task-relevant keypoints and axes, controller compositions, and scene-dependent parameters. Four specialized VLM agents separately decompose the task, construct an abstract task-axis skill, assign controller parameters, and ground the required visual features from RGB-D observations. This abstraction-to-grounding factorization enables zero-shot skill generation without task-specific robot demonstrations, policy training, or fine-tuning. It also keeps intermediate decisions explicit, allowing targeted human feedback to refine an incorrect stage while preserving correct components. We evaluate GTA-2 on 14 real-robot manipulation tasks against a VLA policy pi_{0.5} and two Code-as-Policies baselines using task-axis controllers or conventional robot primitives. GTA-2 achieves an average zero-shot success rate of 73.9%, exceeding the strongest baseline by 31.4 percentage points, while targeted refinement raises GTA-2's average success rate to 90.7%. Project page: https://gta2-project.github.io/

I. INTRODUCTION

GTA-2 addresses the gap between coarse manipulation skills and the scene-specific geometric and control choices required for execution. It uses modular VLM stages to construct explicit, reusable task-axis skills and supports targeted human refinement.

Motivation: Generic manipulation behaviors remain too coarse to specify relevant object parts, end-effector orientation, and geometric or contact constraints.Natural-language task intent alone does not determine these scene-specific choices.
Representation: Task-axis components represent manipulation through reusable object-centric geometric features and controller-level compositions.Examples include combining spiral motion with force control for erasing and positional with rotational control for pouring.
Contribution: GTA-2 constructs explicit task-axis skills from high-level prompts and visual scenes instead of generating robot actions end-to-end.The representation connects language-level task reasoning to controller-level execution through keypoints, axes, controller compositions, and scene-specific parameters.
Method: Four specialized VLM agents decompose tasks, generate abstract skill recipes, assign parameters, and ground visual features for execution.The pipeline separates abstract skill construction from scene-specific grounding.
Refinement: Targeted feedback can revise decomposition, recipes, parameters, or visual groundings while preserving correct components.This supports human-guided adaptation without retraining an end-to-end policy.
Evaluation: The evaluation covers 14 real-robot manipulation tasks against a VLA policy and two Code-as-Policies baselines.Experiments assess zero-shot execution, scene generalization, and targeted human refinement.

II. RELATED WORK

Prior work connects language and robot behavior through end-to-end VLA policies or code and primitive libraries, but these approaches often leave behavior structure implicit or encapsulate grounding and control inside atomic skills. GTA-2 instead constructs structured, grounded skills with VLM agents and reusable task-axis components.

Task-axis controllers: Task-axis controllers express motion, orientation, force, and constraint objectives relative to semantically meaningful keypoints and axes.This provides the finer-grained representation used by GTA-2.
LLM-based plan and code generation: Code-as-Policies generates executable programs by composing APIs or predefined primitives, which may already encapsulate substantial perception, grounding, and control logic.The approach therefore depends on libraries containing behaviors capable of solving the task.
Vision-language-action models: VLA models map visual observations and language directly to actions, but their internal task decomposition and scene reasoning remain largely implicit.This makes intermediate behavior decisions difficult to inspect.
Vision-language-action models: GTA-2 uses VLM agents to construct structured zero-shot skills that can be grounded in the current scene and compiled into robot execution.This complements rather than trains an end-to-end action policy.
GTA-2: GTA-2 maps language, RGB-D observations, and a reusable controller library to an executable robot program without task-specific demonstrations or policy fine-tuning.Its intermediate stages produce abstract skills before scene-specific grounding and compilation.

A. Task-Axis Skill Representation

GTA-2 represents manipulation as prioritized compositions of parameterized controllers grounded in object-centric keypoints and axes. Lifted compositions preserve reusable relationships, while grounding supplies scene-specific metric features and numerical parameters.

Feature representation: Keypoints identify task-relevant locations and axes identify task-relevant directions on objects, the robot, or the world frame.Available features include object locations, tool-frame features, fixed world directions, and derived geometric axes.
Controller representation: A task-axis controller defines a parameterized objective over controlled and reference features, an optional task axis, parameters, and priority.The controller type determines which fields are required and how they are interpreted.
Controller composition: Multiple controllers form a prioritized skill step whose objectives regulate position, orientation, force, motion, or gripper state.Lower-priority commands are projected into higher-priority null spaces to avoid interference.
Example: The erasing example combines constant force along a board normal with spiral motion around the scribble center.Its lifted form uses force and radius placeholders before scene-specific grounding and parameter assignment.
Grounding: A lifted controller becomes grounded when feature definitions resolve to metric values and parameter placeholders receive numerical assignments.Grounding preserves controller types, feature relationships, and priority ordering while adapting values to the current scene.

B. Lifted Skill Synthesis

Lifted skill synthesis converts an instruction into ordered semantic subtasks and maps each subtask to prioritized controller compositions. The resulting representation specifies required relationships and placeholders before metric grounding and numerical assignment.

Task Decomposer: The Task Decomposer converts an instruction into an ordered sequence of semantic subtasks without specifying their controllers.These subtasks expose temporal and semantic structure for subsequent skill generation.
Examples: In the bottle-placement example, subtasks include approaching, grasping, lifting, moving over the box, placing, and releasing.The move-over-box step combines position and axis alignment controllers.
Skill Generator: The Skill Generator maps each semantic subtask to a lifted skill step by selecting controllers, features, parameter placeholders, and priorities.The resulting steps remain abstract because their feature definitions are not yet grounded.
Lifted skill: The full lifted skill preserves subtask order across steps and controller priority order within each step.It specifies controller types and geometric relationships but not metric feature values or numerical parameters.

C. Scene-Specific Skill Grounding

GTA-2 grounds lifted task-axis skills by separately assigning scene-dependent numerical parameters and mapping abstract keypoints and axes to metric values. These operations produce a complete grounded skill while preserving its controller composition, ordering, and priorities.

C. Scene-Specific Skill Grounding: Feature grounding maps lifted keypoints and axes to 3D points or unit directions, while parameter assignment fills placeholders with task- and scene-appropriate numerical values.Together, these operations are called scene-specific skill grounding.
C. Scene-Specific Skill Grounding: The Parameter Setter assigns controller-specific quantities such as offsets, forces, trajectory scales, and timings without changing controller types, feature definitions, or priority ordering.
C. Scene-Specific Skill Grounding: The Vision Module grounds requested features using RGB images and depth, with derived axes computed geometrically and robot- or world-relative features obtained from calibrated frames and forward kinematics.
C. Scene-Specific Skill Grounding: Grounding preserves temporal ordering, controller composition, and relative priorities while replacing abstract feature definitions and parameter placeholders with metric and numerical values.For erasing, the grounded skill includes a 3 N force magnitude and 5 cm spiral radius.

D. Robot Script Compilation and Execution

A deterministic compiler converts each grounded task-axis skill into an executable robot program by instantiating library controllers with grounded features, numerical parameters, and priorities. During execution, ordered controllers produce combined commands that are sent through the robot interface.

D. Robot Script Compilation and Execution: The compiler selects controller implementations from the library and inserts grounded features, numerical parameters, and priority information into the generated robot program.
D. Robot Script Compilation and Execution: During execution, grounded skill steps run in temporal order, while controller commands are combined according to their prescribed priority ordering before transmission through the ROS-based robot interface.

E. Human-Guided Targeted Refinement

GTA-2 routes human feedback to the pipeline stage responsible for an erroneous intermediate output. When upstream outputs change, dependent stages are recomputed while unaffected results are retained.

E. Human-Guided Targeted Refinement: Feedback can target decomposition, controller or feature-definition, numerical-parameter, or geometric-grounding errors at the corresponding pipeline stage.
E. Human-Guided Targeted Refinement: A correction confined to the Parameter Setter reuses the existing task decomposition, controller composition, and feature groundings while revising only affected parameter assignments.
E. Human-Guided Targeted Refinement: When corrections alter upstream outputs, downstream dependent stages are recomputed, while unaffected intermediate results are preserved without retraining or regenerating the entire pipeline.

IV. EXPERIMENTAL RESULTS

GTA-2 is evaluated on 14 real-robot manipulation tasks spanning diverse task types. The evaluation tests zero-shot skill generation, recovery through agent-level feedback, and generalization across scene configurations.

IV. EXPERIMENTAL RESULTS: GTA-2 is evaluated on 14 real-robot tasks covering pick-and-place, articulated-object interaction, contact-rich manipulation, tool use, and constrained motion.
IV. EXPERIMENTAL RESULTS: The evaluation asks how reliably GTA-2 generates executable skills without task-specific examples and how effectively agent-level feedback recovers unsuccessful generations.
IV. EXPERIMENTAL RESULTS: The evaluation also examines how initial and refined skills generalize across scene configurations.

A. Experimental Setup

The evaluation uses a shared controller library and compares GTA-2 with an end-to-end VLA policy and two Code-as-Policies baselines across diverse real-robot manipulation tasks.

Controller library: The controller library supports concurrent position, orientation, force, and gripper objectives across all evaluated tasks.This provides a common task-axis controller interface for the experiments.
Experimental platform: GTA-2 is evaluated on a UR5e robot with RGB observations and point-cloud-derived 3D keypoints and task axes.The setup uses a Robotiq 2F-85 gripper and a fixed ZED 2i stereo camera calibrated to the robot base frame.
Baselines: The comparison includes π0.5, CaP-TAC with task-axis controllers, and CaP-Primitive with conventional motion and force primitives.All methods receive the same task instructions and scene observations.
Baseline design: CaP-TAC directly generates executable code while selecting, grounding, and composing task-axis controllers without GTA-2’s intermediate abstractions.CaP-Primitive instead uses absolute Cartesian motion, gripper actuation, and contact-force motion.
Baseline design: Both Code-as-Policies baselines use Gemini-3.1-Pro and three fixed few-shot demonstrations that exclude the benchmark tasks.The demonstrations cover pick-and-place, force-controlled surface contact, and point traversal.

C. Zero-Shot Generation and Targeted Refinement

GTA-2 generates executable skills without task-specific examples and improves unsuccessful generations through localized feedback, while refined skills generalize across scene configurations.

Zero-shot generation and refinement: Feedback is directed to the responsible module while valid intermediate representations are preserved, evaluating structured refinement rather than full regeneration.This setup tests whether localized corrections can recover unsuccessful generations without discarding already resolved decisions.
Zero-shot generation and refinement: 73.3% of 210 independent runs succeed without feedback, including tasks involving sustained contact and tool use.Human feedback raises cumulative success to 86.2% after one round and 99.0% after two rounds, with all runs succeeding within three rounds.
Zero-shot generation and refinement: Initial corrections primarily target the Skill Generator, then shift to the Vision Module after abstract planning decisions are resolved.The transition matches GTA-2’s abstraction-to-grounding order and makes residual scene-specific errors independently correctable.
Task generalization and baseline comparison: 90.7% average success follows one successful refinement, compared with 73.9% for GTA-2 Zero-Shot and 42.5% for CaP-Primitive.The zero-shot result exceeds the strongest baseline by 31.4 percentage points, and the refined skill remains effective across subsequent scene configurations.
Task generalization and baseline comparison: CaP-TAC outperforms CaP-Primitive on pouring, cutting, and sweeping, where contact constraints, object-relative motion, and tool geometry matter.The comparison indicates that task-axis controllers support contact-rich and tool-use behaviors, while requiring more structured reasoning.

V. LIMITATIONS AND FUTURE WORK

GTA-2's current scope is limited by its task-axis and controller libraries, single-observation grounding, and evaluation on one robot and sensing setup, while its modular design supports future extensions.

GTA-2's skill synthesis is constrained by the expressivity of its task-axis representation and available controller library.Objectives outside the available interface require extending feature or controller libraries.
GTA-2's modular pipeline could accommodate additional feature definitions, controller types, and sensing modalities without changing its overall structure.
Scene-object features are grounded before execution and are not updated online, so motion or slippage can require renewed grounding or refinement.Perceptual grounding remains an important residual source of failure after the manipulation structure is established.
The physical evaluation uses a single tabletop robot, gripper, and sensing configuration, leaving broader generality across embodiments and sensors unestablished.The authors identify additional embodiments and sensing configurations as needed for future evaluation.