PaperlayerView exact revision on arXiv

Source-linked AI summary

GT-VLA: Target-Conditioned Trace Guidance for Generalizable Robotic Manipulation

Ninghan Zhong, Jing-Chen Peng, Sriram Vishwanath

arXiv:2609.31904v1cs.ROcs.LG

TL;DR

VLAs struggle with unseen tasks, configurations, and long-horizon compositions, while external semantic guidance may not reach low-level action generation. GT-VLA turns guidance from an off-the-shelf generalist VLM into traces that condition action generation, and outperforms recent VLA baselines in simulation and on a physical robot.

  • Problem

    VLAs struggle with unseen objects, novel scene configurations, and long-horizon task compositions, and high-level semantic signals are often ignored by low-level controllers.

    Source: A. Vision-Language-Action Models · p. 1, §I · p. 1

  • Method

    GT-VLA converts generalist VLM skill guidance into visual traces that condition a skill-specific trace-and-action Mixture-of-Experts policy.

    Source: §I · p. 1

  • Results

    Across simulation and a physical robot platform, GT-VLA outperforms recent VLA baselines.

    Source: §I · p. 1

  • Takeaways & Limitations

    Ablations show that the intermediate trace interface surpasses direct target guidance for steerability and out-of-distribution performance.

    Source: §V · p. 8

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have shown strong performance on robotic manipulation, but they often struggle to generalize to unseen tasks, configurations, and long-horizon settings. A key challenge is that VLAs overfit to training scenes and fail to follow novel language instructions. Off-the-shelf vision-language models (VLMs) often provide stronger generalization, but cannot directly control robot actions. To combine the common sense of VLMs with VLA control, we propose Guided Trace VLA (GT-VLA), a steerable framework that accepts guidance from an external generalist VLM through trace-conditioned action generation. GT-VLA uses a generalist model to identify semantic guidance for the current skill, converts this guidance into a 2D visual trace, and conditions its action policy on the resulting trace-rendered observation. This design separates semantic target acquisition, trace generation, and low-level action execution, allowing high-level guidance to propagate to robot actions. GT-VLA uses a Mixture-of-Experts architecture with skill-specific trace and action modules for robust execution. We evaluate GT-VLA on LIBERO and a physical robot platform, showing improved generalization over recent VLA baselines in both settings. The code and additional supplemental materials are available on our project website at https://ivaniz.github.io/gt-vla/.

I. Introduction

GT-VLA converts off-the-shelf VLM guidance into visual traces that condition VLA action generation, targeting the gap between semantic guidance and low-level execution. It combines skill-specific trace and action modules with strategies for respecting guidance and handling imperfect traces.

I. Introduction

  • GT-VLA uses a generalist VLM to decompose tasks into skills and provide semantic targets, which become 2D traces conditioning downstream action generation.Source: §I · p. 1
  • Across simulation and a physical robot, GT-VLA outperforms recent VLA baselines, supporting semantic-trace conditioning for steerable, generalizable manipulation.Source: §I · p. 1
  • Its Trace-Action Mixture-of-Experts couples skill-specific trace generation with trace-guided action execution.Data augmentations encourage adherence to semantic guidance and robustness to imperfect generated traces.Source: §I · p. 1

A. Vision-Language-Action Models

  • VLAs perform well in-distribution but struggle with unseen objects, novel configurations, and long-horizon compositions, while visual overfitting can persist despite task decomposition.Source: A. Vision-Language-Action Models · p. 1

B. Runtime Verification and Steering

C. Visual Prompting and Trace Guidance

GT-VLA connects external VLM guidance to low-level action execution through semantic targets and 2D traces. Its skill-based pipeline decomposes tasks, generates target-conditioned traces, and uses them to guide actions.

C. Visual Prompting and Trace Guidance

  • Visual prompting and intermediate representations connect semantic intent to low-level action in prior work, including point and box prompts for zero-shot segmentation.Robotics examples include VLM-generated motion constraints, keypoint affordances, spatial action encodings, and visual latent planning.Source: C. Visual Prompting and Trace Guidance · p. 2
  • GT-VLA differs from prior trace-guidance approaches by explicitly decomposing long-horizon tasks into skills and using an off-the-shelf VLM without fine-tuning.It decouples semantic guidance from trace generation and low-level control.Source: C. Visual Prompting and Trace Guidance · p. 2

III. Method

  • GT-VLA uses sparse guidance from an off-the-shelf generalist VLM, converts it into a dense visual trace, and uses the trace to guide action generation.The architecture routes skills to trace and action experts; a separate MLP predicts skill progress.Source: §III · p. 3, Figure 2 · p. 3

A. Skill Interface and Semantic Target Acquisition

  • A finite library of parameterized manipulation skills lets the VLM turn an instruction and observation into an ordered skill plan.For example, PLACE can be instantiated as PLACE(bowl, table), and the plan breaks a long-horizon instruction into atomic skills.Source: A. Skill Interface and Semantic Target… · p. 3
  • At each skill’s start, the VLM predicts a normalized-image target that represents a spatial goal such as an object, handle, or placement location.The target anchors trace generation and translates sparse high-level guidance into an end-effector path.Source: A. Skill Interface and Semantic Target… · p. 3

B. Target-Conditioned Trace Generation

  • The trace generator predicts the remaining image-space end-effector trajectory for the current skill as a fixed-length sequence of 2D points.Traces are resampled by linear interpolation along their arc length.Source: B. Target-Conditioned Trace Generation · p. 3
  • The trace model uses flow matching, conditioned on the observation, current end-effector position, target, and language combining the task with the current skill.Training uses a skill-annotated demonstration dataset and projected end-effector traces.Source: B. Target-Conditioned Trace Generation · p. 3
  • The current end-effector position constrains the trace’s starting point, while Adaptive RMSNorm conditioning anchors it to the semantic target.Source: B. Target-Conditioned Trace Generation · p. 3

C. Robust Trace-Conditioned Action Learning

GT-VLA trains action generation to use visual traces while handling imperfect guidance and 2D–3D ambiguity. Skill-routed experts separate trace generation and action execution, while a progress head supports skill transitions.

C. Robust Trace-Conditioned Action Learning

  • The action expert maps trace-conditioned observations to 6-DoF robot motion and gripper commands.The trace is rendered onto the visual observation, and the policy is trained to follow it while remaining robust to imperfect traces.Source: C. Robust Trace-Conditioned Action… · p. 4
  • The authors propose three data augmentation techniques to make the action expert respect trace guidance while considering scene geometries and bridge the gap between training demonstrations and imperfect inference traces.Source: C. Robust Trace-Conditioned Action… · p. 4
  • Random scene drop masks the scene while retaining the trace, encouraging action generation to rely on trace guidance rather than memorized visual configurations.Source: C. Robust Trace-Conditioned Action… · p. 4
  • Random trace drop removes the 2D overlay during training, preserving the policy’s ability to reason from the scene and robot state when traces are geometrically ambiguous or inaccurate.Source: C. Robust Trace-Conditioned Action… · p. 4
  • Low-frequency trace perturbation shifts or bends training traces to address the distribution gap between demonstrated traces and generated inference traces.Source: C. Robust Trace-Conditioned Action… · p. 4

D. Trace-Action Mixture-of-Experts Architecture

  • Deterministic skill routing assigns one action expert to each dataset skill category, addressing ambiguity when skills such as PICK and PLACE have similar traces but require different actions.Only the expert for the current skill runs; each is a 300M-parameter transformer.Source: D. Trace-Action Mixture-of-Experts… · p. 4
  • A lightweight flow-matching transformer generates traces using the same skill-routed mixture-of-experts design as the action generator.The trace generator has approximately 60M parameters.Source: D. Trace-Action Mixture-of-Experts… · p. 4
  • An MLP predicts current-skill progress from the shared VLM backbone’s final-layer hidden states, and inference advances when progress reaches a chosen threshold.Source: D. Trace-Action Mixture-of-Experts… · p. 4

E. Inference Algorithm

GT-VLA turns a task prompt into a skill plan and semantic target, then uses periodically refreshed visual traces to guide action execution and determine when to advance skills. The current skill plan is open-loop; closed-loop replanning is left for future work.

E. Inference Algorithm

  • GT-VLA queries a generalist VLM to decompose the prompt into a skill plan and identify a semantic target for each skill.It then executes each skill through nested loops for trace refresh and action execution.Source: E. Inference Algorithm · p. 4
  • A visual trace is regenerated every fupd action chunks, while predicted skill progress is checked against ηc to decide whether to advance.If the threshold is not reached, the policy samples and executes actions, updates the observation, and continues.Source: E. Inference Algorithm · p. 4
  • The high-level skill plan currently runs open-loop; periodic VLM replanning for failure recovery is proposed but not evaluated.The algorithm can be modified for this closed-loop extension, which is left to future work.Source: E. Inference Algorithm · p. 4
  • In the illustrated OOD examples, GT-VLA follows external semantic guidance, while G-VLA and AtomicVLA fail to adapt.GT-VLA completes all skills in the shown four-step real-world task.Source: Figure 3 · p. 5

IV. Experiments

The experiments test whether GT-VLA improves generalization and steerability over recent VLA baselines in simulation and on a physical robot. The simulation protocol uses a clean LIBERO train-test split, while ablations examine the trace stage, expert architecture, and augmentations.

IV. Experiments

  • The evaluation compares GT-VLA with recent baselines on generalization to unseen tasks, object configurations, and task compositions, and on whether explicit trace conditioning improves steerability.It includes simulation and a physical robot platform.Source: §IV · p. 5
  • The baselines include π0.5, AtomicVLA, and SEAL; Gemini 3.1 Pro implements the external VLM for both SEAL and GT-VLA.π0.5 maps vision-language inputs to actions, AtomicVLA uses skill prediction and a MoE, and SEAL uses an external VLM verifier.Source: §IV · p. 5
  • A best-effort LoHo-Manip reproduction is reported only as a supplementary comparison because no official implementation is available.The reproduction is described on the project website.Source: §IV · p. 5
  • The ablations target trace generation, the TA-MoE architecture, and three data augmentations for trace-conditioned learning.The study also uses steerability examples to compare adaptation to unseen configurations.Source: §IV · p. 5

A. Simulation Setup

  • Simulation tests generalization without per-suite in-distribution finetuning, training on Libero-10 and Libero-90 and evaluating on Libero-Goal, Libero-Spatial, and Libero-Object.Methods are trained for 100k iterations with batch size 64 on four GH200 GPUs.Source: A. Simulation Setup · p. 5

B. Simulation Results

GT-VLA leads the reported LIBERO suites and outperforms baselines across five real-world suites; ablations indicate benefits from trace guidance, skill-specific experts, and all three augmentations. Real-world tests cover familiar skills in unseen orders and unseen skill configurations across multiple task horizons.

B. Simulation Results

  • 43.8% average success rate; GT-VLA performs best across all three LIBERO suites, improving over vanilla π0.5, AtomicVLA, and SEAL.The reported gains are 28.0, 9.9, and 16.5 percentage points, respectively.Source: B. Simulation Results · pp. 5–6, Table I · p. 6

C. Real-world Experiment Setup

  • The real-world platform uses a YAM arm with side-view and wrist cameras for long-horizon pick-and-place tasks involving two to six skills.SEAL is excluded because its forward simulation requires a high-fidelity world model or digital twin unavailable on this hardware.Source: C. Real-world Experiment Setup · p. 6
  • Real-world evaluation separates familiar skill configurations in unseen orders from unseen configurations, testing composition-4 and -6 and OOD-2, -4, and -6.Each suite has 10 tasks and 5 trials per task; success requires completing the full episode.Source: C. Real-world Experiment Setup · p. 6

D. Real-world Experiment Results

  • GT-VLA outperforms baselines across all five real-world suites, and its advantage over AtomicVLA widens as task horizon increases in both composition and OOD settings.The authors interpret the pattern as supporting the need for continuous semantic alignment and subtask decomposition in complex manipulation.Source: D. Real-world Experiment Results · p. 6, Figure 5 · p. 6

E. Ablation Studies

  • Removing trace generation or the skill-routed MoE reduces performance; the results indicate semantic targets are more useful when converted into actionable traces.GT-VLA-single slightly outperforms G-VLA, while the gap between GT-VLA-single and GT-VLA indicates a benefit from the MoE structure.Source: E. Ablation Studies · pp. 6–7
  • Removing any one of the three augmentations costs 10.5%–13.2% average performance, and trace guidance has limited benefit unless all three are present.The ablations indicate complementary rather than redundant augmentations, without a clear ranking among them.Source: E. Ablation Studies · pp. 6–7

F. Pilot Studies on Noisy Semantic Guidance

The pilot studies test robustness to noisy semantic targets, swapping the guidance VLM, and failure sources. They report resilience to moderate target noise and competitive performance with unseen guidance models, while identifying downstream failure modes.

F. Pilot Studies on Noisy Semantic Guidance

  • GT-VLA remains robust through r = 32 pixels, generally outperforming baselines, but performance drops sharply at r = 64 pixels.At 32 pixels, targets typically remain on or near the intended object; at 64 pixels they often drift to incorrect objects or background.Source: F. Pilot Studies on Noisy Semantic… · p. 7, Figure 6 · p. 7

G. Pilot Studies on Generalist VLM Swap

  • With learned components frozen, GT-VLA remains competitive with or outperforms baselines when guided by VLMs unseen during training.The tested swaps are Gemini 3.6 Flash, Qwen 3.8 Max, and GPT-5.6 Terra.Source: G. Pilot Studies on Generalist VLM Swap · p. 7

H. Failure Analysis

  • Failures from not following the guidance trace range from 0.6% to 8.3%, while inaccurate traces account for 36.5% of LIBERO failures.On hardware, grasp slips dominate 4-skill task failures at 41.7%.Source: H. Failure Analysis · pp. 7–8
  • Failure profiles remain largely unchanged when semantic targets are perturbed or real-world tasks extend from four to six skills.The authors report that longer or noisier tasks generally fail for similar reasons rather than through new compounding failure modes.Source: H. Failure Analysis · p. 8

V. Conclusion

  • The conclusion reports that GT-VLA outperforms recent baselines on unseen tasks, novel configurations, and long-horizon compositions in simulation and real environments.Ablations also find that the intermediate trace interface surpasses direct target guidance.Source: §V · p. 8

Appendix

The appendix describes skill-specific experts, demonstration annotation, and parameter allocation. Its representation connects skill selection with trace and action experts, while demonstrations supply projected traces and semantic targets for training.

A. Skill Definition

  • LIBERO uses five experts for seven skills, routing place-on/place-in and turn-on/turn-off pairs to shared experts.The real-world experiments use pickup and place categories, corresponding to two experts.Source: A. Skill Definition · p. 9
  • The skill name selects the trace and action experts, while natural-language arguments identify task-relevant objects or regions.This discrete skill representation is described as a compact interface between high-level reasoning and low-level execution.Source: A. Skill Definition · p. 9

B. Data Annotation Pipeline

  • Each demonstration is decomposed into ordered skills, and each skill segment receives an instruction combining the episode instruction with that skill.The future end-effector trajectory is projected into image space as the remaining trace, and Gemini 3.1 Pro supplies a semantic target per skill.Source: B. Data Annotation Pipeline · p. 9

C. Parameter Allocation Details

  • MoE heads use separate dense layers, with attention and AdaRMS conditioning partly shared and other weights growing with the number of experts.The appendix describes Table III as a rough parameter-allocation comparison.Source: C. Parameter Allocation Details · p. 9, Table III · p. 9

D. Approximate Inference Time Latency Breakdown

The appendix reports a supplementary OOD comparison with LoHo-Manip, describes the real-world training demonstrations and evaluation tasks, and discusses inference latency. Its evaluation covers familiar skills in novel compositions and tasks containing unseen skill configurations.

D. Approximate Inference Time Latency Breakdown

  • External generalist VLM calls dominate wall-clock latency, although planning occurs once per task and semantic guidance once per skill.Trace and action modules run locally in tens of milliseconds; using a faster VLM or serving backend may trade semantic accuracy for speed.Source: D. Approximate Inference Time Latency… · p. 9

E. LoHo-Manip Reproduction

  • 22.3% average success for reproduced LoHo-Manip compares with 43.8% for GT-VLA under the same LIBERO OOD protocol.LoHo-Manip was reproduced because no official implementation was available, limiting the comparison.Source: E. LoHo-Manip Reproduction · pp. 9–10, Table V · p. 10

1) Training Task Setup:

  • The real-world training setup nominally yields 300 demonstrations across four settings and three high-level tasks per setting, with 299 episodes collected.The difference from 300 is attributed to human errors.Source: 1) Training Task Setup · p. 10

2) Evaluation Task Setup:

  • Composition suites recombine familiar skill configurations into unseen sequences, whereas OOD suites include novel skill configurations absent from training.The examples include six-skill sequences; each task is evaluated over five episodes with randomized object placements and possible visual distractors.Source: 2) Evaluation Task Setup · p. 10

1) Skill Planning Prompt Template:

The prompt asks a generalist VLM to plan an ordered sequence of atomic skills from the task instruction and initial scene. It constrains the allowed skills and plan syntax, including pickup-before-place ordering.

1) Skill Planning Prompt Template:

  • The planner receives the natural-language task instruction and an image of the initial scene to choose an ordered sequence of atomic skills.It is instructed to use only the allowed skills and follow their syntax.Source: 1) Skill Planning Prompt Template · p. 12
  • The prompt requires pickup before placing an object, with the matching placement immediately following its pickup.It also requires short, specific object descriptions that reuse appearance or color cues from the instruction.Source: 1) Skill Planning Prompt Template · p. 12
  • The plan must be returned as JSON, with a single numbered string for the plan.The prompt asks for the shortest plan that still completes the task.Source: 1) Skill Planning Prompt Template · p. 12

2) Target Acquisition Prompt Template:

The template elicits a skill-specific semantic target from a generalist model and guides it to an image location relevant to pickup or placement.

2) Target Acquisition Prompt Template:

Related papers

Loading 2609.31904v1…