Source-linked AI summary
GeneralVLA: Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planning
Guoqing Ma, Siheng Wang, Zeyu Zhang, Shan Yu, Hao Tang
TL;DR
Robotic VLAs still have limited zero-shot generalization, especially for precise localization and long-horizon 3D planning. GeneralVLA addresses this with a hierarchical system that combines affordance perception, knowledge-guided trajectory planning, and 3D-aware control, reporting zero-shot manipulation and scalable data generation without real-world robotic data collection. Its scope remains bounded by current VLM limitations in precise spatial pose estimation.
Problem
Monolithic VLA models have limited off-domain zero-shot generalization and struggle with fine-grained coordinates and long-horizon 3D manipulation without demonstrations.
Method
GeneralVLA uses ASM for affordance perception, 3DAgent for knowledge-guided 3D trajectory planning, and a low-level policy conditioned on predicted 3D paths.
Results
GeneralVLA achieves high zero-shot accuracy across diverse manipulation tasks, outperforms VoxPoser, and generates high-quality scalable robotic data.
Takeaways & Limitations
The framework supports zero-shot robotic manipulation and robotic data generation without requiring real-world robotic data collection or human demonstration.
Takeaways & Limitations
Current VLM limitations in precise spatial pose estimation constrain GeneralVLA to using VLMs for 2D point estimation.
Abstract
from arXiv · showhide
Large foundation models have shown strong open-world generalization to complex problems in vision and language, but similar levels of generalization have yet to be achieved in robotics. One fundamental challenge is that the models exhibit limited zero-shot capability, which hampers their ability to generalize effectively to unseen scenarios. In this work, we propose GeneralVLA (Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planning), a hierarchical vision-language-action (VLA) model that can be more effective in utilizing the generalization of foundation models, enabling zero-shot manipulation and automatically generating data for robotics. In particular, we study a class of hierarchical VLA model where the high-level ASM (Affordance Segmentation Module) is finetuned to perceive image keypoint affordances of the scene; the mid-level 3DAgent carries out task understanding, skill knowledge, and trajectory planning to produce a 3D path indicating the desired robot end-effector trajectory. The intermediate 3D path prediction is then served as guidance to the low-level, 3D-aware control policy capable of precise manipulation. Compared to alternative approaches, our method requires no real-world robotic data collection or human demonstration, making it much more scalable to diverse tasks and viewpoints. Empirically, GeneralVLA successfully generates trajectories for 14 tasks, significantly outperforming state-of-the-art methods such as VoxPoser. The generated demonstrations can train more robust behavior cloning policies than training with human demonstrations or from data generated by VoxPoser, Scaling-up, and Code-As-Policies. We believe GeneralVLA can be the scalable method for both generating data for robotics and solving novel tasks in a zero-shot setting. Code: https://github.com/AIGeeksGroup/GeneralVLA. Website: https://aigeeksgroup.github.io/GeneralVLA.
I. INTRODUCTION
GeneralVLA addresses limited zero-shot generalization in robotics with a hierarchical VLA that combines affordance perception, knowledge-guided 3D planning, and low-level control. The framework is designed to solve manipulation tasks and generate robotic data without relying on expensive on-robot demonstrations.
- Motivation: Monolithic VLA models rely on large-scale robotics datasets and remain unable to match the off-domain zero-shot generalization of VLMs and LLMs.They also struggle with fine-grained coordinates, long-horizon 3D trajectories, and reuse of experience across tasks.
- GeneralVLA: GeneralVLA uses a hierarchical architecture in which ASM identifies affordance positions, Knowledge-Guided Trajectory Planning produces a 3D end-effector path, and lower-level control executes it.The design separates semantic perception, long-horizon planning, and action prediction.
- Contributions: The framework proposes zero-shot 3D trajectory planning that solves manipulation tasks while generating rich robotic data.It is intended to exploit prior knowledge from foundation models rather than purely robot-collected observation-language-action data.
- Contributions: ASM combines VLM and SAM foundations with iterative refinement, while the knowledge bank captures cross-task skills for trajectory planning.These components target accurate affordance segmentation and reuse of common skills across tasks.
- Results: Experiments report high zero-shot accuracy on diverse manipulation tasks, outperforming methods such as VoxPoser, while generated data is described as high-quality and scalable.The introduction presents this as evidence for both task solving and robotic data generation.
III. THE PROPOSED GENERALVLA
GeneralVLA uses three interconnected models to transform image and task information into precise manipulation actions. ASM refines affordance segmentation, while the downstream planner receives 3D point information and semantics for trajectory planning.
- GeneralVLA’s Framework: GeneralVLA consists of ASM for point-affordance guidance, 3DAgent for 3D path guidance, and a low-level policy conditioned on those paths.The three models form the framework’s perception, planning, and action stages.
- Affordance Segmentation Module: ASM marks object and obstacle affordances as points to support understanding and planning by 3DAgent.It is designed to perceive the scene and identify relevant affordances from camera inputs.
- Affordance Segmentation Module: ASM combines multimodal-language reasoning with SAM segmentation priors to improve the precision of affordance localization.The method addresses weak precise localization by using segmentation refinement after multimodal reasoning.
- Affordance Segmentation Module: The segmentation pipeline encodes the image, decodes the <SEG> representation into a mask, and uses iterative feedback to refine errors.Positive points indicate correctly segmented regions, while negative points indicate segmentation errors.
C. Knowledge-Guided Trajectory Planning
3DAgent uses 3D point information, object semantics, task instructions, and a knowledge bank to plan multi-stage robot trajectories. Its outputs include gripper states and can support obstacle avoidance and long-horizon execution.
- 3DAgent inputs: 3DAgent converts ASM’s 2D points into 3D point information with object semantics before LLM-based trajectory planning.The task instruction, 3D points, and object semantics are jointly provided to the LLM.
- Trajectory planning: The planned trajectory includes open and closed gripper states, supports multiple stages, and can satisfy obstacle-avoidance requirements.The authors report effective spatial-pose understanding when each object has more than three points.
- KnowledgeBank: KnowledgeBank retrieves relevant experiences, constructs knowledge from successful and failed trajectories, and consolidates the resulting items for future use.Successful experiences provide validated strategies, while failures provide counterfactual signals and pitfalls.
- System behavior: Example rollouts demonstrate multi-object, multi-stage performance by combining ASM segmentation, 3DAgent spatial reasoning, and low-level 3D policy execution.The figure describes the combined roles of the three components.
D. Path Guided Low-level policy
The planned 3D path is a coarse trajectory that guides precise grasping and low-level execution. GeneralVLA generates demonstrations in simulation and uses them to train behavior-cloning policies across diverse tasks.
- Path-guided execution: The planned 3D path provides a macroscopic coarse trajectory, while the hybrid grasping module estimates precise grasp poses for robust manipulation.Point clouds and collision filtering help select a task-specific grasp pose near the object center.
- Simulation evaluation: GeneralVLA evaluates zero-shot capability on 14 diverse simulation tasks spanning prehensile and non-prehensile manipulation.The benchmark covers varied object categories, positions, and task horizons in RLBench.
- Simulation evaluation: GeneralVLA outperformed other baselines on 10 out of 14 simulation tasks in task-averaged zero-shot success rate.Each task was evaluated across three seeds with reported standard deviations.
- Demonstration generation: The generated demonstrations contain language instructions, RGB-D trajectory frames, and waypoint gripper poses and states.Ten successful demonstrations were generated per task and filtered using the system’s success condition; missing baseline data could be patched with RLBench demonstrations.
- Behavior cloning: Behavior-cloning policies trained on GeneralVLA data achieved the best performance on 10 out of 12 tasks among autonomous data-generation baselines.The comparison includes VoxPoser and Code-as-Policies, while RLBench human demonstrations are treated as an upper bound.
- Behavior cloning: Policies trained with GeneralVLA or hand-scripted demonstrations differed by only 2.7% on average across all tasks for RVT-2.The policies were evaluated for 50 episodes per task across three seeds.
C. Real-world experiments
GeneralVLA was tested on four language-conditioned real-world manipulation tasks with varying object poses. It generated successful demonstrations for every task, using 3D reasoning for actions such as placing a bottle and opening a drawer.
- Environment and tasks: The real-world evaluation used four language-conditioned tasks: moving a spray bottle, opening a drawer, opening a jar, and sorting objects.Each task was evaluated over 10 episodes with varying object poses across three trials.
- Results: GeneralVLA generated successful demonstrations for each of the four real-world tasks.For spray-bottle movement, 3DLLM estimated bottle height and a suitable placement position; for drawer opening, it determined drawer orientation.
V. ABLATION STUDY
The ablations examine ASM accuracy, data components, and 3DAgent inputs. Results support precise affordance localization, diverse data sources, and sufficiently rich 3D object and obstacle information.
- Point Location Accuracy of ASM: ASM achieved significantly higher point-location accuracy than all baselines on RoboRefIt.Accuracy is the percentage of predicted points within the ground-truth target mask, averaged with standard deviations over three runs.
- Data composition: VQA, LVIS object detection, and object reference on real and synthetic images each significantly contributed to overall RoboRefIt accuracy.The authors highlight LVIS for providing precise semantic and location information.
- Information Required by 3DAgent: 3DAgent trajectory planning requires at least three 3D points per object and information about both target objects and obstacles.These inputs enable orientation reasoning and obstacle avoidance, while reduced settings can fail on tasks such as pulling an umbrella from its bag.
C. Ablation Study of the HGM Module
The HGM ablation examines how multimodal inputs and grasp-selection mechanisms affect manipulation success, while related experiments assess GeneralVLA’s scaling and broader capabilities.
- HGM ablation: RGB images, depth maps, and 3D point ranges are necessary for task completion, while collision detection and optimal pose selection improve success rates.The HGM compares variants that remove RGB, 3D points, collision filtering, or nearest-pose selection.
- Data scaling: 0.539 versus 0.178: GeneralVLA-generated data has a steeper performance-scaling slope than RLBench-generated data.The slopes come from linear fits in the scaling experiment.
- Overall capability: GeneralVLA enables zero-shot robotic manipulation and high-quality data generation without requiring real-world robotic data.The framework uses a hierarchical architecture comprising ASM, 3DAgent, and HGM.
- Limitation: Current VLMs are limited in precise spatial pose estimation, so GeneralVLA uses them for 2D point estimation rather than 3D pose estimation.The paper identifies 3D pose estimation as a direction for future improvement.
- Conclusion: The authors position hierarchical decomposition combined with world knowledge as a promising direction for addressing robotics data scarcity and generalization.This is presented as the paper’s supported conclusion.
APPENDIX
The appendix describes GeneralVLA’s training data, multimodal trajectory-generation pipeline, grasp selection, and simulation evaluation setup.
- VLM fine-tuning dataset details: The fine-tuning dataset combines five sources, including point prediction, LVIS semantic regions, and robot data from Open X-Embodiment.The point prediction source contains 347k samples, while the robot source contributes 100k points.
- Dataflow: GeneralVLA takes RGB images and depth maps, then produces a complete end-effector trajectory by combining a 3D path with an optimal 6D grasp pose.ASM generates 2D pixel coordinates, while HGM filters candidate grasps before final trajectory formation.
- Grasp selection: The grasp-selection procedure uses object 3D spatial range, fused RGB-depth point clouds, collision filtering, and proximity to the object center.These steps determine task-specific grasp, placement, and pre-action poses.
- Simulation setup: Simulation experiments use four cameras positioned at the front, left shoulder, wrist, and right shoulder, with only the wrist camera mounted on the end effector.The setup evaluates 12 modified RLBench tasks and specifies success conditions for trained and zero-shot methods.
A. put block
This appendix entry defines a task that picks up a green block and places it on a red mat, with success determined by the block’s detected placement.
- put block: The task is to pick up the green block and place it on the red mat.The task is listed under the filename put block.py.
- put block: Success is recorded when the target green block is detected on the red mat.The success condition is simulation-defined.
G. take umbrella
This appendix group lists four manipulation tasks, including taking an umbrella from its stand, and specifies their simulation success conditions.
- take umbrella: The take umbrella task requires picking up the umbrella out of the umbrella stand.Success is defined by lifting the umbrella out of the stand.
- Real-world scenarios: Figure 10 presents task execution in four real-world scenarios, with each column representing one task.The figure shows execution images from left to right for the listed tasks.
- Related tasks: The grouped tasks also include sorting mustard, uncapping wine, turning on a lamp, and placing a knife on a chopping board.Each task has a corresponding simulation-defined success condition.
L. push block
The section specifies manipulation tasks, affordance prompts, and 3DAgent trajectory outputs used for zero-shot execution. Tasks include pushing blocks, inserting a block into a Jenga tower, and lifting a cube.
- Tasks: The listed tasks include pushing a red block toward a green target, inserting a green block into a Jenga tower, and picking up a red cube.Success metrics are target placement, insertion, and lifting, respectively.
- ASM Prompt: ASM prompts the model to identify task-relevant objects and output more than three image points describing each object’s affordance position.Coordinates are normalized floats in [0,1], with the image origin at the bottom-left.
- 3DAgent Prompt: The 3DAgent prompt supplies object 3D positions and requests a gripper trajectory composed of spatial points and explicit gripper actions.The output supports OpenGripper and CloseGripper tokens, with trajectories limited to 20 points.
XI. FAILURE ANALYSIS
The failure analysis distinguishes affordance, trajectory-planning, and action-execution failures. In Play Jenga, GeneralVLA succeeds in 82% of cases, with most failures occurring during execution.
- Failure Modes: Affordance failures arise when ASM misunderstands the scene or predicts inaccurate 2D points for target objects.These errors include missing small blocks or assigning an object’s position to the tabletop.
- Failure Modes: Trajectory failures include misunderstanding the language goal, ambiguity in 3D pose or direction, and planned paths that collide with obstacles.The analysis separates these failures from errors in affordance prediction and action execution.
- Failure Modes: The low-level action model may deviate from the predicted trajectory, interact with the wrong object, or fail when an incorrect grasp angle causes an object to slip.These execution problems can directly cause task failure.
- Play Jenga: 82% of Play Jenga cases are successfully completed by GeneralVLA.The analysis attributes this to accurate object localization, 3D pose inference, and grasp execution in successful cases.
- Play Jenga: Most Play Jenga failures occur during action execution, particularly because grasping failures remain possible despite the designed grasping module.ASM and 3DAgent are reported to have low failure rates in this task.