PaperlayerView exact revision on arXiv

Source-linked AI summary

GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Data

Shengliang Deng, Mi Yan, Songlin Wei, Haixin Ma, Yuxin Yang, Jiayi Chen, Zhiqi Zhang, Taoyu Yang, Xuheng Zhang, Wenhao Zhang, Heming Cui, Zhizheng Zhang, He Wang

arXiv:2505.03233v3cs.RO

TL;DR

The paper asks whether VLA grasping models can be pretrained entirely on large-scale synthetic action data instead of costly real-world collections. It constructs SynGrasp-1B and GraspVLA, whose Progressive Action Generation jointly uses synthetic actions and Internet grounding; evaluations report strong zero-shot generalization and few-shot adaptability, with latency and ambiguous-instruction limitations.

  • Problem

    Real-world robotic data collection is costly and labor-intensive, while synthetic data remains underexplored for VLA training.

  • Method

    The paper builds SynGrasp-1B and trains GraspVLA with Progressive Action Generation, combining autoregressive perception and flow-matching action generation across synthetic and Internet data.

  • Results

    GraspVLA demonstrates strong zero-shot generalization and few-shot adaptability across real-world and simulation grasping evaluations, including novel-category objects.

  • Takeaways & Limitations

    Large-scale synthetic action data can support a generalizable grasping VLA model with direct sim-to-real transfer and open-vocabulary behavior.

  • Takeaways & Limitations

    PAG adds around 200ms latency, which may be insufficient for dynamic environments such as fast-moving objects.

Abstract

from arXiv · show

Embodied foundation models are gaining increasing attention for their zero-shot generalization, scalability, and adaptability to new tasks through few-shot post-training. However, existing models rely heavily on real-world data, which is costly and labor-intensive to collect. Synthetic data offers a cost-effective alternative, yet its potential remains largely underexplored. To bridge this gap, we explore the feasibility of training Vision-Language-Action models entirely with large-scale synthetic action data. We curate SynGrasp-1B, a billion-frame robotic grasping dataset generated in simulation with photorealistic rendering and extensive domain randomization. Building on this, we present GraspVLA, a VLA model pretrained on large-scale synthetic action data as a foundational model for grasping tasks. GraspVLA integrates autoregressive perception tasks and flow-matching-based action generation into a unified Chain-of-Thought process, enabling joint training on synthetic action data and Internet semantics data. This design helps mitigate sim-to-real gaps and facilitates the transfer of learned actions to a broader range of Internet-covered objects, achieving open-vocabulary generalization in grasping. Extensive evaluations across real-world and simulation benchmarks demonstrate GraspVLA's advanced zero-shot generalizability and few-shot adaptability to specific human preferences. We will release SynGrasp-1B dataset and pre-trained weights to benefit the community.

1 Introduction

The paper addresses the cost and labor demands of real-world robotic data by exploring fully synthetic VLA pretraining for grasping. It introduces SynGrasp-1B and GraspVLA, combining synthetic actions with Internet data for zero-shot and few-shot generalization.

  • Real-world robotic data collection is costly and labor-intensive, motivating synthetic data as an accessible alternative for VLA training.
  • GraspVLA unifies autoregressive perception and flow-matching action generation through Progressive Action Generation in a Chain-of-Thought process.
  • Joint training on synthetic action data and Internet data supports transfer to semantically diverse objects and open-vocabulary grasping.
  • Experiments report strong zero-shot generalization, few-shot adaptability, and robust performance across real-world and LIBERO simulation settings.

2 Related Work

Related work spans VLA models trained on demonstrations and Internet-scale vision-language data, synthetic-data approaches for robotic learning, and modular or end-to-end grasping systems. These directions motivate scalable, open-vocabulary, and adaptable grasping models.

  • Vision-Language-Action Models: VLA research uses large-scale demonstrations, pretrained vision-language models, action experts, and Internet-scale video data to generate robotic actions.
  • Synthetic Data: Synthetic-data research uses GPU simulation, photorealistic rendering, domain randomization, and motion planning to train or augment robotic grasping systems.
  • Grasping: Grasping methods include open-loop detection, closed-loop reinforcement or imitation learning, and modular vision-language systems for open-vocabulary objects.
  • Grasping: Existing grasping systems face challenges with depth perception, failure recovery, and adapting to specialized task constraints.

3 SynGrasp-1B Dataset Generation

SynGrasp-1B is generated by combining diverse object layouts, planned grasp trajectories, randomized visual rendering, and efficiency-oriented simulation infrastructure. The pipeline is tailored to produce smooth trajectories suitable for imitation learning.

  • Object Assets and Layout Generation: SynGrasp-1B uses 10,680 Objaverse instances from 240 categories, randomly scaled and placed in physically plausible tabletop scenes.
  • Grasp Synthesis and Trajectory Generation: Stable antipodal grasps and collision-free reach-and-lift trajectories are synthesized with grasp algorithms and CuRobo motion planning.
  • Visual Randomization and Rendering: The pipeline renders RGB images with randomized lighting, backgrounds, camera settings, and viewpoints using Isaac Sim photorealistic ray tracing.
  • Efficient Data Generation: Caching, asynchronous writing, and parallel physics simulation and rendering improve billion-frame data-generation efficiency.
  • Tailoring Data for Imitation Learning: Single-step motion planning prioritizes smooth trajectories over planning success rates to reduce hesitation in imitation policies.

4 Model

GraspVLA combines a vision-language model with a flow-matching action expert through Progressive Action Generation. It jointly trains on Internet grounding and synthetic action data, using perception outputs as intermediate steps toward grasp actions.

  • Overall Architecture: GraspVLA combines an autoregressive vision-language backbone with a conditional flow-matching action expert.
  • Progressive Action Generation: Progressive Action Generation addresses synthetic-category limits by integrating Internet grounding with synthetic grasping data.
  • Progressive Action Generation: The VLM predicts 2D target bounding boxes for both datasets, then predicts grasp pose for synthetic data before action generation.
  • Joint Training of VLM and action expert: Internet data supervises bounding-box prediction, while synthetic data supervises bounding boxes, grasp poses, and flow-matching actions.
  • Joint Training of VLM and action expert: The action expert learns chunked end-effector delta actions with flow matching, and the overall loss combines the VLM and action losses.

5 Experiments

Experiments evaluate GraspVLA’s zero-shot generalization, scaling, design choices, and few-shot adaptability across real-world and simulation settings. The results show strong performance across object categories and environmental variations, with efficient adaptation and identifiable trade-offs.

  • 5.1 Zero-Shot Comparison with VLAs in Real World: GraspVLA achieves around 90% on all real-world test sets and significantly outperforms the baselines, demonstrating strong zero-shot generalizability.Performance is comparable across synthetic and web categories, while SPL indicates shorter paths than π0 baselines.
  • 5.2 Zero-Shot Comparison with VLAs in LIBERO Benchmark: GraspVLA’s zero-shot performance on LIBERO surpasses the fine-tuned performance of π0 and OpenVLA.Evaluation covers the Long, Goal, and Object suites after reformulating selected tasks as object-picking tasks.
  • 5.3 Zero-Shot Comparison with AnyGrasp in Real World: GraspVLA maintains consistent performance across language-guided and arbitrary grasping, including transparent objects, whereas AnyGrasp is faster but struggles with transparency.The comparison attributes GraspVLA’s slower inference to its large vision-language backbone.
  • 5.4 Scaling Law: Performance improves steadily with more training frames, while web categories scale more slowly than synthetic categories.The scaling result indicates that web-category generalization requires more training frames.
  • 5.5 Efficient Post-Training: Few-shot post-training evaluates adaptation to rare industrial components, cleanliness-constrained mug grasping, and sequential grasping in dense scenes.Tasks target new vocabularies, task-specific grasp specifications, and grasping order; Table 4 reports superior adaptability to novel tasks.
  • 5.6 Effectiveness of Design Choices: Adding 2D bounding boxes and then grasp pose prediction as intermediate action steps improves web-category success, grasping accuracy, and SPL over the vanilla baseline.PAG-3D substantially reduces hesitation, leading to fewer attempts and shorter trajectories.

6 Conclusion

The paper builds a generalizable grasping VLA from large-scale synthetic data, combining a billion-scale simulated dataset with a design that learns from synthetic actions and Internet grounding data.

  • The study investigates generalizable grasping with large-scale synthetic data and reports strong generalizability for novel-category objects in unseen environments.
  • The approach uses a billion-scale simulated grasping dataset with extensive randomization and photorealistic rendering.
  • The model is designed to learn from synthetic action data and action-free Internet grounding data within one framework.
  • Extensive ablations and comparisons report state-of-the-art performance in table-top grasping.
  • The model scales with the amount of synthetic training data and acquires new grasping behaviors through few-shot post-training.

7 Limitations and Future Work

The paper identifies boundaries in hardware coverage, semantic reasoning, deformable-object handling, task scope, and inference latency, while outlining extensions as future work.

  • Hardware and camera scope: Data generation and evaluation currently use only the Franka Panda arm with front and side views.Adapting the pipeline to other robots and camera configurations is left for future work.
  • Semantic reasoning: GraspVLA struggles with ambiguous instructions such as “pick up food” and “pick up the leftmost object”.The paper suggests scaling vision-language pretraining and exploring architectural changes for semantic reasoning.
  • Deformable objects: Force-closure grasp labels do not account for deformability.The model can still grasp some deformable objects when their initial geometry contains convex regions enabling force closure.
  • Task scope: The current model focuses on grasping, while future work will extend the pipeline to tasks such as pick-and-place and pushing.Reinforcement learning is proposed for more complex non-prehensile manipulation.
  • Inference latency: PAG introduces additional latency of around 200ms on NVIDIA L40s, which may be insufficient for fast-moving objects.The paper identifies distillation and quantization as possible future directions.

B Details about SynGrasp-1B

SynGrasp-1B is a billion-frame synthetic grasping dataset designed to provide broad object, scene, and annotation coverage at substantially lower collection cost than real-world alternatives.

  • Dataset scale and diversity: SynGrasp-1B contains 10 million trajectories of approximately 100 frames each, totaling 1 billion frames, with 10,680 objects across 240 categories.
  • Annotations: The dataset provides precise annotations for camera calibration, bounding boxes, and 3D poses of target objects and grippers.The simulation also supports adding depth maps and segmentation masks when needed.
  • Collection cost: Simulation data generation is presented as more cost-effective than real-world collection across time, space, equipment, and labor.
  • Generation efficiency: 10 million trajectories can be generated in 10 days using 160 NVIDIA 4090 GPUs, compared with around 1,000 trajectories per day for one human operator.
  • Generation cost: The total cost of generating SynGrasp-1B is around $5,000.
  • Future extensions: The pipeline is intended to extend to other robotic arms, manipulation tasks, and larger-scale camera randomization, but these extensions remain future work.

C Details about Data Generation

The data-generation and evaluation pipeline combines randomized object layouts, simulated grasp trajectories, photorealistic rendering, and scaling analyses to test generalization and adaptation.

  • Object processing and layout: Episodes use randomized clutter layouts within a 0.4m by 0.5m table area, with physically plausible object poses and category-specific valid orientations.
  • Camera randomization: The cameras are randomized within a 15 cm radius ball and rotated ±5° around each axis.
  • Adaptation: With 5k additional synthetic trajectories, GraspVLA adapts to new robotic arms, grippers, and camera configurations with minimal fine-tuning.The paper reports rapid deployment on new setups.
  • Scaling analysis: Simulation performance follows a scaling trend similar to real-world performance, while the sim-to-real gap decreases with more training frames.
  • Scaling analysis: Broader training-category coverage improves web-category performance, while increasing instances per category consistently improves both synthetic and web-category performance.

G Details about Experiments on LIBERO Benchmark

The LIBERO evaluation adapts camera setups, task instructions, and success criteria to isolate grasping, then examines robustness across language-conditioned and object-focused tests.

  • Evaluation setup: LIBERO trials count success when the robot grasps and lifts the target object 10 cm, using camera poses aligned with the two-view training setup.The basket is removed when it severely occludes the side view, and the gripper is extended by 2 cm.
  • Evaluation setup: Ambiguous LIBERO-Object descriptions relax success to grasping any object from the target category, such as any can for “alphabet soup.”This criterion addresses scenes where even humans may struggle to identify the named object.
  • Evaluation setup: Non-prehensile and color-discrimination tasks are excluded, limiting the benchmark evaluation to grasping without color-based distinctions.The omitted tasks and modified instructions are documented in the LIBERO-Goal, LIBERO-Object, and LIBERO-Long tables.
  • Evaluation setup: All models receive simplified “pick up a object” instructions to evaluate grasping alone rather than placement, while fine-tuning instructions remain unsimplified.The fine-tuned baselines use official OpenVLA and π0 checkpoints trained on processed LIBERO demonstrations.
  • Results: Both fine-tuned baselines drop significantly under simplified instructions, indicating limited robustness to instruction variations.This comparison is reported in the instruction-format ablation.
  • Evaluation setup: The language-conditioned test set uses Grounding DINO boxes to filter grasp candidates before motion-planned execution, while the broader tests include synthetic, web, and transparent objects.The language-driven set contains 60 trials; arbitrary grasping samples 30 objects, and the transparent set contains five objects.
  • Results: Five baseline failures arise from incorrect language grounding in three trials and merged point clouds for flat objects in two trials.A toy ambulance is misidentified as a charger in one ambiguous top-down view example.
  • Comparison: AnyGrasp and GraspVLA provide complementary solutions: the former is fast, whereas the latter is end-to-end, closed-loop, and adaptable to specialized grasping tasks.Adapting AnyGrasp to open-vocabulary grasping requires segmentation, motion planning, and failure recovery modules.

I Ablation of Camera Views

The camera-view ablation shows that multiple views substantially improve GraspVLA’s performance, while its single-view version still outperforms OpenVLA. The section also bounds this result through sim-to-real design choices and inference-cost analysis.

  • Camera-view ablation: 40% higher performance is achieved by GraspVLA’s single-view implementation than by the OpenVLA baseline.This comparison uses only front-view images for both methods.
  • Camera-view ablation: Approximately 30% lower performance results when GraspVLA uses a single front view instead of its multi-view input.The ablation attributes the comparison to the number of input views while retaining the same single-view baseline setting.
  • Sim-to-real considerations: The sim-to-real gap is addressed through realistic visual rendering and simplified, stability-focused, geometry-driven grasp control.The design uses positional control, discrete gripper commands, low-friction force-closure filtering, and mesh-based grasp poses.
  • Sim-to-real considerations: These sim-to-real strategies may not generalize to tasks requiring fine-grained dynamics understanding, including non-prehensile manipulation.Investigation of such scenarios is left for future work.
  • Inference efficiency: PAG contributes approximately 63% of inference delay through 14 additional generated tokens, despite supporting a high grasp success rate.The reported breakdown is measured on an NVIDIA L40s GPU, and improving PAG efficiency is left for future work.
  • Control implementation: The implementation uses impedance control, cascaded filtering, positional interpolation, and receding-horizon optimization to smooth predicted actions.All evaluations use blocking control, although non-blocking control is used for demonstration recording.

M Failure Analysis

A challenging cluttered-scene evaluation identifies language ambiguity and environmental complexity as the dominant failure sources, alongside grasping and perception failures for difficult objects.

  • Evaluation design: The failure analysis uses cluttered scenes with workspace-wide object placement and stacked objects because the main test set contained only two potentially unrepresentative failures.These scenes are designed to expose primary failure modes under occlusion and complex spatial arrangements.
  • Failure modes: 31% of failures occur when ambiguous instructions match multiple objects, causing the model to hesitate.The analysis suggests longer contextual history as a possible mitigation.
  • Failure modes: 27% of failures arise in highly cluttered scenes where the model misidentifies objects, likely reflecting insufficient training data for such scenarios.The proposed future directions include stronger data augmentation and generative modeling for more complex samples.
  • Failure modes: 21% of failures involve smooth objects slipping during grasping, while 14% involve difficulty grasping occluded targets precisely.The analysis points to tactile feedback for slipping and active perception for occlusion as possible remedies.
  • Failure modes: The remaining 7% of failures consist of minor errors such as early gripper closure or collisions with the environment.Reinforcement learning is identified as a possible avenue for addressing these errors.
Loading 2505.03233v3…