Source-linked AI summary
Gripper-aware Vision Language Action Models
Hanyi Zhang, Zihong Luo, Tianyu Li, Khang Nguyen, Basu Hela, Shreyas Kumar, Ngoc Duy Tran, Feng Dai, Charith Munasinghe, Jorge Peña Queralta, Giovanni Toffetti, Khoa Vo, Ngan Le, Ravi Prakash, Quan Vuong, Tung D. Ta, Long Hu, Anh Nguyen, Baoru Huang
TL;DR
VLAs and their datasets often underrepresent embodiment-dependent gripper strategies, limiting gripper-aware learning. The paper introduces MiGA and GVLA, combining multi-gripper data, structured tokenizer representations, and routed adaptation. GVLA outperforms recent methods in complex grasping and supports stronger few-shot adaptation to new objects, tasks, or grippers.
Problem
Existing VLAs and datasets largely assume gripper invariance and predominantly use parallel-jaw grippers, despite embodiment-dependent grasping strategies.
Method
The paper introduces MiGA, a five-gripper dataset with 103,000 demonstrations, and GVLA, which combines multi-gripper tokenization with routed policy adaptation.
Results
GVLA outperforms recent methods in complex grasping tasks and enables stronger few-shot adaptation to new objects, tasks, or grippers.
Takeaways & Limitations
MiGA and GVLA provide data and policy mechanisms for learning gripper-discriminative representations and gripper-aware manipulation.
Takeaways & Limitations
Simulation modeling is limited for complex soft or multi-DoF gripper motion, while GVLA lacks explicit kinematic and contact modeling for precise intra-type adaptation.
Abstract
from arXiv · showhide
Vision language action models (VLAs) have advanced general purpose robotic grasping and manipulation by enabling robots to interpret visual observations and natural language instructions to generate executable action sequences. However, existing VLAs often implicitly assume gripper invariance, despite grasping strategies being inherently embodiment-dependent. Different gripper types, such as parallel-jaw and suction, usually require distinct interaction strategies to achieve the same grasping objective. Moreover, current datasets for VLAs predominantly rely on parallel-jaw grippers, limiting gripper-aware learning. To address this gap, we introduce MiGA, a multi-gripper-aware dataset spanning five distinct gripper types across multiple robots with 103,000 demonstrations, explicitly capturing strategy divergence under shared task objectives. We further propose GVLA, which combines a new multi-gripper tokenizer with adapter-based policy routing. Our new gripper encoding induces structured embedding information that balances parameter sharing and strategy differentiation, while layer-wise probing confirms meaningful gripper-conditioned representations for VLAs. Intensive experiments in both simulation and real-world robots show that our GVLA outperforms the current baselines across evaluated settings. Our method also improves zero-shot generalization or few-shot adaptation to new objects or unseen tasks, and enable more efficient gripper adaptation.
1 Introduction
Existing VLAs and datasets largely overlook embodiment-dependent gripper strategies, motivating MiGA and GVLA for gripper-aware manipulation learning.
- Motivation: Current VLAs often assume gripper invariance even though different grippers require distinct strategies for identical grasping objectives.Parallel-jaw and suction grippers can require different object-repositioning and grasping procedures.
- MiGA: MiGA provides 103,000 demonstrations spanning five gripper types and explicitly captures strategy divergence under shared task objectives.Each task includes demonstrations from at least three gripper types, yielding 132 gripper-strategy pairs.
- Significance: MiGA is presented as a gripper-aware dataset for training and benchmarking policies that reason about gripper-dependent grasping.It spans multiple robot platforms and both simulation and real-world environments.
- GVLA: Existing gripper tokenization methods struggle to represent meaningful gripper structure, whereas the proposed multi-gripper tokenizer organizes representations by gripper type.The tokenizer encodes domain-, type-, and instance-level gripper information.
- GVLA: GVLA combines the multi-gripper tokenizer with a fine-tuning framework and MoA-based strategy-aware manipulation learning.The contributions describe GVLA as a framework for learning different strategies conditioned on gripper morphology.
2 Related Work
Related work studies gripper representations, embodiment adaptation, and prompt-based routing, but prior datasets and representations leave multi-gripper strategy learning insufficiently addressed.
- Gripper Representation: Prior gripper representations include geometric, diffusion-based, graph-based, contact-centric, and shared eigengrasp approaches.These methods address morphology encoding, synthesis, contact modeling, or cross-gripper transfer.
- Gripper Representation: Some existing methods primarily target grasp-pose transfer among morphologically similar grippers rather than robust gripper tokenization.The paper contrasts this focus with learning structured gripper information for VLAs.
- Prompt-Based Adaptation: Vision-language and robotic studies have explored prompt learning, dynamic routing, multimodal prompts, knowledge preservation, and embodiment-specific soft prompts.These approaches motivate parameter-efficient adaptation and structured conditioning.
3 The MiGA Dataset
MiGA is a multi-gripper dataset designed to expose morphology-dependent strategies through diverse task construction, matched simulation and real-world data, and rich trajectory supervision.
- Gripper Coverage: MiGA covers five gripper types, including parallel-jaw, three-finger, soft, suction, and dexterous five-finger grippers.The grippers differ in closure, compliance, adhesion, and degrees of freedom.
- Data Collection: MiGA combines simulation and real-world collection across multiple robots with multi-view RGB-D observations.Simulation uses NVIDIA Isaac Lab; real-world data uses UFAC-TORY xArm7, Franka Panda, and UR5 robots.
- Task Design: Tasks are designed so identical objectives require different solutions across gripper morphologies and span four task scenarios.The design probes contact feasibility, approach planning, occlusion, and collision constraints.
- Dataset Statistics: The dataset contains 103,000 demonstrations across 36 tasks and five gripper types, with at least three gripper types represented per task.It includes proprioception, gripper-strategy annotations, natural-language descriptions, and approximately 5% failure demonstrations.
- Research Uses: MiGA supports trajectory-level planning and cross-gripper learning through full trajectories, sub-action decompositions, and embodiment-relevant supervision.These resources target both execution policies and high-level planning beyond grasp-pose prediction.
4 Gripper-aware Vision Language Action Model
GVLA extends a pretrained VLA with multi-gripper conditioning and dual adapter routing, combining shared representations with gripper-specific action modulation. Its tokenizer factorizes embodiment information across platform, gripper type, and instance levels, while layer probing places MoA at the final action-expert layer.
- Architecture: GVLA adds a multi-gripper tokenizer and dual MoA to a pretrained VLA backbone for gripper-aware manipulation.The tokenizer provides high-level conditioning, while dual MoA modulates action generation through platform- and gripper-specific adaptations.
- Gripper Representation: Three learnable prompt levels encode platform-specific structure, gripper-type manipulation priors, and instance-level characteristics.The combined platform, gripper-type, and instance tokens are prepended to observation embeddings as conditioned inputs.
- Gripper Representation: The three-level factorization captures shared knowledge across gripper configurations and facilitates adaptation to new gripper instances.Prompts are optimized end-to-end with the backbone under gripper prediction and action losses.
- Adapter Routing: Dual MoA uses platform-aware and gripper-aware routers to select top-k experts whose bottleneck adapters provide additive residual adaptations.The routers return normalized weights and selected expert indices, and the final output sums platform and gripper adaptations.
- Adapter Routing: Layer probing finds low gripper-type sensitivity in early layers but a sharp increase in the final layer, motivating final-layer MoA insertion.The probing metric measures distances between mean action hidden representations for different grippers at each layer.
- Fine-tuning Objective: GVLA is trained with action, gripper-prediction, and load-balance losses to support action generation, gripper-sensitive embeddings, and balanced adapter utilization.The load-balance objective penalizes variance in adapter usage within platform and gripper-type pools.
5 Experiments
The experiments compare GVLA with traditional and VLA-based baselines across task categories, using success rate, prediction error, behavioral divergence, and layer-wise gripper separability. GVLA improves over baselines and its tokenizer produces persistent gripper-discriminative representations.
- Evaluation Setup: The evaluation studies overall performance, gripper-aware transfer, component contributions, and real-robot validation against traditional and VLA-based baselines.Metrics include success rate, prediction error, counterfactual action prediction divergence, and linear probe accuracy.
- Baseline Comparison: Traditional two-stage methods collapse on flat and stacked scenes, while GraspVLA also degrades beyond simple grasp configurations.The comparison highlights limited generalization when methods rely on simple downward grasping behavior.
- Overall Performance: 58.38% average performance is achieved by π0.5, while GVLA improves this baseline by 7.62% and surpasses all competing approaches.The result is reported across four simulation task categories and different gripper types.
L0 L2 L4 L6 L8 L10 L12 L14 L16 Layer
The analyses examine representation separability, cross-object transfer, adaptation, component effects, and real-robot behavior. GVLA supports zero-shot transfer and few-shot adaptation, while failures expose limits in geometric precision and physical or kinematic feasibility.
- Cross-object Generalization: GVLA maintains strong success rates on unseen objects and tasks without task-specific fine-tuning, demonstrating zero-shot cross-object transfer.The evaluation introduces new objects with functional attributes similar to those in training.
- Gripper-Aware Adaptation: GVLA adapts faster to new tasks with the same gripper and supports adaptation to unseen grippers or mixed-gripper data.The same-gripper setting benefits from a shared adapter and underlying knowledge.
- Component Analysis: Removing the gripper-type token worsens prediction error and adaptation, while removing the platform token sharply reduces cross-embodiment adaptation despite comparable prediction error.The instance token has only a slight effect on adaptation, suggesting finer-grained specialization.
- Robot Validation: 10 demonstrations and 20k fine-tuning steps suffice for GVLA to outperform π0.5 across all evaluated real-world tasks on an unseen platform-gripper configuration.The experiment uses a real UR5 arm equipped with a Robotiq 2f-85 gripper.
- Failure Cases: Failures arise from intra-type alignment errors, physical gripper constraints, and kinematically infeasible configurations without recovery.These cases expose limits in fine-grained contact planning and the connection between high-level action prediction and low-level feasibility.
6 Discussion
The discussion presents MiGA and GVLA as responses to gripper-invariant VLA learning, while acknowledging simulation, geometric, and vision-disentanglement limitations. GVLA learns gripper-discriminative representations and improves adaptation, but precise cross-domain transfer remains constrained.
- Limitations: MiGA simulation has limited accuracy for complex soft or multi-DoF gripper motion, constraining fidelity in those settings.This is an environment-level limitation of the dataset.
- Limitations: GVLA lacks explicit kinematic and contact modeling, limiting precise intra-type adaptation; gripper and visual representations also remain partially entangled under domain shift.The paper identifies richer geometric representations and stronger vision disentanglement as needed for robust transfer.
- Contributions: MiGA contains five gripper types and 103,000 demonstrations from simulation and real-world robots, capturing distinct strategies for shared task objectives.The dataset is designed for gripper-aware policy training and benchmarking.
- Contributions: GVLA combines a multi-gripper tokenizer with dual mixture-of-adapter routing to condition action and gripper pathways.The framework is reported to outperform recent methods in complex grasping and support few-shot adaptation.