PaperlayerView exact revision on arXiv

Source-linked AI summary

DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping

Yifan Zhong, Xuchuan Huang, Ruochong Li, Ceyao Zhang, Zhang Chen, Tianrui Guan, Fanlian Zeng, Ka Num Lui, Yuyao Ye, Yitao Liang, Yaodong Yang, Yuanpei Chen

arXiv:2502.20900v5cs.ROcs.AI

TL;DR

Dexterous grasping needs policies that generalize beyond isolated objects and simplified environments. DexGraspVLA combines foundation-model planning and domain-invariant representations with diffusion-based imitation control, achieving over 90% success across unseen cluttered conditions and extending to long-horizon and nonprehensile tasks.

  • Problem

    Dexterous grasping must generalize across diverse objects, environments, clutter, and disturbances, while existing research often assumes isolated objects or simplified settings.

  • Method

    DexGraspVLA uses a pre-trained VLM as a high-level planner and vision foundation models with a diffusion-based imitation controller to learn from domain-invariant representations.

  • Results

    90.8% aggregated success was achieved across unseen objects, backgrounds, and lighting conditions without domain-specific fine-tuning.

  • Takeaways & Limitations

    The framework supports robust generalization in cluttered dexterous grasping and extends to long-horizon prompts, failure recovery, and nonprehensile grasping.

Abstract

from arXiv · show

Dexterous grasping remains a fundamental yet challenging problem in robotics. A general-purpose robot must be capable of grasping diverse objects in arbitrary scenarios. However, existing research typically relies on restrictive assumptions, such as single-object settings or limited environments, showing constrained generalization. We present DexGraspVLA, a hierarchical framework for robust generalization in language-guided general dexterous grasping and beyond. It utilizes a pre-trained Vision-Language model as the high-level planner and learns a diffusion-based low-level Action controller. The key insight to achieve generalization lies in iteratively transforming diverse language and visual inputs into domain-invariant representations via foundation models, where imitation learning can be effectively applied due to the alleviation of domain shift. Notably, our method achieves a 90+% dexterous grasping success rate under thousands of challenging unseen cluttered scenes. Empirical analysis confirms the consistency of internal model behavior across environmental variations, validating our design. DexGraspVLA also, for the first time, simultaneously demonstrates free-form long-horizon prompt execution, robustness to adversarial objects and human disturbance, and failure recovery. Extended application to nonprehensile grasping further proves its generality. Project website: https://dexgraspvla.github.io.

1 Introduction

Dexterous grasping must generalize across diverse objects, environments, clutter, and disturbances, but existing approaches often rely on simplified settings. DexGraspVLA addresses this with hierarchical planning, domain-invariant representations, and imitation-based closed-loop control, achieving broad reported generalization.

  • Motivation: Dexterous grasping remains challenging because real-world policies must handle varied object properties, environmental conditions, clutter, and grasp sequencing.The required variation includes geometry, mass, texture, orientation, lighting, backgrounds, disturbances, and multi-object planning.
  • Motivation: Existing methods commonly assume isolated objects or simplified environments, limiting evidence for general grasping in unseen scenarios.Two-stage methods also depend heavily on calibration and mechanical accuracy, while imitation learning often struggles beyond its training data.
  • Approach: DexGraspVLA combines a pre-trained VLM planner with a diffusion-based controller trained by imitation learning on domain-invariant representations.Foundation models transform visual and linguistic inputs to reduce domain shift before action learning.
  • Results: 90.8% success was achieved across 1,287 unseen object, lighting, and background combinations in a zero-shot cluttered-scene setting.The framework also reports robustness to adversarial objects, human disturbances, and failure recovery.
  • Results: 89.6% success was achieved on free-form long-horizon instructions, while nonprehensile grasping further demonstrated applicability beyond standard dexterous grasping.The introduction reports both embodied long-horizon execution and extension to a distinct grasping setting.

2 Related Work

Prior dexterous grasping work spans modular two-stage pipelines and end-to-end learning, while foundation-model policies increasingly provide visual-linguistic generalization. DexGraspVLA builds on these directions by using foundation models for affordance guidance and a learned action controller.

  • Dexterous Grasping: Two-stage methods generate grasp poses and execute motion plans, benefiting from modularity but remaining vulnerable to disturbances and calibration errors because they are open loop.Pose generation may use sampling, optimization, or regression before motion planning.
  • Foundation Models: Directly fine-tuned VLM policies aim to transfer web-scale vision-language knowledge to robot control, yet still struggle with unseen scenarios and catastrophic forgetting.OpenVLA and π0 are cited as examples of this approach.
  • Foundation Models: DexGraspVLA instead uses a frozen VLM-based planner and foundation-model representations to guide a learned action head for generalizable control.Its overview pairs high-level prompt decomposition with a diffusion-based low-level controller.
  • Dexterous Grasping: End-to-end imitation and reinforcement learning offer closed-loop adaptation, but reinforcement learning faces sim-to-real challenges and imitation learning may not generalize beyond demonstrations.These limitations motivate methods that combine learned control with stronger pretrained representations.

3 Problem Formulation

The task is formulated as sequential language-guided control: observations include camera images and proprioception, and the policy samples actions until the instruction succeeds. Long-horizon prompts require decomposing multiple grasping steps and completing them sequentially.

  • Single-Step Control: A language instruction specifies the target object, while the policy receives wrist-camera images, head-camera images, and 13-dimensional robot proprioception at each timestep.The proprioception consists of arm and hand joint angles.
  • Single-Step Control: Conditioned on these observations and the instruction, the policy samples arm-and-hand joint-angle actions from an action distribution.The action contains separate arm and hand joint targets.
  • Single-Step Control: Execution continues until termination, with binary reward indicating whether the instruction was completed successfully.The policy objective is to maximize expected reward over language instructions and observation trajectories.
  • Long-Horizon Control: Long-horizon prompts such as “clear the table” require the policy to reason about the prompt, decompose it into individual instructions, and complete them sequentially.The formulation extends single-object grasping to multiple ordered grasping steps.

4 Methods

DexGraspVLA uses a hierarchical planner-controller architecture that converts language and visual inputs into domain-invariant affordances and features before diffusion-based imitation learning. The planner iteratively monitors execution, while the controller tracks objects and predicts responsive action chunks.

  • Architecture: DexGraspVLA is a hierarchical architecture consisting of a high-level planner and a low-level controller.The framework is explicitly described as hierarchical and modularized.
  • Planner: The pre-trained Qwen VLM planner converts a user prompt into object-level grasping instructions and dynamically monitors the workflow.It proposes instructions such as grasping a specific object from a broader prompt.
  • Planner: The planner marks each target with a bounding box, providing a consistent object-localization affordance despite diverse language and visual inputs.This representation is intended to alleviate the controller’s learning challenge by reducing domain variation.
  • Controller: SAM initializes a target mask from the bounding box, and Cutie tracks that mask over time in cluttered environments.The controller models actions conditioned on wrist images, robot state, and the tracked mask.
  • Controller: Foundation-model features remain comparatively invariant to distracting visual factors, supporting imitation learning on domain-invariant controller inputs.The method addresses the variability of raw visual observations by converting them into more stable representations.
  • Controller: The controller fuses masks, head and wrist features, and robot state, then uses a DiT diffusion model to predict multi-step action chunks.At inference, iterative denoising and receding-horizon control generate responsive action sequences.
  • Data Collection: Imitation learning is performed on 2,094 successful demonstrations collected from 36 household objects in randomized cluttered scenes.Demonstrations record images, proprioception, object masks, and actions from kinesthetic teaching.

5 Experiments

DexGraspVLA is evaluated in zero-shot real-world settings spanning unseen objects, backgrounds, lighting, baselines, ablations, long-horizon prompts, and nonprehensile grasping. Results show strong generalization, robust closed-loop behavior, and consistent internal representations across environmental variations.

  • Experimental Setup: The evaluation uses a different environment from demonstrations to test zero-shot generalization, with a 7-DoF arm, 6-DoF hand, and two camera views.The platform operates at 20 Hz and includes wrist-mounted and head-mounted cameras.
  • Large-Scale Generalization: The benchmark covers 360 unseen objects, six unseen backgrounds, and three unseen lighting conditions in cluttered scenes containing around six objects.The objects vary in size, weight, geometry, texture, material, and category.
  • Large-Scale Generalization: 90.8% aggregated success is achieved across 360 unseen objects, six unseen backgrounds, and three unseen lighting conditions.Single-attempt success rates are 91.1% on unseen objects, 90.5% on unseen backgrounds, and 90.9% under unseen lighting.
  • Baseline Comparison: DexGraspVLA consistently achieves 90+% success across settings and outperforms fine-tuned VLA baselines, whose performance drops under visual variations.The comparison includes π0, RDT, and OpenVLA variants; the analysis attributes stronger grounding to bounding boxes and better visual detail preservation to DINOv2.
  • Ablation Study: Over 98% success is obtained on seen and unseen single-object grasps, while raw-input ablations fail to generalize reliably under perceptual changes.The results indicate robustness to domain shift and show strong performance without directly learning actions from raw visual inputs.

6 Limitation and Conclusion

DexGraspVLA combines hierarchical planning with foundation-model representations and imitation learning to support broad dexterous grasping generalization. The paper concludes that it remains limited to grasping rather than functional manipulation and does not yet use tactile sensing.

  • Conclusion: DexGraspVLA uses a pretrained VLM as high-level planner and vision foundation models in a low-level controller trained with imitation learning.The system transforms multimodal inputs into domain-invariant representations and learns robust closed-loop policies.
  • Conclusion: The evaluations report over 90% grasping success across thousands of unseen cluttered scenes, alongside long-horizon execution, failure recovery, and nonprehensile grasping.Empirical analysis also finds consistent internal behavior across environmental variations.
  • Limitations and Future Work: The current scope excludes functional grasping, subsequent manipulation, and tactile sensing.Future work targets finer-grained affordance generation and task-oriented control with tactile feedback.

A.1 Details of DexGraspVLA Implementation

DexGraspVLA separates high-level visual-language planning from low-level diffusion control. Its planner converts user prompts and camera observations into object selections, localization, verification, and completion decisions, while the controller learns actions from structured visual representations.

  • Planner: The planner uses a vision-language model for instruction proposal, object localization, grasp verification, and prompt-completion checks.These subtasks connect user prompts and visual observations to sequential grasp execution.
  • Planner: The planner selects the next object by interpreting the user prompt in the current visual context and applying spatial and safety preferences.It prioritizes rightmost, then central, then left objects; avoids blocked objects and destabilizing grasps.
  • Planner: Bounding-box prediction transforms the selected grasping instruction into a consistent object-localization representation for the controller.The localization prompt requires one visible object, valid pixel coordinates, a short label, and an appearance description.
  • Controller: The controller extracts head-camera, wrist-camera, and mask features, projects them into a shared 1024-dimensional space, and concatenates them into observations.Head and wrist images use DINOv2 feature extractors, while masks are processed with a randomly initialized ViT.
  • Controller: The diffusion controller models 64-step, 13-dimensional action chunks and predicts denoising noise from the structured observation sequence.Training uses Immiscible Diffusion and MSE noise-prediction loss; inference iteratively denoises Gaussian noise with DDIM sampling.
  • Controller: The 163M-parameter controller is trained with bfloat16 mixed precision and FusedAdamW for 84 epochs on eight A800 GPUs in less than one day.These implementation choices reduce memory use and accelerate optimization.

A.2 Details of Baseline Implementation

Baseline experiments fine-tune several state-of-the-art vision-language-action models on the authors’ datasets using their official training procedures. The section also specifies instruction construction and an ablation that changes the visual feature extractors.

  • Baselines: Several state-of-the-art VLA baselines are fine-tuned on the authors’ datasets following their official instructions.The compared models include π0, RDT, OpenVLA, and OpenVLA-OFT.
  • Data preparation: Because the datasets lack language annotations, target objects are manually annotated and language instructions are expanded with GPT-4o and Gemini 2.5 Pro.All generated instructions are manually verified.
  • Baseline configurations: π0 uses full-parameter and LoRA fine-tuning with action horizon 50, batch size 256, and 30K training steps.The LoRA ranks are 16 for Gemma and 32 for the action expert.
  • Baseline configurations: RDT uses full-parameter fine-tuning with action horizon 64, batch size 256, and 200K training steps.
  • Baseline configurations: OpenVLA uses LoRA fine-tuning with batch size 16, rank 32, and 60K steps, without action chunking.
  • Ablations: The ablation variants either make DINOv2 trainable or replace it with two trainable small pretrained ViTs using 224 × 224 × 3 inputs and 384-dimensional features.DexGraspVLA (DINOv2-train) changes feature-extractor freezing, while DexGraspVLA (ViT-small) changes the backbone.

A.3 Details of Data Collection

Demonstrations are collected through kinesthetic teaching, then replayed with PD control while recording images, joint angles, and target joint-angle actions. The evaluation environments include varied conditions, and contractors assist with collection.

  • Demonstration collection: Operators guide the robot manually through target-object grasping and placing motions in teaching mode.
  • Demonstration collection: Recorded joint angles become target actions during 20Hz PD-controlled replay, while images and current joint angles are collected as states.
  • Evaluation conditions: The evaluation uses environment conditions for both dexterous and nonprehensile grasping generalization experiments.
  • Data collection workforce: External contractors receive training and fair wages while assisting with data collection.

B.1 The “Zero-Shot” Evaluation Environment

The zero-shot evaluation tests whether demonstrations transfer across separate collection and test rooms, diverse unseen objects, and altered lighting and backgrounds. DexGraspVLA is reported to remain robust in challenging unseen conditions.

  • Environment shift: The test site differs from the data-collection site in layout and background, exposing both cameras—especially the wrist camera—to unseen scenes.The wrist camera encounters altered front and peripheral views during operation.
  • Object diversity: The object evaluation includes 360 unseen objects spanning diverse sizes, weights, geometries, textures, materials, and categories.A 103-object subset is used for unseen-lighting and unseen-background experiments, while all 360 objects are used for the unseen-object evaluation.
  • Nonprehensile evaluation: Nonprehensile grasping uses 32 demonstration objects and 18 unseen evaluation objects covering varied appearances, geometries, sizes, and categories.
  • Generalization results: DexGraspVLA successfully pushes fully white or irregularly shaped objects toward the table edge under complex unseen lighting and background conditions.
  • Internal behavior: The internal-behavior visualization compares raw head-camera images, DINOv2 feature maps, tracked binary masks, and averaged DiT attention maps.The full visualization appears in Figure 9, while Figure 4 presents a cropped tabletop workspace.

C Additional Results

Additional experiments illustrate DexGraspVLA’s long-horizon visual reasoning and prompt-completion checks, while detailed evaluations show strong zero-shot grasping performance and further gains with recovery attempts.

  • Large-scale generalization: DexGraspVLA maintains consistently high success rates across unseen object, lighting, and background combinations in large-scale generalization tests.Additional attempts recover some failed cases, with the Ours@2 and Ours@3 rows showing further performance improvements.
  • Baseline comparison: DexGraspVLA achieves success rates around or above 90% in zero-shot cluttered-scene experiments, while baselines often fail catastrophically, especially in unseen clutter.The comparison is reported for the smaller-scale dexterous grasping experiments summarized in Table 4.
  • Long-horizon execution: In an exemplar long-horizon task, the planner identifies edible targets, verifies grasp outcomes, and checks whether all prompted objects have been removed.The task prompt is to grasp all edible objects, including food and drinks, in a cluttered scene.
  • Visual grounding: The planner also produces target-specific responses, including object descriptions and bounding-box predictions, during the long-horizon interaction.Examples include snack packets and bottles, followed by verification that targets are securely held in the robotic hand.
  • Failure detection: The completion checker correctly identifies incomplete execution when edible objects remain visible on the table.Examples explicitly conclude that the prompt is not fully completed when the bottle and packaged food remain.

D Additional Experiments

Additional experiments separately evaluate DexGraspVLA’s planner for language-grounded bounding-box prediction under environmental distractions. Across 150 tests, it nearly always marks the correct target.

  • Experimental setup: The planner is evaluated under no, background, and lighting distraction conditions using cluttered scenes and textual object prompts.The evaluation includes 30 no-distraction tests and 60 tests each for background and lighting distraction.
  • Metric: Bounding-box accuracy is defined as the proportion of tested objects whose boxes tightly enclose the target object.This metric directly measures whether the planner marks the intended grasp target accurately.
  • Results: 149 of 150 prompts receive successful bounding-box predictions, yielding aggregated accuracy exceeding 99%.Only one bounding box is mislabeled across the full evaluation.
  • Results: Across varied lighting and background conditions, the planner consistently grounds language instructions to correct objects in cluttered scenes.The reported examples include object identification by names such as “Coca Cola” or “milk.”
Loading 2502.20900v5…