Source-linked AI summary
Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis
Sixu Yan, Shikang Wang, Binhua Huang, Xuanlai Tang, Guohua Fan, Fan Huang, Haoxuan Li, Yongkang Li, Yuhan Li, Bencheng Liao, Zeyu Zhang, Wenyu Liu, Hangxin Liu, Xinggang Wang
TL;DR
Robotic grasping must adapt to task context while remaining physically executable across different hands, but many existing approaches tightly couple understanding and grasp policies. AdaRoboVLG separates these functions with a structured interface and composable foundation-model priors, achieving adaptive grasping in cluttered and dynamic real-world settings while supporting cross-hand synthesis.
Problem
Robotic grasping remains difficult because feasible grasp configurations are numerous and grasp strategies depend on surroundings, objectives, and interaction dynamics.
Method
AdaRoboVLG converts task contexts into a structured interface and uses an efficient base policy for hand-specific, physically feasible grasp synthesis.
Results
83.3% overall success rate is achieved across 510 real-world trials, while disturbance testing reaches an 89.7% average success rate.
Takeaways & Limitations
Composable spatial, cognitive, and temporal priors can jointly adapt grasp synthesis to cluttered, functional, and dynamic conditions through the shared interface.
Takeaways & Limitations
Perceptual uncertainty, depth noise, inconsistent grounding, pose drift, and missing tactile feedback constrain robustness and can leave contact failures undetected.
Abstract
from arXiv · showhide
This paper proposes AdaRoboVLG, a task-adaptive Vision-Language-Grasp (VLG) framework that supports generalizable grasp synthesis across different robotic hands. Unlike existing VLG methods that tightly couple foundation models with end-to-end grasp policies, AdaRoboVLG learns an efficient generalizable base policy that generates and evaluates physically feasible grasp candidates through explicit kinematic mapping and force-closure-based stability estimation, while offloading task-dependent understanding to specialized foundation-model modules. These modules provide composable priors that are integrated into the grasp synthesis process, enabling contextually adaptive grasp synthesis without retraining the underlying grasp policy. Through extensive simulation and real-world experiments, we demonstrate that (i) the base policy exhibits efficient learning and strong cross-hand generalization, (ii) the framework effectively incorporates spatial, cognitive, and temporal priors to address three representative grasping challenges without compromising grasp synthesis performance compared to state-of-the-art methods, and (iii) these priors can operate jointly to enable functional grasping in cluttered and dynamic environments. These results indicate that decoupling physical grasp synthesis from task-dependent understanding provides a scalable paradigm for robotic grasping, allowing future advances in foundation models to be directly translated into improved grasp capabilities without redesigning or retraining the underlying grasp policy. Supplementary videos are available at https://adarobovlg.github.io/
I. INTRODUCTION
AdaRoboVLG separates task-dependent understanding from physical grasp synthesis through a structured grasp interface and composable foundation-model priors. The framework supports adaptive grasping across cluttered, functional, and dynamic scenarios while retaining an efficient, cross-hand base policy.
- Motivation: Existing vision-language grasping methods often couple task understanding and grasp execution, while contextual grasping must account for spatial, cognitive, and temporal challenges.These challenges include clutter and occlusion, task objectives, and interaction dynamics.
- Framework: AdaRoboVLG decouples task understanding from physical synthesis through an interface containing object geometry, CGRs, and grasp types.A base policy maps this interface to hand-specific candidates and evaluates stability using force-closure-related representations.
- Composable priors: Spatial, cognitive, and temporal priors construct or update the same interface for clutter-aware, functional, and dynamic grasp generation.The priors can operate independently or jointly, enabling contextually adaptive synthesis without changing the underlying interface.
- Simulation results: 88.4% average success rate is achieved across three robotic hands and three clutter levels on DexGraspNet 2.0.The spatial prior achieves state-of-the-art results on the random and loose splits.
- Real-world results: 83.3% overall success rate is achieved over 510 real-world trials across 102 everyday objects from six categories.On a moving conveyor belt, the system updates grasps online at 5 Hz, and disturbance testing reaches an 89.7% average success rate.
- Implications: Decoupling synthesis from task understanding creates a stable interface through which improved foundation models can be integrated by updating corresponding prior modules.The paper presents this as an alternative to repeatedly retraining end-to-end policies.
III. PROBLEM FORMULATION AND GRASP INTERFACE
The formulation represents adaptive grasping through a structured interface that bridges task context and executable, hand-compatible grasp poses. Contact primitives and grasp types remain hand-agnostic at the interface level, while the base policy performs embodiment-specific synthesis.
- A. Notation and Preliminaries: The grasp representation covers parallel and multi-finger end-effectors and supports functional grasp types through hand-agnostic contact primitives.The listed embodiments include Franka, Robotiq, DH3, Inspire, and Allegro hands.
- A. Notation and Preliminaries: The formulation introduces notation for vectors, matrices, sets, transposes, and set cardinalities before defining adaptive vision-language grasping.The notation section establishes the symbols used throughout the problem formulation.
- B. Problem Formulation and Structured Grasp Interface: A multi-finger grasp pose consists of a hand base pose T in SE(3) and an n-DoF joint configuration q.T contains rotation R and translation t, while q characterizes the hand configuration.
- B. Problem Formulation and Structured Grasp Interface: The goal is to synthesize executable grasp poses compatible with the hand embodiment H and task context C.The grasp set is denoted G.
- B. Problem Formulation and Structured Grasp Interface: Instead of mapping task context directly to poses, the formulation first converts context C into a structured grasp interface I.The interface mediates between task-dependent understanding and physical grasp synthesis.
- B. Problem Formulation and Structured Grasp Interface: The interface contains object geometry, CGRs, and grasp types following a human grasp taxonomy and remaining kinematically compatible with H.CGRs characterize contact primitives for stable grasping, while grasp types provide functional structure.
- B. Problem Formulation and Structured Grasp Interface: The base policy fπ synthesizes executable poses from I and H, while fτ converts task context into I using composable foundation priors.This separates hand-specific physical synthesis from task-dependent interface construction.
- B. Problem Formulation and Structured Grasp Interface: The base grasp policy is organized around policy design, CGR definition, hand-specific mapping, HOI representation, and hand-agnostic evaluation.These components provide the implementation structure for generalizable grasp synthesis.
A. Design Principle
The base policy converts a structured grasp interface into executable hand-specific candidates, then evaluates and ranks them using a hand-agnostic stability model. Explicit kinematic mapping and lookup-table inference support efficient candidate generation across robotic hands.
- The base policy converts the grasp interface into executable grasp poses for a target robotic hand.
- A two-stage process combines hand-specific grasp mapping with hand-agnostic grasp evaluation.
- The mapping function uses object geometry, a CGR descriptor, and a grasp type to generate hand-specific grasp candidates.
- The grasp decision model extracts hand-object contact geometry and predicts a success probability for each candidate.
- Grasp candidates are ranked to select a top-K set that maximizes predicted success.
D. Hand-Object Interaction Representation
The hand-object interaction representation encodes contact-point coordinates and normals to provide a unified interface for evaluating grasp stability across hand embodiments.
- Stable grasping is defined by force closure, meaning the grasp can resist arbitrary external wrenches.
- The HOI representation consists of coordinates and normals for contact points at the hand-object interface.
- Contact pairs are identified by computing object-to-hand surface distances and applying a proximity threshold.
- The resulting representation provides a unified evaluation interface across different hand embodiments.
E. Hand-Agnostic Grasp Evaluation
The hand-agnostic grasp evaluator predicts stability from local contact geometry, while composable foundation priors transform scene and task context into structured grasp constraints for adaptive synthesis.
- E. Hand-Agnostic Grasp Evaluation: A lightweight point-based decision model predicts a scalar stability score from the HOI representation.
- E. Hand-Agnostic Grasp Evaluation: PointTransformer features and contact-aware multi-pooling aggregate variable-sized contact geometry before stability prediction.The pooling mechanism combines maximum, mean, and standard-deviation statistics with learnable weights.
- E. Hand-Agnostic Grasp Evaluation: The decision model is trained with binary cross-entropy using ground-truth labels and predicted grasp-success probabilities.
- Spatial, cognitive, and temporal priors are composed for cluttered, functional, and dynamic grasping tasks.The corresponding transformations are Ms, Mc ◦ Ms, and Mt ◦ Mc ◦ Ms.
- A. Spatial Prior for Cluttered Grasping: The spatial prior extracts scene representations and scene-level CGRs from multi-view observations to provide feasible contact candidates in clutter.
- The cognitive prior combines language reasoning, visual grounding, and 3D lifting to produce task-conditioned target geometry, semantics, and functional CGRs.
1) Scene Semantics Construction:
The framework constructs scene semantics by extracting multi-view DINOv3 features, lifting them into 3D, and combining them with dense CGR predictions to form a task-conditioned grasp interface.
- DINOv3 extracts pixel-level semantic feature maps from each RGB view for scene representation.
- An implicit field lifts multi-view semantic features into 3D descriptors for arbitrary scene points.
- Aggregating point-wise descriptors yields the scene semantic representations RS.
- Contact-GraspNet uses scene geometry and semantic representations to produce dense per-point CGRs, filtered by contact probability δc.
- The resulting CGRs, scene geometry, and grasp types form a task-conditioned interface for clutter-aware grasping and cognitive selection.
1) Grasp Functionality Inference:
The functionality-inference pipeline parses language into target objects, functional parts, and grasp types, grounds them in 3D scene representations, and updates the grasp interface for dynamic targets.
- Grasp Functionality Inference: An LLM-based parser infers target-object, functional-part, and grasp-type specifications from language instructions.
- Grasp Functionality Inference: Retrieval-augmented generation and chain-of-thought reasoning constrain grasp-taxonomy retrieval and decompose task-intent, part-selection, and grasp-requirement inference.
- Grasp Functionality Inference: Visual grounding with Seed-1.8 and SAM3 produces multi-view instance masks that are lifted into 3D to isolate target-object geometry and semantics.
- Grasp Functionality Inference: Functional-region grounding filters CGRs using instance- and function-level consistency, producing spatially feasible candidates for language-guided grasp synthesis.
- Temporal Prior for Dynamic Grasping: For dynamic grasping, SAM3 mask tracking and DINOv3 feature consistency estimate rigid motion and update the interface across observations.
- Temporal Prior for Dynamic Grasping: Propagating the initial target mask over time mitigates sequential segmentation instability and semantic drift during grasping.
2) Object Pose Estimation:
The framework estimates object pose changes by matching semantic descriptors across observations under rigid-body constraints, then transforms the initial contact interface for temporally consistent grasping.
- Object Pose Estimation: SAM3-tracked object observations provide the basis for estimating the target’s real-time rigid transformation in SE(3).
- Object Pose Estimation: Semantic consistency optimization minimizes descriptor discrepancies between the initial object and features queried from the current implicit field.
- Object Pose Estimation: Differentiability enables gradient-based optimization, while rigid-body constraints and distance regularization improve robustness and stability.
- Object Pose Estimation: The estimated rotation and translation directly update the real-time contact interface from the initial contact reference.
- Object Pose Estimation: Because pose error remains within an admissible tolerance, the system directly transforms the initial grasp pose to improve efficiency.
- Training Implementation: Simulation validation covers data collection, base-policy generalization, and spatial, cognitive, and temporal priors.
2) Training Implementation:
Training uses an explicit hand-mapping stage and hand-agnostic grasp evaluation, with HOI representations improving learning and cross-hand generalization in simulated experiments.
- Training Implementation: The base policy separates hand-specific grasp mapping from hand-agnostic grasp evaluation and supports end-effectors with two to five fingers.
- Training Implementation: The representation study compares object-centric HOI, hand-centric HOI, and separated hand-object point-cloud inputs on Allegro data.
- Training Implementation: HOI representations converge faster and achieve lower final training losses than separated representations.
- Training Implementation: Stable HOI representations substantially overlap across DH3, Allegro, and Inspire in the shared embedding space.
- Training Implementation: 82.95%, 88.02%, and 91.07% accuracy are achieved on DH3, Allegro, and Inspire when training jointly across all three hands.
- Spatial Prior Evaluation: The spatial-prior evaluation measures grasp success across Dense, Random, and Loose clutter splits using the DexGraspNet 2.0 protocol.
- Spatial Prior Evaluation: AdaRoboVLG is compared with AnyDexGrasp and AnyDexGrasp* in zero-shot evaluation across DH3, Allegro, and Inspire without retraining.
5) Experimental Results:
AdaRoboVLG combines a generalizable grasp policy with cognitive and temporal modules to synthesize language-guided and dynamically tracked grasps across hands and clutter conditions.
- Base policy and cluttered grasp synthesis: AdaRoboVLG achieves the highest success rates on random and loose DexGraspNet 2.0 splits across three hands, improving over AnyDexGrasp by 3.0% and 9.2%.Performance is less strong on the dense split, but remains more stable across clutter levels.
- Cognitive prior: RAG and CoT substantially improve open-set grasp functionality inference accuracy while reducing reliance on large-scale language-action paired data.The evaluation uses 125 image-instruction pairs spanning 57 manipulation tasks and 80 object categories.
- Language-guided target grasping: AdaRoboVLG achieves the highest language-guided target-grasping success rates on GraspClutter6D and GraspNet-1Billion, exceeding AnyDexGrasp by 2.7% and 2.5%.Compared with D(R, O)-Grasp, average improvements reach 23.7% and 27.1%, respectively.
- Temporal prior: The temporal prior evaluates grasp tracking through MGTA, translation error, and rotation error, using continuous visual observations to update grasp constraints.MGTA measures alignment with ground-truth projections, while translation and rotation errors measure geometric precision.
3) Baseline Models:
Real-world experiments test AdaRoboVLG across platforms, end-effectors, cluttered objects, and failure modes, showing functional grasping and transfer across hardware.
- Experimental platforms: Real-world evaluation spans three robotic platforms equipped with parallel-jaw grippers or 6-DoF ROHand dexterous hands.The platforms are KEENON XMAN-R1, TianJi Marvin, and AgileX Piper.
- Static clutter setup: The static-clutter benchmark contains 102 everyday objects across six categories, with five trials per object under randomized poses.Success requires correct target identification, functional grasping, and collision-free execution.
- Static clutter results: Composing spatial and cognitive priors yields an 83.3% overall success rate across 510 real-world grasping trials.The system selects functional parts and grasp types while planning collision-free grasps among distractors.
- Failure analysis: Functional-part recognition errors are the leading failure mode, accounting for 29.4% of unsuccessful trials.Other failures include grasp-pose estimation at 22.3%, insufficient contact stability at 20.0%, multi-object interference at 16.5%, and grasp-type errors at 11.8%.
- End-effector transfer: AdaRoboVLG transfers functional grasping between dexterous and parallel-jaw end-effectors on the TianJi Marvin platform.The deployment demonstrates hardware transfer without changing the framework’s structured grasp interface.
C. Experiments of Language-Guided Functional Grasping in Dynamic Cluttered Scenes
AdaRoboVLG composes spatial, cognitive, and temporal priors for functional grasping on moving objects, while exposing extensions and explicit limitations for uncertain perception and physical interaction.
- Dynamic cluttered grasping: On a moving conveyor belt, AdaRoboVLG performs language-guided functional grasping while updating grasp poses online at 5 Hz.The experiment uses objects moving at 2 cm/s to 5 cm/s and combines collision-aware spatial reasoning with functional cognitive reasoning.
- Prior composition: The dynamic results demonstrate joint operation of spatial, cognitive, and temporal priors for real-world grasping.The temporal module maintains target feature consistency and continuously updates grasp poses during perturbations.
- Tracking robustness: Under four human-induced disturbances, the system achieves an average grasping success rate of 89.7%.The disturbances include out-of-view recovery, fast movement, continuous occlusion, and simultaneous translation and rotation.
- Extensibility: Bimanual sorting and transparent-object grasping are supported by adding task-allocation and depth-completion modules.The extensions distribute targets between arms or recover missing geometry before constructing structured grasps.
- Limitations: The framework remains sensitive to perceptual uncertainty, lacks tactile feedback for verifying contact conditions, and does not explicitly model dependencies with other manipulation actions.These limitations affect geometric accuracy, closed-loop stability assessment, and sequential manipulation in heavily cluttered scenes.