Source-linked AI summary
Learning Dexterous Grasping with Object-Centric Visual Affordances
Priyanka Mandikal, Kristen Grauman
TL;DR
Dexterous grasp learning is difficult because high-dimensional hands require efficient guidance without costly human action demonstrations. GRAFF embeds an object-centric image affordance model in closed-loop reinforcement learning, and reports faster, more effective, robust, and more generalizable grasping across objects.
Problem
High-dimensional dexterous hands make pure reinforcement learning sample-inefficient, while expert state-action demonstrations are costly and image-based affordances are needed for generalization.
Method
GRAFF predicts human-favored object affordance regions from images and uses them within a closed-loop RL policy for dexterous grasping.
Results
3× faster training is achieved while producing better-quality grasps, robustness to sensor and actuation noise, and generalization to unseen objects across 40 objects.
Takeaways & Limitations
Object-centric visual affordances provide a human prior while allowing the agent to discover grasp strategies through closed-loop experience.
Abstract
from arXiv · showhide
Dexterous robotic hands are appealing for their agility and human-like morphology, yet their high degree of freedom makes learning to manipulate challenging. We introduce an approach for learning dexterous grasping. Our key idea is to embed an object-centric visual affordance model within a deep reinforcement learning loop to learn grasping policies that favor the same object regions favored by people. Unlike traditional approaches that learn from human demonstration trajectories (e.g., hand joint sequences captured with a glove), the proposed prior is object-centric and image-based, allowing the agent to anticipate useful affordance regions for objects unseen during policy learning. We demonstrate our idea with a 30-DoF five-fingered robotic hand simulator on 40 objects from two datasets, where it successfully and efficiently learns policies for stable functional grasps. Our affordance-guided policies are significantly more effective, generalize better to novel objects, train 3 X faster than the baselines, and are more robust to noisy sensor readings and actuation. Our work offers a step towards manipulation agents that learn by watching how people use objects, without requiring state and action information about the human body. Project website: http://vision.cs.utexas.edu/projects/graff-dexterous-affordance-grasp
I. INTRODUCTION
Dexterous hands enable fine-grained, functional manipulation but make learning difficult because of high-dimensional control and costly human demonstrations. GRAFF replaces person-centric trajectories with object-centric visual affordances embedded in reinforcement learning, improving efficiency and generalization.
- Dexterous hands offer fine-grained manipulation and compatibility with objects designed for human hands.
- 24 DoF hand models create high-dimensional state and action spaces, making pure reinforcement learning sample-inefficient.
- GRAFF predicts object regions amenable to human interaction and rewards an RL agent for touching those regions.
- The image-based affordance prior lets the policy discover grasp strategies through closed-loop experience while generalizing to unseen object instances.
- Experiments use a 30 DoF AdroitHand simulator and image annotations, which require lighter supervision than state-action demonstrations.
- Across 40 objects, GRAFF improves grasp quality, robustness, sample efficiency, and generalization, including a 3× training speedup without state-action demonstrations.
II. RELATED WORK
Prior grasping work uses analytical planning, closed-loop RL, object-specific tasks, or expert demonstrations. GRAFF instead injects human-derived object-centric affordances into closed-loop RL to improve dexterous grasping efficiency and generalization.
- Grasping with planning: Analytical grasping approaches use object pose, shape, gripper configuration, and friction to plan grasps, while learning methods often estimate pose before planning.
- Reinforcement learning for closed-loop grasping: Closed-loop RL updates grasp strategies from visual or contact feedback rather than separating grasp synthesis from motion planning.
- Reinforcement learning for closed-loop grasping: GRAFF adds an object-centric affordance prior learned from human grasps to closed-loop RL, targeting sample efficiency in dexterous action spaces.
- Learning manipulation with imitation: Unlike object-specific manipulation systems, GRAFF targets grasping and lifting across objects, including novel categories.
- Visual affordances: Compared with ContactGrasp, GRAFF learns a closed-loop RL policy and predicts affordances for unseen objects rather than sampling model-based poses for each object.ContactGrasp takes about 4 hours to sample poses for each unseen object, whereas the trained GRAFF policy runs in real time on new objects.
III. APPROACH
GRAFF uses two stages: image-based prediction of object-centric grasp affordances followed by dynamic RL policy learning. The policy combines these affordances with visual and hand-state inputs in a simulated 30 DoF dexterous-hand environment.
- GRAFF first trains a network to predict affordance regions from static images, then trains a dynamic grasping policy using those learned affordances.
- Training uses a simulated tabletop environment with a 30 DoF dexterous hand.
- Affordance anticipation is illustrated through training images generated from ContactDB thermal maps and predictions on seen and novel objects.
A. Affordance Anticipation From Images
The affordance module learns to infer functional grasp regions from images using human contact data and segmentation. Its predictions cover novel objects and viewpoints, providing the visual prior used to guide dexterous grasping.
- The perception model infers object-centric grasp affordance regions from static images, replacing demonstration trajectories with image-based supervision.
- Thermal image contact training data: ContactDB supplies thermal contact maps from human grasps, including use and hand-off intents, for household objects.
- Thermal image contact training data: Representative contact maps are selected by clustering high-contact mesh points across participants, then rendered into image-affordance pairs.
- Thermal image contact training data: The dataset contains approximately 15k rendered training pairs split 80:10:10 across training, validation, and test sets.
- Image affordance prediction model: Affordance learning is formulated as binary per-pixel segmentation using a Feature Pyramid Network with an ImageNet-pretrained ResNet-50 backbone.
- Image affordance prediction model: 80.4% IoU segmentation accuracy is achieved on the ContactDB test split, with meaningful predictions for novel objects and viewpoints.
B. Dexterous Grasping using Visual Affordances
The method combines RGB-D vision, an inferred affordance map, proprioception, and hand-to-object distances in an actor-critic reinforcement-learning policy. Affordance-guided reward shapes exploration toward object regions suitable for grasping without requiring full object state or dynamics.
- An actor-critic model estimates state values and samples actions for the dexterous grasping policy.The policy is formulated as a finite-horizon discounted MDP with stochastic actions.
- The policy receives RGB-D images, an inferred affordance map, proprioception, and distances between the hand and affordance region.These inputs are processed as visuomotor state components for closed-loop control.
- The affordance map is backprojected into 3D points and tracked throughout each episode, avoiding access to the object mesh, mass, or full object state.The experiments also examine substantial tracking failures, while the described setup assumes perfect affordance-region tracking.
- The architecture independently processes visual-affordance inputs and proprioceptive-contact inputs before sending them to an actor-critic network.The network samples actions and estimates state values from these streams.
- The total reward combines success, affordance, and entropy terms to guide learning and exploration.The affordance term incentivizes exploration of object areas within the predicted affordance region.
IV. EXPERIMENTS
Experiments evaluate GRAFF on seen and novel objects against pure RL baselines and DAPG, including noisy sensing and actuation, physical-property changes, and training efficiency. GRAFF achieves functional grasps, generalizes to unseen objects, remains robust under perturbations, and reaches comparable success with one-third the training samples.
- Experimental setup: GRAFF is evaluated on 16 seen ContactDB objects and 24 novel 3DNet meshes, including five categories absent from ContactDB.The comparisons include two pure RL baselines without affordance priors and DAPG, which uses object-specific motion-glove demonstrations.
- Experimental scope: The experiments focus on closed-loop dexterous grasp acquisition in simulation rather than end-effector pose estimation.The evaluation uses a multi-fingered hand and excludes methods designed only for pose estimation or parallel-jaw grasp execution.
- Grasping performance: GRAFF successfully grasps seen and novel objects at functional regions, whereas baselines sometimes fail or grasp non-functional regions.Examples include handles and other human-preferred regions for objects such as pans, mugs, teapots, knives, and scissors.
- Quantitative comparison: GRAFF outperforms pure RL baselines and DAPG across reported grasping metrics, including under noisy sensing and actuation.DAPG provides full expert motion trajectories, while GRAFF uses object-centric affordances instead.
- Physical-property robustness: GRAFF remains fairly robust across large variations in object mass and scale not encountered during training.The authors attribute this robustness to the policy’s preference for stable human-preferred grasp regions.
- Training efficiency: 3× speedup: GRAFF reaches the pure RL baselines’ 30% success rate with 50M samples instead of 150M.The comparison corresponds to approximately 50 versus 150 hours of robot experience, using one policy across ContactDB objects.
V. CONCLUSION
GRAFF learns dexterous grasping with object-centric visual affordances, improving learning speed and generalization to unseen visually related objects. The authors view these results as encouraging evidence for manipulation agents using more distant human supervision.
- GRAFF uses an image-based affordance model to focus closed-loop reinforcement learning on good places to grasp.
- The approach improves learning speed and enables policies to generalize to unseen, visually related objects.
- Future work includes expanding affordance modeling, representing multimodal viable regions, and extending manipulation beyond grasping.