PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Universal Grasping

Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu, Billy Yan, Irmak Guzey, David Fouhey, Dandan Shan, Lerrel Pinto

arXiv:2606.17054v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Dexterous robots lack the generality of human grasping partly because they lack diverse real-world grasp data. HUG learns natural human grasps from egocentric RGB-D data and achieves 66.7% success on 30 unseen test objects, beating baselines by +23% and +34%.

  • Problem

    Dexterous robot grasping remains less general than human grasping, while robots lack the diverse real-world grasping experience humans accumulate daily.

  • Method

    HUG trains a flow-matching model on 1M egocentric human grasps to generate diverse MANO grasps from one RGB-D image and retarget them across robot hands.

  • Results

    66.7% tabletop success on 30 unseen test objects, exceeding Dex1B by +23% and CAP by +34%.

  • Takeaways & Limitations

    HUG transfers human-like grasps zero-shot across new stereo cameras, robot embodiments, and uncontrolled households.

  • Takeaways & Limitations

    HUG models only right-handed grasps with fixed canonical hand shape and executes real-world rollouts open-loop without visual feedback.

Abstract

from arXiv · show

Humans can grasp objects effortlessly, whereas multi-fingered robots are far from this level of generality. We argue that the most natural source of robot grasping data is from humans, who pick up thousands of objects every day. We present HUG, a flow-matching model that generates diverse human grasps for any user-specified object in a single RGB-D image captured from a stereo camera. Using smart glasses, we first collect 1M-HUGs, an egocentric dataset of human grasps spanning 1M frames (27.8 hrs) and 6,707 object instances across 41 buildings. Next, to model the distribution of natural human grasps, our novel flow-matching model fuses RGB and depth observations to output a grasp parameterized by wrist translation, wrist rotation, and MANO hand pose. Predicted grasps can be retargeted to various robot hands, enabling zero-shot grasping in everyday scenes. To standardize evaluation, we build a new simulated benchmark, HUG-Bench, of 90 unseen objects from five geometric categories and various sizes, with metric-scale 3D meshes. We evaluate HUG in the real world on the 30-object test set of HUG-Bench across multiple stereo cameras, robot embodiments, and household environments. HUG outperforms the state-of-the-art grasping baselines by +23% and +34% on our challenging object set. Code, data, benchmark, checkpoints, and an interactive demo are released on our website: https://grasping.io/

1 Introduction

HUG addresses the lack of diverse real-world grasping data by learning natural human grasps from egocentric RGB-D observations and retargeting them across robot hands. It contributes the 1M-HUGS dataset, the HUG flow-matching model, and HUG-BENCH, achieving strong real-world grasping performance.

  • Motivation: Current dexterous robot-grasping models lack the diverse real-world experience humans accumulate while grasping arbitrary objects.Prior synthetic methods face the sim-to-real gap, while teleoperation is tedious and cannot cover the open world.
  • Method: HUG uses a flow-matching model to generate diverse MANO grasps from a single stereo-camera RGB-D image and retargets them to robot hands without per-embodiment training.The framework is trained on in-the-wild human grasps to model naturally occurring, reliably executable grasps.
  • Dataset: 1M-HUGS contains 1M egocentric image-grasp pairs from 6,707 recordings captured across 41 buildings.The recordings use Aria Gen 2 and include natural human grasps, MANO-fit hand poses, and metric depth.
  • Benchmark: HUG-BENCH comprises 90 challenging unseen objects across five geometric categories and three size bins, reconstructed as metric-scale meshes from real objects.It evaluates grasping in both simulation and the real world.
  • Results: 66.7% tabletop success on HUG-BENCH’s 30-object real-world test set surpasses baselines by +23% and +34%, while in-the-wild success reaches 62.0%.The system generalizes zero-shot across stereo cameras, robot hands, and household environments.

2 Related Work

Prior grasp prediction work performs well for two-fingered grippers but remains difficult for multi-fingered hands, where simulation-based methods are common because dexterous teleoperation data is hard to collect. Recent work instead learns robot behaviors from human and other non-robot datasets, motivating scalable egocentric grasp data such as 1M-HUGS.

  • Robotic object grasp prediction: Two-fingered grasp prediction has achieved strong real-world performance using self-supervised collection and large-scale datasets, whereas multi-fingered grasping remains harder.The passage contrasts established two-fingered systems with the greater difficulty of dexterous multi-fingered hands.
  • Robotic object grasp prediction: Simulation-based reinforcement learning and generative grasp synthesis dominate prior multi-fingered approaches because dexterous teleoperation data is difficult to collect.These approaches still struggle, according to the related-work discussion.
  • Robot learning from non-robot datasets: 1M-HUGS provides 1M egocentric frames spanning 6,707 object instances, with synchronized views, metric depth, object masks, and MANO hand poses with wrist transformations.Each entry records these modalities in the camera frame.
  • Robot learning from non-robot datasets: Recent robot-learning methods use non-teleoperated datasets, enabled by motion tracking, hand-object reconstruction, and interaction synthesis.Earlier in-domain human datasets offered rich 3D annotations but tied collection to deployment, limiting scalability.

3 1M-HUGS Dataset

1M-HUGS is an egocentric dataset of natural human grasps collected with smart glasses in everyday environments, producing diverse RGB and stereo-grayscale training pairs. Its protocol combines multi-view recording, automated and human curation, and frame-level filtering across thousands of recordings and buildings.

  • Collection protocol: 1M-HUGS records natural human grasps in everyday environments using egocentric smart glasses, unlike datasets collected in simulation or laboratory settings.The dataset captures dynamic views by back-propagating grasp information across preceding no-hand frames.
  • Collection protocol: Each recording captures 15–30 seconds of static-scene views from diverse head viewpoints before the wearer reaches with the right hand to grasp the object.Recordings use Aria Gen 2 glasses with synchronized RGB and stereo grayscale views, 6-DoF camera poses, and 3D hand landmarks.
  • Curation: Automated curation identifies the grasped object, propagates its mask across frames, selects the grasp frame with stability and proximity heuristics, and requires human web-based review.The pipeline uses a vision-language model and SAM3 for object identification and mask propagation.
  • Dataset statistics and labels: 6,707 recordings across 41 buildings cover an estimated ∼1.5K unique objects, with scenes spanning kitchens, bedrooms, and hundreds of distinct environments.The collection protocol grasps whatever objects are available within each building.
  • Frame filtering: 2M training entries remain after filtering, comprising 1M RGB and 1M grayscale frames, with grayscale frames sharing timesteps with RGB frames for monochrome-camera generalization.Surviving entries are cropped and resized to 224×224 after applying object-mask, depth-confidence, grasp-intersection, landmark-bounds, and no-hand criteria.

4 Method

HUG is a flow-matching model that generates diverse human grasps from an RGB-D image and a target-object click. It represents grasps explicitly and combines RGB, metric point-cloud, and query information before flow-based prediction.

  • Model overview: HUG generates diverse natural grasps for a user-specified object from a single stereo-camera RGB-D image.The model is trained on real-world human grasps.
  • Grasp representation: The predicted grasp is a 99-dim state comprising wrist translation, 6D wrist rotation, and 15 MANO joint rotations.A 2D click (u, v) identifies the target, while depth back-projection produces a metric 3D query point.
  • RGB-PC fusion: RGB and point-cloud streams are fused by projecting point-cloud centroids into RGB features and refining 256 fused tokens with a 4-layer transformer.The RGB encoder is frozen DINOv2-Base, while PointNeXt processes 4096 points sampled within a 0.3 m crop around the query.
  • Flow transformer: The flow transformer separately embeds translation, wrist rotation, and finger pose into 512-dim tokens and processes them with 6 DiT blocks conditioned on scene tokens and timestep.Separate tokens reduce over-mixing between geometrically distinct grasp components and balance their gradient signals.
  • Training objective: Training combines velocity-prediction MSE with MANO landmark supervision, using λv =1 and λ3D =20 and weighting geometric loss by (1−t).The geometric term is concentrated on near-clean diffusion steps where the recovered clean state is meaningful.

5 Experiments

Experiments evaluate HUG on the challenging HUG-BENCH benchmark in simulation and on 30 real-world test objects. HUG achieves 73.0% simulated test success and 66.7% real-world success, outperforming modality baselines and prior robot grasping methods.

  • 5.1 HUG-BENCH: HUG-BENCH contains 90 difficult objects across five geometric categories and three size bins, including 30 unseen test objects.The test split has two objects per category-size combination.
  • 5.2 Simulation: Simulation evaluates 10 grasps per object using success rate and fingertip contact error during open-loop grasp-and-lift rollouts.Success requires the object to leave the table after a 0.5 m lift; lower contact error is better.
  • 5.2 Simulation: 73.0% test SR is achieved by HUG, compared with 94.0% for the human grasp oracle; validation SR is 71.5% versus 90.3%.The oracle is an upper bound affected by tracking error, asset inaccuracies, and open-loop failures.
  • 5.2 Simulation: RGB+PC outperforms single-modality variants, reaching 73.0% test SR versus 70.7% for PC-only and 29.7% for RGB-only.Test FC error is 14.6 mm for the full model, 22.1 mm for PC-only, and 109 mm for RGB-only.
  • 5.2 Simulation: From 25K to 1M frames, test SR rises from 33% to 73% while FC error falls from 54.2 mm to 14.6 mm, without saturation.Validation and test curves track tightly, indicating transfer to held-out objects without overfitting.

6 Conclusion

HUG learns dexterous robot grasping from 1M natural human grasps, predicts MANO hand grasps from single RGB-D images, and retargets them to multiple dexterous hands without per-hand training. HUG-BENCH evaluates these capabilities on unseen everyday objects through simulation and real-robot trials.

  • Conclusion: HUG learns dexterous robot grasping entirely from 1M natural human grasps in the 1M-HUGS egocentric dataset.The dataset contains 1M natural human grasps.
  • Conclusion: HUG predicts a MANO hand grasp from a single RGB-D image and retargets it to multiple dexterous hands without per-hand training.Retargeting occurs at deployment.
  • Conclusion: HUG-BENCH evaluates HUG on unseen everyday objects with metric-scale meshes using both simulation and real-robot trials.The benchmark pairs simulation with real-robot trials.

7 Limitations … B.2 Quality Control

HUG’s limitations include right-hand-only, fixed-shape modeling and possible human-to-robot retargeting failures. The appendix documents MANO fitting, dataset curation and quality control, model and benchmark implementation, and simulation and real-world evaluation procedures.

  • 7 Limitations: HUG models only right-handed grasps with MANO shape fixed canonically, excluding left-handed, bimanual, and hand-specific morphology.Object rotations are varied during evaluation to reduce bias toward favorable poses.
  • 7 Limitations: Human-to-robot retargeting can fail for some target robot hands.
  • Appendix of “Human Universal Grasping”: The appendix covers MANO parameter optimization, 1M-HUGS curation, HUG implementation, HUG-BENCH asset generation, and simulation and real-world evaluation.The listed sections include camera generalization, training details, object statistics, grasp-quality metrics, retargeting, and qualitative observations.
  • A MANO Parameter Optimization: Aria2mano fits full articulated MANO pose, shape, and mesh to sparse 21-landmark Aria skeletons using per-frame anatomical optimization.It optimizes 15 finger-joint axis-angle parameters and 10 MANO shape dimensions, while fixing wrist orientation and translation to Aria estimates.
  • A MANO Parameter Optimization: <2 mm average fingertip error is achieved by L-BFGS optimization with anatomical constraints, fingertip-weighted landmark loss, and temporal warm starts.The optimizer uses strong-Wolfe line search, history size 10, and a closure-call budget of 20 for warm-started frames.
  • B 1M-HUGS Dataset Curation with aria2mano: 1M-HUGS curation converts 15–30-second Aria recordings of stationary objects into verified grasp entries through object identification, segmentation, grasp-frame selection, and preparation.Each recording captures a single right-hand grasp, with the wearer moving around the object before grasping without lifting it.
  • B.1 Automated Labeling: The first three labeling stages are automated using VLM object identification, SAM3 mask propagation, and heuristic grasp-frame selection.Selection requires valid MANO tracking, hand motion below 0.01 m per frame, fingertip-to-mask distance below 100 px, and placement within the recording’s last 10 seconds.
  • B.2 Quality Control: Every recording undergoes web-app review because automatic labeling is not 100% accurate.Reviewers can correct masks, reselect grasp frames, approve recordings, or mark them as having no stable grasp; approval sets a checked flag for dataset preparation.

B.3 Dataset Preparation … C.2 Camera Generalization

The paper prepares HUG data with geometric, depth-validity, object-mask, and hand-shape consistency filters, then implements localized point-cloud encoding and camera-independent geometric processing. These choices support focused receptive fields, consistent MANO supervision, and transfer across stereo cameras without retraining.

  • B.3 Dataset Preparation: Training frames are filtered by temporal position, annotation exclusion, depth validity, object-mask presence, and grasp-hand landmark overlap.Shared filters retain indices in [20, grasp−10), require at least 60% valid depth-confidence pixels, and enforce object-mask and landmark conditions.
  • B.3 Dataset Preparation: RGB images are cropped to the depth field of view, reduced to a 1440×1440 center window, resized to 224×224, and stored with updated intrinsics.After cropping, roughly 99.9% of depth is valid; approximately 0.1% parallax holes are filled by nearest neighbor.
  • B.3 Dataset Preparation: HUG fixes MANO shape β to one canonical hand while recomputing labels across 1M-HUGS and HUG-BENCH for consistent training and evaluation.The canonical hand is exported as an MJCF and used for every simulation grasp rollout.
  • C HUG Model Implementation Details: The implementation details cover receptive-field cropping, camera generalization, modality ablations, training dynamics, and hyperparameters.These topics define the scope of the model-implementation section beyond the main-text method description.
  • C.1 Point Cloud Crop Radius: The back-projected point cloud is cropped to points within radius r = 0.3 m of the 3D query point and sampled to Np = 4096 points.The crop bounds the receptive field and concentrates points around the target object rather than distributing them across the full image.
  • C.1 Point Cloud Crop Radius: The cropped point cloud produces N = 256 tokens, with metric centroids encoded using Fourier features to better capture geometric detail.The radius determines the model’s spatial scope and the largest object region represented by the crop.
  • C.2 Camera Generalization: Camera intrinsics K are used only for back-projection and inverse projection, not provided as learned inputs.This geometric design enables transfer across stereo cameras with different intrinsics without retraining, including Aria-to-ZED deployment and successful Realsense D415/D435 tests.

C.3 Training Details and Hyperparameters · D HUG-BENCH Asset Generation with aria2mesh · D.1 Metric-scale 3D Object Reconstruction

The paper specifies HUG’s architecture, training, inference, and parameter-count configurations, while aria2mesh reconstructs 90 everyday objects into metric-scale, physics-ready simulation assets. The reconstruction pipeline combines egocentric Aria views, Multi-view SAM3D, depth and camera geometry, manual verification, and mesh processing to produce benchmark assets in minutes.

  • C.3 Training Details and Hyperparameters: HUG’s architecture, training, and inference hyperparameters are reported in Table 4.MANO-fitting settings are provided separately in Appendix A.
  • C.3 Training Details and Hyperparameters: HUG’s per-module parameter counts are reported in Table 5.The paper also presents training curves for all ablation configurations.
  • D HUG-BENCH Asset Generation with aria2mesh: HUG-BENCH contains 90 everyday objects with metric-scale meshes and physics-ready simulation assets reconstructed from Aria Gen 2 recordings.The released aria2mesh pipeline generates these benchmark assets and supports per-object statistics for the full set.
  • D.1 Metric-scale 3D Object Reconstruction: Each object begins with five egocentric Aria Gen 2 views fused by training-free Multi-view SAM3D with injected camera intrinsics, extrinsics, and stereo depth.This establishes the initial 3D reconstruction from the recorded views.
  • D.1 Metric-scale 3D Object Reconstruction: The reconstructed meshes are pose-optimized and gravity-aligned, then manually verified and edited against SLAM semidense point clouds and dense stereo depth in Viser.These steps correct object scale and pose before final mesh processing.
  • D.1 Metric-scale 3D Object Reconstruction: Alpha Wrap, implemented in PyMeshLab, makes each mesh watertight before convex decomposition produces a simulation asset.The passage describes this as part of the final asset-generation pipeline.
  • D.1 Metric-scale 3D Object Reconstruction: Metric-scale, simulation-ready assets are produced in minutes, enabling practical benchmark expansion and construction of task-specific object sets.This speed is presented as the practical benefit of the reconstruction pipeline.

D.2 Object Statistics … E.2 The Human Grasp Oracle

The paper combines a 90-object HUG-BENCH inventory with simulated MANO-hand rollouts and analyzes both the human grasp oracle’s realism and its sensitivity to spatial error. The evaluation shows that successful grasping requires precise placement, with tolerance depending on object size.

  • D.2 Object Statistics: Table 6 catalogs all 90 HUG-BENCH objects by size class, geometric category, mass, volume, meshes, and best-checkpoint success rates.The 30 test objects appear first and include real-world results; validation objects are simulation-only.
  • D.2 Object Statistics: The simulation reports per-object ground-truth human-oracle and HUG success rates, while real-world rates cover tabletop and in-the-wild settings.Tabletop uses ZED + xArm + Ability; in-the-wild uses Aria + YOR + WUJI.
  • E Simulation Grasping Evaluation: The evaluation complements success rate with per-object simulation results for all 90 objects and qualitative predicted grasps in HUG-BENCH scenes.The per-object GT Sim SR and Sim SR columns are reported in Table 6, while qualitative examples appear in Figure 4.
  • E.1 Simulated MANO Hand: Simulation evaluation uses a physics-ready capsule MANO hand baked from the fixed-shape MANO model, with the palm represented as a convex mesh.The model assigns mesh vertices to bones and represents each phalanx with a capsule spanning adjacent joints.
  • E.1 Simulated MANO Hand: The simulated hand has 0.4 kg total mass, a 6-DoF free wrist, ball-joint fingers, critically damped actuators, and softer palm contact compliance.The softer palm compliance addresses recorded grasps whose palms penetrate rigid object meshes by 1–2 cm.
  • E.2 The Human Grasp Oracle: The human grasp oracle does not achieve 100% success because of hand-tracking error, rigid non-articulated objects, and an open-loop pre-grasp-to-lift trajectory.Occluded fingers may be tracked too open or too closed, making recorded grasps slightly loose or tight.
  • E.2 The Human Grasp Oracle: Small objects reach near-zero success by 3–4 cm per axis, whereas large objects retain about 25% at 3 cm and 10% at 4 cm before vanishing by 6 cm.The size-dependent tolerance indicates that the benchmark rewards precise placement, although larger objects provide more stable contact regions.

E.3 Dense Grasp Quality Metrics · F Real-World Grasping Evaluation

The paper supplements fingertip contact error with penetration, intersection, and miss-distance metrics, while cautioning that static metrics require interpretation against success rate. It also evaluates real-world grasping in tabletop and in-the-wild settings, including robot-hand retargeting and per-object success rates.

  • E.3 Dense Grasp Quality Metrics: The evaluation reports maximum penetration depth and hand-object intersection volume at grasp closure, complementing fingertip contact error.Penetration depth is computed over hand surface samples using signed surface distance, while intersection volume is |H∩O|.
  • E.3 Dense Grasp Quality Metrics: Table 7 evaluates extended penetration and contact metrics on HUG-BENCH using each ablation model’s best-val-SR checkpoint.It reports per-grasp mean ± SE over 600 validation and 300 test grasps, with object counts over 60 validation and 30 test objects; human grasp is an oracle upper bound.
  • E.3 Dense Grasp Quality Metrics: Penetration depth and volume alone can misrepresent grasp quality because low values may accompany unstable contact or averaged non-intersecting grasps.For example, a grasp tangent to the palm may be unstable, while grasps 10 cm inside and outside the object average to 0 penetration depth.
  • E.3 Dense Grasp Quality Metrics: The revised metrics report the percentage of penetrating grasps, penetration averages only for intersecting grasps, and nearest miss distance otherwise; lower is better for all three.These metrics are reported on ablation models in Table 7, extending the main ablation across the same evaluation splits.
  • E.3 Dense Grasp Quality Metrics: Static penetration metrics should be interpreted cautiously because rigid simulated meshes can label correct grasps of soft objects, such as pillows, as highly penetrating.The recommended reading compares each row’s penetration metrics with its SR rather than minimizing penetration alone.
  • F Real-World Grasping Evaluation: The real-world evaluation covers tabletop and in-the-wild settings, retargets predicted MANO grasps to Ability and WUJI robot hands, and examines qualitative trends and dexterity over a gripper.Per-object success rates are reported in the Tabletop SR and Wild SR columns of Table 6.

F.1 Real-World Evaluation Design · F.2 Retargeting to Robot Hands · F.3 Qualitative Observations

The real-world evaluation uses fixed, consecutive open-loop trials without real-world tuning, while HUG retargets MANO grasps to multiple robot hands. Qualitative results identify object geometry, environmental variation, and hand dexterity as key factors affecting grasp success.

  • F.1 Real-World Evaluation Design: HUG-BENCH evaluation runs 300 consecutive test trials, each with one grasp prediction and one open-loop execution, without cuts, retries, or real-world tuning.The checkpoint is selected by best simulated HUG-BENCH validation success rate, and the test objects are not used to tune the model or execution strategy.
  • F.1 Real-World Evaluation Design: Tabletop trials place each object at one location with slight position and rotation variation, while cluttered in-the-wild settings test robustness beyond tabletop constraints.The tabletop setup avoids clutter for repeatability and comparison with Dex1B.
  • F.2 Retargeting to Robot Hands: HUG’s predicted MANO grasp is retargeted to the Ability hand with AnyTeleop and to the WUJI hand with WUJI retargeting.The paper compares the MANO hand and robot hands in Figure 10.
  • F.2 Retargeting to Robot Hands: Robot-hand fingertips are aligned to MANO using a single fixed offset estimated from simulation, with no correction for WUJI and a translation correction for Ability.Offsets are expressed in each robot hand’s coordinate frame, whose origin is at the palm.
  • F.3 Qualitative Observations: Rounded, convex objects sized to fit the hand are easiest because they afford many stable enveloping grasps.Examples include pears, pineapples, and hacky sacks.
  • F.3 Qualitative Observations: Objects too large to wrap or irregular and articulated objects are hardest to grasp open-loop.Examples include footballs, wipe dispensers, nail clippers, and headphones.
  • F.3 Qualitative Observations: HUG remains robust across lighting, object rotation, viewpoint, color or grayscale stereo cameras, and reflective surfaces with noisier stereo depth.This robustness is attributed to 1M-HUGS spanning many environments, lighting conditions, and grayscale frames.
  • F.3 Qualitative Observations: Dexterous enveloping grasps outperform grippers selectively: they cage large or irregular objects, while antipodal grippers slip, and pinches can lose heavy objects to gravitational torque.The comparison is against CAP on identical objects, and the benefit of dexterity depends on object properties.
Loading 2606.17054v1…