PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains

Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, Cewu Lu

arXiv:2212.08333v2cs.RO

TL;DR

Existing grasp-detection methods are time-consuming, produce non-dense predictions, and largely focus on static scenes. AnyGrasp unifies dense spatial grasp perception with temporal tracking for parallel-gripper manipulation, achieving 93.3% average accuracy on over 300 unseen objects and over 900 mean picks per hour.

  • Problem

    Existing sampling-based methods are time-consuming, cannot generate dense predictions, and mainly address static rather than dynamic grasp detection.

  • Method

    AnyGrasp directly estimates dense 7-DoF grasp configurations and associates grasp correspondences across observations for temporally smooth tracking, using whole-scene geometry and center-of-gravity awareness.

  • Results

    93.3% average grasping accuracy was achieved across over 300 unseen objects, matching human performance, while the system reported over 900 mean picks per hour.

  • Takeaways & Limitations

    The system provides robust, efficient spatially dense and temporally smooth visual grasp perception for a two-finger parallel gripper across challenging real-world scenarios.

  • Takeaways & Limitations

    The method cannot adjust grasp poses using visual or tactile feedback and is evaluated here only with a two-finger parallel gripper.

Abstract

from arXiv · show

As the basis for prehensile manipulation, it is vital to enable robots to grasp as robustly as humans. Our innate grasping system is prompt, accurate, flexible, and continuous across spatial and temporal domains. Few existing methods cover all these properties for robot grasping. In this paper, we propose AnyGrasp for grasp perception to enable robots these abilities using a parallel gripper. Specifically, we develop a dense supervision strategy with real perception and analytic labels in the spatial-temporal domain. Additional awareness of objects' center-of-mass is incorporated into the learning process to help improve grasping stability. Utilization of grasp correspondence across observations enables dynamic grasp tracking. Our model can efficiently generate accurate, 7-DoF, dense, and temporally-smooth grasp poses and works robustly against large depth-sensing noise. Using AnyGrasp, we achieve a 93.3% success rate when clearing bins with over 300 unseen objects, which is on par with human subjects under controlled conditions. Over 900 mean-picks-per-hour is reported on a single-arm system. For dynamic grasping, we demonstrate catching swimming robot fish in the water. Our project page is at https://graspnet.net/anygrasp.html

I. INTRODUCTION

AnyGrasp targets robust, fast, spatially dense, and temporally continuous grasp perception for parallel-gripper robots in unstructured and dynamic scenes. It combines unified perception and temporal association with center-of-gravity awareness, and reports human-level bin-picking performance.

  • Motivation: Existing approaches assume object models, simplify grasping to planar detection, or sample and evaluate candidates slowly, leaving dense and dynamic prediction underexplored.These limitations restrict real-world applicability, prediction density, computation speed, or temporal grasp tracking.
  • Method: AnyGrasp is a unified system for fast, accurate, 7-DoF, temporally continuous grasp pose detection with a parallel gripper.Its geometry module predicts dense grasp configurations from monocular observations, while temporal association links poses across observations.
  • Method: The system incorporates object center-of-gravity awareness and generation-association learning for dynamic 7-DoF grasp configuration prediction.These design choices target grasp stability and temporal correspondence.
  • Results: The library reports over 900 mean picks per hour and demonstrates robustness across challenging scenarios after training on real-world data from 144 objects.The experiments include depth-sensing noise and dynamic scenes involving swimming robot fish.
  • Analysis: The paper analyzes real versus simulated data, annotation density, scene variance, and other dataset-design factors for reproducible grasping research.The authors provide the library and example code in supplementary materials.

B. Continuous Action Learning for Grasping

This section formulates parallel-jaw grasp detection as selecting successful 7-DoF poses and relates the objective to continuous grasp prediction across spatial and temporal domains.

  • Problem Definition: The section’s design objective is to support grasp detection across both spatial and temporal dimensions rather than only static pose prediction.The formulation introduces environment, perception, and grasp-pose variables at time t.
  • Problem Definition: AnyGrasp represents a parallel-jaw grasp pose with rotation, translation, and minimum suitable gripper width, covering all parallel-gripper degrees of freedom.The representation is referred to as a 7-DoF grasp configuration.
  • Problem Definition: The objective is to predict a fixed number of grasp poses that maximize the probability of successful object lifting.The formulation seeks abundant candidates so different poses are available for execution.
  • Problem Definition: Temporal grasp prediction constrains paired poses across observations to remain within a tolerance distance in the grasped target’s coordinate system.The tolerance δ determines when two grasp poses are treated as the same pose.

B. Spatial-Continuous Learning

AnyGrasp learns dense grasp quality over a scene and associates these poses across time, using global geometry, center-of-gravity cues, and collision awareness to support smooth dynamic grasping.

  • Spatial-Continuous Learning: The geometry module directly estimates grasp quality across the R^6 space from a single-view point cloud instead of sampling candidates.This spatial-continuous formulation allows feasible grasp poses to be queried at target locations in most cases.
  • Spatial-Continuous Learning: Global neighboring geometry, object center-of-gravity cues, and obstacle information provide inputs for evaluating grasp quality and stability.The network predicts normalized vertical distance from the gripper plane to the grasped object’s center of gravity.
  • Spatial-Continuous Learning: Implicit collision detection using mesh and point-cloud information removes many unreachable grasp poses and reduces computation time.This avoids relying on expensive explicit collision detection during grasp selection.
  • Temporal-Continuous Learning: Temporal-continuous learning keeps grasp poses close across frames in the object coordinate system to support smooth, target-consistent gripper movement.The relevant distance is measured in SE(3).
  • Temporal-Continuous Learning: The generation-association method produces dense poses and a many-to-many temporal association matrix, avoiding the sparse proposals of earlier sampling-based tracking.Dense poses around a selected target support both temporal continuity and grasp quality.
  • Data: Real-world training is motivated as an alternative to simulation-based grasp learning, whose inference can require expensive high-precision depth cameras.The paper emphasizes real perception and analytic labels for dense supervision.

A. Data Collection and Annotation

AnyGrasp extends grasp training data with analytic labels for grasp stability and temporal correspondence, using real scenes, object models, and multi-view observations.

  • Data collection: 168 extra scenes containing 104 new objects extend the GraspNet-1Billion training data.Objects are scanned, placed randomly, and observed from 256 viewpoints per scene.
  • Data collection: Dense grasp poses are computed from 3D object meshes using analytic antipodal scores and projected into scenes using annotated object 6D poses.Collision detection is also applied during annotation.
  • Annotation: Three extra labels support small-object grasping, stability estimation, and grasp-pose tracking.The added labels include a 0.5-centimeter approach depth, stable scores, and association labels.
  • Annotation: The stable score is the normalized perpendicular distance from the gripper plane to the object’s center of gravity under a vertical transport assumption.Lower scores indicate more balanced gravity moments and greater disturbance tolerance; the object is modeled as a uniform-density solid.
  • Annotation: Temporal labels associate grasp poses across adjacent viewpoints using a distance that combines translation and rotation differences.Grasps on different objects are assigned infinite distance, with wmax = 0.01m and γ = 0.1.

B. Grasp Perception Model Details

AnyGrasp combines dense geometric grasp prediction with a temporal association module that tracks corresponding grasp poses across observations.

  • Architecture: The model contains geometry processing and temporal association modules for dense grasp prediction and cross-frame tracking.The architecture takes a partial point cloud and produces grasp poses plus correspondence features.
  • Geometry processing: The geometry module decomposes grasp configuration into grasp point, view, in-plane rotation, approach depth, and width.A backbone extracts point features, while MLP blocks predict objectness, graspability, pose scores, and widths.
  • Geometry processing: The modified output layer adds a fifth approach depth and stable-score predictions for grasp poses at different angles.The network predicts 12 × 5 × 2 + 12 values per sampled point.
  • Geometry processing: During inference, grasp scores are multiplied by (1 − stable score), then poses are reparameterized as 7-DoF configurations and ranked.The top-n poses are retained for subsequent processing.
  • Temporal association: Each grasp pose receives a feature vector built from geometric cues and texture or color information for temporal correspondence.Cosine similarities between feature vectors form the correspondence matrix across frames.
  • Temporal association: Supervised contrastive learning pulls together features for grasp poses whose cross-frame pose distance is at most σ.The training correspondence matrix is M × M, with σ = 0.1 and temperature τ = 0.1.

C. Training Details

Training uses fixed point-cloud and feature dimensions, staged optimization, geometric augmentation, and GPU post-processing for collision and gripper stability.

  • Training configuration: Point clouds are voxel-downsampled to 0.005m, with M = 1024 grasp poses and C = 256 feature dimensions per scene.Cylinder grouping uses K = 16.
  • Training configuration: Both modules are trained on the extended GraspNet-1Billion dataset with Adam, an initial learning rate of 0.001, and poly decay power 0.9.Training uses one Nvidia GTX 2080 Ti GPU.
  • Optimization and augmentation: The geometry module is first trained from scratch with scene flips, rotations within Uniform[−30°, 30°], and translations across X/Y and Z.Its batch size is 4.
  • Optimization and augmentation: After geometry convergence, its weights are frozen while the temporal association module is trained jointly using neighboring-viewpoint point-cloud pairs.Training additionally removes scene objects with probability 0.2.
  • Post-processing: Collision detection is applied to the top-100 predicted poses because the network’s obstacle awareness is not a hard constraint.A gripper-centering process also addresses sequential fingertip contact that can push objects away.
  • Post-processing: GPU matrix computation performs the two post-processing steps for 100 poses in 80 ms.The execution phase outputs the top-100 ranked poses for a grasp planner.

V. EXPERIMENTAL SETUP

Experiments evaluate AnyGrasp on static bin picking against humans and on dynamic fish catching, using separate camera, robot, workspace, and execution procedures.

  • Static scene setup: Static experiments use a UR5 arm, overhead camera, RealSense D415 and D435 sensors, and a Robotiq-85 gripper.The fingertips use publicly available soft table-tennis rubber.
  • Static scene setup: Human comparison uses the same two-finger jaw opening width and fingertip material, with volunteers restricted to open-loop visual grasping.Tactile adjustment after contact is not allowed.
  • Dynamic scene setup: Dynamic grasping uses a Flexiv Rizon arm with a wrist-mounted RealSense L515 camera and extended 3D-printed parallel jaws.The in-hand camera avoids overhead-camera occlusion during tracking.
  • Static procedure: Each bin-picking attempt uses a single-view point cloud, abundant predicted grasps, and a workspace limit; trials end after ten pose-estimation failures.Objects scattered outside the workspace are manually returned to the plane.
  • Dynamic procedure: The dynamic procedure selects an initial grasp, tracks associated poses, stores ten poses in a temporary buffer, and predicts a future grasp pose.The robot pre-grasps 3.5 cm backward along the z-axis to avoid disturbing fish motion during servoing.
  • Dynamic procedure: Grasping begins when pose differences satisfy thresholds of Δt ≤ 5.5cm, Δt_xOy ≤ 2cm, and ΔR ≤ 20°.The robot returns to its initial state if the tracked pose leaves the workspace or camera view.

C. Evaluation Metric

The paper distinguishes attempt-centric and object-centric success rates, using the former as its primary metric while reporting both for comparison with prior work.

  • Attempt-centric success rate is the ratio of successful grasp attempts to total grasp attempts.It measures per-attempt grasping ability and is commonly used in prior literature.
  • Object-centric success rate is the ratio of successfully grasped objects to total objects.It measures adaptation across objects and is less strict because it ignores the number of attempts per object.
  • AnyGrasp uses attempt-centric success rate but reports both metrics to compare with previous work using object-centric success rate.Table I reports success rates for methods including AnyGrasp and DexNet 4.0.

VI. EXPERIMENTAL RESULTS

Experiments evaluate AnyGrasp across diverse static objects, sensors, adversarial shapes, thin fragments, and efficiency conditions, with strong grasping accuracy and throughput.

  • Evaluation scope: The experiments evaluate generalization across objects and sensors, grasp accuracy against humans, temporal consistency, and whole-system efficiency.The evaluation includes static-scene tests, sensor comparisons, adversarial objects, fragment cleaning, and dynamic tasks.
  • Efficiency: Over 900 MPPH was achieved with a single UR5 arm and Robotiq gripper, compared with 300 MPPH from a previous dual-arm system.The perception system predicts grasp poses in 100 ms, with total grasp decision time below 200 ms.
  • Sensor robustness: AnyGrasp maintained good performance with D435 depth sensing despite deviations reaching up to ±5 mm.The experiments compare grasping under depth noise from two frequently adopted cameras.
  • Adversarial objects: Accuracy decreased on the challenging adversarial object set, although humans remained stable.The authors attribute this partly to repeating failed trials without using feedback from each trial.
  • Fragments cleaning: AnyGrasp generated accurate grasp poses for thin fragments in a broken-pot cleaning task, where fragments were usually less than 3 mm thick.The task combines thin targets with noisy depth perception and requires accurate pose estimation.

B. Dynamic Scenes

In dynamic scenes, AnyGrasp tracks grasp correspondences while updating target poses during motion and catches swimming robot fish, but failures arise from water friction, prediction error, grasp quality, and correspondence switches.

  • Dynamic grasping: 75.5% average success rate was achieved when catching swimming robot fish in a fish tank.The procedure selects a target, updates the grasp pose while servoing, closes the gripper, and lifts the fish.
  • Failure analysis: Nearly half of failures occurred when fish slipped despite a good grasp pose because of low water friction.Other failures involved predicted poses falling ahead of or behind the fish, poor grasp quality, and correspondence switches.
  • Failure analysis: Correspondence switches between close, similar fish can corrupt the history buffer and produce an incorrect future grasp pose.Changing fish speed can also make historical momentum outdated, causing future-pose prediction errors.
  • Comparison: AnyGrasp achieved a higher average success rate than the nearest-grasp heuristic, 75.5% versus 62.5%.The heuristic also took 12.7% more time than AnyGrasp on successful grasps on average.
  • Efficiency: The grasp perception system runs at 7 Hz on an Nvidia 2060 GPU.This runtime supports repeated pose updates during dynamic tracking.

VII. DISCUSSION

Real-world training outperforms simulated-depth training, including noisy simulation, on benchmark and real-world grasping evaluations. Center-of-gravity-aware grasping also reduces observed in-hand slippage.

  • Training data: Real-data training retains a large advantage over noisy simulated-depth training on GraspNet-1Billion, especially for novel scenes and high-score grasps.Adding Gaussian noise improves simulated-data evaluation but does not close the gap with direct real-data training.
  • Training data: Real-world bin-picking performance decreases substantially when the model is trained on simulated data with Gaussian depth noise.The simulated-data model can also fail to generate grasp poses when only a few objects remain, which the authors attribute to domain shift.
  • Implications: The authors conclude that frequently used sim-to-real techniques are insufficient for grasping and call for improved approaches.Their analysis motivates treating real-world data as important for robust grasp perception.
  • Stability: 16 in-hand slippages occurred without the stable score, compared with 11 when the stable score was considered, across 25 attempts per condition.The counts include 3 failed grasps without the stable score and 2 with it.

C. Dense Supervision Strategy

Dense grasp-pose annotations are as important as training-image quantity in the reported analyses, while scene diversity is especially consequential. The paper also identifies practical boundaries for temporal and closed-loop grasping.

  • C. Dense Supervision Strategy: Over 10M grasp poses are generated per object in GraspNet-1Billion, far denser than the 100 to 34k poses reported for comparison datasets.The study evaluates how this annotation density affects model performance.
  • C. Dense Supervision Strategy: Downsampling grasp-pose density by 10× produces performance degradation similar to reducing training images by 10×, with larger losses on novel-object scenes.The same similarity between pose and image reductions remains when both dimensions are downsampled by 50×.
  • C. Dense Supervision Strategy: Downsampling training scenes causes larger performance degradation, and using only 2 training scenes prevents convergence.These results suggest diverse scenes are particularly important for training.
  • Temporal tracking: Grasp-pose tracking remains stable for a deformable, textureless tube under severe noise, whereas object 6D pose tracking cannot handle that scenario.The comparison motivates tracking grasp poses directly rather than relying on object pose tracking.
  • Closed-loop limitations: Closed-loop perception is fragile to gripper-induced occlusion, and tactile perception for slip detection and in-hand adjustment remains outside this work.The authors identify occlusion-robust closed-loop perception as an open question.
  • Scope: The system is limited to visual grasping with a two-finger parallel gripper and cannot adjust grasp poses using visual or tactile feedback.The paper points to transfer to different robotic hands as future research.

APPENDIX

The appendix documents a large real-world object evaluation and supplementary robot experiments. The test set excludes objects with substantial transparent or black surfaces because current depth sensing performs poorly on them.

  • A. Object Collection: Over 300 objects were collected from supermarkets, hardware stores, and toy stores for daily-scenario evaluation.The selection required each object to have a graspable region smaller than the gripper width.
  • A. Object Collection: Objects with large transparent or black surfaces were excluded because current depth sensors cannot provide good predictions on those materials.The paper notes that RGB-based methods address this issue only partially.
  • Supplementary experiments: The supplementary material contains over 12 hours of recorded real-robot experiments, with videos speeded up according to their importance.The authors state that experiment statistics are reported for the supplementary videos.
  • Supplementary experiments: Supplementary demonstrations cover AnyGrasp and DexNet4.0 on daily and adversarial objects using RealSense D415 and D435 cameras.The listed videos also include human-subject experiments on daily and adversarial objects.
  • Supplementary experiments: Additional demonstrations show fragment cleaning, robot-fish catching, and AnyGrasp trained solely in simulation and tested in the real world.These videos correspond to supplementary demonstrations S9, S10, and S11.
Loading 2212.08333v2…