Source-linked AI summary
EGAD! an Evolved Grasping Analysis Dataset for diversity and reproducibility in robotic manipulation
Douglas Morrison, Peter Corke, Jürgen Leitner
TL;DR
Robotic grasp detection needs diverse training data and evaluation that reflects physical grasping across varied objects. EGAD addresses this with over 2000 evolved objects and a 49-object reproducible 3D-printable evaluation set, which reveals performance variation and algorithm limitations across object difficulty.
Problem
Visual grasp detection lacks sufficiently diverse training data and standardized evaluation, while offline metrics may not represent real-world performance.
Method
EGAD uses evolutionary algorithms, CPPN-based shape generation, and search-space organization by shape complexity and grasp difficulty to create diverse objects.
Results
58% overall grasp success was achieved across EGAD’s 49 evaluation objects, decreasing from 69% for the easiest seven objects to 40% for the most difficult seven.
Takeaways & Limitations
The 49-object evaluation set enables reproducible testing across complexity and difficulty and helps identify limitations of state-of-the-art grasping algorithms.
Takeaways & Limitations
The evaluated GG-CNN system is limited to top-down 4-DoF grasps, which restricts grasp options for some objects.
Abstract
from arXiv · showhide
We present the Evolved Grasping Analysis Dataset (EGAD), comprising over 2000 generated objects aimed at training and evaluating robotic visual grasp detection algorithms. The objects in EGAD are geometrically diverse, filling a space ranging from simple to complex shapes and from easy to difficult to grasp, compared to other datasets for robotic grasping, which may be limited in size or contain only a small number of object classes. Additionally, we specify a set of 49 diverse 3D-printable evaluation objects to encourage reproducible testing of robotic grasping systems across a range of complexity and difficulty. The dataset, code and videos can be found at https://dougsm.github.io/egad/
I. INTRODUCTION
Robotic grasping requires diverse objects for training and evaluation, but existing datasets and physical test sets often lack sufficient diversity, standardization, and reproducibility. EGAD addresses these gaps with over 2000 evolved objects and 49 reproducible 3D-printable evaluation objects.
- Visual grasp detection needs diverse object datasets to train systems that generalize to unknown conditions.
- Existing evaluation commonly relies on researchers’ intuitive selections of household objects, making algorithm comparison difficult.
- EGAD contains over 2000 generated objects spanning shape complexity, grasp difficulty, and geometric diversity for training and evaluation.
- EGAD is released as mesh files and Dex-Net database files, with over 1 million precomputed grasp poses and support for vision-based training datasets.
- 49 diverse EGAD objects are specified as a reproducible, 3D-printable real-world testing suite.
II. RELATED WORK
Visual grasp detection faces two linked challenges: obtaining diverse, high-quality data and evaluating systems with standardized, reproducible methods. Existing physical and virtual datasets provide useful resources but remain limited in diversity, adoption, or comparability.
- Visual grasp detection requires diverse training data and standardized, reproducible evaluation methods.
- Physical datasets such as YCB provide scanned household objects but are limited in size and object diversity for training.
- Virtual datasets such as the Columbia Grasping Dataset and Dex-Net support machine-learning training using collections of 3D meshes.
- EGAD extends generated-object approaches toward geometrically diverse objects with a gradient of grasp difficulty and shape complexity.
- Surveyed evaluation practice often uses random household objects, making comparisons difficult because object selection depends on researcher intuition.
C. Evaluation for Robotic Grasping
Robotic grasping evaluation remains difficult because physical datasets are not widely adopted and offline metrics may not reflect real-world performance. EGAD organizes evaluation around a diverse, reproducible object space to support more informative comparisons.
- Researchers often use random household objects rather than established physical datasets, making algorithm comparisons difficult.
- EGAD’s method organizes objects across shape complexity and grasp difficulty while retaining geometrically diverse objects in each search-space cell.
- 3D-printable adversarial objects improve reproducibility, while EGAD expands this idea to a larger set spanning varied complexity and difficulty.
- Offline Cornell and Jacquard evaluations can produce false positives and false negatives, depend on closed-source simulation, and may not predict real-world performance.
III. DATASET GENERATION
EGAD is generated with evolutionary algorithms that search a discretized space defined by shape complexity and grasp difficulty. CPPN-generated meshes are assigned to cells, and MAP-Elites preserves geometrically diverse objects until the space is filled.
- Evolutionary algorithms generate objects diverse in geometry, shape complexity, and grasp difficulty in a high-dimensional design space.
- The method discretizes shape complexity and grasp difficulty into a two-dimensional search space for object placement.
- CPPNs encode objects from which 3D meshes are obtained, after which complexity and difficulty metrics assign each mesh to a search-space cell.
- MAP-Elites repeatedly samples and evolves objects, retaining the most geometrically diverse candidates while filling the search space.
1) Shape Complexity:
The paper measures shape complexity from the entropy of a probability distribution over mesh-vertex angular deficits, while estimating grasping difficulty from sampled grasp quality.
- Shape Complexity: Shape complexity uses morphological complexity, an information-theoretic measure reported to correlate with human judgments of shape complexity.
- Shape Complexity: Angular deficits are computed at mesh vertices from the internal angles of incident triangles and histogrammed over [−2π, 2π).
- Shape Complexity: Shape complexity is the entropy of the normalized angular-deficit probability distribution.
- Grasp Difficulty: The grasp-difficulty feature is the 75th percentile of robust Ferrari-Canny quality scores from sampled antipodal grasps generated with Dex-Net.
- Geometric Diversity: Geometric diversity is based on distances derived by inverting Topology Matching similarity and averaging distances to the k most similar meshes.
B. Evolutionary Algorithm
The dataset-generation process evolves CPPN-encoded 3D shapes with NEAT and uses MAP-Elites to populate a multidimensional archive while preserving diverse solutions.
- Evolutionary Algorithm: CPPNs encode 3D volumes by querying neural networks at spatial coordinates, thresholding outputs into voxel grids, and converting them into smoothed triangle meshes.
- Evolutionary Algorithm: NEAT evolves CPPNs through mutations to architectures and parameters plus crossover between CPPN architectures, increasing shape complexity and diversity over time.
- Evolutionary Algorithm: MAP-Elites is used to reduce susceptibility to local minima and insufficient exploration of the search space.
- Evolutionary Algorithm: MAP-Elites initializes random CPPNs, evaluates generated objects, assigns them to search-space cells, and iteratively evolves randomly sampled occupants.
IV. EVOLVED GRASPING ANALYSIS DATASET
EGAD is generated by evolutionary algorithms to populate a structured object search space spanning shape complexity and grasp difficulty. The dataset contains 2331 objects, covering 93% of the defined search space, with diversity increasing alongside shape complexity.
- The dataset is generated using the method and parameters described for EGAD object creation and analysis.
- EGAD contains 2331 objects uniformly distributed across a 25 × 25-cell search space, achieving 93% coverage.Some extreme combinations, such as simple-but-difficult or complex-but-easy objects, could not be filled.
- Object diversity ranges from 0.07 to 0.27, with a mean of 0.19, and generally increases with shape complexity.
- Many objects are symmetric across multiple axes because the CPPN uses distances to coordinate planes and the origin as inputs.This symmetry makes grasping difficulty less sensitive to object orientation.
B. Dataset Comparison
EGAD covers the robotic grasping object space more broadly than YCB and Dex-Net 2.0 while providing task-oriented meshes, grasp labels, and customization tools. Its formats support both grasping research and reproducible dataset construction.
- YCB fills 3% of the object search space, while Dex-Net 2.0 covers 37%.
- EGAD matches YCB’s object diversity mean of 0.19 despite containing orders of magnitude more objects.
- Dex-Net 2.0 exhibits lower geometric diversity than EGAD because its models use few semantic classes and many simple or similar shapes.
- EGAD provides robotic-manipulation meshes labelled by complexity and difficulty across both dimensions, with watertight and cohesive geometry.
- The Dex-Net-compatible format prelabels each object with up to 100 antipodal grasps and a robust grasp-quality metric.
- Released code supports custom object datasets by changing grasp sampling methods or imposing geometric constraints on generated shapes.
D. Evaluation Set
EGAD defines a 49-object, 3D-printable evaluation set spanning grasping complexity and difficulty, with standardized scaling for reproducibility. The set is used to evaluate GG-CNN and expose strengths and limitations of visual grasp detection.
- 49 EGAD objects form a 3D-printable evaluation set chosen to provide a spread of grasping difficulty and complexity.
- The evaluation objects are organized in a 7×7 grid from A0, simplest and easiest, to G6, most complex and difficult.
- Object size relative to gripper width affects graspability nonlinearly depending on geometry.
- Each object is scaled so its minimum bounding-box dimension is no more than 80% of gripper width for uniform, comparable experiments.
- The evaluation set is used to assess GG-CNN and provide a template for reporting strengths and limitations of visual grasp detection.
B. Equipment
Experiments evaluated GG-CNN on 49 diverse objects using repeated depth-based grasp attempts, revealing performance trends and object-specific strengths and limitations. The results also show how grasp depth, orientation, gripper width, object geometry, and fingertip compliance affect success.
- Experimental setup: 20 attempts were performed on each of 49 evaluation objects, totaling 980 grasp attempts, with success defined by lifting and holding an object 40cm above the table.Each attempt used a fixed-view depth image and executed GG-CNN’s best predicted grasp.
- Results: 69% success for the seven easiest objects fell to 40% for the seven most difficult, with 58% overall across all objects.The evaluation set exposed a difficulty-related performance trend while producing lower success than earlier GG-CNN reports.
- Results: Objects C1 and D1 performed much worse than neighboring objects, whereas E5, F4, F5, and G3 performed better despite their difficulty.Per-object analysis exposed outliers that a single average success rate would conceal.
- Failure modes: GG-CNN often produced grasps that were too shallow because grasp depth was computed relative to the grasp center, causing failures on several objects.The authors suggest encoding required grasp depth in the input or prediction.
- Failure modes: GG-CNN’s grasps were often perpendicular to an object’s major axis, while long-way grasps could provide more stable contact on objects such as E0.The paper attributes this orientation bias to a lack of similar training examples and suggests improved training data.
- Failure modes: Top-down 4-DoF grasping limited performance on objects such as C2, motivating 6-DoF grasps aligned with graspable object parts.The limitation arises because some objects offer few viable top-down grasp approaches.
- Grasp and gripper effects: Predicting gripper width enabled precise narrow grasps on B1, G3, and F5, while depressions in B5 and E1 caused collisions and failures.The evaluation also identified compliant silicone fingertips as supporting stable grasps on uneven, protruding surfaces such as E5 and D6.
VI. CONCLUSION
EGAD provides a large, evolved collection of diverse 3D objects for robotic grasping, alongside a printable evaluation set for reproducible testing. Applying the evaluation set to GG-CNN exposed limitations that simpler household-object sets had not revealed.
- Dataset: EGAD contains over 2000 evolved 3D objects spanning shape complexity and grasp difficulty for training and evaluating robotic grasping and manipulation.The authors characterize the dataset as filling this space uniformly relative to similar datasets limited in size and diversity.
- Evaluation set: The 49-object evaluation set is 3D-printable and supports reproducible testing across a wide range of object complexity and grasp difficulty.The set is intended to standardize physical evaluation of grasping algorithms.
- Implications: Using EGAD revealed limitations of the state-of-the-art GG-CNN algorithm that had not previously been identifiable with simpler household-object sets.The authors propose using these insights to improve baseline performance and study how diverse training data affects robustness.