Source-linked AI summary
Jacquard: A Large Scale Dataset for Robotic Grasp Detection
Amaury Depierre, Emmanuel Dellandréa, Liming Chen
TL;DR
Robotic grasp prediction needs large amounts of labeled data, but creating such data is difficult and costly. The paper addresses this gap with Jacquard, a large-scale synthetic RGB-D dataset whose grasp labels come from simulated trials. Across evaluation metrics including real robot trials, Jacquard-trained predictors generalize better than predictors trained on a small human-labeled dataset.
Problem
Robotic grasping networks require huge amounts of labeled data, while producing accurate labels is difficult and often impracticable in robotics.
Method
The paper generates Jacquard by simulating physical grasping environments from CAD models, testing grasp candidates in physics simulation, and recording successful positions in RGB-D scenes.
Results
Jacquard-trained grasp predictors achieve better generalization on unseen objects than the same network trained on human-labeled data, including evaluation with simulated and real grasp trials.
Takeaways & Limitations
A synthetic dataset with diverse objects and grasp positions can train a deep neural network to predict grasp locations from scene images.
Abstract
from arXiv · showhide
Grasping skill is a major ability that a wide number of real-life applications require for robotisation. State-of-the-art robotic grasping methods perform prediction of object grasp locations based on deep neural networks. However, such networks require huge amount of labeled data for training making this approach often impracticable in robotics. In this paper, we propose a method to generate a large scale synthetic dataset with ground truth, which we refer to as the Jacquard grasping dataset. Jacquard is built on a subset of ShapeNet, a large CAD models dataset, and contains both RGB-D images and annotations of successful grasping positions based on grasp attempts performed in a simulated environment. We carried out experiments using an off-the-shelf CNN, with three different evaluation metrics, including real grasping robot trials. The results show that Jacquard enables much better generalization skills than a human labeled dataset thanks to its diversity of objects and grasping positions. For the purpose of reproducible research in robotics, we are releasing along with the Jacquard dataset a web interface for researchers to evaluate the successfulness of their grasping position detections using our dataset.
I. INTRODUCTION
Robotic grasp prediction remains difficult because precise image-based localization requires labeled data that is costly to obtain at scale. The paper introduces Jacquard, a simulated dataset designed to improve grasp detection and generalization, especially on unseen objects.
- Robotic grasp planning requires precise image-based localization, because even a one-pixel prediction error can determine grasp success or failure.
- Existing deep neural grasp predictors depend on supervised labeled data obtained through human labeling, robot trials, analytic computation, or physics simulation.
- Millions of labels can be generated analytically or through simulation, but these approaches generally require matching CAD models to objects in images.
- Jacquard automates labeled-image generation by simulating an environment close to a physical setup and introduces simulated grasp trial evaluation.
- Jacquard contains diverse objects with multiple labeled grasps and yields better prediction of grasp locations on unseen objects than training the same DNN on human-labeled data.
II. RELATED WORK
Robotic grasp prediction has progressed from CAD-based planning toward image-based deep learning, but existing datasets vary in scale, accessibility, and prediction scope. Jacquard addresses these limitations with a larger RGB-D dataset containing diverse objects and grasp annotations.
- CAD-based grasp planning: Early grasp prediction used known 3D object models and matched query objects against databases such as the Columbia Grasp Database.The Columbia database contained more than 230k grasps, but this approach did not use images.
- Image-based prediction: Deep learning and inexpensive depth sensors shifted grasp prediction toward RGB-D images and datasets collected through physical trials.
- Synthetic datasets: DexNet-2.0 provided 6.7 millions depth images and achieved a 93% grasp-outcome prediction success rate, but its GQ-CNN required externally generated grasp candidates.
- Synthetic datasets: A related simulated approach trained predictions over predefined gripper-position grids using 1000 objects, but its data were not publicly released.
- Jacquard dataset: Jacquard contains more than 11k objects with RGB and realistic stereo-vision depth information, expanding beyond the scale of earlier publicly available alternatives.
- Human-labeled datasets: The Cornell dataset contained 885 RGB-D images of 240 objects and 8019 hand-labeled grasp rectangles, making it small relative to typical deep-learning datasets.Its limited scale may impair generalization across images or object configurations.
III. MODELLING ROBOTIC GRASP
The paper models a successful parallel-plate grasp as a five-dimensional rectangle representation in an RGB-D image. Depth supplies the scene information needed for the approach and plate height, while gripper geometry constrains the rectangle dimensions.
- Grasp definition: A good grasp is defined as one that lets a parallel-plate gripper lift the object and move it away from the table.
- Grasp representation: The grasp representation uses rectangle center (x, y), size (h, w), and orientation θ relative to the image’s horizontal axis.
- Image-coordinate formulation: Representing grasps in image coordinates avoids requiring explicit physical-scene information, because depth determines plate z position and the approach vector.
- Gripper constraints: For a real robot, h and w are fixed and bounded by the parallel gripper’s shape.
IV. JACQUARD DATASET
Jacquard generates RGB-D images and successful grasp labels from CAD models through simulation. Applying the pipeline to ShapeNetSem produces a large dataset spanning thousands of objects, varied scenes, and many annotated grasp positions.
- Dataset construction: The dataset-generation method converts CAD models into simulated images and ground-truth grasp labels to address data starvation.
- Dataset scale: Applying the process to ShapeNetSem produced more than 50k images of 11k objects and 1 million unique successful grasp positions.
- Scene generation: Each scene uses a randomly rotated and translated white plane texture to vary the background.
- Scene generation: Objects are rescaled to longest-side lengths between 8 and 90 cm and assigned masses based on their sizes before being dropped.
- Simulation pipeline: Rendering and physics simulation operate as independent modules, with up to five scenes created for each object to provide different views.
B. Image rendering
The dataset-generation pipeline renders synthetic RGB-D images and produces grasp annotations by testing candidate grasps in physics simulation. Candidate sampling is biased toward promising image regions to reduce attempts while preserving diverse grasp locations.
- RGB and true depth images are rendered with Blender, while projected-pattern stereo processing produces noisy depth and an object-background mask.
- Grasp annotations are generated by sampling random candidates, testing them with a 2 cm jaw, and retesting successful candidates across gripper sizes.Each successful grasp location is associated with between one and five jaw sizes.
- Successful simulated grasps are lifted, moved, and dropped before nearby duplicate positions are removed.
- A non-uniform probability distribution samples candidates more frequently near aligned image edges, avoiding many empty-area grasps.
- The sampling strategy reduces annotation attempts by orders of magnitude while retaining grasp-location diversity.The passage links this diversity to deep-learning-oriented methods.
D. Assessment criterion of successful grasp predictions
The paper contrasts rectangle-based evaluation with SGT, a simulation-based criterion that tests whether a predicted grasp successfully lifts and moves the object. SGT addresses cases where geometric agreement with one annotation does not reflect physical grasp success.
- Rectangle metrics can produce visually false positives and false negatives relative to human judgment.Figure 4 illustrates both types of misclassification.
- SGT rebuilds the scene and performs the predicted grasp in simulation under the same conditions used to generate annotations.
- Under SGT, a prediction is good when the simulated robot successfully lifts and moves the object away.
- SGT is closer to real-world grasp evaluation because an object may have multiple successful grasp locations, including unannotated ones.
V. EXPERIMENTS AND RESULTS
The experiments evaluate Jacquard through cross-dataset grasp prediction and real-robot trials. The supplied passage identifies these as the paper’s two experimental series but does not report their outcomes.
- The experiments comprise cross-dataset grasp prediction using Cornell and Jacquard and evaluation of grasp predictions with a real grasping robot.
A. Training setup
Training uses an off-the-shelf AlexNet adapted for RGB-D inputs, with convolutional weights pretrained on ImageNet and fully connected layers trained from scratch. Prediction error is measured against the closest annotation.
- AlexNet uses ImageNet-pretrained convolutional weights and fully connected layers trained from scratch.
- RGB-D input is supported by normalizing depth values and duplicating blue filters in the first pretrained convolutional layers.
- The learning rate changes to 0.00005 after the first 75k iterations.
- Network error is the Euclidean distance between the prediction and the closest annotation.G denotes all image annotations, and ĝ denotes the network prediction.
- Data augmentation translates, rotates, and mirrors images, while synthetic images receive varied backgrounds such as cardboard, paper, wood, and grass.
B. Cross-dataset evaluation
Cross-dataset evaluation shows that Jacquard-trained CNNs transfer effectively to Cornell, while Cornell-trained CNNs generalize poorly to Jacquard’s broader object and grasp diversity.
- The experiments use Cornell’s 885 RGB-D images and Jacquard’s 15k RGB-D images spanning 3k selected objects.
- 81.92% accuracy on Cornell was achieved by Alexnet trained on Jacquard, close to the 86.88% baseline.
- 54.28% accuracy was achieved by Alexnet trained on Cornell and tested on Jacquard, a 20-point decrease from its baseline.
- 42.76% SGT accuracy for Cornell-trained Alexnet trailed the Jacquard-trained network’s 72.42% accuracy by 30 points.
- Jacquard’s diversity of objects and grasp locations supports better generalization by trained CNNs.
C. Evaluation of grasp predictions using a real grasping robot
Real-robot trials compare grasp predictions from Alexnet models trained on Cornell and Jacquard using varied everyday and industrial objects. The Jacquard-trained model achieves a higher successful-grasp rate.
- 78.43% of Jacquard-trained grasp predictions succeeded on the real robot.The test used 15 everyday objects and 13 industrial components.
- The Jacquard-trained model’s real-robot success rate was 6 points higher than its 72.42% SGT accuracy on Jacquard.
- Figure 6 contrasts Cornell-trained predictions in the top row with Jacquard-trained predictions in the bottom row.
- Most failed real grasps were unstable: objects were lifted successfully but dropped during robot movement.
- 60.46% of Cornell-trained grasp predictions succeeded, mostly because of bad rectangle localization.
VI. CONCLUSIONS
The paper presents Jacquard as a large-scale simulated RGB-D grasp dataset with localized annotations, and reports improved generalization over a small hand-labeled dataset.
- Jacquard generates realistic RGB-D data with localized grasp annotations from simulation.
- The dataset successfully trains a deep neural network to predict grasp positions in images.
- Jacquard-trained grasp predictors generalize much better than the same network trained with a small hand-labeled grasp dataset.
- Future work targets grasp-prediction quality assessment and more complex scenes with multiple objects.