Grasp2Vec: Learning Object Representations from Self-Supervised Grasping

1 Introduction

• Structured representations may disentangle persistent objects and their identities, while self-supervision lets robots use experience collected autonomously.The introduction contrasts this goal with end-to-end perception tailored to individual control tasks.Source: §1 · pp. 1–2
• Grasping supplies paired observations: the robot removes an object, then observes both the changed scene and the object in its gripper.Figure 1 depicts scene-embedding differences matching object embeddings, with object-embedding similarity then used as a grasp reward.Source: §1 · p. 1, Figure 1 · p. 2
• The method uses object persistence: removing an object should change the scene embedding by the feature value of that object.It trains a convolutional feature extractor to capture object identity and represent sets of objects without human supervision.Source: §1 · p. 2
• Embedding similarity provides a grasping reward signal that is reported to be as good as manually labeling whether the correct object was grasped.The policy is conditioned on an object feature vector or image.Source: §1 · p. 2
• Grasp2Vec outperforms alternative unsupervised methods in a simulated goal-conditioned grasping benchmark and supports object localization, instance detection, and user-commanded grasping.The same grasping experience can be relabeled with representation-based goals to train a policy.Source: §1 · p. 2

2 Related Work

• Prior interactive-learning work uses egomotion or poking [6] [7] [8] [9], while Grasp2Vec preserves object identity and scene object-set structure in one feature vector.The paper also distinguishes its arithmetic consistency objective and policy-learning application from action-recognition work.Source: §2 · p. 2
• Prior grasping systems detect successful grasps without human supervision [14] [4] [16], but target-object grasping has used class labels [17] or human labels and simulation transfer [18] [19].The real and simulated setups are illustrated in Figure 2.Source: §2 · p. 3, Figure 2 · p. 3
• Grasp2Vec avoids labeling whether the desired goal was achieved, a challenge that prior unintentional-reward methods [20] [21] do not address outside simulation.The authors describe their instance-grasping approach as completely label-free.Source: §2 · p. 3
• Compared with concurrent depth-image descriptor work [22], Grasp2Vec reports high accuracy for identifying, localizing, and grasping objects across viewpoint changes, heavy occlusion, and deformation.The comparison is stated by the authors in relation to their quantitative results.Source: §2 · p. 3

3 Grasp2Vec: Representation Learning from Grasping

• Each grasping episode records a pre-grasp scene, post-grasp scene, and image of the grasped object; their embeddings are trained so the scene-feature difference matches the object feature.The difference is also required to be far from embeddings of objects that were not grasped.Source: §3 · p. 3
• The scene and outcome encoders use ResNet features and global average pooling to produce single vectors from spatial feature maps.The figure describes tied scene-encoder weights for pre- and post-grasp images and applies an n-pairs loss.Source: §3 · p. 3, Figure 3 · p. 4
• The ReLU-constrained spatial features are non-negative, so adding objects can grow a scene representation without one object being the inverse of another.Spatial feature maps can encode object position through the receptive fields that produce each feature.Source: §3 · p. 4
• The n-pairs objective pulls each scene-difference/object pair together while pushing unpaired minibatch examples apart.Other positives in the minibatch act as negatives for each anchor-positive pair.Source: §3 · p. 4
• Because the loss is asymmetric for anchors and positives, the method evaluates the embeddings in both orders.The experiments use λ = 0.0005 and batch size B = 16.Source: §3 · p. 4

4 Self-Supervised Goal-Conditioned Grasping

• The policy receives the current image and goal as state, uses Cartesian gripper motions, and earns a terminal reward of 1 only for grasping the specified object.All other time steps receive reward 0.Source: §4 · p. 4
• Training must handle early episodes that rarely grasp the target and infer rewards without object labels, assuming the system can detect a grasp but not identify the object.A successful grasp of any object supplies the indiscriminate reward used to form Q-learning tuples.Source: §4 · pp. 4–5
• Embedding similarity labels a successful grasp with the similarity between the commanded goal and the grasped-object embedding.The reward is the dot product of their normalized embeddings.Source: §4 · pp. 5–6
• Posthoc labeling adds a successful episode as a positive example for the object actually grasped, even when it was not the commanded goal.This supplies a learning signal when the policy has not yet grasped the desired object.Source: §4 · p. 6
• Auxiliary goal augmentation relabels a successful transition with a second sampled goal and rewards it by that goal’s embedding similarity to the grasped object.For baselines without embeddings, the relabeled reward is set to 0.Source: §4 · p. 6

5.1 Grasp2Vec embedding analysis.

• Scene-feature differences retrieve the grasped object, with high retrieval accuracy on training simulated objects and all real objects.The evaluation counts retrieval as correct when the nearest-neighbor outcome image contains the grasped object; real-image-trained embeddings generalized better than simulated-image-trained embeddings.Source: Table 1 · p. 6, §5.1 · p. 7
• Grasp2vec localizes unseen objects at almost 80% accuracy without position labels, versus 15% for the ImageNet baseline.Localization uses the object embedding against each pixel's spatial scene feature; a prediction counts only when the maximum activation falls on the correct object.Source: Table 1 · p. 6, §5.1 · p. 7

5.2 Simulated instance grasping

• All grasp2vec-similarity reward models achieve at least 78% instance-grasping success on seen simulated objects.Table 2 averages 700 trials and evaluates test objects at the same checkpoint used for training-object results.Source: §5.2 · p. 7, Table 2 · p. 8
• Grasp2vec similarity performs on par with oracle labels on seen objects and outperforms the oracle on new objects, without object-identity labels.Posthoc labeling exceeds indiscriminate grasping but remains below the true-label upper bound; adding auxiliary goal supervision worsens performance, while an autoencoder reward performs no better than indiscriminate grasping.Source: §5.2 · p. 7
• Policies trained on single goals grasp one of two averaged goal embeddings with 51.9% seen-object and 42.9% unseen-object success.The authors attribute generalization to composite goals to grasp2vec's additive semantics.Source: §5.2 · pp. 7–8

5.3 Real-world instance grasping.

• Without additional on-policy training, real-world localization-guided grasping succeeds on 80.8% of training objects and 62.9% of test objects.The method localizes the goal, executes an indiscriminate grasp, and drops and re-localizes if wrist-camera similarity falls below 0.7.Source: §5.3 · p. 8

6 Discussion

• The discussion presents grasp2vec as a scene-as-object-sets representation where adding or removing objects corresponds to arithmetic operations in embedding space.The authors identify semantic information, spatial object-centric relational reasoning, and planning compound skills as future directions.Source: §6 · p. 8

• Figure 6 illustrates object recall by showing the pregrasp scene, grasped object, embedding-difference outcome, and postgrasp scene, with successes and failures marked.Source: Figure 6 · p. 11
• Figures 7–9 show localization of a deformable bag in clutter, recognition from a different pose, and recognition by parts when the goal is occluded by the gripper.Source: A Qualitative Detection Results · p. 11, Figure 7 · p. 11, Figure 8 · p. 12
• Figure 10 shows training objects localized with the correct colors but a similar, incorrect shape.Source: Figure 10 · p. 13
• Figure 11 shows a failed grasp incorrectly labeled successful, producing an empty and meaningless localization goal.Source: Figure 11 · p. 13
• Figure 12 reports that an untrained model lacks the detection property, while ImageNet features fail localization, likely because scene and outcome images differ in resolution and pose.Source: Figure 12 · p. 13

• The Bullet simulation used a 7-DoF Kuka arm, with 44 unique training objects and 15 mutually exclusive evaluation objects, all scanned mugs, bottles, cups, or bowls.Source: B Simulation Experiment Details · p. 14
• Scenes contained up to six distinct objects sampled without replacement and dropped at random poses, where objects could bounce against one another.Source: B Simulation Experiment Details · p. 14
• Each episode first collected ten objects with indiscriminate grasping, then switched to on-policy instance grasping using those images as goals; data collection ran across 1,000 simulated robots per experiment.Source: B Simulation Experiment Details · p. 14

• The real-world setup used roughly 500 training objects and 42 unseen evaluation objects, without restricting object categories.Source: C Real-World Experiment Details · p. 14
• Six objects were placed randomly in each bin; after a grasp, the object was returned, bins were refreshed about twice daily, and six to seven robots ran in parallel.Source: C Real-World Experiment Details · p. 14
• Figure 13 depicts the objects used to evaluate performance on unseen test objects.Source: Figure 13 · p. 15