Source-linked AI summary
Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation
Peter R. Florence, Lucas Manuelli, Russ Tedrake
TL;DR
Robots need an object representation that is task-agnostic, broadly applicable, 3D-informed, and learnable without human supervision. The paper introduces Dense Object Nets, self-supervised dense descriptors trained from RGBD data and extended to multi-object settings. They support consistent descriptors across configurations, class-level grasp transfer, and instance-specific manipulation for varied objects.
Problem
Existing grasping methods do not address specific objects or other tasks, while task-specific learning may require many trials to generalize across configurations and tasks.
Method
Dense Object Nets learn dense pixelwise object descriptors from self-supervised RGBD correspondences using contrastive loss, with cross-object loss enabling distinct multi-object descriptors.
Results
The approach supports consistent descriptors for varied potentially non-rigid objects, class-general descriptors, instance-specific descriptors, and robotic grasps across deformation, clutter, and object instances.
Takeaways & Limitations
Dense Object Nets provide a learned object representation that can support multiple manipulation behaviors, including specific-point grasping and transferring grasps across objects in a class.
Abstract
from arXiv · showhide
What is the right object representation for manipulation? We would like robots to visually perceive scenes and learn an understanding of the objects in them that (i) is task-agnostic and can be used as a building block for a variety of manipulation tasks, (ii) is generally applicable to both rigid and non-rigid objects, (iii) takes advantage of the strong priors provided by 3D vision, and (iv) is entirely learned from self-supervision. This is hard to achieve with previous methods: much recent work in grasping does not extend to grasping specific objects or other tasks, whereas task-specific learning may require many trials to generalize well across object configurations or other tasks. In this paper we present Dense Object Nets, which build on recent developments in self-supervised dense descriptor learning, as a consistent object representation for visual understanding and manipulation. We demonstrate they can be trained quickly (approximately 20 minutes) for a wide variety of previously unseen and potentially non-rigid objects. We additionally present novel contributions to enable multi-object descriptor learning, and show that by modifying our training procedure, we can either acquire descriptors which generalize across classes of objects, or descriptors that are distinct for each object instance. Finally, we demonstrate the novel application of learned dense descriptors to robotic manipulation. We demonstrate grasping of specific points on an object across potentially deformed object configurations, and demonstrate using class general descriptors to transfer specific grasps across objects in a class.
1 Introduction
The paper proposes Dense Object Nets as a self-supervised, dense visual representation for robotic manipulation. It targets object-specific and class-level manipulation across viewpoints, configurations, and potentially non-rigid objects.
- The paper introduces dense visual descriptions as a representation for robotic manipulation.Dense Object Nets are deep neural networks that provide dense, pixelwise object descriptions.
- The system learns consistent object representations entirely through robot self-supervision, without human supervision during training.
- Dense descriptors support grasping specific locations across deformed configurations, instance-specific grasping in clutter, and grasp transfer across objects in a class.
- Self-supervised descriptors can be learned for 47 objects across 3 classes in approximately 20 minutes.
- Modifying the loss function and sampling procedure yields either descriptors that generalize across object classes or descriptors distinct for each instance.
2 Related Work
The related work spans learned visual descriptors, self-supervised visual learning for robots, and task-specific robot learning. Dense Object Nets instead provide a task-agnostic, dense representation based on 3D reconstruction and pixelwise correspondences.
- Prior dense descriptor methods learn visual correspondences, but the cited prior work does not involve robots or uniformly avoid human annotations.
- The paper asks for the current visual state of the robot’s world rather than how the world changes after robot actions.
- Unlike coarse semantic segmentation based on image-level background subtraction, this work uses 3D reconstruction-based change detection and dense pixelwise correspondences.
- Unlike end-to-end reinforcement learning and grasping methods aimed at specific tasks or grasping any item, the paper learns a general task-agnostic visual representation.
3 Methodology
The method learns dense pixelwise object descriptors from RGBD videos using 3D-based correspondences and contrastive loss. Object masking, augmentation, hard-negative scaling, and multi-object training support consistent, distinct representations across configurations, classes, and clutter.
- Self-Supervised Pixelwise Contrastive Loss: Dense Object Nets map each RGB image to a dense descriptor image, assigning every pixel a D-dimensional descriptor.Training uses Siamese image pairs, with descriptors pulled together for matching pixels and separated by margin M for non-matches.
- Self-Supervised Pixelwise Contrastive Loss: 3D reconstruction identifies matching pixels as projections of the same reconstructed vertex, providing dense self-supervision without human labels.The training pair is sampled from one RGBD video, and correspondences are generated through the reconstruction.
- Object-Centric Descriptors: Object-centric training combines 3D change-detection masks, background randomization, data augmentation, and loss balancing to improve consistency across challenging configurations.Background randomization discourages descriptors from encoding object location relative to scene elements.
- Object-Centric Descriptors: Hard-negative scaling reweights non-match loss by the number of violating negatives after the hard-negative rate falls below 1%.The procedure uses N_hard-negatives rather than all sampled non-matches for adaptive scaling.
- Multi-Object Dense Descriptors: Multi-object descriptors use cross-object loss, direct training on cluttered scenes, or synthetic layered scenes with occluded matches pruned.Synthetic layering creates combinatorial multi-object scenes and varied occlusion types from single-object scenes.
4 Experimental
The experimental system collects RGBD videos with a robot-mounted RGBD camera and trains descriptors from several image-comparison types. Static-scene reconstructions provide pixel correspondences through raycasting and reprojection.
- Data Collection and Pre-Processing: Data collection uses a 7-DOF Kuka IIWA LBR robot arm carrying a Primesense Carmine 1.09 RGBD sensor.Forward kinematics and camera calibration provide reliable camera poses during automated collection.
- Training Dense Descriptors: Training samples single-object, cross-scene, multi-object, and synthetic multi-object image comparisons, each with matches and non-matches.The procedure randomly selects comparison types and then samples correspondence pairs for each.
5 Results
Dense Object Nets produce descriptors consistent across viewpoints, configurations, and substantial deformation, while training choices support either multi-object separation or class-level generalization. These representations enable correspondence-based manipulation, including grasping specified points on objects and transferring grasps across instances.
- 5.1 Single-Object Dense Descriptors: Dense descriptors remain consistent across viewpoints and configurations for moderately deformable and low-texture objects.The object set includes plush toys, shoes, mugs, and hats, and uses a common network architecture and training parameters.
- 5.1 Single-Object Dense Descriptors: Object masking and hard-negative scaling improve correspondence precision, while randomization prevents descriptor inconsistency caused by memorizing object orientation or background.Orientation randomization and background domain randomization are especially important for some low-texture objects and smaller datasets.
- 5.2 Multi-Object Dense Descriptors: Cross-object loss separates multiple objects into distinct descriptor-space subsets, but larger descriptor dimensions may be needed to recover correspondence precision.Three-dimensional descriptors often suffice for single objects, whereas distinct multi-object networks frequently require more dimensions.
- 5.3 Selective Class Generalization or Instance Specificity: Training on similar instances with consistent descriptors generalizes across classes despite differences in color, texture, deformation, and underlying shape.Class-consistent descriptors were demonstrated for hats, shoes, and mugs, including unseen hats and sufficiently similar shoes.
- 5.4 Example Applications to Robotic Manipulation: Grasping Specific Points: Specific training can preserve instance identity even for similar objects, enabling grasping a designated point on one object in clutter.The paper contrasts this instance-specific behavior with class-general descriptors that transfer corresponding grasps across instances.
- 5.4 Example Applications to Robotic Manipulation: Grasping Specific Points: The system identifies corresponding pixels in new scenes and uses point-cloud lookup with geometric grasping to execute the selected grasp.A reference-image pixel specifies the desired point, and the robot searches for a valid descriptor match before grasping.
6 Conclusion
Dense Object Nets provide self-supervised visual object representations for manipulation, extending to diverse objects, multiple objects, and class-general descriptors.
- Dense Object Nets provide visual object representations that are useful for robotic manipulation and require only robot self-supervision.
- 47 objects across 3 classes demonstrate automated and reliable descriptor learning at scale for varied objects.
- Contrastive techniques train multi-object networks that map different objects to different descriptor-space regions while retaining non-distinguishing network performance.
- The approach also learns class-general descriptors that generalize across different object instances.
Appendix A Experimental Hardware
The experimental system collected data with an RGBD camera mounted on the end effector of a 7-DOF robot arm and used a parallel-jaw gripper.
- All data was collected using an RGBD camera mounted on the end effector of a 7 DOF robot arm.
- The robot was a Kuka IIWA LBR equipped with a Schunk WSG 50 parallel jaw gripper.
- A Primesense Carmine 1.09 RGBD sensor was precisely calibrated for both intrinsics and extrinsics.
Appendix B Experimental Setup: Data Collection and Pre-Processing
Data collection reconstructs static scenes from robot-mounted RGBD scans, then varies object configurations and preprocesses frames by camera-pose separation.
- A scanning pattern captures approximately 2100 RGBD frames over about 70 seconds for each static scene.Forward kinematics are used during the scanning procedure, according to the passage.
- Images are downsampled until consecutive camera poses differ by at least 5cm of translation or 10 degrees of rotation.
- Downsampling leaves approximately 315 images per scene.
- Between RGBD videos, objects are moved through varied configurations and lighting may also be changed.The robot can autonomously rearrange objects, although human intervention was used for many collections.
- For multiple objects, a human currently switches objects and indicates which scenes correspond to each object.The passage notes that this identification could potentially be automated using object continuity.
Appendix C Grasping Pipeline
The grasping pipeline uses geometric point-cloud methods to choose grasps when descriptors are unavailable or insufficient for full gripper orientation.
- Visual descriptors cannot guide grasps during bootstrapping before visual learning has occurred.
- Descriptors help determine where to grasp but do not constrain the gripper’s 6DOF orientation.
- The pipeline therefore employs simple geometric point-cloud techniques for grasp selection.
Appendix D Network Architecture and Training Details
The network uses a pretrained stride-8 ResNet with bilinear upsampling, while training samples multiple image-comparison types and match sets. Training takes approximately 13 minutes, with a new object requiring about 20 minutes including data collection.
- Network Architecture: A 34-layer, stride-8 ResNet pretrained on ImageNet is bilinearly upsampled to produce full-resolution 640x480 images.
- Training Procedure: Training randomly selects among four image-comparison types according to specified probabilities, then samples matches and non-matches for each type.The comparison types are Single Object Within Scene, Different Object Across Scene, Multi Object Within Scene, and Synthetic Multi Object.
- Training Procedure: 3500 Adam optimization steps take approximately 13 minutes, and training a new object takes about 20 minutes including collection of a handful of scenes.The optimizer uses weight decay 1e−4 and learning rate 1e−4, reduced by 0.9 every 250 iterations.