Source-linked AI summary
Self-supervised 6D Object Pose Estimation for Robot Manipulation
Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl, Dieter Fox
TL;DR
Robot learning is bottlenecked by the cost of annotating real-world training data, especially for accurate self-supervised 6D pose labels. The proposed robot system collects accurately annotated views, continuously creates new object configurations through interaction, and improves segmentation, pose estimation, and grasping performance.
Problem
Training deep neural networks for robotic perception requires substantial data, while automatically producing accurate real-world 6D pose annotations remains challenging.
Method
A robot-mounted RGB-D camera collects multi-view images while PoseRBPF-based initialization and tracking use robot forward kinematics as a motion prior to generate pose annotations; the robot also grasps or pushes objects to create new scenes.
Results
Self-supervised learning consistently improves segmentation and 6D pose estimation, while significantly improving grasping success rate and grasping time.
Takeaways & Limitations
The system enables continuous collection of real-world training data and more reliable robot manipulation from 6D object pose estimation.
Abstract
from arXiv · showhide
To teach robots skills, it is crucial to obtain data with supervision. Since annotating real world data is time-consuming and expensive, enabling robots to learn in a self-supervised way is important. In this work, we introduce a robot system for self-supervised 6D object pose estimation. Starting from modules trained in simulation, our system is able to label real world images with accurate 6D object poses for self-supervised learning. In addition, the robot interacts with objects in the environment to change the object configuration by grasping or pushing objects. In this way, our system is able to continuously collect data and improve its pose estimation modules. We show that the self-supervised learning improves object segmentation and 6D pose estimation performance, and consequently enables the system to grasp objects more reliably. A video showing the experiments can be found at https://youtu.be/W1Y0Mmh1Gd8.
I. INTRODUCTION
The paper addresses the cost and difficulty of obtaining accurate real-world training annotations for 6D pose estimation. It presents an interactive robot system that self-annotates images, continuously collects new scenes, and improves perception and grasping.
- Real-world annotation is time-consuming and labor-intensive, while synthetic data may not transfer reliably because of the simulation-to-reality domain gap.
- Self-supervision requires automatically generated annotations whose accuracy can be guaranteed, a challenge especially difficult for 6D object pose.
- The system uses a hand-mounted RGB-D camera, robot motion, pose initialization and tracking to collect accurately annotated multi-view images.
- The robot grasps or pushes objects after tracking to create new configurations and continuously generate training data.
- Self-annotated data improves segmentation, pose-estimation reconstruction quality and pose accuracy, while enabling high-success-rate model-based grasping.
II. RELATED WORK
Prior 6D pose methods include feature matching, template matching, synthetic-data training and real-world dataset collection. This work extends self-supervised robot perception to directly annotate 6D object poses through interactive data collection.
- 6D pose estimation recovers an object's 3D translation and 3D rotation from images, using methods ranging from feature or template matching to deep networks.
- Synthetic-only methods can fail when real objects differ from simulation in texture or lighting, motivating real-world data collection.
- Existing real-world approaches generate pose labels by tracking camera motion around static scenes, while generally learning from fixed datasets.
- Because robots can interact with the world, planned data collection can improve self-supervised learning efficiency.
- This work applies interactive robot perception to self-supervised 6D pose estimation, beyond prior focus on segmentation, detection and pixel correspondences.
III. SELF-SUPERVISED 6D OBJECT POSE ESTIMATION SYSTEM
The system combines hand-mounted RGB-D sensing, pose initialization, tracking, robot interaction and self-supervised training. It initializes all object poses from a view selected to reduce occlusion effects before tracking and data collection.
- System Overview: A hand-mounted RGB-D camera collects images from different viewpoints as the robot arm moves around workspace objects.
- System Overview: Pose initialization estimates all object 6D poses from one RGB-D image and repeats evaluation and re-initialization until poses are satisfactory.
- Pose Initialization: The system automatically moves to a high-elevation viewpoint for new scenes, reducing the effect of inter-object occlusions during single-image initialization.
- Pose Initialization: PoseCNN assigns object classes pixelwise, uses Hough voting to locate object centers and boxes, and produces segmentation masks for subsequent processing.
2) 3D Rotation and 3D Translation Estimation:
PoseRBPF estimates object translation and rotation with particles and repeated RGB-D filtering, after which depth-derived object points support continuous SDF-based pose refinement.
- PoseRBPF represents each particle with a 3D translation and a full distribution over 3D rotation, using translation coordinates tied to image center and camera depth.
- Particle translations are sampled from detected object centers and observed depth, and each particle defines a target-object region of interest for auto-encoder likelihood evaluation.
- Repeated filtering on the same RGB-D image drives particle convergence, allowing the system to extract expected rotation and translation estimates.
- The refinement mask combines predicted segmentation, rendered pose-based segmentation and depth agreement before back-projecting object pixels into a point cloud.
- Continuous optimization aligns the object point cloud with the model's signed distance function while regularizing translation.
4) Pose Evaluation:
The pose evaluation component checks estimated object poses by comparing rendered and real RGB-D observations. It uses RGB and depth metrics to detect failures and reinitialize erroneous estimates.
- Pose evaluation compares the rendered RGB-D image with the real RGB-D image.
- The system renders an object from its estimated pose to create a synthetic RGB-D image for evaluation.
- For RGB, it compares auto-encoder codes from corresponding rendered and measured color RoIs using cosine distance.
- The system declares failure when the RGB similarity falls below s∗ or the depth error exceeds e∗, then reinitializes the object pose.
C. The Pose Tracking Module
The pose tracking module collects RGB-D video while the robot moves its camera around initialized objects. It tracks poses using propagated particles, filtering, and later refinement, while evaluating data quality before saving annotations.
- The robot moves an RGB-D camera around initialized objects while PoseRBPF tracks their poses in a continuous video sequence.
- Particles are first propagated using the camera transformation obtained from the robot’s forward kinematics.
- Rotation distributions are shifted according to pitch, yaw, and roll changes using bilinear interpolation of shifted grids.
- Joint-encoder noise and hand-eye calibration bias can make forward-kinematics propagation inaccurate, so the system filters particles using the current RGB-D image.
- Data are saved only when pose-evaluation metrics exceed thresholds, after which poses receive additional SDF-based refinement.
D. The Robot Interaction Module
The robot changes object configurations after capturing a scene so the system can collect new images with varied object poses. It uses pushing and grasping interactions to reposition objects.
- After capturing a scene, the robot interacts physically with objects to generate new configurations and continue data collection.
- Pushing moves an object in a randomly chosen direction and radius while directing it toward a workspace-safe target.
- Grasping places an object at a different location using 100 precomputed parallel-jaw grasps sampled from robust grasps.
E. The Self-Supervised Training Module
The self-supervised training module fine-tunes segmentation, detection, and pose-estimation networks using collected images and estimated poses. It bootstraps learning with single-object scenes before using cluttered scenes with varied poses and occlusions.
- Collected images and estimated object poses are used to fine-tune the system’s neural networks for improved pose estimation.
- The segmentation and detection networks and PoseRBPF auto-encoders are initially trained only with synthetic data.
- Curriculum learning first collects single-object training images, then fine-tunes segmentation for all objects and cluttered scenes.
- Table I reports semantic-segmentation F1 scores for 20 YCB objects, highlighting scores below 60%.
- The cluttered-scene stage collects examples with different object poses and occlusions for further network fine-tuning.
- During around 12 robot hours, the system collected 497 scenes, 6,541 RGB-D images, and 22,851 object instances with accurate 6D poses.
B. Evaluation Metrics
The system evaluates semantic segmentation with pixel-wise F1 and 6D pose estimation with ADD and ADD-S, while also measuring grasping performance and initialization speed.
- Semantic segmentation is evaluated using pixel-wise F1 score.
- The pose-estimation evaluation is reported on a test set.
- 6D pose accuracy is evaluated using ADD and ADD-S metrics.ADD compares corresponding transformed model points, while ADD-S uses nearest-point matching for model points.
- Grasping evaluation measures success rate and the time required for pose initialization and grasping.
D. Semantic Segmentation
Self-collected real images improve segmentation over synthetic-only training, helping the system adapt to real-object appearance differences and improve downstream pose estimation.
- Using 20 % real training data increases the overall segmentation F1 score by 25 %.More real data consistently improves performance.
- Fine-tuning improves segmentation for objects affected by textureless surfaces, reflections, or differences between real and modeled textures.These appearance differences make synthetic training difficult to transfer to real environments.
- Fine-tuned networks succeed in an example where synthetic networks fail at segmentation and pose estimation.
- Fine-tuned segmentation networks and autoencoders improve segmentation and pose estimation together.
F. Real Robot Grasping Experiments
The robot evaluates model-based grasping in clutter across six environments and 30 trials, using estimated poses to pick and place YCB objects.
- The system initializes scene objects, re-initializes detected failures, and uses estimated poses to grasp and place target objects.Objects are placed into a bin along a predefined trajectory after pickup.
- Five grasping examples are successful and one is failed in the cluttered-environment illustration.
- The experiment uses six cluttered environments containing five YCB objects each, with 30 grasping trials in total.Fourteen graspable YCB objects are selected.
- Self-annotated-data fine-tuning improves grasping success rate by 85.7 %, reduces initialization duration by 55.3 %, and reduces overall grasping duration by 27.3 %.
- The reported outcomes support the conclusion that self-supervised learning improves grasping success and grasping time.