Source-linked AI summary
SegICP: Integrated Deep Semantic Segmentation and Pose Estimation
Jay M. Wong, Vincent Kee, Tiffany Le, Syler Wagner, Gian-Luca Mariottini, Abraham Schneider, Lei Hamilton, Rahul Chipalkatty, Mitchell Hebert, David M. S. Johnson, Jimmy Wu, Bolei Zhou, Antonio Torralba
TL;DR
Robotic manipulation requires faster and more reliable perception of task-relevant objects in cluttered environments. SegICP integrates CNN-based semantic segmentation with multi-hypothesis point-cloud registration for real-time 6-DOF pose estimation. On a motion-capture-annotated benchmark, it reports 1 cm position error and < 5◦ angle error, with performance varying by segmentation model and sensor calibration.
Problem
Existing object identification and pose-estimation systems can be non-robust in clutter, slower than real time, insufficiently accurate, or unable to handle high occlusion.
Method
SegICP combines RGB-D semantic segmentation, multi-hypothesis point-cloud registration, and a parallelized architecture using model-to-scene alignment scoring.
Results
1 cm position error and < 5◦ angle error are achieved on successful scenes, while SegNet reaches 77% benchmark performance versus 66% for DilatedNet.
Takeaways & Limitations
SegICP provides real-time object segmentation and pose estimation without an initial pose seed, with performance influenced by sensor technology and calibration.
Abstract
from arXiv · showhide
Recent robotic manipulation competitions have highlighted that sophisticated robots still struggle to achieve fast and reliable perception of task-relevant objects in complex, realistic scenarios. To improve these systems' perceptive speed and robustness, we present SegICP, a novel integrated solution to object recognition and pose estimation. SegICP couples convolutional neural networks and multi-hypothesis point cloud registration to achieve both robust pixel-wise semantic segmentation as well as accurate and real-time 6-DOF pose estimation for relevant objects. Our architecture achieves 1cm position error and <5^\circ$ angle error in real time without an initial seed. We evaluate and benchmark SegICP against an annotated dataset generated by motion capture.
I. INTRODUCTION
SegICP addresses perception systems that struggle with clutter, speed, and accuracy by integrating semantic segmentation with real-time 6-DOF pose estimation. Its parallelized pipeline combines multi-hypothesis registration, automated motion-capture annotation, and benchmarking.
- The approach targets cluttered, partially occluded, slow, and inaccurate object identification and pose-estimation systems.These limitations motivate the integrated perception pipeline.
- SegICP integrates deep semantic segmentation with model-based pose estimation for relevant objects in RGB-D scenes.The architecture is highly parallelized and provides semantic labels with corresponding poses.
- The pipeline estimates each object’s 6-DOF pose with 1 cm position and 5◦ angle error.The overview figure presents RGB and depth inputs alongside segmentation and pose outputs.
- SegICP introduces a registration-quality metric for autonomous pose initialization across multiple hypotheses.The paper also includes an automatic motion-capture data-collection framework and benchmarking procedure.
II. RELATED WORK
Prior work separates pixel-wise object recognition from 6-DOF pose estimation, while ICP-based methods can be slow, initialization-sensitive, and vulnerable to occlusion. SegICP advances this line by combining segmentation, multi-hypothesis registration, and parallel processing.
- Semantic segmentation preserves each object’s pixel location, unlike recognition systems that output only bounding boxes.Pixel-wise labels retain spatial information useful for robotic applications.
- CNNs enable dense pixel-wise semantic segmentation, extending image-classification feature learning to per-pixel predictions.Earlier approaches used hand-engineered features and smoothing methods such as CRFs.
- Pose estimation complements 2D recognition by estimating each identified object’s full 6-DOF pose.Prior approaches include template matching, RGB-D features, and parts-based models, with limitations for ambiguous or low-feature geometries.
- SegICP advances prior segmented point-cloud fitting with a registration metric, multi-hypothesis initialization, and simultaneous processing of objects.The cited prior system required about 1 s per object and reported 5 cm and approximately 15◦ errors.
- ICP registration commonly requires initialization near the global optimum and is not robust to partial or full occlusions.These limitations motivate SegICP’s multi-hypothesis registration strategy.
III. TECHNICAL APPROACH
SegICP receives RGB-D data and produces semantic labels plus relative object poses through an object-library and point-cloud-matching architecture. It tracks poses in real time while maintaining robustness during occlusions and segmentation outliers.
- SegICP matches cropped scene point clouds against scanned 3D models from an object library to estimate detected-object poses.The system uses RGB-D input and associates each semantic label with a pose relative to the sensor.
- ∼70 ms per frame and 270 ms during initialization are reported with 1 cm position error and < 5◦ angle error.A Kalman filter supports tracking through prolonged occlusions and potential segmentation outliers.
A. Semantic Segmentation by Neural Networks
The segmentation stage generates depth masks tailored to pose estimation rather than treating segmentation as an isolated recognition task. SegICP compares SegNet and DilatedNet architectures for this purpose.
- SegICP experiments with SegNet and DilatedNet to generate semantic masks over depth maps for accurate pose estimation.The segmentation architectures are evaluated specifically for their influence on the downstream pose pipeline.
B. Multi-Hypothesis Object Pose Estimation
SegICP estimates object poses by registering segmented scene point clouds against multiple candidate model crops, then tracking the best alignment over time.
- The segmented object cloud is matched against a corresponding downsampled mesh-model point cloud for pose estimation.The semantic label selects the corresponding model from the object library.
- Acquisition Phase: Acquisition evaluates multiple rendered model crops, initialized at the scene-cloud median, while tracking refines each pose hypothesis.This design reduces sensitivity to segmentation outliers and poor ICP local minima.
- Acquisition Phase: The alignment metric scores the fraction of candidate-crop points with unique matches in the segmented scene cloud within τ meters.Matching uses radius searches and prevents multiple candidate points from sharing a scene-point correspondence.
- Acquisition Phase: The proposed metric distinguishes registrations that ICP fitness and IoU fail to separate, particularly for highly symmetric objects.Figure 3 visualizes candidate model crops and their alignment scores.
- Tracking Phase: The highest-scoring candidate initializes tracking, which prunes the scene cloud and fuses registration measurements with odometry in a Kalman filter.Tracking maintains each object’s 6-DOF pose and velocities during camera motion and potential occlusions.
C. Automatically Annotating Training Data
The authors create training annotations with a motion-capture system that marks cameras and objects, producing segmentation and pose labels from RGB-D inputs.
- Active markers mounted on RGB-D cameras and objects enable motion-capture-based pose measurement for automatic annotation.The setup includes markers on the PR2’s Kinect1 and Kinect2.
- The automatic labeling system converts RGB and encoded depth images into object segmentations and axis-angle poses.Figure 5 shows the inputs in the top row and generated labels in the bottom row.
- SegNet and DilatedNet outputs illustrate alternative segmentation results for the same Kinect1 RGB image, with SegNet appearing tighter.The comparison uses identical input imagery.
IV. EVALUATION
SegICP is benchmarked on a motion-capture dataset containing 1,246 annotated object poses.
- 1,246 annotated object poses obtained with the motion-capture system form the benchmarking dataset.
A. Semantic Segmentation Results
The evaluation compares SegNet and DilatedNet segmentation and examines how their mask quality affects SegICP pose estimation.
- SegNet and DilatedNet are trained on more than 7,500 annotated images and evaluated as alternative semantic-segmentation architectures.SegNet uses an encoder-decoder design, whereas DilatedNet uses dilated convolutions for multi-scale context.
- SegNet appears to achieve higher precision, while DilatedNet is designed for increased recall through dilated convolution modules.
- Figure 7 compares pose-estimation errors produced from annotated, SegNet, and DilatedNet segmentation masks.Annotated segmentation is derived from motion capture.
- Segmentation quality directly affects the point-cloud mask and therefore the performance of the downstream point-to-pose registration pipeline.The evaluation asks whether IoU, precision, or recall best predicts pose quality.
B. Pose Estimation Results
SegICP achieves accurate pose estimation on the benchmark, but performance depends strongly on segmentation quality, sensor technology, calibration, and point-cloud integrity.
- About 80% of benchmarked instances achieve 1 cm position error and < 5° angle error, while failures exceed 5 cm or 15° thresholds.The benchmark contains 1246 object pose annotations.
- SegNet reaches 77% success, outperforming DilatedNet at 66% on the benchmark.The comparison uses annotated segmentation, SegNet, and DilatedNet outputs to assess segmentation influence on pose estimation.
- Kinect1 instances achieve 90%, 73%, and 72% success with annotated, SegNet, and DilatedNet masks, respectively.The structured-light Kinect1 subset contains 466 instances.
- With Kinect2, success is 85%, 80%, and 62% for annotated, SegNet, and DilatedNet masks, respectively.The lower DilatedNet result is attributed to excessive false positives and poor cropped scene clouds.
- Reflective, symmetric objects and RGB-depth miscalibration produce deformed or misaligned point clouds that degrade ICP registration.Time-of-flight deformation is illustrated for reflective surfaces, while RGB-depth calibration affects point-cloud cropping.
V. CONCLUSION
SegICP integrates parallel semantic segmentation with pose estimation to provide accurate, real-time perception without a close initial pose guess. The approach also supports motion-capture data collection, while larger object sets and task-motion planning remain future work.
- The system processes multiple objects simultaneously at 70–270 ms in tracking and acquisition modes, respectively.Its parallel design does not require an initial guess sufficiently close to the solution.
- The framework uses motion capture to collect annotated segmentation and pose data and is being extended toward larger object sets and integrated task-motion planning.These extensions target complex interactions in unstructured environments.