Source-linked AI summary
RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter
Max Schwarz, Anton Milan, Arul Selvam Periyasamy, Sven Behnke
TL;DR
Robotic manipulation in clutter requires reliable perception of diverse, occluded objects in constrained and unstructured scenes. The paper combines detection, segmentation, RGB-D depth fusion, and pretrained features, then evaluates them in bin-picking and disaster-response settings, where the experiments support reliable perception across both tasks.
Problem
Robotic manipulation requires precise perception of diverse, partially occluded objects among distractors and in restricted or unstructured scenes.
Method
The paper combines object detection and semantic segmentation with RGB-D depth fusion and transfer learning from pretrained features.
Results
The approach was demonstrated in bin-picking and disaster-response settings, with experiments showing validity and encouraging performance in detection and segmentation.
Takeaways & Limitations
Transfer learning supports applying deep-learning perception to real-world manipulation with little additional annotation across two challenging settings.
Takeaways & Limitations
Performance remains sensitive to input resolution, domain-specific tuning, and object properties such as small size and shininess.
Abstract
from arXiv · showhide
Autonomous robotic manipulation in clutter is challenging. A large variety of objects must be perceived in complex scenes, where they are partially occluded and embedded among many distractors, often in restricted spaces. To tackle these challenges, we developed a deep-learning approach that combines object detection and semantic segmentation. The manipulation scenes are captured with RGB-D cameras, for which we developed a depth fusion method. Employing pretrained features makes learning from small annotated robotic data sets possible. We evaluate our approach on two challenging data sets: one captured for the Amazon Picking Challenge 2016, where our team NimbRo came in second in the Stowing and third in the Picking task, and one captured in disaster-response scenarios. The experiments show that object detection and semantic segmentation complement each other and can be combined to yield reliable object perception.
1 Introduction
The paper targets precise perception for robotic manipulation in cluttered, diverse, and partially structured environments. It combines deep-learning perception with transfer learning and evaluates the approach in bin-picking and disaster-response settings.
- Robotic manipulation requires object detection and pixel-wise semantic segmentation for precise scene understanding.
- Cluttered manipulation scenes contain diverse objects, chaotic arrangements, spatial restrictions, and, in disaster response, highly varying backgrounds and unknown objects.
- Transfer learning adapts pretrained initial layers while fine-tuning task-specific layers, enabling deep networks to learn from little additional annotated data.
- The paper develops two transfer-learning-based perception methods, integrates them into a real robotic system, and fuses three depth sources to improve depth measurements.
- The methods are evaluated in bin-picking and disaster-response scenarios, demonstrating their validity and generality across the two settings.
2 Related Work
Prior work applies deep learning, RGB-D sensing, segmentation, pose estimation, and specialized grasping to robotic perception and manipulation. The paper adapts DenseCap- and OverFeat-based approaches with transfer learning for bin-picking and disaster-response settings.
- Deep learning uses multilayer convolutional architectures and has advanced image classification, object detection, and semantic segmentation.
- The paper adapts OverFeat and DenseCap to detect and segment objects for robotic perception and manipulation in bin-picking and disaster-response tasks.
- DenseCap-based detection extracts CNN features, samples 1000 region proposals using objectness, and processes regions of arbitrary size and shape.
- Some prior bin-picking studies simplify conditions by assuming uniform, geometric, textured, hollow, or ferrous objects that support specialized grasping.
- Related robotic systems combine RGB-D perception with industrial manipulators, suction or pinch grasping, segmentation, pose estimation, and point-cloud registration.
- One compared system used over 130,000 training images, about three orders of magnitude more than this paper's approach.
3 Methods
The paper develops complementary object-detection and semantic-segmentation methods for robotic perception, using pretrained CNN features and multiple ways to incorporate RGB-D information. Their combination integrates detection posteriors with segmentation while accommodating limited annotated data and imperfect depth geometry.
- Perception methods: Object detection outputs bounding boxes and object classes, while semantic segmentation provides pixel-wise object-category labels.
- Transfer learning: Both methods leverage pretrained CNNs and adapt only task-specific components because robotic training data and time are limited.
- Object detection: The detection method extends DenseCap by extracting CNN features, sampling 1000 region proposals, and replacing caption generation with classification.
- Depth integration: Depth is incorporated through RGB-D connected-component proposals, HHA features concatenated with CNN maps, or Cross Modal Distillation from pretrained RGB features.
- Semantic segmentation: Semantic segmentation offers finer localization than bounding boxes, supporting manipulation when occlusions or small objects make coarse localization unreliable.
- Combining detection and segmentation: The combined posterior multiplies detection and segmentation information after Gaussian rendering of detections, yielding a consistent performance increase despite conditional-independence and shape-approximation assumptions.
4 Application to Bin-Picking
The bin-picking system combines a stationary UR10-based robotic platform, dual-camera RGB-D sensing, depth fusion, and complementary deep-learning perception methods. It performed successfully in APC 2016, with strong but imperfect stowing and picking outcomes, while experiments showed benefits from depth, fusion, and combining detection with segmentation.
- 4.1 Robotic System: The APC system used a stationary UR10 arm, custom extendable endeffector, rotary joint, and suction gripper for shelf-bin manipulation.The linear actuator extended 37 cm and supported front and top grasps; the suction cup was 3 cm in diameter.
- 4.2 RGB-D Preprocessing: Dual-camera RGB-D sensing corrected view-dependent depth artifacts and fused three aligned depth streams, while rejecting pixels with disagreements above 5 cm.The fused inputs were the two RGB-D depth streams and stereo disparity from the two RGB cameras.
- 4.3 Overall Results: The system performed both APC 2016 tasks successfully, stowing 11 of 12 items and picking 10 of 12 items.A misrecognized eraser caused the stowing shortfall, while three dropped items affected the picking result and final score.
- 4.4 Object Detection and Semantic Segmentation: The softmax detector outperformed the fixed-network SVM design, and data augmentation slightly improved performance.The SVM was particularly weak on cross-image detection ranking, as reflected by mAP.
- 4.4 Object Detection and Semantic Segmentation: Depth improved detection performance, with Cross Modal Distillation producing the best depth features and TGV fusion outperforming color-guided smoothing.The fused two-camera stream also showed an advantage over the alternative depth-processing method.
- 4.4 Object Detection and Semantic Segmentation: Combining the finetuned detector with semantic segmentation yielded a small but consistent performance increase, although small and shiny objects remained difficult.Segmentation struggled especially on objects such as toothbrushes and scissors, with F1 below 0.5 in the reported difficult cases.
5 Application to Disaster Response
The pipeline was adapted from bin-picking to disaster-response scenes with severe clutter and unstructured backgrounds. Higher input resolution substantially improved detection, while segmentation remained effective despite visually similar distractors.
- Dataset and adaptation: The disaster-response dataset extends the bin-picking pipeline to an unrelated domain with cluttered backgrounds and unknown objects.The dataset was collected in the CENTAURO project for autonomous perception and manipulation in disaster response.
- Evaluation setup: Evaluation uses five-fold cross-validation with Iterative Stratified Sampling, after cropping RGB frames to the region with valid Kinect v2 depth.The cropped resolution is 1470 × 1035 from the original 1920 × 1080 RGB image.
- Model adaptation: The disaster-response detector replaces HHA features with raw-depth Cross Modal Distillation because some scenes lack a reliably estimable ground plane.The segmentation network is also adapted from the bin-picking architecture.
- Detection results: 97.3% mAP is achieved at 1470×1035 input resolution, compared with 83.4% mAP using the APC configuration.An intermediate resolution reaches 94.7% mAP with 550 ms prediction time, whereas full resolution takes about 1 s per image.
- Segmentation and interpretation: Small objects, rather than highly cluttered backgrounds, are identified as the main difficulty for this dataset.The segmentation method achieves good results even when backgrounds contain many visually similar objects.
6 Lessons Learned
Transfer learning worked effectively for real-life robotic perception with few annotated images, and finetuning pretrained architectures outperformed the considered CNN+SVM alternative. The experiments also show that performance depends on domain-specific tuning, especially input resolution and feature choices.
- Transfer learning: Transfer learning enabled effective real-life robotic applications with only a few annotated images.The authors identify this as a path toward one-shot or few-shot learning.
- Transfer learning: Finetuning pretrained architectures was beneficial compared with the classical CNN+SVM combination in the considered scenarios.This result held despite the limited training data.
- Domain adaptation: Input resolution is an important detection-performance parameter, requiring adaptation to the task and available processing time.The disaster-response experiments specifically exposed this trade-off.
- Domain adaptation: State-of-the-art deep-learning techniques still require substantial manual hyperparameter tuning for target-domain adaptation.The authors also note that HHA encoding can help in some settings but is not generally applicable across domains.
7 Conclusion
The paper adapts two deep-learning image-understanding methods to robotic perception and demonstrates them in bin-picking and disaster-response settings. Transfer learning supports these applications with limited annotation, while future work aims to integrate detection and segmentation into one trainable network.
- Conclusion: The system adapts two deep-learning image-understanding methods to robotic perception and demonstrates them in two application settings.The settings are APC 2016 bin-picking and disaster response.
- Conclusion: Transfer learning allows deep-learning perception to be applied to real-world manipulation without excessive training-image annotation.The APC scenario combines many categories, narrow workspaces, and shiny or textureless surfaces.
- Conclusion: In disaster response, the adapted methods achieve encouraging performance in both object detection and semantic segmentation despite severe clutter and unstructured backgrounds.The authors collected and annotated a domain-specific dataset to validate the adaptation.
- Future work: Future work will integrate detection and segmentation into a single network for simultaneous end-to-end training, alongside public release of the APC code and collected data.The planned integration is presented as future work rather than as part of the current system.