PaperlayerView exact revision on arXiv

Source-linked AI summary

A Self-supervised Learning System for Object Detection using Physics Simulation and Multi-view Pose Estimation

Chaitanya Mitash, Kostas E. Bekris, Abdeslam Boularias

arXiv:1703.03347v2cs.ROcs.CV

TL;DR

Robotic object detection depends on large labeled datasets, while rendered training scenes may not reflect real poses and occlusions. This paper combines physics-based synthetic scene generation with robot-driven multi-view labeling of real images. The combined approach improves reported detection and pose-estimation results over comparison training processes.

  • Problem

    CNN object detection needs large labeled datasets, while rendered synthetic images can differ from real scenes in object poses and occlusions.

    Source: §I · p. 1

  • Method

    The system generates labeled synthetic scenes using physics simulation, then uses a robot’s multi-view pose estimates to label real images for further training.

    Source: §I · p. 2, §III · p. 3

  • Results

    On Shelf&Tote, self-training produced a 12% performance boost and outperformed training with approximately 15,000 manually labeled real images; pose estimation exceeded the prior system by 25%.

    Source: C. Evaluating Self-learning · p. 6, D. Evaluating the detector for 6DoF… · p. 6

  • Takeaways & Limitations

    Physics-based synthetic scenes and autonomous real-image labeling together yield a success ratio that outperforms current state-of-the-art systems in robot vision.

    Source: §VI · p. 6

    Synthetic data may not always generalize to real-world lighting conditions, motivating additional labeled data from a robotic setup.

    Source: §VI · p. 6

Abstract

from arXiv · show

Progress has been achieved recently in object detection given advancements in deep learning. Nevertheless, such tools typically require a large amount of training data and significant manual effort to label objects. This limits their applicability in robotics, where solutions must scale to a large number of objects and variety of conditions. This work proposes an autonomous process for training a Convolutional Neural Network (CNN) for object detection and pose estimation in robotic setups. The focus is on detecting objects placed in cluttered, tight environments, such as a shelf with multiple objects. In particular, given access to 3D object models, several aspects of the environment are physically simulated. The models are placed in physically realistic poses with respect to their environment to generate a labeled synthetic dataset. To further improve object detection, the network self-trains over real images that are labeled using a robust multi-view pose estimation process. The proposed training process is evaluated on several existing datasets and on a dataset collected for this paper with a Motoman robotic arm. Results show that the proposed approach outperforms popular training processes relying on synthetic - but not physically realistic - data and manual annotation. The key contributions are the incorporation of physical reasoning in the synthetic data generation process and the automation of the annotation process over real images.

I. INTRODUCTION

Robotic object detection needs large labeled datasets, while rendered synthetic scenes can have unrealistic poses and occlusions. The paper proposes physics-based synthetic data generation and robot-driven self-labeling of real images to reduce manual effort and improve detection in robotic setups.

I. INTRODUCTION

  • CNN-based detection relies on large labeled datasets, whose creation is labor-intensive and often requires new data for robotics-specific objects and environments.Existing shelf datasets use human annotation or incremental object placement with foreground masking.Source: §I · p. 1
  • Rendered synthetic data can differ from real testing images, with unrealistic object poses and naively simulated occlusions.Prior approaches varied texture, lighting, and shape to address the synthetic-to-real disparity.Source: §I · p. 1
  • The proposed system aims to reduce data-generation effort and improve bounding-box object detection in robotic setups.Its physics-based simulator uses camera calibration, object models, and shelf or table localization to create scenes with realistic object configurations.Source: §I · p. 2
  • A robot uses a detector trained on simulated scenes to estimate poses from multiple views and label real images, adding them to training over time.Labels from a view where the detector succeeds are used to annotate the same scene from more difficult viewpoints, whose transformations are known from robot motion.Source: Figure 1 · p. 1, §I · p. 2

II. RELATED WORK

Related work spans CNN detection and segmentation, model-based pose estimation, synthetic datasets, and self-supervised learning; this paper combines these components in its training process. Its synthetic-data pipeline uses calibrated scenes, physics simulation, rendering, and projected bounding-box labels to train a detector.

II. RELATED WORK

  • The paper’s novelty is a training process combining synthetic-data generation with real-data augmentation through self-learning.Related work includes CNN detection and segmentation, pose estimation with features, templates, synthesis or segmentation followed by point-cloud registration, and synthetic and self-supervised training approaches.Source: §II · p. 2

III. PHYSICS-AWARE SYNTHETIC DATA GENERATION

  • The simulator uses camera poses, camera intrinsics, a localized resting surface, and 3D object models to generate labeled training scenes.The camera pose can be computed by forward kinematics; shelf localization uses RANSAC to estimate edges and planes.Source: §III · pp. 2–3
  • Objects are simulated under gravity until their poses stabilize, then rendered under varied lighting and projected into 2D bounding-box labels.The physics engine handles falling, bouncing, collisions, and interpenetrations; lighting varies in location, intensity, and color.Source: §III · p. 3, Figure 2 · p. 3
  • The generated synthetic dataset trains a Faster-RCNN detector using a deep VGG network architecture.Source: §III · p. 3

IV. SELF-LEARNING VIA MULTI-VIEW POSE ESTIMATION

The self-learning pipeline uses detections from multiple views to build object point clouds and estimate 6DoF poses, then projects high-confidence poses back to label real images. It iteratively reconfigures scenes to expand the labeled dataset.

IV. SELF-LEARNING VIA MULTI-VIEW POSE ESTIMATION

  • High-confidence 6DoF poses estimated from multi-view detections are projected back onto the real images to produce bounding-box labels across views.The scene is recreated in simulation at the estimated poses, and its bounding boxes are transferred to the real images.Source: §IV · pp. 3–4, Figure 3 · p. 4
  • The detector can be imprecise or fail in some views, so only detections above a confidence threshold proceed to pose estimation.The paper notes possible simulation bias in texture or poses as a source of detection errors.Source: §IV · p. 4
  • The pipeline aggregates depth points from confident detections across views, cleans the object point clouds, and registers each cloud to its 3D model.SUPER4PCS with ICP was found most applicable because it is robust to outliers and provides a confidence measure.Source: §IV · p. 4
  • A robot iteratively reconfigures scenes with pick-and-place actions, moving randomly selected objects to predefined workspace configurations.The resulting scenes are labeled and used to retrain the detector.Source: §IV · pp. 3–4

V. EVALUATION

The evaluation section compares synthetic-data generation techniques and examines self-learning, with object detection assessed using Intersection-Over-Union (IoU).

V. EVALUATION

  • The evaluation compares synthetic-data generation techniques, studies self-learning, and uses Intersection-Over-Union (IoU) for object detection.It also applies the trained detector to 6DoF pose estimation.Source: §V · p. 5

A. Datasets

The system is evaluated on the Shelf&Tote benchmark and on a manually labeled tabletop dataset collected with a Motoman-mounted camera.

A. Datasets

  • The Shelf&Tote evaluation uses 148 shelf scenes, 11 APC objects, 2,220 images, and 229 unique object poses.The scenes vary in lighting and clutter; objects lacking sufficient depth information were excluded.Source: A. Datasets · p. 5
  • The tabletop test set contains 41 cluttered scenes, 11 APC objects, and 473 manually labeled detection instances.An Intel RealSense camera mounted on a Motoman robot captured the scenes from multiple views.Source: A. Datasets · p. 5

B. Evaluating the Object Detector trained in Simulation

This subsection examines how object-pose distributions affect training by evaluating synthetic-data generation techniques on Shelf&Tote.

B. Evaluating the Object Detector trained in Simulation

  • Synthetic-data generation techniques are compared on Shelf&Tote to study the effect of object-pose distribution on training.The results are presented in Table I, but the supplied passage does not report its values.Source: B. Evaluating the Object Detector… · p. 5

1) Generating training data using test data distribution:

This subsection samples synthetic training poses according to their similarity to test-data pose distributions, with the aim of establishing an upper bound for simulated-data performance.

1) Generating training data using test data distribution:

  • Test-pose density is estimated with Kernel Density Estimation to guide training-data generation.The procedure starts by uniformly simulating scenes and recording each object's pose.Source: 1) Generating training data using test… · p. 5
  • Scenes are weighted by pose matches within 15° rotation and 5 cm translation, then weights are normalized to sample training poses.A scene's weight sums the objects whose poses match at least one pose in their corresponding test distribution.Source: 1) Generating training data using test… · p. 5
  • 69% accuracy was achieved by a Faster-RCNN trained on scenes sampled from the test-pose distribution.The subsection frames this as an upper-bound estimate for a detector trained with simulated images.Source: 1) Generating training data using test… · p. 5

2) Uniformly sampled synthetic data:

This alternative renders object views and pastes them onto shelf backgrounds, but does not ensure physically realistic object poses and performs poorly on the target task.

2) Uniformly sampled synthetic data:

3) Generating training data with physics-aware simulation:

Physics-aware simulation performs strongly in the reported setups, while variation in lighting only partly addresses synthetic-data bias.

3) Generating training data with physics-aware simulation:

C. Evaluating Self-learning

The self-learning pipeline uses multi-view pose estimation to label real images and retrain the detector, improving performance on Shelf&Tote.

C. Evaluating Self-learning

  • 12% performance boost was achieved, outperforming 's process using approximately 15,000 manually labeled real images.The passage attributes the advantage mostly to labeling objects placed in clutter; Fig. 5 depicts improvement from adding system components.Source: C. Evaluating Self-learning · p. 6, Figure 5 · p. 6
  • High-confidence estimated poses are projected to known camera views to label real-scene bounding boxes before retraining.The table-top process reconfigures scenes by pick-and-place manipulation and generated 140 scenes with four objects per image.Source: C. Evaluating Self-learning · p. 6

D. Evaluating the detector for 6DoF Pose estimation

The proposed system improves pose estimation against the APC Team MIT-Princeton system under the stated pose-success metric. The discussion attributes the broader gains to physically realistic simulation combined with self-learning on real images, while noting that simulated data alone remains a future goal.

D. Evaluating the detector for 6DoF Pose estimation

  • After self-learning, the detector predicts more precise bounding boxes and detects objects better from novel views.Source: Figure 6 · p. 6

VI. DISCUSSION

  • Physics-based simulation produces images with object-pose distributions similar to real scenes, which helps increase simulated-data training success and reduce manual-labeling requirements.Source: §VI · p. 6
  • Because synthetic data may not generalize to real-world lighting, the system uses high-confidence detections across multiple views for pose estimation and additional real-image labels.The combined physical-reasoning and self-learning approach is reported to outperform current state-of-the-art robot-vision systems in success ratio.Source: §VI · p. 6
  • Achieving similar detection and pose-estimation quality using only simulated data remains future work; semantic segmentation is also proposed for future training extensions.Source: §VI · p. 6

D. Evaluating the detector for 6DoF Pose estimation

Related papers

Loading 1703.03347v2…