PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthesizing Training Data for Object Detection in Indoor Scenes

Georgios Georgakis, Arsalan Mousavian, Alexander C. Berg, Jana Kosecka

arXiv:1702.07836v2cs.CVcs.RO

TL;DR

Indoor object detectors need costly annotations spanning viewpoints, lighting, occlusion, clutter, and backgrounds. The paper generates synthetic training images by compositing textured object instances into real scenes with geometry- and semantics-informed placement and depth-based scaling. Across its evaluations, augmented training improves detector performance, while the discussion reports a 79.0% versus 98.2% recognition-accuracy contrast for cross-background testing.

  • Problem

    CNN-based object detectors require large amounts of annotated data covering viewpoint, lighting, occlusion, clutter, and background variation.

  • Method

    The paper superimposes cropped textured object images into real scenes, using scene geometry, semantic segmentation, and depth to inform placement and scale.

  • Results

    Augmenting real training data with synthetic composites increases the performance of Faster R-CNN and SSD detectors.

  • Takeaways & Limitations

    The approach supports scalable detector training with significantly less annotated data and is more effective when placements use semantic and geometric context.

Abstract

from arXiv · show

Detection of objects in cluttered indoor environments is one of the key enabling functionalities for service robots. The best performing object detection approaches in computer vision exploit deep Convolutional Neural Networks (CNN) to simultaneously detect and categorize the objects of interest in cluttered scenes. Training of such models typically requires large amounts of annotated training data which is time consuming and costly to obtain. In this work we explore the ability of using synthetically generated composite images for training state-of-the-art object detectors, especially for object instance detection. We superimpose 2D images of textured object models into images of real environments at variety of locations and scales. Our experiments evaluate different superimposition strategies ranging from purely image-based blending all the way to depth and semantics informed positioning of the object models into real scenes. We demonstrate the effectiveness of these object detector training strategies on two publicly available datasets, the GMU-Kitchens and the Washington RGB-D Scenes v2. As one observation, augmenting some hand-labeled training data with synthetic examples carefully composed onto scenes yields object detectors with comparable performance to using much more hand-labeled data. Broadly, this work charts new opportunities for training detectors for new objects by exploiting existing object model repositories in either a purely automatic fashion or with only a very small number of human-annotated examples.

I. INTRODUCTION

Indoor object detection supports service-robot tasks but requires costly annotations for cluttered scenes. This work proposes automated, scene-informed synthetic training data and evaluates combining it with real data.

  • Service robots rely on detecting and searching for household objects during fetch-and-delivery tasks.
  • Object detection in indoor scenes must handle clutter, lighting variation, pose variation, and small object scale.
  • CNN-based detectors require many labeled images covering varied backgrounds, poses, bounding boxes, and object segmentations.
  • The proposed system superimposes cropped object images onto real background scenes, using scene understanding to choose meaningful positions.
  • Depth and semantic information guide object placement, while the study also evaluates combining synthetic and real images for detector training.
  • Scene-informed positioning improves detector performance, and augmented training is reported to outperform training with only real data.

II. RELATED WORK

Prior work used hand-designed features, CAD models, and rendered synthetic images, often in simpler settings or with randomized placement. This work instead performs informed compositing in real background scenes.

  • Object Detection: Traditional cluttered-scene detectors commonly used sliding windows, hand-designed features, and discriminative classifiers.
  • Synthetic Data: Earlier synthetic-data approaches varied CAD-model projections, orientations, textures, or backgrounds for CNN training.
  • Synthetic Data: These prior approaches often used simpler backgrounds and proposal strategies with less prominent dependence on scene context.
  • Synthetic Data: The proposed method composites object-centric images into real scenes at informed locations rather than random positions.
  • Synthetic Data: Informed compositing is intended to help CNN detectors produce higher-quality object proposals instead of relying on unsupervised bottom-up techniques.

III. APPROACH

The approach generates composite training images by placing textured object instances into real scenes using support surfaces, semantics, geometry, depth, and controlled blending. Object scale is adjusted to scene depth and overlap is constrained.

  • Synthetic Set Generation: Manual annotation must cover viewpoint, lighting, occlusion, and clutter variations, while models may generalize poorly across environments and backgrounds.
  • Synthetic Set Generation: The method uses cropped textured object images from recognition datasets and superimposes them into real scenes at varied positions and scales.
  • Selective Positioning: Support surfaces are estimated from scene geometry and checked with semantic segmentation to identify valid placement regions.
  • Selective Positioning: Objects are randomly selected and posed, then placed on support surfaces with appropriate context and backgrounds.
  • Selective Scaling and Blending: Object dimensions are scaled using the selected background depth relative to the median depth of the object’s training images.The passage defines z̄ as median training-image depth, z as selected-position depth, and ŵ and ĥ as scaled dimensions.
  • Selective Scaling and Blending: Object overlap is limited to 40%, and masked seamless cloning blends objects with backgrounds to reduce illumination and contrast differences.

B. Object Detectors

The paper evaluates Faster R-CNN and SSD, two end-to-end CNN detectors with different architectures for proposing and localizing objects.

  • The experiments use Faster R-CNN and Single-Shot Multibox Detector (SSD) as state-of-the-art object detectors.
  • Faster R-CNN combines a Region Proposal Network with a second module that classifies proposals and refines bounding boxes.
  • SSD uses feature maps at multiple resolutions to detect objects of different sizes, providing a speed advantage at similar accuracy.
  • Faster R-CNN’s shared feature map creates problems for small objects, while both detectors struggle with small objects in cluttered settings.

IV. EXPERIMENTS

The experiments test synthetic compositing strategies, mixtures of synthetic and real training images, and synthetic train-test settings across two RGB-D datasets.

  • Three experiment sets evaluate compositing strategies, synthetic-to-real transfer, varying synthetic/real proportions, and synthetic training and testing.
  • The study uses the GMU-Kitchen Scenes and Washington RGB-D Scenes v2 datasets to evaluate detectors trained on composited images.
  • The first experiments vary object positioning, scaling, and blending before evaluating detectors on real scenes.
  • The final experiment uses synthetic data for both training and testing to examine over-fitting to superimposition artifacts.

A. Datasets and Backgrounds

The evaluation combines kitchen and tabletop RGB-D scenes with cropped object instances, synthetic backgrounds, and real-data baselines for detector training.

  • Table I and Table II report average precision for SSD and Faster R-CNN on GMU-Kitchens and WRGB-D, respectively.
  • GMU-Kitchens contains 9 kitchen videos with 11 BigBird object instances, supplemented by roughly 7000 NYU Depth V2 background frames.
  • After removing images without detected support surfaces, the GMU-Kitchens background set contains roughly 5000 images.
  • BigBird object masks are refined with GraphCut to clean jagged or incomplete boundaries caused by imperfect depth-based masks.
  • WRGB-D provides indoor tabletop scenes with five object categories and roughly 4600 generated synthetic training images.

B. Synthetic to Real

Synthetic-to-real experiments compare positioning, blending, and scaling choices across SSD and Faster R-CNN, revealing dataset- and detector-dependent effects.

  • The experiments compare random or selective positioning, simple or blended superimposition, and random or selective scale.
  • Selective positioning and scaling are evaluated as alternatives to randomly placing and scaling objects in synthetic scenes.
  • Table III reports RPN recall on both datasets at two IoU thresholds using 3000 proposals per image.
  • 10.3% and 9.6% improvements over RP-SI-RS were obtained by SP-BL-SS for SSD and Faster R-CNN, respectively, on GMU-Kitchens.
  • On WRGB-D, SSD performed better with RP-SI-RS, whereas Faster R-CNN improved 26.1% with SP-BL-SS.

C. Synthetic+Real to Real

Combining selective synthetic composites with real training data generally improves detector performance over real-only training, even with limited real annotations. Gains are especially evident for SSD and with increasing synthetic-data quantities.

  • Synthetic+Real to Real: With synthetic data plus 10% of real data, detection accuracy is higher or comparable to training on real data alone.The 10% setting corresponds to approximately 400 GMU-Kitchens images and 600 WRGB-D images.
  • Synthetic+Real to Real: 6%: SSD performance increases on GMU-Kitchens when synthetic data is combined with 10% of the real training data.
  • Synthetic+Real to Real: 2.3% lower: Faster R-CNN performance on GMU-Kitchens with synthetic data plus 10% real data, but 1.3% higher with 50% real data.
  • Synthetic+Real to Real: SSD shows the largest gain when detectors are trained with both real and synthetic data, while increasing synthetic quantity produces continuing SSD improvement.Faster R-CNN reaches top performance when half of the synthetic data is used with all real data.

D. Synthetic to Synthetic

Synthetic-to-synthetic evaluation reveals substantial overfitting for some placement strategies, especially when synthetic scenes are complex. The gap between synthetic and real testing is smaller for selective blending and for the less complex WRGB-D scenes.

  • Synthetic to Synthetic: The experiment splits RP-SI-RS and SP-BL-SS synthetic sets in half to create synthetic training and test sets.
  • Synthetic to Synthetic: Over 90%: RP-SI-RS achieves synthetic-test results on GMU-Kitchens, reaching almost 100% with Faster R-CNN.Despite these synthetic-test results, RP-SI-RS is the least performing synthetic set in the synthetic-to-real experiment.
  • Synthetic to Synthetic: 17.3% for SSD and 23.4% for Faster R-CNN: SP-BL-SS shows smaller synthetic-to-real accuracy gaps than RP-SI-RS on GMU-Kitchens.
  • Synthetic to Synthetic: Both synthetic training sets achieve similar results on synthetic test sets for WRGB-D.The passage attributes this to WRGB-D having less complex scenes than GMU-Kitchens.

E. Additional Discussion

Cross-background differences help explain why detectors trained on synthetic data perform worse on real scenes. Recognition remains high across BigBird camera views but drops when testing on GMU-Kitchens backgrounds.

  • Additional Discussion: 98.2%: VGG recognition accuracy when trained on BigBird images from cameras 1, 3, and 5 and tested on cameras 2 and 4.
  • Additional Discussion: 79.0%: recognition accuracy when VGG trained on all BigBird images is tested on GMU-Kitchens cropped images with real backgrounds.The comparison indicates a substantial background difference between NYU V2 and GMU-Kitchens.

V. CONCLUSION

The paper concludes with an automated procedure for generating synthetic training data using scene geometry and semantic segmentation to position and scale objects. Augmenting training with these images increases detector performance, while alternative domain adaptation remains an avenue for reducing annotation needs.

  • V. CONCLUSION: The method scales with the number of objects of interest and possible backgrounds, supporting robotics applications.
  • V. CONCLUSION: Alternative domain adaptation approaches could further reduce the amount of human annotation required.
  • V. CONCLUSION: The procedure uses scene geometry and semantic segmentation to make informed decisions about object positions and scales.
  • V. CONCLUSION: Augmented training sets increase performance for two state-of-the-art object detectors.
Loading 1702.07836v2…