Source-linked AI summary
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection
Debidatta Dwibedi, Ishan Misra, Martial Hebert
TL;DR
Instance detection needs extensive annotations for new environments, motivating a low-effort alternative to manual dataset construction. The paper composes object masks with real backgrounds and uses blending variation to promote patch-level realism. The resulting synthetic data is competitive with real and curated data, and combining it with just 10% real annotations can outperform using all available real annotations in a cross-domain setting.
Problem
Rapid instance-detection deployment is hindered by the large, costly annotation sets required for new environments and objects.
Method
The method extracts object masks, pastes them onto real backgrounds, and varies blending so detectors focus on object appearance rather than local artifacts.
Results
In a realistic cross-domain setting, combining synthesized data with just 10% of available real annotations outperforms training with all real annotations.
Takeaways & Limitations
Patch-based realism is sufficient for training region-proposal detectors, while synthetic data provides complementary information and diverse viewpoint and scale coverage.
Abstract
from arXiv · showhide
A major impediment in rapidly deploying object detection models for instance detection is the lack of large annotated datasets. For example, finding a large labeled dataset containing instances in a particular kitchen is unlikely. Each new environment with new instances requires expensive data collection and annotation. In this paper, we propose a simple approach to generate large annotated instance datasets with minimal effort. Our key insight is that ensuring only patch-level realism provides enough training signal for current object detector models. We automatically `cut' object instances and `paste' them on random backgrounds. A naive way to do this results in pixel artifacts which result in poor performance for trained models. We show how to make detectors ignore these artifacts during training and generate data that gives competitive performance on real data. Our method outperforms existing synthesis approaches and when combined with real images improves relative performance by more than 21% on benchmark datasets. In a cross-domain setting, our synthetic data combined with just 10% real data outperforms models trained on all real data.
1. Introduction
The paper addresses the annotation burden of instance detection with a simple synthesis strategy that composes object masks and real backgrounds. It argues that patch-level realism, artifact robustness, and viewpoint diversity can produce effective training data with little effort.
- Instance detection deployment requires thousands of diverse, annotated images for new environments and objects, making rapid collection impractical.
- The approach extracts object masks and pastes them onto random background scenes to create annotated training images.
- Patch-level realism is prioritized because detectors rely more on local region features than global scene layout.
- Naive compositing creates pixel artifacts that alter learned features and reduce detection performance.
- Varying blending while preserving scene and object placement makes detectors ignore blending artifacts and improves training.
- The synthesized data is effective because varied viewpoints and scales improve visual coverage, especially when test scenes differ from training scenes.
2. Related Work
Prior work spans handcrafted features, rendered scenes, and real-image composition for instance detection and related vision tasks. This paper emphasizes local feature robustness over global scene consistency when synthesizing detector training data.
- Instance detection research includes local-feature matching and shape-based methods, while learned ConvNet detectors generalize across feature-rich and feature-poor objects.
- Modern detection methods support real-time robotics applications, while related work also applies deep learning to specific-object pose estimation.
- Synthetic-data approaches render 3D objects or scenes against sampled backgrounds for detection, pose estimation, and related tasks.
- The synthesis spectrum ranges from single objects on random backgrounds to fully rendered scenes; this work composes real object and background images.
- The paper differs from approaches centered on global consistency by training detectors to become invariant to local discrepancies.
- Instance detection distinguishes fine-grained instances and viewpoints, unlike generic object detection, which distinguishes broader categories.
3. Background
Instance detection localizes particular objects rather than generic categories, but conventional dataset curation introduces diversity biases and costly annotation. These constraints hinder rapid collection as scenes and instances grow.
- Instance detection identifies a particular object, such as a brand-specific cereal box or cup, whereas generic detection identifies an entire category.
- Traditional dataset construction requires curation and manual annotation, both of which are unsuitable for rapidly gathering instance annotations.
- Even within kitchens, curation can lack diversity and create biases that do not hold in test settings.
- Manual annotation requires additional time and expense as the number of images and instances increases.
4. Approach Overview
The proposed approach rapidly collects instance-detection data by combining diverse object images with scene backgrounds. It is competitive with manual curation while requiring little time and no human annotation.
- The method is competitive with manual curation while requiring little time and no human annotation.
- It seeks broad visual coverage of each instance across viewpoints, scales, orientations, and lighting conditions.
- Object-instance images should cover diverse viewpoints, while scene images provide backgrounds and may come from known test environments.
5. Approach Details and Analysis
The approach builds synthetic instance-detection data by extracting object masks, pasting them onto real backgrounds, and varying blending and augmentation settings. Empirical analysis shows that blending reduces artifacts, while controlled blending variation and diverse transformations improve detector performance.
- Data collection and synthesis: Object masks are automatically extracted from instance images before pasted objects are composed with real background scenes.The approach collects multi-view object images and uses automated foreground/background segmentation to obtain masks.
- Blending: Direct pasting creates boundary artifacts that cause detection algorithms to focus on local imperfections and degrade performance.These artifacts can produce poor detector performance despite appearing subtle at the pixel level.
- Blending: Blending smooths pasted-object boundaries and introduces image variations that improve trained-detector performance.Poisson blending, for example, smooths edges and adds lighting variations, although the results are not visually perfect.
- Blending: 8 AP points improvement comes from training on identical scenes with identical object placements while varying only the blending type, compared with no blending.This setup encourages invariance to subtle blending artifacts.
- Data Augmentation: 3 AP points come from 2D rotation, while 3D rotation contributes more than 4 AP points in the GMU Dataset evaluation.The augmentation also varies occlusion and truncation to expose detectors to partially visible objects and diverse viewpoints.
6. Experiments
Experiments show that the synthesized data is competitive with real training data, complements it, and improves cross-domain detection when real annotations are scarce.
- Experimental setup: Approximately 6000 synthetic images were generated from object instances and random backgrounds with randomized scale, rotation, position, occlusion, and truncation.The evaluation used BigBIRD instances overlapping with the GMU and Active Vision datasets.
- Failure modes: Human-annotated training data misses certain viewpoints because of poor viewpoint coverage and train-test biases, whereas synthetic data provides more diverse viewpoint coverage.The experiments attribute reduced viewpoint errors to this broader coverage.
- GMU evaluation: Synthetic-data training is competitive with real-image training and outperforms the evaluated existing synthetic-data approach.Combining synthetic and real data further improves performance for all synthetic image datasets.
- Qualitative results: The model trained on synthetic data detects occluded and truncated objects that the real-data model misses, while mixed training removes false positives caused by real-only training.These qualitative comparisons span GMU Kitchen Scenes and the Active Vision Dataset.
- Active Vision evaluation: In cross-dataset evaluation, combining just 10% of real images with synthetic data achieves higher mAP than training on all real images alone.Synthetic training also reduces viewpoint-related errors, and adding synthetic data to real data gives a further 10 AP-point improvement.
7. Discussion and Future Work
The paper concludes that simple patch-based synthesis can efficiently create useful instance-detection data, especially by diversifying viewpoints and scales. It also identifies combining patch-level synthesis with methods modeling global consistency or realism as future work.
- Discussion: The method synthesizes annotated instance-detection images using randomized blending and diverse instance viewpoints and scales.The approach requires little time and no human annotation compared with manual curation.
- Discussion: Patch-based realism is sufficient for training region-proposal-based detectors, and the synthesized data captures information complementary to hand-curated datasets.The cross-domain experiments combine 10% real annotations with synthetic data to outperform all-real training.
- Future Work: Future work should combine the method with approaches that model global scene consistency or image realism.The paper presents this as a direction for combining complementary synthesis strategies.
Appendix
The appendix presents a few randomly selected samples from the synthesized images.
- Appendix: The appendix shows randomly chosen examples from the synthesized image dataset.The figure is illustrative rather than a quantitative evaluation.
- Appendix: The displayed samples represent images generated by the paper’s synthesis procedure.
- Appendix: The figure provides qualitative examples of the generated training images.