Source-linked AI summary
On Pre-Trained Image Features and Synthetic Images for Deep Learning
Stefan Hinterstoisser, Vincent Lepetit, Paul Wohlhart, Kurt Konolige
TL;DR
Object detection requires large, expensive labeled datasets, and synthetic images introduce a domain gap with real images. The paper freezes feature extractors pretrained on real images while training the remaining detector layers on simply rendered synthetic data. Across modern detectors and extractors, this approach performs well, with results close to real-only training and no apparent need for complicated scene composition.
Problem
Object detectors need huge amounts of expensive labeled data, while synthetic images offer free labels but typically suffer from a domain gap that can deteriorate performance on real images.
Method
The paper freezes feature extractor weights pretrained on real images and adapts only the remaining detector layers using synthetic data generated with simple rendering.
Results
Experiments across Faster-RCNN, RFCN, and Mask-RCNN with InceptionResnet and Resnet extractors show good performance, close to approaches trained on real data only.
Takeaways & Limitations
Simple rendering with frozen real-image feature extractors can support synthetic-only training, making real-data generation and expensive manual labeling pipelines redundant within the demonstrated scope.
Takeaways & Limitations
The approach assumes patch-level realism, and performance varies across cameras because different cameras have different image statistics.
Abstract
from arXiv · showhide
Deep Learning methods usually require huge amounts of training data to perform at their full potential, and often require expensive manual labeling. Using synthetic images is therefore very attractive to train object detectors, as the labeling comes for free, and several approaches have been proposed to combine synthetic and real images for training. In this paper, we show that a simple trick is sufficient to train very effectively modern object detectors with synthetic images only: We freeze the layers responsible for feature extraction to generic layers pre-trained on real images, and train only the remaining layers with plain OpenGL rendering. Our experiments with very recent deep architectures for object recognition (Faster-RCNN, R-FCN, Mask-RCNN) and image feature extractors (InceptionResnet and Resnet) show this simple approach performs surprisingly well.
1. Introduction
Modern object detectors need extensive labeled data, while synthetic images offer free labels but suffer from a synthetic-to-real domain gap. The paper proposes freezing real-image-pretrained feature extractors and training the remaining detector layers on synthetic images.
- Leading object detectors rely on convolutional networks and require huge amounts of labeled training data that are time consuming and expensive to create.
- Synthetic images provide free labels, but mismatched real-world and rendering statistics create a domain gap that can deteriorate transfer performance.
- The proposed approach splits modern detectors into a feature extractor and detector-specific remaining layers, then freezes the extractor pretrained on real images.
- Training only the remaining layers enables state-of-the-art Faster-RCNN, RFCN, and Mask-RCNN detectors to be trained purely on synthetic data.
- Different cameras produce different image statistics and therefore different performance levels after synthetic retraining, with the proposed approach significantly boosting performance for those cameras.
2. Related Work
Prior work combines real and synthetic data, uses elaborate realistic rendering or scene composition, or applies transfer learning, but these approaches retain real-data requirements or engineering burdens. The paper positions synthetic-only training with simpler rendering as an alternative to those limitations.
- Methods mixing real and synthetic data improve detection but still require real data, including real masked patches or specialized synthetic-model generation pipelines.
- Photo-realistic rendering, complex scene composition, and physics engines can achieve good results but require significant resources and elaborate, domain-specific engineering.
- Graphics-based rendering approaches can suffer from the domain gap between synthetic and real images and may require expensive manual post-processing or labeling.
- Composing masked object patches into real images reduces the domain gap, but limits new viewpoints and illumination settings and depends on potentially error-prone segmentation.
- GAN-based approaches remain hard to train and have mainly demonstrated usefulness for regression rather than detection applications.
- Transfer-learning approaches use source and target domains jointly or through coupled predictors, whereas this paper applies one feature extractor across domains without real images of the target objects.
3. Method
The method generates labeled synthetic object images with simple rendering and adapts modern detectors by freezing real-image-pretrained feature extractors while training the remaining layers.
- 3.1. Synthetic Data Generation Pipeline: The pipeline samples object poses and scales, places objects in cluttered backgrounds, and computes tightly fitting bounding boxes from CAD models and poses.Pose rotations cover the pose space through recursively divided icosahedron vertices and sampled in-plane rotations; scale is sampled logarithmically.
- 3.1. Synthetic Data Generation Pipeline: Plain OpenGL rendering uses Phong shading, perturbed illumination, Gaussian noise, and Gaussian blurring to integrate rendered objects with backgrounds.The pipeline also varies background channels and image orientation to increase background variability.
- 3.2. Freezing a Pre-Trained Feature Extractor: Modern detectors can be decoupled into a meta-architecture and a feature extractor, with the remaining layers handling classification and localization.The paper considers Faster-RCNN, R-FCN, and Mask-RCNN, using feature extractors such as VGG, ResNet, and InceptionResNet.
- 3.2. Freezing a Pre-Trained Feature Extractor: The approach freezes feature-extractor weights pre-trained on real images and trains only the detector’s remaining layers on synthetic images.In practice, the frozen extractors include InceptionResNet v2 and ResNet101, while other network parts remain trainable.
4. Experiments
The evaluation uses ten varied objects, two cameras, and separate synthetic and manually labeled real-image datasets designed to test detection in challenging environments.
- Dataset: The evaluation dataset contains synthetic and real images of 10 different objects and considers two cameras.The authors note that camera quality influences recognition results.
- Dataset: The selected objects vary in colors, textures, 3D shapes, material properties, and application fields.The set includes industrial objects, household objects, and toys, represented by textured 3D CAD models.
- Dataset: For each camera, training datasets contain approximately 20K images and evaluation datasets approximately 1K manually labeled real-world images.Evaluation images contain objects in heavy background clutter and changing illumination, with varied poses.
4.3. Freezing the Feature Extractor
Freezing feature extractors pre-trained on real images substantially improves synthetic-only detector training, bringing performance close to real-data training in most cases.
- Performance: Training feature extractors on synthetic images performs poorly and completely fails for the AsusXtionPROLive camera.Using frozen real-image-pretrained extractors makes synthetic-trained detectors almost as good as real-trained detectors, except for ResNet101 with AsusXtionPROLive images.
- Performance: Up to 95% of the performance of detectors trained on real data was obtained when detector training used synthetic images with frozen real-image-pretrained feature extractors.Training Faster-RCNN on synthetic images without freezing performed significantly worse than training on real data.
- Feature Analysis: Feature distances between corresponding real and synthetic images are more clustered around 0 with frozen real-image-pretrained extractors than with synthetic-image-finetuned extractors.The comparison uses Euclidean distances between features from paired images showing the same object under the same pose.
4.4. Freezing the Feature Extractor at Different Layers
Freezing the full feature extractor performs best, while freezing only intermediate layers can sharply reduce performance for the AsusXtionPROLive camera.
- 4.4. Freezing the Feature Extractor at Different Layers: Freezing the full feature extractor always performs best compared with freezing at different intermediate layers.For AsusXtionPROLive, intermediate-layer freezing results in a dramatic performance loss.
4.5. On Finetuning the Feature Extractor
Finetuning the pre-trained feature extractor harms detection performance, even after extended training, whereas freezing the full extractor performs best.
- Freezing the full feature extractor performs best across InceptionResnet and Resnet101 experiments.
- Unfreezing the feature extractor after 400K, 900K, or 1200K frozen training steps is evaluated for both cameras and architectures.
- After 1200K frozen training steps, finetuning the extractor significantly degrades performance despite detection performance beginning to plateau.
4.6. Ablation Experiments
Ablations of the synthetic-image pipeline identify boundary blurring as especially useful, while noise, lighting variation, and blending provide limited or camera-dependent gains.
- Blurring the rendered object and adjacent background pixels gives a huge performance boost.
- Adding noise or random light color produces little improvement, with their influence depending on the camera.
- No blending, Gaussian blurring, and Poisson blending do not produce significant performance improvements in the blending experiments.
- Using background images from another camera leaves PtGreyBlackfly results approximately unchanged and appears to improve AsusXtionPROLive results.
4.7. RFCN and MASK-RCNN
The freezing approach generalizes beyond Faster-RCNN: RFCN and Mask-RCNN trained only on synthetic images show significant performance gains, with Mask-RCNN producing reasonable masks in clutter.
- Table 1 reports outcomes for InceptionResnet and Resnet101, with experiments generally performed on synthetic data only.
- Blurring is identified as a useful synthetic-rendering operation for improving results in experiments using InceptionResnet.
- RFCN trained only on synthetic data gains significant performance improvements when its feature extractor is frozen.
- Mask-RCNN trained only on synthetic data also receives a significant performance boost from freezing the feature extractor.
4.8. Qualitative Results
Qualitative results show synthetic-only training with frozen feature extractors detecting multiple objects in cluttered scenes, across varied poses, illumination conditions, and object appearances.
- Qualitative examples include varied poses, heavy background clutter, and illumination changes across the considered objects.
- Faster-RCNN detects several objects in highly cluttered scenes with multiple instances and arbitrary poses.
- Mask-RCNN detects objects and produces reasonable masks in highly cluttered environments under various poses.
- The method detects objects with similar shapes and colors in challenging environments after synthetic-only training.
5. Conclusion
Freezing a pre-trained feature extractor enables state-of-the-art object detectors to train using synthetic data only, with results close to real-data-only approaches. Simple rendering and rendered 3D CAD models can support broad viewpoint coverage without elaborate scene composition or expensive real-data labeling.
- Freezing a pre-trained feature extractor enables state-of-the-art object detectors to train on synthetic data only.
- The resulting performance is close to approaches trained on real data only.
- Simple rendering is sufficient for good performance, while complicated scene composition does not seem necessary.
- Rendered 3D CAD models support object detection from all possible viewpoints, reducing the need for real-data generation and expensive manual labeling.