Source-linked AI summary
On Rendering Synthetic Images for Training an Object Detector
Artem Rozantsev, Vincent Lepetit, Pascal Fua
TL;DR
When training images are scarce and do not cover object poses and lighting, detector training faces a data gap. The paper estimates rendering parameters from real seed images and a coarse 3D model, reporting improved detector performance from the resulting synthetic images.
Problem
Training images may be scarce and fail to cover the full range of object shapes, poses, and lighting conditions.
Source: §1 · p. 2Method
The method estimates rendering parameters from real seed images and a coarse 3D model, using similarity guided by the target detector's features.
Source: §1 · pp. 3–4Results
Optimized rendering parameters produce synthetic images that significantly improve object-detector performance when used for training.
Source: §6 · p. 25Takeaways & Limitations
The best similarity criterion depends on the target detection method.
Source: §1 · p. 4This augmentation approach assumes the original training set is diverse enough, since the range of synthetic images it can produce is limited.
Source: §1 · p. 3
Abstract
from arXiv · showhide
We propose a novel approach to synthesizing images that are effective for training object detectors. Starting from a small set of real images, our algorithm estimates the rendering parameters required to synthesize similar images given a coarse 3D model of the target object. These parameters can then be reused to generate an unlimited number of training images of the object of interest in arbitrary 3D poses, which can then be used to increase classification performances. A key insight of our approach is that the synthetically generated images should be similar to real images, not in terms of image quality, but rather in terms of features used during the detector training. We show in the context of drone, plane, and car detection that using such synthetically generated images yields significantly better performances than simply perturbing real images or even synthesizing images in such way that they look very realistic, as is often done when only limited amounts of training data are available.
1. Introduction
The introduction motivates expanding scarce real training data with synthetic examples and presents automated rendering-parameter estimation guided by detector-relevant similarity.
1. Introduction
- Training data can be scarce and fail to cover the full range of object shapes, poses, and lighting conditions, particularly for drone detection.The introduction notes that training videos for this application are rare and do not cover all such conditions.Source: §1 · p. 2
- Simple augmentation assumes the original training set is sufficiently diverse, while small perturbations may be inadequate and require care.The range of synthetic images available from such augmentation is limited.Source: §1 · p. 3
- A prior 3D-car synthesis approach improved training performance but omitted artifacts such as motion blur and applied only to relatively simple geometry.The introduction also describes other prior methods as either not modeling acquisition artifacts or requiring considerable manual interaction.Source: §1 · p. 3
- The method estimates rendering parameters from a small set of real images and a coarse 3D object model, then varies object pose to generate synthetic training images.The resulting composites are post-processed to resemble real images, and the fitted parameters can be reused to produce large datasets.Source: §1 · p. 3
- Similarity should reflect detector-training features rather than visual appeal, and the best similarity function depends on the target detection method.The authors say they demonstrate this dependence for three representative detection methods.Source: §1 · p. 4
- The authors claim their automated synthetic-image approach improves detection performance and outperforms the discussed state-of-the-art techniques across drone, plane, and car detection.They state the result holds irrespective of the detector used.Source: §1 · p. 4
2. Related Work
Prior work uses synthetic images across several computer-vision tasks, but existing approaches leave gaps in modeling imaging artifacts or tailoring synthesis to detector performance. This work instead optimizes rendering parameters for classification performance, not visual realism.
2. Related Work
- Deformations and added noise have been used to augment OCR samples, while synthetic image patches have also been used for training.Characters and patches are easier to synthesize than natural images because they do not exhibit their full complexity.Source: §2 · pp. 4–5
- Complete depth images generated from 3D human models trained classifiers to recover human 3D pose from Kinect output, enabling arbitrarily large datasets but limiting video use because they lack motion blur and lighting effects.Source: §2 · p. 5
- Other work creates realistic synthetic images by placing real pedestrian silhouettes over varied backgrounds, but is specific to pedestrian detection.The method also requires considerable manual interaction.Source: §2 · p. 5
- Earlier 3D-model synthesis estimates material and lighting properties, but omits artifacts such as motion blur and requires precise registration.This approach uses real training images and a 3D model.Source: §2 · p. 5
- The authors optimize synthetic-image rendering parameters for classification performance rather than visual realism.They note that prior approaches do not estimate image realism or suitability for the application itself.Source: §2 · pp. 5–6
3. Generating Synthetic Images
The image-generation pipeline combines a coarse 3D model with real images and estimates rendering parameters for each seed image. It then reuses capture settings while varying pose to create many detector-training images.
3. Generating Synthetic Images
- The pipeline uses a coarse CAD model and a small set of real images, with backgrounds taken from the training sequence or reconstructed by filling object cutouts.The model roughly captures the target geometry.Source: §3 · p. 6
- Five pose parameters place the 3D model in the image, while a multi-scale detector removes the need to vary object scale.The excerpt identifies three orientation parameters and begins describing the remaining pose parameters.Source: §3 · p. 6
- Post-processing adjusts object boundaries with Gaussian blur and simulates motion blur using anisotropic Gaussian blur in the motion direction.The boundary blur models mixed object and background intensities at sensor edges; motion blur uses two kernel standard deviations and a motion angle.Source: §3 · pp. 6–7
- The pipeline also adds Gaussian noise to inserted-object pixels and varies diffuse material reflection to change object color and lighting.It does not model specularities, which the authors describe as a natural extension.Source: §3 · p. 7
- Pose and capture settings are difficult to tune because they are correlated, especially object pose with motion-blur direction and boundary blur.After estimating settings for seed images, the method retains capture parameters and randomizes pose to generate large synthetic datasets.Source: §3 · pp. 7–8
4. Optimizing the Rendering Parameters
The method fits rendering parameters independently to each real image by matching a rendered object against that image and its background. It selects image-similarity measures based on detector features, because the best match depends on the detection method.
4. Optimizing the Rendering Parameters
- For each real image, the method renders the target model on its background and optimizes parameters to minimize the distance between real and synthetic images.The optimization is performed independently because capture conditions may differ across images.Source: §4 · p. 8
- Simulated annealing optimizes the rendering parameters, with pose initialized from a manually supplied object center and capture parameters initialized randomly.On the authors’ 40 × 40 images, optimization takes a few seconds per image.Source: §4 · p. 8
- Capture parameters depend on viewing conditions such as lighting and weather, and their observed distributions are non-Gaussian.The paper therefore optimizes each image separately rather than assuming one shared parameter setting.Source: §4 · pp. 8–9, Figure 3 · p. 9
4.1. Image Similarity Measures
- The best image-similarity function depends on the target detection method, so the authors base each function on features used by its detector.They consider DPM, AdaBoost, and CNN detectors.Source: §4.1 · p. 9
- The tested similarities include HoG-vector distance for DPM, weak-learner gradient energy for AdaBoost, and CNN-layer activations for the CNN detector.For AdaBoost, the weak learners are either random or selected by AdaBoost on seed images.Source: §4.1 · pp. 9–10
5. Results
The experiments evaluate the rendering approach on UAV, aircraft, and car detection, including a comparison with prior synthetic-view generation on PASCAL VOC 2007. The authors report that appropriately generated synthetic data improves performance across detection methods and generalizes beyond drones.
5. Results
- The UAV dataset tests a single drone across low-resolution images with varied backgrounds, lighting, and pose.The paper uses it to investigate the effects included in the rendering pipeline.Source: §5 · pp. 11–12
- The Aircraft dataset tests generalization beyond drones using planes photographed across changing backgrounds and weather and lighting conditions.The authors also compare against a recent synthetic-view-generation approach on the PASCAL VOC 2007 car class.Source: §5 · pp. 12–13
- Across the evaluated object classes, appropriately generated synthetic training images improve performance regardless of the detection method.The authors present UAV, aircraft, and car detection experiments, with the car evaluation on PASCAL VOC 2007.Source: §5 · pp. 12–13
5.1. Gauging the Various Components of the Approach using the UAV Dataset
The section evaluates the approach’s components on the UAV dataset, examining image perturbations, rendering effects, parameter optimization, and sample counts. It reports that synthetic training performs best with an appropriate similarity measure and that 12 real seed images can outperform training on 100 real images, while Euclidean similarity can hurt performance.
5.1.1. Experimental Setup
- The UAV dataset contains 2,000 images captured by one UAV filming another in varied environments and lighting; training and testing videos use different environments.Detectors are trained with real and synthetic data and tested on real data only.Source: §5.1 · p. 13, §5.1.1 · pp. 13–14
5.1.2. Comparing against simply Perturbing the Real Images
- The approach significantly outperforms training-image perturbations that combine rotation, translation, mirroring, blurring, and noise.The comparison uses the optimal number of synthetic images and appropriate distance measure for each method.Source: §5.1.2 · pp. 14–15, Table 1 · p. 15
5.1.3. Relative Importance of the Various Rendering Effects
- Correctly modeling each rendering effect improves final performance, although the effects’ relative impact varies by detector.The tested effects are object-boundary blurring, motion blurring, random noise, and material properties.Source: Figure 8 · p. 16, §5.1.3 · p. 16, §5.1.2 · p. 16
5.1.4. Importance of Optimizing over the Rendering Parameters
- Optimized capture parameters yield better performance than randomly selected parameters, and the optimization-based approach brings a significant improvement.Random parameters were drawn from uniform ranges bounded by the optimized parameter values.Source: Figure 9 · p. 17, §5.1.3 · p. 17, §5.1.4 · p. 17
5.1.5. Influence of the Number of Synthetic and Real Images
- The best synthetic-to-real training ratio depends on the detector: AdaBoost performs best at 100:1, DPM at 50:1, and CNN at 15–20:1.Each detector’s synthetic samples use parameters obtained with its corresponding similarity function.Source: §5.1.5 · p. 18
- With 12 real seed samples, synthetic training data lets AdaBoost outperform a detector trained with about eight times as many real images.Increasing the number of real seed images also improves final performance.Source: §5.1.5 · p. 18
- Synthetic images improve performance over real-only training only up to a point; too many can reduce performance by drowning out the real images.The authors recommend using a validation set to determine the optimal synthetic-to-real ratio.Source: §5.1.5 · p. 18, Figure 10 · p. 19
5.1.6. Optimal Performance
- The optimal-performance experiments used each detector’s optimal synthetic-sample count and also estimated optimal counts using Euclidean distance.The supplied passage describes the comparison setup but does not report its results.Source: §5.1.6 · p. 19
- Each detection method performed best when trained on synthetic images generated using its appropriate similarity measure.The passage identifies this as confirmed by Table 2, but no table values are supplied here.Source: §5.1.6 · p. 19
- Using as few as 12 real seed images, synthetic training outperformed a detector trained with 100 images when each detector used its corresponding similarity measure.Figure 11 compares detector performance for different numbers of real seed images.Source: §5.1.6 · p. 20, Figure 11 · p. 20
- Using Euclidean distance as the similarity measure was ineffective and yielded worse results than using no synthetic images.The comparison is stated in the context of Section 4.1.Source: §5.1.6 · p. 20
- The best performance came from a DPM trained on both real and synthetic images, although CNN outperformed DPM when neither used synthetic images.Source: §5.1.6 · p. 20
5.2. Detecting Multiple Kinds of Aircrafts
For aircraft detection, combining real and synthetic training images achieves about 90% detection accuracy, versus approximately 65% with real images only. The experiments use 100 real images of three aircraft models and add generated samples.
5.2. Detecting Multiple Kinds of Aircrafts
- About 90% detection accuracy is achieved with real and synthetic images, versus approximately 65% with real images only.The test set comprises eight video sequences showing aircraft in different environments and weather conditions.Source: §5.2 · p. 21
- The training data starts with 100 real images covering three aircraft models and adds 100 synthetic images per real seed, yielding 10,000 positive examples.The real images have corresponding background images.Source: §5.2 · pp. 20–21
- Adding synthetic images substantially improves performance in all cases, while the best results depend on the detector’s corresponding similarity measure.For AdaBoost, the plotted alternatives use dL_WL and dR_WL similarity measures.Source: §5.2 · p. 22
5.3. Comparing against another recent Image-Based Synthesis Approach
The approach outperforms on the PASCAL VOC car dataset while using a simple rendering pipeline. Its color rendering varies pose and lighting, and the results suggest visual realism is not essential for detector training.
5.3. Comparing against another recent Image-Based Synthesis Approach
- The related approach [12] registers a CAD model and back-projects image texture to assign material properties to visible model parts before generating new views.It supplements real training images with the resulting synthetic views.Source: §5.3 · p. 23
- Because the approach requires background images, the authors create them by cutting cars from seed images and filling the empty areas with content-aware texture filling.They show sample images of the resulting backgrounds.Source: §5.3 · p. 23
- For color rendering, the method optimizes parameters independently on RGB channels, combines the rendered channels, and varies car pose and light direction—unlike.The paper says the car in Fig. 18(b) looks unrealistic because the same properties are applied to all components.Source: §5.3 · p. 24
- Although shared material properties make some color renders look unrealistic, the authors report satisfying results and conclude that visual polish is not critical for their purpose.They note that a more sophisticated model could address the shared-material issue.Source: §5.3 · pp. 24–25
- The method applies to low-resolution images with limited texture, and omitting color reduces performance by only 1–2%.The authors describe this drop as not very large.Source: §5.3 · p. 24
- Table 4 reports that the approach outperforms, and even its simplistic rendering is sufficient to do so.The table compares DPM performance on the PASCAL VOC car dataset.Source: §5.3 · pp. 24–25, Table 4 · p. 25
6. Conclusion
The conclusion argues that optimizing a simple rendering pipeline can produce synthetic images that significantly improve object-detector performance when used for training. The parameter-optimization scheme could also help manage richer rendering pipelines and support using more sophisticated graphics and post-processing effects.
6. Conclusion
- Optimizing a simple rendering pipeline produces synthetic images that significantly improve object-detector performance when used for training.The authors present parameter optimization as the core contribution.Source: §6 · p. 25
- The parameter-optimization scheme can manage the large numbers of parameters that a more complex rendering pipeline could contain.The authors frame this as opening the way to more sophisticated computer graphics and post-processing effects.Source: §6 · p. 26
- The authors identify reducing the cumbersome need for large numbers of real images as a motivation for generating synthetic training images.They describe using more sophisticated graphics and post-processing to make synthetic images closer to real ones.Source: §6 · p. 26