Source-linked AI summary
Learning Data Augmentation Strategies for Object Detection
Barret Zoph, Ekin D. Cubuk, Golnaz Ghiasi, Tsung-Yi Lin, Jonathon Shlens, Quoc V. Le
TL;DR
Object detection has scarce, costly annotations, and the best way to augment images while preserving bounding-box consistency is unclear. The paper learns specialized augmentation policies through discrete, reinforcement-learning-based search and finds that they improve detection across datasets and models, especially when training data are limited. It also reports that these policies can outperform other tested regularization methods, while extending beyond the evaluated detection setting remains a stated direction.
Problem
Object detection has fewer and costlier labeled examples than classification, while augmentation must handle distortions to images, bounding boxes, and object sizes.
Method
The paper searches discrete, annotation-aware augmentation policies with a reinforcement-learning controller and applies the learned policies during training only.
Results
Learned augmentation improves detection across data sizes, architectures, and datasets, with larger gains on small-data and harder small-object or higher-precision tasks.
Takeaways & Limitations
Learned augmentation policies can transfer across detection datasets and models and independently outperform the tested regularization methods.
Takeaways & Limitations
The discussion frames extension to semantic and instance segmentation and point-cloud domains as future work rather than an evaluated result here.
Abstract
from arXiv · showhide
Data augmentation is a critical component of training deep learning models. Although data augmentation has been shown to significantly improve image classification, its potential has not been thoroughly investigated for object detection. Given the additional cost for annotating images for object detection, data augmentation may be of even greater importance for this computer vision task. In this work, we study the impact of data augmentation on object detection. We first demonstrate that data augmentation operations borrowed from image classification may be helpful for training detection models, but the improvement is limited. Thus, we investigate how learned, specialized data augmentation policies improve generalization performance for detection models. Importantly, these augmentation policies only affect training and leave a trained model unchanged during evaluation. Experiments on the COCO dataset indicate that an optimized data augmentation policy improves detection accuracy by more than +2.3 mAP, and allow a single inference model to achieve a state-of-the-art accuracy of 50.7 mAP. Importantly, the best policy found on COCO may be transferred unchanged to other detection datasets and models to improve predictive accuracy. For example, the best augmentation policy identified with COCO improves a strong baseline on PASCAL-VOC by +2.7 mAP. Our results also reveal that a learned augmentation policy is superior to state-of-the-art architecture regularization methods for object detection, even when considering strong baselines. Code for training with the learned policy is available online at https://github.com/tensorflow/tpu/tree/master/models/official/detection
1. Introduction
Object detection needs augmentation strategies that account for scarce, costly annotations and bounding-box geometry. The paper proposes learned policies specialized for detection and reports gains across datasets and architectures.
- Object detection has fewer examples and costlier labeled data than image classification, making augmentation especially important.
- The proposed transformations operate on whole images, geometrically modify images with corresponding box changes, or alter content inside bounding boxes.They are used during training but not at test time.
- The paper designs a search method that combines and optimizes augmentation policies using operations specific to bounding-box annotations.
- Learned policies produce consistent accuracy gains across detection architectures and datasets, exceeding state-of-the-art results on COCO for a single model.The paper also reports competitive results on PASCAL VOC.
- Learned augmentation is particularly advantageous for small datasets by providing strong regularization against over-fitting on small objects.
2. Related Work
Prior augmentation methods are often tailored to particular datasets or architectures, while learned policies aim to derive strategies directly from data. This paper focuses that automated approach on the additional geometric and annotation complexities of object detection.
- Vision-model augmentation strategies are often specific to a dataset or machine-learning architecture.Examples include elastic distortions for MNIST and cropping or mirroring for natural-image classification.
- Existing vision augmentations include cropping, erasing, noise addition, adversarial occlusion patterns, and cut-and-paste object insertion.
- Learned augmentation approaches have generated new samples or transformed existing samples in classification settings.Examples use sample merging, Bayesian generation, learned feature-space transformations, and related data-driven methods.
- Object detection augmentation is harder because annotations are scarcer and image distortions affect bounding-box locations and object sizes.
- The paper uses validation accuracy to search for detection augmentation procedures intended to generalize across datasets, dataset sizes, backbones, and detection algorithms.
3. Methods
The method formulates augmentation-policy search as discrete optimization over annotation-aware operations, with policies sampled and applied during training. A reinforcement-learning controller searches policies using child detectors trained on a COCO subset and validation mAP as reward.
- Object-detection augmentation search is formulated as discrete optimization while maintaining consistency between distorted images and bounding-box annotations.Bounding boxes also enable operations that act only on contents inside each box.
- An augmentation policy is an unordered set of K sub-policies, with one sub-policy selected randomly during training.
- The search space uses K = 5 sub-policies, each containing N = 2 sequential operations with discretized probability and magnitude parameters.The policy representation is illustrated through sampled sub-policies applied to an image.
- The operation library includes color transformations, geometry-changing transformations, and transformations applied only within bounding boxes.Geometric operations modify bounding-box size and location to remain consistent with the image.
- A recurrent neural-network controller uses reinforcement learning with PPO to search the discrete policy space.
- Child models are trained from scratch on 5K COCO images, and the controller receives mAP on a held-out validation set of 7392 images as reward.The search trains 20K policies using 400 TPUs over 48 hours.
4. Results
The learned augmentation policy produces consistent detection gains across architectures, supports transfer across models and datasets, and is especially effective with limited data and small objects.
- Learned policy: The most frequent operations in strong policies are Rotate, Equalize, and BBox Only TranslateY.Rotate transforms the whole image and bounding boxes, while the resulting boxes enlarge to include rotated objects.
- Systematic gains: +1.6 to +2.3 mAP improvements are achieved across several ResNet backbones on COCO with RetinaNet.
- Systematic gains: +2.3% mAP is obtained by combining color, geometric, and bounding-box-specific operations, compared with +0.8 mAP from color operations alone.Adding geometric operations raises the boost to +1.9 mAP before bounding-box-specific operations are included.
- Transferability: +1.5% mAP is obtained when the policy transfers to AmoebaNet-D with NAS-FPN across a different architecture, detector, image size, and training procedure.The transfer experiment changes from 640 to 1280 pixels and from training from scratch to ImageNet pre-training.
- State-of-the-art detection: 50.7 mAP is achieved on COCO by a single-stage, single-model system requiring one inference at one spatial scale.The result combines learned augmentation with increased image resolution and more detection anchors.
- Limited data and object size: Learned augmentation yields larger improvements on smaller datasets and can improve mAP by more than 70% relative to baseline on 5,000 training samples.For small objects, the policy with 9,000 examples outperforms the baseline using 15,000 images and is almost as effective as doubling the dataset size.
- Limited data and object size: The policy produces relatively larger gains for AP75 than AP50, particularly benefiting precise bounding-box alignment and small-object detection.The authors associate this pattern with learning fine spatial details in bounding-box position.
5. Discussion
The learned augmentation policy improves object detection most when data are scarce and detection is harder, while reducing weight norms and making added regularization unnecessary. Its transferability suggests a potential alternative to acquiring additional annotated data, though extension beyond detection remains future work.
- Learned augmentation improves accuracy across all data sizes, with larger gains on small training sets and harder small-object or higher-precision detection tasks.
- The learned policy makes other tested regularization methods unnecessary when combined with it, as Input Mixup, Manifold Mixup, and Dropblock neither helped nor hurt performance.
- Applying learned augmentation policies to other perceptual domains, including semantic and instance segmentation or point-cloud featurizations, is identified as future work.
- Learned augmentation policies may offer an efficient alternative to acquiring additional human-annotated data, particularly for models trained with limited data.
- The learned augmentation policy further decreases model weight L2 norms, alongside the decrease associated with increasing training-set size.
A. Appendix
The appendix lists the transformations available during policy search and specifies the learned sub-policies used during training, including their application probabilities and magnitudes.
- Table 6 lists the image transformations available to the controller during search and the magnitude range predicted for each transformation.
- Table 7 specifies each learned sub-policy’s operation probabilities and magnitudes, with one sub-policy selected uniformly at random for each image in a mini-batch.