Source-linked AI summary
A-Fast-RCNN: Hard Positive Generation via Adversary for Object Detection
Xiaolong Wang, Abhinav Shrivastava, Abhinav Gupta
TL;DR
Large-scale datasets may miss rare occlusions and deformations needed for invariant object detection. The paper jointly learns an adversary that generates difficult feature-space examples with the detector, improving Fast-RCNN performance on VOC07 and VOC2012.
Problem
Large-scale datasets may rarely contain some occlusions and deformations, making it difficult to learn detector invariance to these variations.
Method
The method jointly learns an adversarial network that creates difficult occlusions and deformations in convolutional feature space while the detector adapts to classify them.
Results
The full approach improves Fast-RCNN by 2.3% mAP on VOC07 and 2.6% mAP on VOC2012.
Takeaways & Limitations
Adversarial learning provides a way to learn invariance to rare occlusions and deformations by generating difficult examples during detector training.
Takeaways & Limitations
Adversarial examples can resemble other object categories, causing over-generalization and false positives.
Abstract
from arXiv · showhide
How do we learn an object detector that is invariant to occlusions and deformations? Our current solution is to use a data-driven strategy -- collect large-scale datasets which have object instances under different conditions. The hope is that the final classifier can use these examples to learn invariances. But is it really possible to see all the occlusions in a dataset? We argue that like categories, occlusions and object deformations also follow a long-tail. Some occlusions and deformations are so rare that they hardly happen; yet we want to learn a model invariant to such occurrences. In this paper, we propose an alternative solution. We propose to learn an adversarial network that generates examples with occlusions and deformations. The goal of the adversary is to generate examples that are difficult for the object detector to classify. In our framework both the original detector and adversary are learned in a joint manner. Our experimental results indicate a 2.3% mAP boost on VOC07 and a 2.6% mAP boost on VOC2012 object detection challenge compared to the Fast-RCNN pipeline. We also release the code for this paper.
1. Introduction
Object detectors typically rely on large datasets to learn invariance, but rare occlusions and deformations may remain uncovered because these variations follow a long-tail distribution. The paper therefore proposes generating difficult feature-space examples adversarially so the detector can learn from them.
- Motivation: Large-scale datasets are used to expose detectors to illumination, deformation, occlusion, and other intra-class variations.The approach assumes sufficient examples can help classifiers model these invariances.
- Motivation: Rare occlusions and deformations may occur too infrequently to be represented in large-scale datasets.Examples include uncommon object-based occlusions and rare animal poses.
- Motivation: Larger datasets are unlikely to scale as a complete solution because long-tail statistics make uncommon variations difficult to sample.Generating realistic rare cases also requires examples of those cases, while enumerating all occlusions and deformations is not scalable.
- Approach: The proposed adversary generates hard occlusion and deformation examples in convolutional feature space rather than directly generating pixels.It adapts as the detector improves, and the paper reports empirical detection gains from this strategy.
2. Related Work
Related work improves object detection through stronger architectures, contextual reasoning, and better use of data. This paper follows the data-utilization direction but generates hard examples instead of only mining them from existing data.
- Detection improvements: One research direction improves detection by changing the base architecture, including the use of deeper networks.Examples include ResNet, Inception-ResNet, and ResNeXt.
- Detection improvements: A second direction uses contextual reasoning, proxy tasks, and top-down mechanisms to improve detector representations.Related methods incorporate segmentation, skip-network features, context, and finer details.
- Data utilization: A third direction seeks to improve performance by exploiting training data more effectively, including hard-example mining.Prior work searches existing data for difficult examples during region-based ConvNet training.
- Data utilization: This paper instead generates hard positive examples by adding occlusions and deformations to existing dataset instances.Adversarial networks predict variations likely to cause Fast-RCNN misclassification.
3. Adversarial Learning for Object Detection
The method generates difficult occlusion and deformation examples in a restricted feature space and trains the detector and adversary together. The detector learns from adversarial examples while the adversary seeks features on which detection is difficult.
- Adversarial setup: The method restricts generation to occlusions and deformations in feature space instead of generating complete images in pixel space.This targets the desired invariances while avoiding unrestricted image generation.
- Adversarial setup: ASDN receives RoI-pooled image-patch features, predicts an occlusion or dropout mask, and passes masked features to Fast-RCNN classification layers.The mask drops selected feature values to create adversarial examples.
- Detector objective: The original detector loss combines classification loss with bounding-box localization loss for foreground classes.The detector outputs class predictions and bounding-box locations for each proposal.
- Joint learning: The adversarial network is trained to predict features on which the detector fails, while adversarial examples are included alongside original examples in detector training.The detector and adversary therefore optimize opposing objectives over generated features.
4. A-Fast-RCNN: Approach Details
A-Fast-RCNN augments Fast-RCNN with adversarial networks that generate hard occlusions and deformations in feature space during training. ASDN creates occlusions, ASTN creates deformations, and joint training makes the detector more robust to both.
- Framework overview: The framework extends Fast-RCNN with adversarial feature generation for occlusions and deformations, followed by end-to-end detection training.The detector outputs object classes and bounding boxes, using SoftMax and regression losses.
- Adversarial Spatial Dropout: ASDN learns spatial masks that drop out feature-map activations, creating occluded object features that are difficult for Fast-RCNN to classify.The mask is applied across all channels at selected spatial locations.
- Adversarial Spatial Dropout: ASDN pre-training selects spatial windows producing the highest detector loss, then uses them as ground truth for learning adversarial dropout masks.The detector is first trained for 10K iterations, after which ASDN is trained with the detector fixed.
- Adversarial Spatial Dropout: ASDN converts its continuous heatmap into stochastic binary masks by sampling high-probability pixels rather than directly thresholding every location.The procedure selects top-probability pixels and randomly samples among them to introduce diversity during training.
- Adversarial Spatial Transformer: ASTN learns feature deformations that make recognition difficult, using differentiable spatial transformation and independently rotating feature-channel blocks.The implementation constrains rotations to 10° clockwise or anticlockwise and estimates four angles for four channel blocks.
- Unified framework: ASDN and ASTN can be trained jointly: ASDN first drops activations after RoI pooling, then ASTN deforms the modified features, improving detector robustness.The two adversarial networks are learned simultaneously with Fast-RCNN rather than using fixed generation policies.
5. Experiments
Experiments evaluate adversarial occlusion and deformation generation across architectures, datasets, ablations, categories, and comparison with OHEM. Jointly learning the adversarial networks with Fast-RCNN improves detection performance over the corresponding baselines, while also revealing category-specific gains and over-generalization failures.
- Experimental setup: Experiments use PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO, with most ablations conducted on VOC 2007 and a comparison against OHEM.VOC uses trainval for training and test for evaluation; COCO uses trainval35k and minival.
- PASCAL VOC 2007 results: 2.3% mAP boost: the VGG16 full model reaches 71.4% mAP versus 69.1% for the baseline on VOC 2007.ASTN reaches 69.9%, ASDN 71.0%, and their combination 71.4%.
- PASCAL VOC 2007 results: Adversarial training improves Fast-RCNN with ResNet-101 from 71.8% mAP to 73.6% mAP.The reported improvement is observed with a substantially deeper architecture as well as AlexNet and VGG16.
- Category-based analysis: The combined adversarial processes help some categories, including car and aeroplane, but adversarial training can create false positives through over-generalization toward other object categories.The category analysis reports that some classes helped by the combined transformations were hurt by occlusion or deformation alone.
6. Conclusion
The paper argues that rare occlusions and deformations limit dataset-based invariance learning and proposes adversarial feature modification to generate difficult examples. Experiments report significant detection improvements on VOC and COCO.
- Rare occlusions and deformations may remain uncovered even in large-scale datasets because they follow a long-tail distribution.
- The proposed adversarial network jointly learns with the detector, modifying features to mimic occlusions and deformations that hinder classification.
- Adversarial learning provides a significant boost in detection performance on VOC and COCO datasets.