Source-linked AI summary
Adversarial Complementary Learning for Weakly Supervised Object Localization
Xiaolin Zhang, Yunchao Wei, Jiashi Feng, Yi Yang, Thomas Huang
TL;DR
Weakly supervised object localization must identify integral objects from image-level labels, despite classifiers tending to focus on sparse discriminative parts. ACoL proves that localization maps can be selected from last-layer feature maps and uses two adversarial classifiers with feature erasing to discover complementary regions end to end. The method reports Top-1 localization error of 45.14% on ILSVRC and is presented as state of the art.
Problem
Weakly supervised object localization seeks object locations from image-level labels, but classification networks often highlight sparse discriminative parts instead of integral target objects.
Method
ACoL jointly trains two adversarial classifiers, using one classifier's localization to erase feature-map regions so the counterpart discovers complementary object regions during the forward pass.
Results
Top-1 localization error is 45.14% and Top-5 localization error is 30.03% on ILSVRC 2016 in the weakly supervised setting.
Takeaways & Limitations
ACoL produces integral object localization through end-to-end joint training, with localization maps obtained directly from selected class-specific feature maps.
Takeaways & Limitations
Adversarial erasing requires several independently trained classification networks, increasing training time and computing-resource requirements.
Abstract
from arXiv · showhide
In this work, we propose Adversarial Complementary Learning (ACoL) to automatically localize integral objects of semantic interest with weak supervision. We first mathematically prove that class localization maps can be obtained by directly selecting the class-specific feature maps of the last convolutional layer, which paves a simple way to identify object regions. We then present a simple network architecture including two parallel-classifiers for object localization. Specifically, we leverage one classification branch to dynamically localize some discriminative object regions during the forward pass. Although it is usually responsive to sparse parts of the target objects, this classifier can drive the counterpart classifier to discover new and complementary object regions by erasing its discovered regions from the feature maps. With such an adversarial learning, the two parallel-classifiers are forced to leverage complementary object regions for classification and can finally generate integral object localization together. The merits of ACoL are mainly two-fold: 1) it can be trained in an end-to-end manner; 2) dynamically erasing enables the counterpart classifier to discover complementary object regions more effectively. We demonstrate the superiority of our ACoL approach in a variety of experiments. In particular, the Top-1 localization error rate on the ILSVRC dataset is 45.14%, which is the new state-of-the-art.
1. Introduction
Weakly supervised object localization uses image-level labels to avoid bounding-box annotation, but conventional classifiers often focus on sparse discriminative parts rather than integral objects. ACoL addresses this with jointly trained adversarial classifiers that erase discovered regions and mine complementary ones during a single forward pass.
- Motivation: WSOL learns object locations from image-level labels, avoiding the expensive bounding-box annotations required by fully supervised methods.This reduces annotation labor compared with fully supervised counterparts.
- Motivation: Classification-based localization often over-relies on category-specific discriminative features and fails to densely localize integral target-object regions.Such networks may recognize a cat from its head without covering the entire object.
- Prior limitations: Adversarial erasing discovers integral regions by training additional classifiers on images with discriminative regions erased, but requires several independent networks.Randomly hiding input patches also lacks high-level guidance and cannot guarantee discovery of new object regions.
- ACoL method: ACoL uses two adversarial classifiers in one end-to-end network: one identifies discriminative regions, while the other classifies erased features to discover complementary regions.The resulting localization maps are fused to obtain integral target regions.
- ACoL method: ACoL obtains localization maps by selecting class-specific feature maps during the forward pass, rather than using post-inference processing.This supports a simpler end-to-end training procedure.
- Results: Top-1 localization error is 45.14% and Top-5 localization error is 30.03% on ILSVRC 2016 in the weakly supervised setting.The paper reports these as state-of-the-art results.
2. Related Work
Related work develops cheaper weakly supervised detection and localization from image-level supervision, alongside fully supervised detection and weakly supervised segmentation approaches. Prior methods use classification, pooling, masking, clustering, external location annotations, or segmentation-specific objectives.
- Fully supervised detection: Fully supervised detection includes one-stage multiscale or sliding-window methods such as OverFeat, SSD, and YOLO, as well as two-stage Faster-RCNN.These approaches predict object boundaries or bounding boxes with architectures designed for detection efficiency or accuracy.
- Weakly supervised detection and localization: Weakly supervised detection and localization use image-level supervision as a cheaper alternative to bounding-box annotation.Examples include multi-label classification with max-pooling MIL for coarse localization and classification-score analysis with masked regions.
- Weakly supervised detection and localization: Other weakly supervised localization methods generate self-taught hypotheses through clustering or adapt generic object features using extra images with location annotations.These methods differ in whether they rely on masked classification scores, clustering, or additional annotated images.
- Weakly supervised segmentation: Weakly supervised segmentation predicts pixel-level labels using methods such as progressive annotation learning and SEC's seeding, expansion, and boundary-constrain losses.These approaches target segmentation rather than only coarse object localization.
3. Adversarial Complementary Learning
ACoL combines a proven direct feature-map method for localization with two adversarial classifiers that erase and recover complementary object regions. The branches are trained jointly and their maps are fused into an integral localization map.
- 3.1. Revisiting CAM: Directly selecting class-specific feature maps from the last convolutional layer produces localization maps with the same quality as CAM during the forward pass.The approach avoids CAM's separate post-processing step while preserving comparable maps.
- 3.1. Revisiting CAM: A 1×1 convolution followed by global average pooling and softmax provides the class-specific feature maps used for localization.The convolutional output maps are aligned with target classes, enabling direct map extraction.
- 3.2. The proposed ACoL: The two classifiers discover different but complementary object regions, whose normalized localization maps are fused to obtain integral target regions.The fused map is produced after both branches generate maps from their respective feature inputs.
- 3.2. The proposed ACoL: ACoL uses Backbone, Classifier A, and Classifier B; Classifier A identifies discriminative regions, while Classifier B receives features with those regions erased.Erasure replaces the identified feature values with zeros after thresholding Classifier A's localization map.
- 3.2. The proposed ACoL: The complete ACoL procedure is trained end-to-end with both classifiers optimized using cross-entropy loss.Training updates the Backbone and both classifier parameter sets through back-propagation.
- 3.2. The proposed ACoL: At test time, fused maps are resized and thresholded to produce bounding boxes covering the largest connected foreground area.The bounding-box procedure follows the same strategy used in CAM-based comparison.
4. Experiments
Experiments evaluate ACoL on ILSVRC and CUB-200-2011 for classification and localization. ACoL improves localization over baselines, with complementary-region discovery supported by threshold and branch ablations.
- Experiment setup: ACoL is evaluated on ILSVRC 2016 and CUB-200-2011 using classification and localization accuracy, with localization measured by bounding-box IoU exceeding 50%.ILSVRC provides 1.2 million training images and 50,000 validation images; CUB-200-2011 provides 5,994 training and 5,794 test images.
- Classification: The modified ACoL networks achieve slightly better classification results than GAP variants and remain comparable to the original GoogLeNet and VGGnet.The comparison covers GoogLeNet-ACoL, VGGnet-ACoL, GoogLeNet-GAP, and VGGnet-GAP.
- Classification: 28.1% is the lowest classification error achieved by VGGnet-ACoL among methods without bounding-box annotations on CUB-200-2011.The comparison is reported in Table 2 for the fine-grained recognition dataset.
- Localization: ACoL outperforms all baselines on ILSVRC localization and improves GoogLeNet-GAP by 4.92% in Top-1 error on CUB-200-2011.VGGnet-ACoL is better than VGGnet-GAP, and GoogLeNet-ACoL is better than GoogLeNet-HaS-32.
- Localization: 45.14% Top-1 error and 30.03% Top-5 error are obtained on ILSVRC after combining ACoL localization with ensemble DPN classification and top-three predictions, respectively.The corresponding Top-5 error before top-three selection is 38.45%.
- Ablation study: The best ILSVRC Top-1 localization error occurs at erasing threshold δ = 0.6; larger or smaller thresholds perform worse.A large threshold may fail to encourage useful regions, whereas a small threshold may introduce background noise.
- Ablation study: Adding a third classifier produces no significant classification or localization improvement, indicating that two branches are usually sufficient.The third classifier erases feature maps using fused maps from Classifiers A and B.
- Ablation study: 37.04% Top-1 localization error is achieved using ground-truth labels, surpassing other approaches and supporting complementary-region localization.This evaluation removes the influence of classification results.
5. Conclusion
ACoL uses a mathematically supported feature-map selection strategy and two adversarial classifiers to locate complementary object parts under weak supervision. Experiments show that it mines integral regions and outperforms state-of-the-art localization methods.
- 5. Conclusion: The paper proves that object localization maps can be obtained by selecting from feature maps.This provides theoretical support for identifying object regions during the forward pass.
- 5. Conclusion: ACoL uses two adversarial classifiers to locate different object parts and discover complementary regions belonging to the same objects or categories.The method is trained in a weakly supervised manner.
- 5. Conclusion: Extensive experiments show that ACoL mines integral object regions and outperforms state-of-the-art localization methods.The conclusion summarizes the reported empirical findings.