Source-linked AI summary
ContextLocNet: Context-Aware Deep Network Models for Weakly Supervised Localization
Vadim Kantorov, Maxime Oquab, Minsu Cho, Ivan Laptev
TL;DR
Weakly supervised localization must infer object extents from image-level labels, while prior methods often focus on discriminative regions rather than precise boundaries. The paper introduces additive and contrastive context-aware CNN models built on ROI-based processing. On PASCAL VOC, the approach improves weakly supervised localization and detection, with contrastive S reaching 36.3% mAP, while limitations remain in multiple-object scenes and some context-design choices.
Problem
Weakly supervised localization must learn object extents and representations from image-level labels, yet prior approaches often fail to locate precise boundaries.
Method
The paper introduces additive and contrastive CNN guidance models that use surrounding context regions with ROI-based processing to refine localization.
Results
36.3% mAP: contrastive S outperforms previous WSL methods using selective search object proposals.
Takeaways & Limitations
Semantic contrast provides an effective cue for more accurate object boundaries and reduces sensitivity to shrinking detections to discriminative object parts.
Takeaways & Limitations
When many same-class objects occur close together, the method tends to detect them as a single object.
Abstract
from arXiv · showhide
We aim to localize objects in images using image-level supervision only. Previous approaches to this problem mainly focus on discriminative object regions and often fail to locate precise object boundaries. We address this problem by introducing two types of context-aware guidance models, additive and contrastive models, that leverage their surrounding context regions to improve localization. The additive model encourages the predicted object region to be supported by its surrounding context region. The contrastive model encourages the predicted object region to be outstanding from its surrounding context region. Our approach benefits from the recent success of convolutional neural networks for object recognition and extends Fast R-CNN to weakly supervised object localization. Extensive experimental evaluation on the PASCAL VOC 2007 and 2012 benchmarks shows hat our context-aware approach significantly improves weakly supervised localization and detection.
1 Introduction
Weakly supervised localization seeks object extents and representations from image-level labels, but remains less accurate than strongly supervised methods. ContextLocNet introduces additive and contrastive context guidance to refine localization boundaries.
- Weakly supervised object localization learns target-object representations and spatial extents using only image-level labels.
- Strong bounding-box or mask supervision is difficult to obtain at scale and can contain subjective annotation biases.
- WSL performance on standard benchmarks remains considerably lower than strongly supervised counterparts despite CNN-based progress.
- Additive guidance aggregates ROI and context class activations, while contrastive guidance computes their difference to support or separate candidate objects.
- ContextLocNet asks whether surrounding visual context can guide localization without supervision for object or context regions.
- The proposed CNN architectures build on ROI pooling and use context to refine detections, with additive guidance limiting boundary expansion and contrastive guidance limiting contraction to object parts.
2 Related Work
Related WSL methods commonly formulate localization as multiple instance learning and increasingly use CNN architectures or features. Contextual information has improved detection and some WSL systems, but remains less explored for weakly supervised localization.
- Many WSL methods localize bounding boxes with visually consistent appearance despite cluttered backgrounds, often using multiple instance learning.
- Recent WSL approaches use CNN architectures or features, but several promising methods are not trained end-to-end.
- CNN-based WSL methods include global max pooling, global average pooling, multi-fold MIL, semantic clustering, and background modeling.
- Contextual information is widely used for object detection but has received relatively little attention in weakly supervised or unsupervised localization.
- Prior context-based WSL work uses background descriptors or contextual-region alignment to improve modeling or discover visual object clusters.
3 Context-Aware Weakly Supervised Network
The network extends ROI-based CNNs with context and frame pooling, then uses additive or contrastive guidance in a localization stream alongside classification. These streams produce ROI-level class scores whose aggregation supports weakly supervised object localization.
- Overview: The architecture uses VGG-F convolutional features and ROI pooling, with separate classification and localization streams for each object proposal.The classification stream produces ROI class scores, while the localization stream processes ROI and context-region features.
- Convolutional and ROI Pooling Layers.: Context pooling extracts an outer region around each ROI, while frame pooling extracts an internal frame and produces feature maps shaped like context pooling.The external-to-internal side ratio for context and frame pooling is fixed at 1.8, and frame-shaped maps have zeros in the center.
- Two-Stream Network.: The two-stream model multiplies classification scores by softmaxed localization scores for each ROI and class, then sums ROI scores into image-level class scores.Training uses a hinge loss for multi-label image classification, normalized by the number of classes and examples.
- 3.2 Additive Model: The additive model combines independently transformed ROI and context features by summing their outputs to favor regions semantically compatible with surrounding context.Its context branch discourages small interior detections and large detections extending beyond object boundaries.
- 3.3 Contrastive Model: The contrastive model subtracts a shared context activation from an ROI or frame activation, encouraging detections that stand out from their surrounding context.The contrastive branches compensate for one another to localize object boundaries and discourage detections on object interiors.
- 3.3 Contrastive Model: The symmetric contrastive model replaces ROI pooling with frame pooling so shared localization layers process feature maps of the same shape.The classification stream retains the original ROI pooling in this variant.
4 Experimental Evaluation
The method is evaluated on PASCAL VOC 2007 and 2012 using detection mAP and localization CorLoc on standard dataset splits.
- PASCAL VOC 2007 contains 2,501 training, 2,510 validation, and 4,952 test images across 20 object classes.
- PASCAL VOC 2012 contains the same 20 object classes as VOC 2007 and is approximately twice larger for both splits.
- mAP measures detection using the standard intersection-over-union criterion, while CorLoc measures localization accuracy on a training set.
- CorLoc is computed per class on positive images and counts highest-scoring candidates overlapping a ground-truth box with IoU > 0.5.
- mAP and CorLoc are evaluated on the test and trainval splits, respectively.
Implementation Details.
The implementation uses Selective Search regions, Torch-based ROI pooling, a pretrained VGG-F network, and SGD training with specified schedules and augmentation.
- VOC 2007 and VOC 2012 use Selective Search windows, with ROI pooling implemented in Torch using Fast R-CNN-based code.
- The models use a VGG-F network ported to Torch and are trained with cuDNN on an NVIDIA Titan X GPU.
- The context-to-internal-rectangle side ratio is fixed to 1.8 for context and frame pooling types.
- Training uses SGD with momentum 0.9, batch size 1, 30 epochs, and learning rates of 10^-5 followed by 10^-6.
- Experiments retain Selective Search ROIs whose width and height exceed 20 pixels and apply scale jittering.
Parameters.
On VOC 2007, the proposed models are compared with weakly supervised baselines using CorLoc and detection mAP, with contrastive S achieving the strongest reported result.
- The VOC 2007 comparison evaluates recent weakly supervised object detection methods in Table 1.
- The original and reimplemented WSDDN-SSW-S results match up to 1%, indicating minor effects from their differing loss and softmax choices.
- 36.3% mAP is achieved by contrastive S, outperforming previous WSL methods using Selective Search proposals.
- Combining EdgeBox proposals, objectness rescaling, regularization, and ensembling improves a related result to 39.3%.
- The additive model improves localization and detection over the WSDDN-SSW-S* baseline.
Context Branch Helps.
Separate context processing improves weakly supervised localization and detection, while the contrastive S architecture performs better than contrastive A.
- The additive model reaches 33.3% mAP, surpassing the context-padding model’s 30.9% mAP.
- Contrastive A improves localization and detection but performs slightly worse than the additive model.
- Processing context in a separate branch helps localization in the weakly supervised setup.
- Contrastive S processes same-shaped frame and context feature maps, whereas contrastive A processes different-shaped ROI and context maps.
- Contrastive S greatly improves CorLoc and mAP over contrastive A.
Contrastive Model with Frame Pooling.
The contrastive S model is evaluated on VOC 2012, while the supplied table passages identify VOC 2007 comparisons and methodological observations. Additional pooling options did not significantly improve results, and some additive-model changes degraded performance.
- Contrastive Model with Frame Pooling.: Contrastive S results on VOC 2012 are summarized in detection AP and CorLoc tables.The supplied passages identify Tables 4 and 5 for these results but provide no numerical values.
- Contrastive Model with Frame Pooling.: Additional region-pooling types did not significantly improve the reported results.The authors also note that alternative context pooling may provide improvements.
- Contrastive Model with Frame Pooling.: Sharing weights or replacing context pooling with frame pooling degraded additive-model performance.
Qualitative Results.
Qualitative examples indicate that the method more often localizes whole objects than prior methods that focus on discriminative parts, while multiple nearby same-class objects remain difficult to separate.
- Qualitative Results.: The method often finds whole person and animal extents, whereas previous methods tend to localize head regions.This pattern is reported for classes with localized discriminative parts.
- Qualitative Results.: When many same-class objects occur close together, the method tends to detect them as a single object.This remains a typical weakly supervised localization failure mode.
5 Conclusions
The paper presents context-aware deep models for weakly supervised localization that use semantic contrast to refine object boundaries. On VOC 2007 and 2012, the approach improves over baselines, while remaining complementary to other techniques.
- 5 Conclusions: The proposed localization subnetwork integrates visual contextual information to refine detected-object boundaries.
- 5 Conclusions: Semantic contrast is reported as an effective cue for obtaining more accurate object boundaries.
- 5 Conclusions: Qualitative results show reduced sensitivity to shrinking toward discriminative object parts.
- 5 Conclusions: The method demonstrates significant improvements over baselines on the VOC 2007 and 2012 benchmarks.
- 5 Conclusions: The approach is described as complementary to previously explored ideas and potentially combinable with other techniques.