Source-linked AI summary
Attentive Contexts for Object Detection
Jianan Li, Yunchao Wei, Xiaodan Liang, Jian Dong, Tingfa Xu, Jiashi Feng, Shuicheng Yan
TL;DR
Object detectors often underuse global and local context beyond proposal interiors, motivating AC-CNN's integration of both forms of contextual information into region-based detection. Its attention-based recurrent and multi-scale contextualized subnetworks improve Fast-RCNN performance on PASCAL VOC benchmarks.
Problem
Existing object detectors often use only information within proposals, leaving useful global and surrounding local context insufficiently exploited for challenging objects.
Method
AC-CNN integrates an attention-based recurrent global contextualized sub-network and a multi-scale local contextualized sub-network into a region-based CNN detector.
Results
2.0% and 2.2% mAP gains over Fast-RCNN were reported on VOC 2007 and VOC 2012, respectively.
Takeaways & Limitations
Attention-guided global context and multi-scale local context provide a practical way to contextualize region-based object detectors.
Abstract
from arXiv · showhide
Modern deep neural network based object detection methods typically classify candidate proposals using their interior features. However, global and local surrounding contexts that are believed to be valuable for object detection are not fully exploited by existing methods yet. In this work, we take a step towards understanding what is a robust practice to extract and utilize contextual information to facilitate object detection in practice. Specifically, we consider the following two questions: "how to identify useful global contextual information for detecting a certain object?" and "how to exploit local context surrounding a proposal for better inferring its contents?". We provide preliminary answers to these questions through developing a novel Attention to Context Convolution Neural Network (AC-CNN) based object detection model. AC-CNN effectively incorporates global and local contextual information into the region-based CNN (e.g. Fast RCNN) detection model and provides better object detection performance. It consists of one attention-based global contextualized (AGC) sub-network and one multi-scale local contextualized (MLC) sub-network. To capture global context, the AGC sub-network recurrently generates an attention map for an input image to highlight useful global contextual locations, through multiple stacked Long Short-Term Memory (LSTM) layers. For capturing surrounding local context, the MLC sub-network exploits both the inside and outside contextual information of each specific proposal at multiple scales. The global and local context are then fused together for making the final decision for detection. Extensive experiments on PASCAL VOC 2007 and VOC 2012 well demonstrate the superiority of the proposed AC-CNN over well-established baselines. In particular, AC-CNN outperforms the popular Fast-RCNN by 2.0% and 2.2% on VOC 2007 and VOC 2012 in terms of mAP, respectively.
1 Introduction
Existing region-based detectors often rely on proposal interiors, while AC-CNN integrates attended global context and multi-scale local context to improve object detection. Experiments on PASCAL VOC show gains over Fast-RCNN.
- Region-based CNN detectors typically classify proposals using interior information, which can be insufficient for low-resolution, small-scale, or heavily occluded objects.
- Global context can provide useful co-occurrence cues, but irrelevant background may introduce noise and hurt detection performance.
- AC-CNN contextualizes region-based detectors by combining attention-based global context with multi-scale inside and outside local context.
- The multi-scale contextualized sub-network captures inside and outside information around each proposal to enhance its feature representation.
- The attention-based recurrent sub-network identifies discriminative image locations and combines their feature maps to produce global contextual features for proposal recognition.
- 2.0% and 2.2% mAP gains over Fast-RCNN were reported on VOC 2007 and VOC 2012, respectively.
2 Related Work
Related work established region-based CNNs as a standard object-detection pipeline and explored contextual information and LSTM-based attention in visual recognition.
- R-CNN detects objects by applying a deep CNN to classify region proposals, while Fast R-CNN and Faster R-CNN improve detection accuracy and computational efficiency.
- The region-based detection pipeline uses RoI pooling to extract proposal features, followed by joint localization and classification through multi-task loss functions.
- LSTM-based visual-attention methods had been applied to tasks including image captioning, video description, people detection, and action recognition.
3 Attention to Context Convolution Neural Network (AC-CNN)
AC-CNN extends the Fast-RCNN framework with multi-scale local context and attention-based global context. These contextual features are used jointly for classification, while local context alone supports bounding box regression.
- 3 Attention to Context Convolution Neural Network (AC-CNN): AC-CNN builds on VGG-16 and Fast-RCNN, adding multi-scale local and attention-based global context sub-networks.The image first produces a convolutional feature cube, which feeds both context-aware components.
- 3.2 Attention-based Contextualized Sub-network: The global-context branch uses three stacked LSTM layers to recurrently generate an attention map over feature locations.The map assigns weighted probabilities to locations, allowing attended features to be computed from feature slices.
- 3.2 Attention-based Contextualized Sub-network: The attention map selectively combines all locations into a global attention-based feature, which is transformed into FG for proposal-level detection.The global feature passes through two fully connected layers after attention-based pooling.
- 3.3 Learning with Multi-task Loss Function: AC-CNN concatenates local and global features for classification and uses only the local feature for bounding box regression.The joint loss combines classification and regression objectives, with the regression term applied to positive proposals.
4 Experimental Results
Experiments on VOC 2007 and VOC 2012 show that AC-CNN improves Fast-RCNN detection by combining attention-based global context with multi-scale local context. Ablations indicate that both contextual subnetworks and selected scale settings contribute to performance.
- Global Context Attention Method: The attention-based recurrent global-context model outperforms average pooling, whose use decreases performance by 0.4%.The recurrent model highlights positively correlated regions while suppressing noisy background context.
- Contributions of Each Sub-network: Removing either the global-context or multi-scale local-context subnetwork reduces performance by 0.6%.AC-CNN minus L uses only attention-based global context, whereas AC-CNN minus G uses only multi-scale local context.
- Effectiveness of Multi-scale Setting: Removing either additional scale 0.8 or 1.8 decreases detection performance, while adding scale 2.7 provides further improvement.The selected 0.8+1.2+1.8 setting balances detection accuracy with computational consumption of time and GPU memory.
- No Global Context for Bounding Box Regression: Removing global context from bounding box regression improves performance by 0.1%.The model therefore uses concatenated local and global features for detection classification but not for bounding box regression.
- Detection Error Analysis: AC-CNN reduces false positives for challenging categories, with 5% improvements for both bottle and pottedplant.The error analysis attributes these gains to improved handling of small objects and examines localization and category-confusion errors.
5 Conclusion
AC-CNN contextualizes region-based object detectors with attention-based global and multi-scale local subnetworks. Experiments on VOC 2007 and VOC 2012 demonstrate significant improvement from exploiting contextual information.
- AC-CNN integrates stacked-LSTM global attention with three-scale inside-and-outside local context in a unified detector.
- Experiments on VOC 2007 and VOC 2012 show significant detection improvement from exploiting contextual information.