Source-linked AI summary
Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks
Sean Bell, C. Lawrence Zitnick, Kavita Bala, Ross Girshick
TL;DR
Object detectors need both fine-grained object details and surrounding context, whereas prior approaches mainly use information near the region of interest. ION combines multi-scale skip pooling inside the ROI with spatial RNN context outside it, achieving state-of-the-art results across PASCAL VOC and COCO and particularly improving small-object detection.
Problem
Prior detection approaches mainly use information near the region of interest, despite the need for both fine-grained object details and surrounding context.
Method
ION combines multi-scale skip pooling from lower-level convolutional layers with spatial RNNs that propagate contextual information horizontally and vertically across images.
Results
ION achieves state-of-the-art results on PASCAL VOC and COCO and improves detection particularly for small objects.
Takeaways & Limitations
Context and multi-scale features are complementary, with improvements especially significant for small objects and other historically difficult object types.
Takeaways & Limitations
The architecture preserves existing VGG16 layer shapes to use ImageNet pre-training.
Abstract
from arXiv · showhide
It is well known that contextual and multi-scale representations are important for accurate visual recognition. In this paper we present the Inside-Outside Net (ION), an object detector that exploits information both inside and outside the region of interest. Contextual information outside the region of interest is integrated using spatial recurrent neural networks. Inside, we use skip pooling to extract information at multiple scales and levels of abstraction. Through extensive experiments we evaluate the design space and provide readers with an overview of what tricks of the trade are important. ION improves state-of-the-art on PASCAL VOC 2012 object detection from 73.9% to 76.4% mAP. On the new and more challenging MS COCO dataset, we improve state-of-art-the from 19.7% to 33.1% mAP. In the 2015 MS COCO Detection Challenge, our ION model won the Best Student Entry and finished 3rd place overall. As intuition suggests, our detection results provide strong evidence that context and multi-scale representations improve small object detection.
1. Introduction
ION combines contextual information outside each region of interest with multi-scale features inside it to improve object detection, especially for difficult and small objects.
- Current detectors often rely only on information near an object’s region of interest, limiting the types and accuracy of detectable objects.
- Multi-scale skip pooling combines lower-level convolutional features across spatial resolutions and abstraction levels, preserving fine-grained information important for small objects.
- Spatial RNNs propagate spatially varying contextual information horizontally and vertically across the image, with at least two layers enabling image-wide propagation.
- ION classifies thousands of candidate regions by concatenating multi-scale and contextual features before fully connected classification layers.
- Context and multi-scale features are complementary, with gains especially pronounced for historically difficult categories, small objects, and heavily occluded chairs.
- 79.2% mAP on PASCAL VOC 2007, 76.4% mAP on VOC 2012, and 24.9% mAP on COCO establish state-of-the-art results across three datasets.
2. Prior work
ION builds on region-based ConvNet detection and combines established spatial RNN and skip-layer ideas with multi-scale, context-aware representations.
- ION extends the R-CNN line of region-based ConvNet detectors, using Fast R-CNN as its experimental foundation.
- Spatial RNNs run laterally over feature maps as a lower-complexity alternative to convolutions, supporting tasks such as semantic segmentation and image classification.
- Skip-layer connections route lower-layer activations directly to higher layers, and ION L2-normalizes activations before combining them.
- Prior models also use skip connections for semantic segmentation and instance segmentation, including fully convolutional networks and hypercolumn features.
- ION’s four-directional IRNN architecture uses ReLU recurrent transitions initialized to the identity and 1x1 convolutions for efficient recurrence.
3. Architecture: Inside-Outside Net (ION)
ION builds an object descriptor from multi-layer ROI features and spatial context computed by stacked four-directional IRNNs. The architecture combines skip pooling, normalization, and contextual recurrence, with experiments examining simplifications and auxiliary segmentation supervision.
- Architecture: ION processes each image with one deep ConvNet, stores convolutional feature maps, and computes context features once using stacked four-directional IRNNs.The context features match the dimensions of conv5 and are later combined with ROI features.
- Multi-scale ROI features: Multi-layer ROI pooling combines features across spatial resolutions and abstraction levels, then concatenates and reduces them to the shape required by fc6.Pooled features are concatenated along channels and reduced with a 1x1 convolution.
- Variants and simplifications: The design explores ReLU IRNNs initialized with identity recurrent matrices, efficient recurrent updates, dropout, and semantic-segmentation supervision.Segmentation supervision adds an auxiliary softmax loss, while removing the recurrent matrix has a surprisingly small impact.
- Context features with IRNNs: Four IRNNs move right, left, down, and up across conv5, producing direction-specific hidden states concatenated at each spatial location.The input-to-hidden transition is implemented as a shared 1x1 convolution, removing six convolutional layers with a negligible −0.1 mAP effect.
- Context features with IRNNs: The first four-directional IRNN summarizes nearby features around each position, while the second makes every output cell depend on every input cell.This yields context features that are both spatially varying and global.
4. Results
ION achieves strong detection results across PASCAL VOC and MS COCO, with especially large gains for small objects. Experiments examine architectural choices, training settings, contextual features, and bounding-box refinement.
- PASCAL VOC 2007: 79.2% mAP is achieved on VOC 2007 test by the best ION model.The reported result uses the best model configuration.
- PASCAL VOC 2012: 76.4% mAP is obtained on the more challenging VOC 2012 test set, exceeding the next best submission by several points.The detector is reported as most accurate for most categories.
- MS COCO: 24.9% mAP is obtained on COCO, improving over the 20.5% mAP Fast R-CNN baseline trained with the same images and schedule.The cited baseline is the authors’ re-trained Fast R-CNN model.
- MS COCO: 31.0% test-competition and 31.2% test-dev scores are achieved by the single-model competition submission, later rising to 33.1% test-dev.The submission won Best Student Entry and placed third overall; the later improvement used left-right flipping and adjusted training parameters.
- Object-size analysis: Small-object COCO average precision rises from 4.1% to 7.0%, while average recall rises from 7.3% to 10.7%.The paper also reports similarly large size-based improvements on VOC 2007.
5. Design evaluation
The design evaluation tests normalization, segmentation supervision, contextual mechanisms, and IRNN configurations. It identifies a two-layer, four-directional IRNN with 512 hidden units as the preferred context design, while segmentation supervision adds about one mAP point without test-time cost.
- Multi-scale feature combination: Naively concatenating ROI features from different layers fails because their feature amplitudes differ substantially, causing unstable learning.The evaluation therefore considers normalization before combining multi-layer ROI features.
- Segmentation loss: About +1 mAP results from adding semantic segmentation loss to the IRNN training setup, with no change in test-time runtime.The segmentation output is used only during training and can be removed at test time.
- Context mechanisms: Two-layer, four-directional IRNNs use fewer parameters and outperform global average pooling and additional convolutional-layer alternatives on VOC 2007.This architecture is selected for computing context features in the remaining experiments.
- IRNN architecture: Removing the recurrent Whh transition performs almost as well as learning it and saves a large matrix multiplication.The authors attribute the near-equivalence to sufficient context in the input-to-hidden and hidden-to-output connections.
- Other variations: Other IRNN and pooling variations perform about the same or no better than the main method.The evaluated variants include directional separation and pooling from additional IRNN or convolutional outputs.
6. Conclusion
ION improves object descriptions by combining context outside the ROI with normalized multi-layer ROI pooling inside it. Extensive design experiments support state-of-the-art performance on PASCAL VOC and COCO, particularly for small objects.
- Conclusion: ION combines a 2x stacked 4-directional IRNN for context with normalized multi-layer ROI pooling for object description.The architecture evaluates design choices including layer combinations, segmentation loss, feature normalization, and IRNN variants.
- Conclusion: ION achieves state-of-the-art results on both PASCAL VOC and COCO and is particularly effective at improving detection of small objects.The conclusion connects the reported benchmark gains with the architecture’s context and multi-scale design.
Appendix: 2015 MS COCO Competition
The 2015 MS COCO challenge submission used a single ConvNet with improved proposals, longer training, and refined localization. It won Best Student Entry, placed third overall, and later reached 33.1% test-dev mAP.
- Challenge submission: 31.0% mAP on 2015 test-challenge and 31.2% on 2015 test-dev earned ION Best Student Entry and third place overall.The challenge submission used a single ConvNet without ensembling.
- Challenge configuration: The challenge configuration combined MCG and RPN proposal boxes, added two 512x3x3 convolutional layers, trained longer, and used two rounds of box regression with weighted voting.These changes were introduced to improve the submitted system under the challenge schedule.
- Scope caveat: The authors note that many challenge choices were driven by the deadline and may therefore be suboptimal.This qualifies how the competition configuration should be interpreted.
- Proposal boxes: RPN1 and RPN2 achieved 44.1% and 44.3% average recall, respectively, versus 41.7% for selective search and 51.6% for MCG.Despite lower average recall than MCG, the RPN configurations produced comparable detection results to MCG.
Post-competition improvements
After the competition, the authors improved the model through longer training, flipped-image evaluation, and more proposal boxes. The resulting model ran at 5.5 seconds per image on a single Titan X GPU, excluding proposal generation.
- Longer training: +0.4 mAP on minival came from further training with conv1 and conv2 frozen.This adjustment used 80k additional iterations.
- Longer training: +0.7 mAP on minival came from another 160k iterations with no layers frozen.The reported gain followed the preceding additional training stage.
- Test-time augmentation: +0.8 mAP on minival came from averaging predictions from the original and left-right flipped images.The merge averages both softmax scores and box-regression shifts after flipping predictions back.
- Proposal boxes: Using 4000 proposal boxes performed +0.1 mAP better than using 3000 boxes for the competition model.The 4000 boxes comprised 2000 RPN2 and 2000 MCG proposals.
- Runtime: The post-competition model ran in 5.5 seconds per image on a single Titan X GPU, excluding proposal generation.Most of the slowdown came from left-right flipping.