Object detection via a multi-region & semantic segmentation-aware CNN model

Spyros Gidaris; Nikos Komodakis

Source-linked AI summary

Object detection via a multi-region & semantic segmentation-aware CNN model

Spyros Gidaris, Nikos Komodakis

arXiv:1505.01749v3May 7, 2015cs.CVcs.LGcs.NE

TL;DR

Object detection needs representations that capture diverse object appearance while supporting accurate localization. The paper proposes a multi-region CNN with weakly supervised segmentation-aware features and iterative CNN-based box refinement, achieving 78.2% mAP on VOC2007 and 73.9% on VOC2012. The approach surpasses the reported previous state of the art, although adjacent-object detection and missing dataset annotations remain limitations.

Problem
The paper addresses object representation and accurate object localization as two critical aspects of object detection.
Method
A multi-region CNN learns diverse, localization-sensitive features, including weakly supervised semantic segmentation-aware features, and combines them with iterative CNN bounding-box regression.
Results
78.2% mAP on VOC2007 and 73.9% mAP on VOC2012 surpassed the previous state of the art by a significant margin.
Takeaways & Limitations
The combined representation and localization refinement scheme achieves excellent object-detection results with high localization accuracy.
Takeaways & Limitations
Accurately detecting multiple adjacent object instances remains difficult, and some correctly detected instances are absent from VOC2007 ground-truth annotations.

Abstract

from arXiv · show

We propose an object detection system that relies on a multi-region deep convolutional neural network (CNN) that also encodes semantic segmentation-aware features. The resulting CNN-based representation aims at capturing a diverse set of discriminative appearance factors and exhibits localization sensitivity that is essential for accurate object localization. We exploit the above properties of our recognition module by integrating it on an iterative localization mechanism that alternates between scoring a box proposal and refining its location with a deep CNN regression model. Thanks to the efficient use of our modules, we detect objects with very high localization accuracy. On the detection challenges of PASCAL VOC2007 and PASCAL VOC2012 we achieve mAP of 78.2% and 73.9% correspondingly, surpassing any other published work by a significant margin.

1. Introduction

The paper targets object detection by enriching object representation and improving localization. It proposes a multi-region CNN with segmentation-aware features and iterative box refinement, achieving strong PASCAL VOC results.

Object detection requires returning tightly enclosing bounding boxes for all instances of specified object types.
The paper addresses two critical aspects of detection: object representation and accurate object localization.
The multi-region CNN diversifies appearance cues by focusing separate components on object appearance, parts, context, boundaries, and semantics.
The representation is designed to remain sensitive to inaccurate localization, countering CNN invariances associated with max-pooling.
Segmentation-aware CNN features are jointly learned weakly supervised, without requiring ground-truth object segmentations or additional annotation.
The localization system alternates candidate-box scoring with coordinate refinement using CNN recognition and bounding-box regression models.
78.2% mAP on VOC2007 and 73.9% mAP on VOC2012 surpassed the previous state of the art by a significant margin.

2. Related Work

The paper builds on deep CNN detection systems and related contextual, segmentation-based, and SPP-Net approaches. It distinguishes itself through multiple region-specific features, unified segmentation-aware learning, deep box regression, and box voting.

Compared with conceptually similar work, this approach diversifies appearance factors across multiple regions and integrates CNN-based segmentation-aware features in one unified architecture.
The method further uses deep CNN bounding-box regression and box voting after non-maximum suppression.
The Multi-Region CNN architecture presents multiple regions and can be extended with activation-map and region-adaptation modules for segmentation-aware features.
Contemporary SPP-Net methods modify network depth, simplify training, use multi-task objectives, or add class-independent proposal prediction.

3. Multi-Region CNN Model

The Multi-Region CNN analyzes candidate boxes through dedicated components focused on complementary object, border, central, and contextual regions. Concatenating these region-specific features yields a richer representation intended to diversify discrimination and improve localization sensitivity.

Architecture: The Multi-Region CNN assigns dedicated network components to different regions of a candidate object.The architecture first computes image activation maps, then adapts pooled activations from regions defined relative to candidate box B.
Architecture: Each region adaptation module projects its region onto activation maps, pools the enclosed activations, and produces a high-level feature whose outputs are concatenated.This concatenation forms the candidate-box representation.
Role on detection: The model aims to diversify discriminative factors by limiting each component’s visual field to a region or its immediate surroundings.The intended result is a richer and more robust representation based on complementary appearance cues.
Region components: The region set includes the original box, half boxes, central regions, border rings, and a contextual ring surrounding the candidate box.These shapes target whole-object appearance, object parts, central content, boundaries, and nearby context.
Role on detection: 64.1% mAP versus 62.9% mAP was obtained when a rectangular border ring replaced a same-sized normal box, supporting the border-region design.The comparison was conducted on PASCAL VOC2007 trainval training and test evaluation.
Role on detection: The architecture is designed to remain sensitive to inaccurate localization, particularly through regions focused on object borders.This addresses localization errors that can persist despite strong classification capability.

4. Semantic Segmentation-Aware CNN Model

The model extends the Multi-Region CNN with semantic segmentation-aware features learned jointly from detection data. It uses adapted activation-map and region-adaptation modules, trained weakly from bounding boxes rather than additional segmentation annotations.

Model extension: The extended model adds semantic segmentation-aware CNN features to diversify the features encoded by the Multi-Region CNN.The extension is motivated by the connection between segmentation and detection and by reported usefulness of segmentation cues for detection.
Model extension: Semantic segmentation-aware features use adapted activation-maps and region-adaptation modules within the unified architecture.The two module types are added as counterparts to the original Multi-Region CNN components.
Feature fusion: The semantic segmentation-aware and Multi-Region CNN features are concatenated so the network jointly learns both feature types during training.This combines appearance-focused and segmentation-aware representations in the resulting network.
Activation maps: The activation-maps module is an FCN trained to predict class-specific foreground probabilities.After training, its final classification layer is removed and the remaining network extracts semantic segmentation-aware activation maps.
Weak supervision: The FCN is trained weakly using artificial foreground masks generated from detection bounding boxes, without additional segmentation annotations.Bounding boxes are projected onto the FCN’s last-hidden-layer spatial domain to create training targets.
Region adaptation: A single region enlarged by a factor of 1.5 adapts the semantic activation maps for detection, avoiding repeated appearance regions for efficiency.The enlarged region includes semantic information from around the candidate box.

5. Object Localization

The localization scheme iteratively scores candidate boxes and refines their coordinates with CNN regression, then merges results and applies suppression and voting. This produces candidate sets that jointly maintain high object recall and improved localization.

CNN region adaptation module for bounding box regression: An additional region adaptation module predicts bounding boxes from shared activation maps rather than serving object recognition directly.Enlarging each candidate box by a factor of 1.3 helps predict instances farther from initial candidates and improves prediction accuracy.
Iterative Localization: The recognition model scores candidate boxes while a CNN regression model refines their coordinates across successive localization iterations.The scheme starts from selective search proposals and forms candidate detections by scoring and regressing each iteration's boxes.
Iterative Localization: Boxes scoring below threshold τs are rejected to reduce the computational burden of subsequent iterations.The retained detection sets are designed to preserve high object recall while improving localization.
Bounding box voting: Candidate detections from all iterations are merged before final suppression and box voting.Multiple regression steps concentrate generated boxes around objects, enabling overlapping boxes to vote on final coordinates.
Bounding box voting: Non-maximum suppression uses a 0.3 IoU threshold, while voting neighbors overlap the detection by more than 0.5 IoU and contribute with nonnegative score weights.The final bounding-box coordinates are refined from the overlapping candidate detections.

6. Implementation Details

The system uses VGG-based CNN components with shared convolutional features and separately adapted region modules for recognition, segmentation-aware representation, and bounding-box regression. Training uses ImageNet initialization, selective fine-tuning, and proposal-based SVM supervision, while multiscale processing and feature caching support the implementation.

Model setup: All CNN models use publicly available 16-layer VGG models, with convolutional layers shared across the system's models.Only the fully connected layers are fine-tuned, while convolutional weights are preserved from ImageNet pretraining.
Training/Test Time: In practice, two localization iterations were sufficient for convergence, and τs = −2.1 retained about 250 proposals per image across classes.The threshold was selected using the average number of proposals per image as the criterion.
Multi-Region CNN model: The Multi-Region CNN uses convolutional VGG layers producing 512 channels and separately fine-tuned fully connected region adaptation modules.The max-pooling layer after the final convolution is omitted, and rectangular-ring regions use inner and outer boxes.
Semantic Segmentation-Aware CNN model: The semantic segmentation-aware CNN reshapes VGG fully connected layers into convolutional layers and reduces fc7 output channels from 4096 to 512.Its region adaptation module uses spatially adaptive max-pooling on a 9 × 9 grid followed by a fully connected layer.
Classification SVMs: Classification SVMs treat ground-truth boxes as positives and selective-search proposals with IoU below 0.3 as negatives, using hard-negative mining.This follows the stated training principles for the classification SVMs.
Multiscale processing: The models use multiscale image processing, with seven scales for the Multi-Region and bounding-box-regression CNNs and three for the segmentation-aware CNN.At testing, each model selects a single scale based on the target region's area.
Training/Test Time: Feature caching stores conv5_3 and fc7 segmentation-aware features on an SSD, while reported per-image runtime is around 30 seconds.On a Titan GPU, training the region adaptation modules, segmentation activation maps, and linear SVM takes approximately 12 hours, 4 days, and 16 hours, respectively.

7. Experimental Evaluation

The system improves object detection through multi-region and segmentation-aware representations, iterative localization, and evaluation of detection and localization performance on PASCAL VOC benchmarks.

Module evaluation: 66.2% mAP from MR-CNN exceeds R-CNN with VGG-Net at 62.0% and the Original candidate box region alone at 61.7%.MR-CNN also slightly exceeds R-CNN with VGG-Net and bounding box regression at 66.0%.
Module evaluation: 67.5% mAP is reached by adding semantic segmentation-aware CNN features to the Multi-Region CNN model.The segmentation-aware features add 1.3 points over MR-CNN.
Iterative localization: 74.9% mAP results when the recognition model is coupled with CNN bounding box regression under the iterative localization scheme.Performance rises from 67.5% to 74.9% on VOC2007 train+val training data.
Localization evaluation: At an IoU threshold of 0.7, the overall system exceeds the compared approach, while the experiments assess localization capability.The same evaluation reports localization performance for the proposed modules and overall system.
Error analysis: The error analysis categorizes false positives as poor localization, similar-object confusion, other-object confusion, or background or unlabelled-object triggers.Figure 8 compares these error categories across the baseline, MR-CNN without segmentation-aware features, and the overall system.
PASCAL VOC2012: VOC2012 results reach 69.1% and 70.7% mAP for two training configurations, setting the state of the art on that test set for those training sets.The overall system combines MR-CNN, segmentation-aware features, and iterative CNN-based bounding box regression.
Extra-data comparison: With extra data, the methods rank first and second on the VOC2012 comp4 leaderboard, with the top entry 3.5 points ahead of the third.The comparison uses selective-search and EdgeBox proposals for training and EdgeBox proposals at test time.

8. Qualitative Results

Qualitative results illustrate detections and identify remaining difficulties, especially accurately detecting multiple adjacent object instances and handling missing annotations.

Detection visualizations: The qualitative figures mark true positives with blue boxes, false positives with red boxes, and ground truth boxes with green.These color conventions are used for detections shown in figures 12–14.
Failure cases: Accurately detecting multiple adjacent object instances remains difficult for the approach in many cases.Figures 9 and 10 present examples of this failure type and other failure cases.
Missing annotations: Some object instances are correctly detected but absent from the PASCAL VOC2007 ground-truth annotations.Figure 11 presents examples of these nonannotated instances.

9. Conclusions

The paper concludes that its object-detection representation combines diversified regional appearance factors with semantic segmentation-aware features and performs strongly with iterative localization refinement.

Conclusion: The proposed representation diversifies discriminative appearance factors across object regions and encodes semantic segmentation-aware features.The representation is used with a CNN-based localization refinement scheme.
Conclusion: Combined with CNN-based localization refinement, the representation achieves excellent results that surpass the state of the art by a significant margin.