PaperlayerView exact revision on arXiv

Source-linked AI summary

Crafting GBD-Net for Object Detection

Xingyu Zeng, Wanli Ouyang, Junjie Yan, Hongsheng Li, Tong Xiao, Kun Wang, Yu Liu, Yucong Zhou, Bin Yang, Zhe Wang, Hui Zhou, Xiaogang Wang

arXiv:1610.02579v1cs.CV

TL;DR

Candidate-box classification must combine local detail with surrounding context, since partial objects can be ambiguous and feature scales contribute complementary cues. GBD-Net passes gated messages between multi-resolution features. It was evaluated on three detection datasets, and on PASCAL VOC2007 it achieved 77.2% mAP versus 73.1% for BN+FRCN.

  • Problem

    Candidate boxes covering only parts of objects can have similar local cues, while local detail and surrounding context provide complementary information for classification.

    Source: §I · pp. 1–2

  • Method

    GBD-Net uses bidirectional convolutional message passing between multi-resolution support-region features and gates transmission according to the input sample.

    Source: §I · p. 2

  • Results

    GBD-Net was evaluated on ImageNet, PASCAL VOC2007, and Microsoft COCO; on VOC2007 it achieved 77.2% mAP versus 73.1% for BN+FRCN.

    Source: 1) PASCAL VOC2007 dataset · p. 8, §VI · p. 10

  • Takeaways & Limitations

    The proposed general layer design can be placed after convolutional layers and was validated on three object-detection datasets.

    Source: §VI · p. 10

Abstract

from arXiv · show

The visual cues from multiple support regions of different sizes and resolutions are complementary in classifying a candidate box in object detection. Effective integration of local and contextual visual cues from these regions has become a fundamental problem in object detection. In this paper, we propose a gated bi-directional CNN (GBD-Net) to pass messages among features from different support regions during both feature learning and feature extraction. Such message passing can be implemented through convolution between neighboring support regions in two directions and can be conducted in various layers. Therefore, local and contextual visual patterns can validate the existence of each other by learning their nonlinear relationships and their close interactions are modeled in a more complex way. It is also shown that message passing is not always helpful but dependent on individual samples. Gated functions are therefore needed to control message transmission, whose on-or-offs are controlled by extra visual evidence from the input sample. The effectiveness of GBD-Net is shown through experiments on three object detection datasets, ImageNet, Pascal VOC2007 and Microsoft COCO. This paper also shows the details of our approach in wining the ImageNet object detection challenge of 2016, with source code provided on \url{https://github.com/craftGBD/craftGBD}.

I. INTRODUCTION

Candidate boxes can be difficult to classify from local cues alone, while surrounding context and local detail provide complementary evidence. GBD-Net addresses this by passing gated messages between multi-resolution features, in contrast to prior approaches that concatenate part or context features.

I. INTRODUCTION

  • Partial-object boxes can share similar local visual cues, and judging occlusion requires information from larger surrounding regions.Surrounding regions also provide background and nearby-object context, while large CNN inputs can lose fine details such as ear shape.Source: §I · p. 1
  • GBD-Net adaptively models interactions between contextual and local cues during feature learning and feature extraction.Its bi-directional convolutions pass detailed local messages to larger patterns and contextual messages in the opposite direction.Source: §I · p. 2
  • Gate functions transmit a detected pattern's message only for samples where it is useful and block it for others.For example, human-face cues can indicate that a rabbit ear is artificial and should not strengthen evidence for a rabbit head.Source: §I · p. 2
  • The paper presents a detection pipeline integrating region proposals, feature representation learning, context modeling, and model averaging, and evaluates GBD-Net on three datasets.The evaluation datasets are ImageNet, PASCAL VOC2007, and Microsoft COCO.Source: §I · p. 2

II. RELATED WORK

  • GBD-Net is complementary to prior part- and context-based approaches that concatenate features or scores, and its gating addresses message control not investigated in those works.The authors describe the network as complementary to approaches in,.Source: §II · p. 3

III. GATED BI-DIRECTIONAL CNN

The paper builds GBD-Net on the Fast R-CNN detection pipeline. It extracts features for multiple support regions per candidate box, passes messages between them under gate control, then classifies the resulting features.

A. Fast RCNN pipeline

  • Fast R-CNN generates candidate boxes and feature maps, pools each box to a common feature size, then predicts class labels and box refinements.The pooled features go through convolutional, pooling, and fully connected layers for classification and localization refinement.Source: A. Fast RCNN pipeline · p. 3

B. Framework overview

  • For each candidate box, GBD-Net uses ROI pooling to obtain features from same-center regions of different sizes, optionally processes them with CNN layers, and applies gated bidirectional message passing before classification.The BN-net is the baseline; considering one support region and one branch reduces the overview model to a BN-net.Source: B. Framework overview · p. 3
  • The exemplar implementation applies ROI pooling after the sixth inception module and sends the resulting features through the gated bidirectional network.The paper also tried placing ROI pooling and GBD-Net after different BN-net layers, including after the input image.Source: B. Framework overview · p. 3

C. Roi-pooling of features with different resolutions and support regions

ROI pooling creates same-sized features from candidate-centered regions with different context extents and resolutions. The bidirectional structure passes messages between neighboring features so local and contextual patterns can validate one another.

C. Roi-pooling of features with different resolutions and support regions

  • Each candidate box is padded by scale p, and ROI pooling maps the resulting region to a fixed-size feature map.The implementation uses p values of −0.2, 0.2, 0.8, and 1.7, producing 14 × 14 × 608 features for BN-net.Source: C. Roi-pooling of features with… · p. 4
  • With pooled features held at the same size, larger p gives lower resolution for the original box but more surrounding context; smaller p gives higher resolution and less context.In Fast R-CNN, ROI pooling warps features rather than image pixels.Source: C. Roi-pooling of features with… · p. 4

1) Bi-direction structure:

  • The bidirectional structure connects features from the smallest region to the largest and in the reverse direction, so each resolution/context receives messages from its neighbors.The smallest-support feature receives no incoming message in one direction, and the largest-support feature receives none in the other.Source: 1) Bi-direction structure · p. 4
  • High-resolution, small-context features can validate larger-context features, while low-resolution, large-context features can validate smaller-context features.The paper illustrates this with rabbit-ear details supporting a rabbit-head feature and the rabbit head also supporting the ear feature.Source: 1) Bi-direction structure · p. 4
  • Unlike a bidirectional RNN, GBD-Net does not share convolutional parameters across resolution/context connections because the relationships differ at different levels.Features from both directions are integrated and remain complementary for later classification.Source: 1) Bi-direction structure · p. 4

2) Gate functions for message passing:

Gate functions make message transmission sample-dependent rather than identical for every candidate box. The gates use visual evidence from co-located regions to control how strongly messages are passed.

2) Gate functions for message passing:

  • Gate functions adapt message passing for individual candidate boxes instead of transmitting messages identically for all boxes.The gate filters are convolutional, with a separate learned filter for each feature channel.Source: 2) Gate functions for message passing · p. 4
  • Convolutional gate filters use nearby visual cues to determine message rates, with 3 × 3 filters allowing cues such as a girl's face to suppress a misleading rabbit-ear message.The sigmoid nonlinearity keeps message-passing rates in the range (0,1).Source: 2) Gate functions for message passing · p. 5
  • The gate function uses features from co-located regions to set each message rate, and a zero gate blocks the message.The gating operation uses a sigmoid and element-wise multiplication.Source: 2) Gate functions for message passing · p. 5

E. The modified GBD structure (GBD-v2)

GBD-v2 changes how contextual messages are merged and added to the input. A constant scale controls message magnitude, which the authors report improves detection accuracy.

E. The modified GBD structure (GBD-v2)

  • Max pooling merges the directional features, saving the memory and computation required by the previous convolution-based merge.The earlier structure concatenated the features and convolved them to produce the merged output.Source: E. The modified GBD structure (GBD-v2) · p. 5
  • An identity mapping adds the merged messages to the input to refine it, while avoiding a drastic output change that could hinder adapting later pretrained layers.The previous structure required careful initialization of convolution parameters and gate functions; identity mapping allowed simple initialization using.Source: E. The modified GBD structure (GBD-v2) · p. 5
  • The authors empirically found that scaling merged contextual messages with β improves detection accuracy.The constant controls the messages’ magnitude.Source: E. The modified GBD structure (GBD-v2) · p. 5

F. Implementation details, training scheme, and loss function

The framework predicts classification and bounding-box regression from h3 representations. Its training loss combines classification and localization objectives.

F. Implementation details, training scheme, and loss function

  • Classification and bounding-box regression predictions are based on the h3 representations.The CNN is initialized from a pretrained model except for randomly initialized filters in the gated bi-directional structure.Source: F. Implementation details, training… · p. 5
  • The training loss sums cross-entropy for classification and smoothed L1 loss for bounding-box regression.The localization term applies to samples with y ≥ 1.Source: F. Implementation details, training… · p. 5
  • The predicted class probability is t_c and the predicted bounding-box offset is t_v; λ is set to 1 and network parameters are learned by back-propagation.Source: F. Implementation details, training… · p. 5

G. Discussion

GBD-Net’s placement is independent of roi-pooling, and an extreme configuration applies roi-pooling to raw pixels while placing message passing in the last convolution layer. The ImageNet testing pipeline combines proposals, GBD-Net classification, model averaging, and postprocessing.

G. Discussion

  • GBD-Net can be placed independently of roi-pooling; in an extreme configuration, roi-pooling extracts raw-pixel regions and message passing occurs in the last convolution layer.In this configuration, Fast RCNN is reduced to RCNN, cropping surrounding regions from raw pixels rather than feature maps.Source: G. Discussion · p. 5

A. Overview at the testing stage

  • The 2016 ImageNet framework uses an improved CRAFT version for candidate-box generation, then GBD-Net for box classification.The testing overview also averages multiple deep-model outputs and postprocesses scores with whole-image classification, while bounding-box voting adjusts locations using neighboring boxes.Source: A. Overview at the testing stage • · p. 6

B. Candidate box generation

  • The final ImageNet submission uses Craft-v3, an improved proposal version following the earlier Craft-v2.The paper describes the two proposal versions and details Craft-v3.Source: B. Candidate box generation · p. 6

1) Craft-V1 and V2:

The paper describes successive CRAFT proposal versions and ImageNet box-classification configurations, including a 269-layer ResNet baseline. It also reports gains from roi-pooling-based object-centric pretraining over image-centric pretraining.

1) Craft-V1 and V2:

  • Craft-V1 and Craft-V2 use a two-stage cascade: an RPN generates about 300 proposals, then a two-category Fast RCNN classifier distinguishes objects from background.The versions differ in pretraining: Craft-V1 uses 1000-class image classification and Craft-V2 uses RPN pretraining.Source: 1) Craft-V1 and V2 · p. 6

2) Craft-v3:

  • Craft-v3 adds random-crop training, multi-scale testing, balanced RPN samples, and LocNet proposals complementary to Craft-based proposals.Source: 2) Craft-v3 · p. 6

C. Box classification with GBD-Net

1) The baseline ResNet-269 model:

  • The ResNet-269 baseline increases the stacked blocks in conv3_x, conv4_x, and conv5_x over ResNet-152 and places roi-pooling after layer 234.The basic blocks use identity mapping, and stochastic depth is used during training.Source: 1) The baseline ResNet-269 model · p. 6

2) Adapt the pretraining for roi-pooling:

  • On ImageNet val2, image-centric BN-net pretraining achieved 49.4% mAP, versus 48.4% mAP for object-centric pretraining after fine-tuning on train+val1.The authors note that object-centric pretraining for RCNN does not match Fast RCNN’s relative contextual region because Fast RCNN uses convolution over the whole image.Source: 2) Adapt the pretraining for roi-pooling · pp. 6–7
  • For roi-pooling pretraining, the method includes a 32-pixel context around each object box and randomly shakes the target box for augmentation.Fast RCNN’s feature warping differs from RCNN’s image warping, and its fixed receptive-field context changes relative to boxes of different sizes.Source: 2) Adapt the pretraining for roi-pooling · pp. 6–7
  • Object-centric pretraining with roi-pooling gained 1.5% mAP for BN-net and around 1.0% mAP for ResNet-101 over image-centric classification pretraining.Source: 2) Adapt the pretraining for roi-pooling · p. 7

E. Technical details on improving performance

The section describes inference and scoring choices used to improve detection performance, including multi-scale testing, global-context score fusion, and model averaging.

1) Multi-scale testing:

  • Multi-scale testing uses image-pyramid shorter sides of 400, 500, 600, 700, and 800, but averaging box scores across scales showed no obvious improvement.The longer side is capped at 1000 pixels.Source: 1) Multi-scale testing · p. 7

2) Left-right flip:

  • At test time, scores and estimated box locations from the original and horizontally flipped images are averaged.Training images and their candidate boxes are also flipped for augmentation.Source: 2) Left-right flip · p. 7

3) Bounding box voting:

  • Bounding-box voting refines the highest-scoring neighborhood box using boxes with IoU above 0.5, weighted by their scores.Source: 3) Bounding box voting · p. 7

4) Non-maximum suppression (NMS) threshold:

5) Global context:

  • Global context combines 200-class image-classification scores with 200-class detection scores by weighted averaging, with weights greedily searched on val1.The classification model is finetuned from an image-centric pretrained CNN.Source: 5) Global context · p. 7

6) Model ensemble:

  • Six models are selected by greedy search from ten on ImageNet Det val2, then their scores and box-regression results are averaged.Source: 6) Model ensemble · p. 7

V. EXPERIMENTAL RESULTS

The experiments evaluate GBD-Net on VOC2007, COCO, and ImageNet, and examine implementation choices including where to place roi-pooling. Results report improvements over baselines on the three datasets and a placement trade-off between speed and accuracy.

A. Implementation details

  • The gated structure is added after BN-net’s sixth inception module or ResNet-269’s 234th layer, with pretrained baseline layers and randomly initialized GBD layers.The framework is based on fast RCNN; inference overhead from gated connections is less than 40%.Source: A. Implementation details · pp. 7–8

1) PASCAL VOC2007 dataset:

  • On VOC2007, GBD-Net obtains 77.2% mAP versus 73.1% for the BN+FRCN baseline.Evaluation uses the 07+12 trainval set and the test set.Source: 1) PASCAL VOC2007 dataset · p. 8

2) Microsoft COCO object detection dataset:

  • On COCO closed-test data, GBD-Net reaches 27.0% AP and 45.8% AP50, improving AP by 2.6 points over BN+FRCN.The evaluation uses MCG region proposals; the baseline reports 24.4% AP and 39.3% AP50.Source: 2) Microsoft COCO object detection… · p. 8

3) ImageNet object detection dataset:

1) Investigation on placing roi-pooling at different layers:

  • Placing roi-pooling after image pixels gives 51.4% mAP versus 48.9% after inception module 4d, but is slower because CNN computation cannot be shared.The authors attribute the accuracy difference to more diverse, complementary features from pixel-level pooling.Source: 1) Investigation on placing… · p. 8

2) Investigation on gate functions:

The component studies examine gating, feature-region sizes, combining GBD-Net with multi-region detection, and framework variants. They report that gating aids training, multiple resolutions and contexts improve detection, and GBD-v2 outperforms GBD-v1 in the tested setup.

2) Investigation on gate functions:

  • With random initialization, message-passing layers without gates reach 42.3% mAP; initialized from trained GBD-Net, they reach 48.2%, while the authors report ungated training is hard.The authors connect the difficulty to exploding or vanishing gradients as message passing increases nonlinearity.Source: 2) Investigation on gate functions · p. 8

3) Investigation on using different feature region sizes:

  • Increasing a single region’s padding from 0.2 to 1.7 harms detection, while integrating multiple resolutions and contexts substantially improves it as their number increases.The authors attribute the single-region decline to lost resolution and background clutter.Source: 3) Investigation on using different… · p. 8

4) Investigation on combination with multi-region:

  • Combining GBD-Net with multi-region BN reaches 51.2% mAP, 2.3 points above GBD-Net and 3.9 above multi-region BN.The study reports the improvement from the combination as complementary to the multi-region approach.Source: 4) Investigation on combination with… · p. 8

D. Investigation on GBD and GBD-v2

  • On the same BN-net, GBD-v2 improves mAP by 3.7% versus 2.6% for GBD-v1, and is selected as the default because it performs better and is easier to train.The reported GBD-v2 configuration uses roi-pooling at module 4d for BN-net and layer 234 for ResNet-269, with gating and padding values -0.2, 0.2, 0.8, and 1.7.Source: D. Investigation on GBD and GBD-v2 · p. 9

E. Investigation on other components in the detection framework

  • Random cropping and multi-scale testing add 0.74% and 1.1% recall, respectively; multi-scale training adds 0.7% recall, and 1:1 positive-negative sampling adds 0.6%.The component study uses left-right flips during training across evaluated approaches, but not at test time unless specified.Source: 1) Region proposal · p. 9, E. Investigation on other components… · p. 9

1) Region proposal:

  • Craft-v1, Craft-v2, and Craft-v3 reach 46.3, 48.4, and 49.4 mAP on ImageNet val2; Craft-v3 adds 1.9% final detection AP over Craft-v2 with ResNet-269+GBD-Net.Craft-v3 combines two proposal methods to provide 300 proposals per image.Source: 1) Region proposal · p. 9

2) Pretraining scheme:

The section reports pretraining, GBD and baseline variants, ensemble and component results, and false-positive analysis. It concludes that GBD-Net was validated on ImageNet, Pascal VOC2007, and Microsoft COCO.

2) Pretraining scheme:

  • Including ROI pooling makes object-centric pretraining effective for detection fine-tuning, yielding a 1.5% absolute mAP increase; without ROI pooling, it performs worse than image-centric pretraining.Source: 2) Pretraining scheme · pp. 9–10

3) The GBD structure:

  • GBD improves BN-net by 3.7% mAP with Craft-V2 and 4.2% with Craft-V3; β=1 yields a 1.6% mAP drop, while β=0.1 performs best.Source: 3) The GBD structure · p. 10

4) Baseline deep models:

5) Model ensemble:

6) Components improving performance:

F. Analysis of false positive types

Related papers

Loading 1610.02579v1…