PaperlayerView exact revision on arXiv

Source-linked AI summary

Single-Shot Refinement Neural Network for Object Detection

Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei, Stan Z. Li

arXiv:1711.06897v3cs.CV

TL;DR

One-stage detectors are efficient but usually less accurate than two-stage detectors. RefineDet refines anchors before object detection and reports the best published results on three detection benchmarks.

  • Problem

    One-stage detectors offer high computational efficiency but usually trail two-stage detectors in accuracy, with class imbalance identified as a main reason.

    Source: §1 · p. 1

  • Method

    RefineDet filters negative anchors and coarsely adjusts their boxes before object detection, with a transfer connection block carrying ARM features to the ODM.

    Source: §1 · p. 2

  • Results

    RefineDet achieves the best published performance on VOC 2007, VOC 2012, and COCO: 85.8% mAP, 86.8% mAP, and 41.8% AP, respectively.

    Source: §7 · p. 11

  • Takeaways & Limitations

    RefineDet reports performance better than two-stage approaches while maintaining the high efficiency of one-stage approaches.

    Source: §1 · p. 2

    Due to computational-resource shortages, RefineDet was trained only at 320 × 320 and 512 × 512 input sizes; larger inputs may improve accuracy.

    Source: §5.1.1 · p. 5

Abstract

from arXiv · show

For object detection, the two-stage approach (e.g., Faster R-CNN) has been achieving the highest accuracy, whereas the one-stage approach (e.g., SSD) has the advantage of high efficiency. To inherit the merits of both while overcoming their disadvantages, in this paper, we propose a novel single-shot based detector, called RefineDet, that achieves better accuracy than two-stage methods and maintains comparable efficiency of one-stage methods. RefineDet consists of two inter-connected modules, namely, the anchor refinement module and the object detection module. Specifically, the former aims to (1) filter out negative anchors to reduce search space for the classifier, and (2) coarsely adjust the locations and sizes of anchors to provide better initialization for the subsequent regressor. The latter module takes the refined anchors as the input from the former to further improve the regression and predict multi-class label. Meanwhile, we design a transfer connection block to transfer the features in the anchor refinement module to predict locations, sizes and class labels of objects in the object detection module. The multi-task loss function enables us to train the whole network in an end-to-end way. Extensive experiments on PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO demonstrate that RefineDet achieves state-of-the-art detection accuracy with high efficiency. Code is available at https://github.com/sfzhang15/RefineDet

1. Introduction

RefineDet combines one-stage efficiency with a two-stage-inspired detection design, using interconnected anchor-refinement and object-detection modules plus a feature-transfer block to pursue accurate, efficient detection.

1. Introduction

  • Two-stage detectors lead on accuracy using sampling heuristics, cascade regression, and two-stage features; one-stage detectors are efficient but typically less accurate, partly due to class imbalance.Source: §1 · p. 1
  • The ARM removes negative anchors to reduce classifier search space and coarsely adjusts anchor locations and sizes; the ODM further refines boxes and predicts multi-class labels.Source: §1 · p. 2
  • A transfer connection block carries ARM features to the ODM for location, size, and class prediction, while a multi-task loss trains the full network end to end.The transferred features support the ODM's more challenging prediction tasks.Source: §1 · p. 2
  • RefineDet achieves 85.8% mAP on VOC 2007 and 86.8% mAP on VOC 2012 with VGG-16, and 41.8% AP on MS COCO test-dev with ResNet-101, outperforming previously best published one- and two-stage results.The paper also reports state-of-the-art results on the three benchmarks overall.Source: §1 · p. 2
  • RefineDet runs at 40.2 FPS with 320 × 320 input and 24.1 FPS with 512 × 512 input on an NVIDIA Titan X GPU.Source: §1 · p. 2

2. Related Work

Related work traces detection from hand-crafted sliding-window systems to CNN detectors, contrasting proposal-based two-stage accuracy with efficient one-stage methods whose accuracy still trails.

Classical Object Detectors

  • Early detectors used hand-crafted features and classifiers on dense sliding windows; Viola and Jones used Haar features and AdaBoost cascades, while DPM maintained top PASCAL VOC results for years.Source: §2 · pp. 2–3

Two-Stage Approach

  • Two-stage detectors generate sparse proposals before CNN-based region and class prediction; the approach achieved dominant performance on challenging benchmarks, prompting architectural, training, contextual, and multilevel advances.Source: §2 · p. 3

One-Stage Approach

  • One-stage detectors directly predict classes and locations for efficiency, but their accuracy still trails two-stage methods.OverFeat [38], YOLO [34], YOLOv2 [35], SSD [30], DSSD [13], and DSOD [39] exemplify this line. YOLOv2 added batch normalization, a high-resolution classifier, and anchor-box convolutional prediction; SSD assigns scale-specific anchors across layers, DSSD adds deconvolutional context, DSOD learns detectors from scratch, and other methods [24] [28] address class imbalance through loss or classification redesign.Source: §2 · p. 3

3. Network Architecture

RefineDet is a feed-forward detector built from an anchor refinement module (ARM) and an object detection module (ODM). The ARM filters negative anchors and coarsely adjusts them; the ODM uses the refined anchors for localization and classification.

Network Architecture

  • The ARM filters negative anchors and coarsely adjusts their locations and sizes, while the ODM uses refined anchors to predict object locations and multi-class labels.The ARM adapts ImageNet-pretrained VGG-16 or ResNet-101 features; the ODM uses TCB outputs and 3 × 3 convolutional prediction layers.Source: §3 · p. 3

Transfer Connection Block

  • TCBs transfer ARM features from anchor-associated maps to the ODM and fuse high-level context by deconvolution, element-wise summation, and convolution.The added convolution is used to ensure feature discriminability for detection.Source: §3 · p. 3

Two-Step Cascaded Regression

  • The ARM predicts four offsets and two foreground-confidence scores per cell to create refined anchors; the ODM predicts c class scores and four offsets per anchor relative to those refined anchors.This two-step cascade replaces SSD’s direct use of regularly tiled default boxes and is described as improving detection, especially for small objects.Source: §3 · pp. 3–4

Negative Anchor Filtering

  • For ODM training and inference, RefineDet discards refined anchors whose negative confidence exceeds θ = 0.99, passing only hard negatives and positives to the ODM.The filtering mechanism is designed to reject well-classified negatives early and mitigate class imbalance.Source: §3 · p. 4

4. Training and Inference

RefineDet is trained end to end with matched multiscale anchors, hard-negative mining, and joint ARM/ODM losses. At inference, the ARM filters and refines anchors before the ODM ranks detections and non-maximum suppression produces final results.

Data Augmentation

  • Training images are randomly expanded and cropped, with photometric distortion [20] and flipping, following augmentation strategies in [30].Source: §4 · p. 4

Backbone Network

  • The detector uses ILSVRC-pretrained VGG-16 or ResNet-101 backbones, adds convolutional or residual layers, and normalizes VGG feature scales with learned L2 norms.It can also use pretrained Inception V2, Inception ResNet, or ResNeXt-101; VGG-16’s conv4_3 and conv5_3 norms are initialized to 10 and 8.Source: §4 · p. 4

Anchors Design and Matching

  • RefineDet matches each ground-truth box to its best-overlap anchor, then matches anchors above 0.5 Jaccard overlap, using four feature strides and three aspect ratios.The four strides are 8, 16, 32, and 64 pixels; each layer’s anchor scale is four times its stride, with aspect ratios 0.5, 1.0, and 2.0.Source: §4 · p. 4

Hard Negative Mining

  • Hard-negative mining selects negative anchors with the highest losses to keep the negative-to-positive ratio below 3:1, mitigating foreground-background imbalance.This replaces using all negatives or randomly selecting negatives during training.Source: §4 · p. 4

Loss Function

  • RefineDet’s loss combines ARM binary classification and anchor regression with ODM multi-class classification and box regression.It uses cross-entropy for binary classification, softmax for multi-class classification, and smooth L1 for regression; regression is ignored for negative anchors, and classification loss is zero when its module has no positive anchors.Source: §4 · pp. 4–5
  • The ARM predicts objectness and refines anchors, while the ODM uses retained refined anchors to predict object categories and more accurate locations and sizes.The ARM filters anchors whose negative confidence is below a threshold before passing them to the ODM.Source: §4 · p. 5

Inference

  • At inference, the ARM filters anchors by negative-confidence threshold and refines the remaining boxes; the ODM outputs 400 detections before per-class NMS retains 200.NMS uses a Jaccard overlap of 0.45.Source: §4 · p. 5

Optimization

  • Training initializes added layers with Xavier for VGG-16 or zero-mean Gaussian weights (standard deviation 0.01) for ResNet-101, then fine-tunes with SGD, batch size 32, momentum 0.9, and weight decay 0.0005.The initial learning rate is 10^-3, with dataset-specific decay policies.Source: §4 · p. 5

5. Experiments

Across PASCAL VOC 2007/2012 and MS COCO, RefineDet achieves strong detection accuracy with fast inference. Component ablations show gains from negative-anchor filtering, cascaded regression, and transfer connections.

5.1.1 Run Time Performance

  • 24.8 ms (40.3 FPS) at 320 × 320 and 41.5 ms (24.1 FPS) at 512 × 512 are RefineDet’s reported inference speeds.The paper describes RefineDet as the first real-time method above 80% mAP on VOC 2007 and reports the best accuracy-speed trade-off; only YOLO and SSD300 are slightly faster than RefineDet320, with lower accuracy by 16.6% and 2.5%, respectively. Training was limited to 320 × 320 and 512 × 512 inputs because of computational resources.Source: §5.1.1 · pp. 5–6, Table 1 · p. 6

5.1. PASCAL VOC 2007

  • 83.8% mAP (RefineDet512+) and 83.1% mAP (RefineDet320+) are the reported multi-scale VOC 2007 results.At single scale, RefineDet320 achieves 80.0% mAP and RefineDet512 achieves 81.8% mAP; RefineDet512 surpasses all one-stage methods and performs better than most two-stage methods, except CoupleNet.Source: §5.1 · p. 5, Table 1 · p. 6

5.2. PASCAL VOC 2012

  • 83.5% mAP (RefineDet512+) and 82.7% mAP (RefineDet320+) are the reported multi-scale VOC 2012 results.At single scale, RefineDet320 achieves 78.1% mAP and RefineDet512 achieves 80.1% mAP; the 512 × 512 result surpasses all one-stage methods and is slightly below CoupleNet’s 80.4%.Source: Table 1 · p. 6, §5.2 · p. 7

5.3. MS COCO

  • 41.8% AP is RefineDet’s best MS COCO result, surpassing the reported one- and two-stage approaches.The single-model result is 0.2% higher than the five-model G-RMI ensemble; it also exceeds umd det by 1.0% AP and RetinaNet800 by 2.7% AP.Source: Table 2 · p. 7, §5.3 · p. 8
  • 38.6% AP (RefineDet320+ with ResNet-101) and 41.8% AP (RefineDet512+ with ResNet-101) are the reported multi-scale MS COCO results.With VGG-16, RefineDet320 achieves 29.4% AP and RefineDet512 achieves 33.0% AP; with ResNet-101 at single scale, the corresponding results are 32.0% AP and 36.4% AP.Source: Table 2 · p. 7, §5.3 · p. 8

5.1.2 Ablation Study

  • 0.5% mAP is lost without negative-anchor filtering, 2.2% without two-step regression, and the transfer connection block improves mAP by 1.1%.The comparisons are 80.0% versus 79.5%, 79.5% versus 77.3%, and 77.3% versus 76.2%, respectively. The transfer connection block passes discriminative ARM features and large-scale context to improve detection accuracy.Source: §5.1.2 · pp. 6–7, Table 3 · p. 7

5.4. From MS COCO to PASCAL VOC

  • 85.2% mAP (RefineDet512) on VOC 2007 and 85.0% mAP (RefineDet512) on VOC 2012 are reported after fine-tuning on VOC from MS COCO pretraining.Multi-scale testing raises the VOC 2007 results to 85.6% and 85.8%, and the VOC 2012 results to 86.0% and 86.8%, for RefineDet320+ and RefineDet512+, respectively; the paper reports top mAP scores on both datasets.Source: Table 4 · p. 8, §5.4 · p. 8

6. Conclusions

RefineDet combines anchor refinement and object detection modules in an end-to-end detector. Experiments on PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO demonstrate state-of-the-art detection accuracy with high efficiency.

6. Conclusions

  • RefineDet uses an ARM to filter negative anchors and coarsely adjust them, then an ODM to refine locations and sizes and predict multi-class labels.The whole network is trained end to end with a multi-task loss.Source: §6 · p. 8
  • Experiments on PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO demonstrate state-of-the-art detection accuracy with high efficiency.Source: §6 · p. 8
  • Future work will apply RefineDet to pedestrian, vehicle, and face detection and introduce attention to further improve performance.Source: §6 · p. 8

7. Complete Object Detection Results

RefineDet reports the best published results on PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO, while small-object detection remains challenging and larger inputs trade speed for improved performance.

7. Complete Object Detection Results

  • Increasing input size from 320 × 320 to 512 × 512 can improve small-object performance, but burdens inference speed and is only a temporary solution; small-object detection remains challenging.Source: §7 · p. 11
  • RefineDet achieved the best published results: 85.8% mAP on the PASCAL VOC 2007 test set, 86.8% mAP on the PASCAL VOC 2012 test set, and 41.8% AP on the MS COCO test-dev set.The results are reported in Tables 5, 6, and 7, respectively.Source: §7 · p. 11

8. Qualitative Results

Qualitative results show RefineDet detections on PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO, including examples involving occlusions, truncations, inter-class interference, and clustered backgrounds.

8. Qualitative Results

  • Figures 3–5 show detections on the PASCAL VOC 2007 test set, PASCAL VOC 2012 test set, and MS COCO test-dev, respectively.Source: §8 · p. 11
  • Only detected bounding boxes with scores larger than 0.6 are displayed, and their colors indicate object categories.Source: §8 · p. 11
  • The method works well with occlusions, truncations, inter-class interference, and clustered backgrounds.Source: §8 · p. 11

9. Detection Analysis on PASCAL VOC 2007

On PASCAL VOC 2007, RefineDet detects objects with high recall and fewer false positives than SSD, and is robust across object sizes and aspect ratios.

9. Detection Analysis on PASCAL VOC 2007

  • Recall was around 95%-98%, higher under weak 0.1-Jaccard criteria; versus SSD, RefineDet reduced localization, background, and similar-category false positives.The paper attributes lower localization errors to two-step cascade regression, lower background confusion to ARM negative-anchor filtering, and lower similar-category confusion to ARM binary-classification and ODM multi-class features.Source: §9 · p. 11, Figure 6 · p. 14
  • RefineDet was robust to object sizes and aspect ratios, though performance still had room to improve.The two-step cascade diversifies default anchor scales and aspect ratios in ARM, helping ODM regress extra-small, extra-large, extra-wide, and extra-tall objects.Source: §9 · p. 11, Figure 7 · p. 14

Related papers

Loading 1711.06897v3…