PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Advances in Deep Learning for Object Detection

Xiongwei Wu, Doyen Sahoo, Steven C. H. Hoi

arXiv:1908.03673v1cs.CVcs.LGcs.MM

TL;DR

Object detection requires both semantic recognition and spatial localization, and traditional approaches faced limitations in proposals, features, and separately optimized stages. This paper systematically surveys deep learning-based detection frameworks across components, learning strategies, applications, and benchmarks, reporting substantial progress over traditional algorithms and reviewing state-of-the-art benchmark results.

  • Problem

    Object detection requires understanding semantic cues and spatial information, while traditional detectors had constrained proposals, hand-crafted features, and separately optimized pipeline stages.

  • Method

    The paper comprehensively reviews deep learning-based object detection and organizes contributions into detection components, learning strategies, and applications and benchmarks.

  • Results

    Deep learning-based algorithms outperformed traditional detection algorithms by huge margins, and the survey reports state-of-the-art generic detection results on public benchmarks.

  • Takeaways & Limitations

    The survey provides a systematic review of recent object detection advances and discusses future directions for the field.

  • Takeaways & Limitations

    Traditional detection progress on Pascal VOC had become incremental, with only minor gains from complicated ensemble systems.

Abstract

from arXiv · show

Object detection is a fundamental visual recognition problem in computer vision and has been widely studied in the past decades. Visual object detection aims to find objects of certain target classes with precise localization in a given image and assign each object instance a corresponding class label. Due to the tremendous successes of deep learning based image classification, object detection techniques using deep learning have been actively studied in recent years. In this paper, we give a comprehensive survey of recent advances in visual object detection with deep learning. By reviewing a large body of recent related work in literature, we systematically analyze the existing object detection frameworks and organize the survey into three major parts: (i) detection components, (ii) learning strategies, and (iii) applications & benchmarks. In the survey, we cover a variety of factors affecting the detection performance in detail, such as detector architectures, feature learning, proposal generation, sampling strategies, etc. Finally, we discuss several future directions to facilitate and spur future research for visual object detection with deep learning. Keywords: Object Detection, Deep Learning, Deep Convolutional Neural Networks

1. Introduction

Object detection extends visual recognition from image-level classification to localizing and labeling individual objects. This survey reviews deep learning advances through detection components, learning strategies, applications, benchmarks, and future directions.

  • Object detection assigns categorical labels while localizing each object instance with bounding boxes.
  • Traditional Detection: Traditional detectors used proposal generation, feature extraction, and region classification as separate pipeline stages.Proposals searched likely object locations, feature descriptors encoded regions, and classifiers assigned categorical labels.
  • Deep Learning Transition: Deep learning methods substantially outperformed traditional detection algorithms and produced remarkable progress after convolutional neural networks succeeded in image classification.
  • Modern Detection Frameworks: Two-stage detectors commonly achieve better detection performance, whereas one-stage detectors are more time-efficient and applicable to real-time detection.
  • Survey Scope: The survey organizes recent work into detection components, learning strategies, and applications and benchmarks.It covers architectures, proposal generation, feature learning, optimization, applications, evaluation metrics, and state-of-the-art generic detection results.

2. Problem Settings

Object detection identifies target objects in images using categorical labels and precise spatial localization. Its formalization includes annotated objects, a parameterized detector, loss optimization, and evaluation through localization and detection metrics.

  • The task assigns each object instance a target-class label and a precise bounding box or pixel mask.
  • Each image contains Mi annotated objects from C categories, with categorical and spatial labels for every object.
  • A detector f parameterized by θ maps each image to predictions sharing the format of its ground-truth annotations.
  • Training uses a loss function whose regularization term is weighted by the trade-off parameter λ.
  • IoU measures localization quality, while mAP over C classes evaluates generic detection and FPS measures inference efficiency.
  • Real-time video detection requires rapid processing, with 20 FPS commonly considered real-time.

3. Detection Components

The survey organizes detection components around the detection paradigm and distinguishes bounding-box localization from mask-level algorithms. It also covers backbones, proposal generators, and feature representations.

  • Detection components begin with the choice between bounding-box-level and mask-level detection settings.
  • The two major detection paradigms are two-stage detectors and one-stage detectors.
  • Within these paradigms, detectors may combine diverse deep-learning backbones, proposal generators, and feature-representation modules.

3.1. Detection Settings

Object detection includes vanilla bounding-box detection and instance segmentation with pixel-wise masks. The latter demands more precise spatial processing, although both settings share most components and similar IoU-based evaluation.

  • Vanilla object detection localizes objects with rectangular bounding boxes and requires only bounding-box annotations.
  • Instance segmentation localizes each object with a pixel-wise mask rather than a rough rectangle.
  • Pixel-level prediction makes instance segmentation more sensitive to spatial misalignment and increases its spatial-information requirements.
  • Evaluation is nearly identical across settings, except that IoU is computed on mask predictions for instance segmentation.
  • Most detection components can be shared between bounding-box detection and instance segmentation despite their differences.

3.2. Detection Paradigms

Two-stage detectors generate proposals before classifying and refining them, whereas one-stage detectors predict without a separate proposal stage. The surveyed evolution trades accuracy, speed, proposal quality, and trainability across frameworks.

  • Two-stage Detectors: Two-stage detectors first generate a sparse set of proposals, then encode proposal features and predict object classes.
  • One-stage Detectors: One-stage detectors omit a separate proposal-generation stage and typically consider all image positions directly.
  • Detection Paradigms: Two-stage detectors often achieve state-of-the-art benchmark results but generally have lower inference speeds, while one-stage detectors are faster but relatively less accurate.
  • Two-stage Detectors: R-CNN generates around 2,000 Selective Search proposals, extracts proposal features separately, and applies SVM classification with bounding-box regression.
  • Two-stage Detectors: R-CNN improved detection performance to 53.7% mAP but incurred duplicated computation, multi-stage optimization, and weak proposals in complex contexts.
  • Two-stage Detectors: SPP-net accelerated inference by computing one whole-image feature map and pooling fixed-length region features across multiple spatial scales.
  • Two-stage Detectors: Fast R-CNN used ROI pooling with backpropagation and jointly optimized feature extraction, classification, and box regression end-to-end.
  • One-stage Detectors: YOLO reached 45 FPS, or 155 FPS with a simplified backbone, while SSD achieved accuracy comparable to Faster R-CNN with real-time inference.

3.3. Backbone Architecture

Deep convolutional neural networks provide the standard initialization and feature-extraction backbone for object detectors, but classification-oriented designs can conflict with detection’s spatial and multi-scale requirements. Modern backbones address these needs through architectural changes including residual connections, grouped convolutions, efficient mobile designs, multi-scale processing, and detection-specific high-resolution features.

  • Pretrained classification networks became the default initialization strategy for object detectors because their convolutional weights provide richer semantic information.
  • Basic CNN Architecture: A CNN combines convolution, pooling, nonlinear activation, and fully connected layers, and optimizes the resulting network with gradient-based methods.
  • CNN Backbone for Object Detection: Residual connections reduce optimization difficulty by directly propagating values and gradients, enabling substantially deeper networks while retaining performance improvements from increased depth.
  • CNN Backbone for Object Detection: ResNeXt uses grouped convolutions to reduce computation and memory cost while maintaining comparable classification accuracy and richer semantic representations.
  • CNN Backbone for Object Detection: MobileNet targets mobile platforms by significantly reducing computation and parameter count without significant classification-accuracy loss.
  • CNN Backbone for Object Detection: GoogleNet captures multi-scale features by applying 1 × 1, 3 × 3, and 5 × 5 convolutions to the same feature map and aggregating their outputs.
  • CNN Backbone for Object Detection: Classification backbones are sub-optimal for detection because downsampling favors large receptive fields and spatial invariance, whereas detection requires high-resolution localization and multi-scale representations.
  • CNN Backbone for Object Detection: DetNet was designed specifically for detection, preserving high-resolution feature maps with dilated convolutions and predicting on multiple scales.

3.4. Proposal Generation

Proposal generation supplies candidate regions for classification and localization, with methods spanning traditional vision cues, supervised anchors, and keypoint formulations. Traditional methods offer simplicity and recall but struggle on challenging datasets, while anchor-based methods improve learnability yet retain manually designed priors.

  • Proposal generators produce candidate bounding boxes that subsequent detection stages classify and localize, and the survey groups them into four methodological categories.
  • One-stage detectors treat each image region as a potential proposal and estimate class and bounding-box coordinates at every location.
  • Traditional Computer Vision Methods: Traditional proposal methods use low-level cues, superpixel merging, or foreground-background segmentation; Selective Search was widely used because of its efficiency and high recall.
  • Traditional Computer Vision Methods: These traditional methods cannot be jointly optimized with the detection pipeline and struggled to generate high-quality proposals on MSCOCO.
  • Anchor-based Methods: Anchor-based methods match ground-truth boxes to predefined anchors, then use classification and regression branches to estimate objectness and refine box locations.
  • Anchor-based Methods: Anchor priors are manually designed across scales and aspect ratios, so their heuristic choices may be suboptimal and dataset-dependent.
  • Anchor-based Methods: RefineDet cascades localization-offset learning over progressively refined anchors, improving anchor quality and final prediction accuracy in a data-driven manner.
  • Keypoint-based Methods: Keypoint-based proposal methods form corner-based and center-based families, with CenterNet identified as a significant improvement and a promising future direction.

3.5. Feature Representation Learning

Feature representation learning addresses scale and aspect-ratio variation in object detection through three categories: multi-scale feature learning, contextual reasoning, and deformable feature learning. The survey highlights four multi-scale paradigms and describes how feature integration supports detection across object sizes.

  • Feature representation learning is critical because objects occur in complex environments with large scale and aspect-ratio variance.
  • The survey organizes feature representation learning into multi-scale feature learning, contextual reasoning, and deformable feature learning.
  • Multi-scale Feature Learning: Multi-scale feature learning has four paradigms: Image Pyramid, Prediction Pyramid, Integrated Features, and Feature Pyramid.
  • Image Pyramid: Image pyramids resize inputs to multiple scales and use multiple detectors, but their testing procedure can be computationally expensive.
  • Integrated Features and Prediction Pyramid: Integrated Features combine shallow spatial information with deep semantic information, while Prediction Pyramids make scale-specific predictions from multiple feature maps.
  • Feature Pyramid: Feature Pyramid Networks combine multi-scale features through top-down lateral connections and achieved state-of-the-art results in learning multi-scale features.

4. Learning Strategy

Training object detectors is harder than image classification because it must jointly optimize localization and classification. The survey therefore reviews learning strategies addressing imbalance, localization, acceleration, and related efficiency challenges.

  • Object detection jointly optimizes localization and classification, making robust detector training more difficult than image classification.
  • The survey identifies imbalance sampling, localization, and acceleration as issues requiring learning strategies for effective and efficient detectors.
  • The reviewed learning strategies include data augmentation, imbalance sampling, cascade learning, and localization refinement.

4.1. Training Stage

Training-stage strategies increase useful training variation, address class and difficulty imbalance, refine localization, and use cascaded processing to improve detector learning. The survey also notes practical limitations of separate refinement components and high-IoU localization.

  • Data Augmentation: Data augmentation expands training variation through operations including flips, rotation, random crops, expansion, and color jittering.
  • Imbalance Sampling: Class imbalance arises because most candidate proposals are background, while difficulty imbalance reflects the challenge of classifying hard objects against easy background proposals.
  • Imbalance Sampling: Hard negative sampling selects difficult negative proposals while maintaining a fixed foreground-background ratio to use informative background examples.
  • Imbalance Sampling: Focal loss suppresses easy-sample gradient signals so training focuses more on hard proposals, while GHM also avoids negative effects from outliers.
  • Localization Refinement: Localization refinement uses bounding-box regressors and auxiliary methods such as iterative regression and LocNet to improve preliminary predictions.
  • Localization Refinement: High-IoU requirements can classify insufficiently overlapping predictions as false positives even when they are correct under lower IoU requirements.
  • Localization Refinement: Separate auxiliary localization components can prevent joint optimization within the detection pipeline.
  • Cascade Learning: Cascade learning uses coarse-to-fine stages that reject easy negatives first and pass harder proposals to later classifiers.

4.2. Testing Stage

Testing-stage strategies remove duplicate predictions, accelerate inference, and improve robustness across scales. NMS removes highly overlapping boxes, whereas Soft-NMS decays scores to preserve clustered-object predictions; efficiency remains important for real-world deployment.

  • Dense detectors produce duplicate predictions that cannot be directly evaluated, motivating testing-stage strategies for duplicate removal and inference acceleration.
  • Duplicate Removal: NMS sorts boxes by confidence, keeps the highest-scoring box, and removes boxes whose IoU with it exceeds a predefined threshold.
  • Duplicate Removal: NMS can suppress a prediction for a nearby clustered object when its overlap with the selected box exceeds the threshold.
  • Duplicate Removal: Soft-NMS decays an overlapping box’s confidence score continuously instead of eliminating it, and improved performance on common benchmarks.
  • Model Acceleration: Detector acceleration includes efficient backbone architectures, offline model optimization, and deployment toolkits such as TensorRT.
  • Other Testing Strategies: Testing techniques such as image pyramids and horizontal flipping improve handling of different object scales but increase computation cost and may not suit real-world applications.

5. Applications

The survey presents face and pedestrian detection as applications requiring specialized designs because their properties differ from generic object detection. Deep-learning methods address scale variation, context, structure, and difficult samples through adapted architectures and training strategies.

  • Face Detection: Face detection differs from generic detection in scale range, blur, occlusion, and structural properties, motivating specialized algorithms.Faces can be much larger in scale range than generic objects and commonly involve blur and occlusion.
  • Face Detection: Deep-learning face detectors extend generic frameworks such as Fast R-CNN and SSD while emphasizing robust feature representations.Multi-scale feature learning is widely used to handle extreme scale variance.
  • Face Detection: Multi-scale architectures detect faces across sizes by combining feature maps, assigning scale ranges, or predicting on larger maps for small faces.Examples include S3FD, feature agglomeration, SSH, and multiple RPNs, though one lightweight multi-RPN method handled scale variance slowly.
  • Face Detection: Context-aware methods improve face detection by modeling surrounding human-body information and combining contextual and facial features.PyramidBox uses context anchors, a low-level feature pyramid, context-sensitive prediction, and data-anchor sampling.
  • Face Detection: Specialized loss functions, hard-negative mining, cascades, and position-sensitive pooling adapt generic detectors to face-specific variation.Reported methods include center loss, cascaded processing, and facial-part reweighting; several achieve state-of-the-art results on FDDB and WIDER benchmarks.
  • Other Applications: The applications discussion also covers pedestrian detection, including a novel Repulsion Loss for detecting pedestrians in crowds, and logo detection in e-commerce.The survey identifies logo detection as an important e-commerce research topic.

6. Detection Benchmarks

The survey reviews datasets and evaluation metrics for generic, face, and pedestrian detection. Benchmarks vary in scale, categories, annotations, difficulty settings, and metrics, with detection accuracy and inference speed both considered.

  • Evaluation Metrics: Detection evaluation considers both accuracy and inference speed, using mAP across tasks and additional IoU- and scale-specific scores for MSCOCO.VOC and ImageNet use an IoU threshold of 0.5 for mAP, whereas MSCOCO reports six scores covering IoU thresholds and object scales.
  • Face Detection Benchmarks: Face evaluation uses mAP at IoU 0.5 for WIDER FACE and PASCAL FACE, while FDDB reports TPR at 1k false positives with box- or ellipse-level annotations.FDDB therefore supports two annotation types and a threshold-specific true-positive-rate measure.
  • Pedestrian Detection Benchmarks: Pedestrian benchmarks include Caltech, ETH, INRIA, CityPersons, and KITTI, evaluated with log-average miss rate or mAP depending on the dataset.For CityPersons, INRIA, and ETH, log-average miss rate over nine FPPI points is used; KITTI uses mAP and difficulty levels.

7. State-of-the-art for Generic Object Detection

VOC and MSCOCO are common benchmarks for generic object detection, but they differ substantially in scene density and object-scale distributions. MSCOCO is more challenging because it contains many small objects across broad scale ranges.

  • Benchmark Scope: Pascal VOC2007, VOC2012, and MSCOCO are among the most commonly used datasets for evaluating detection algorithms.The survey reports recent benchmark results on VOC2007, VOC2012, and MSCOCO.
  • MSCOCO: MSCOCO contains nearly 10 objects per image, with most objects being small and distributed across large scale ranges.This distribution makes MSCOCO a very challenging detection task.

8. Concluding Remarks and Future Directions

The survey organizes recent object-detection advances and identifies open directions in proposal generation, context, AutoML, benchmarks, low-shot learning, backbones, and continual or large-batch training. It emphasizes that future progress requires efficient designs, broader benchmarks, and detection-specific methods.

  • Future Directions: Efficient proposal generation remains important because anchor-based methods have limitations and anchor-free approaches can incur high computation costs.The survey identifies scalable and efficient proposal generation as a future research direction.
  • Future Directions: Context can help or impede detection, but the survey reports limited work on using contextual information correctly and calls effective incorporation promising.Objects have relationships with surrounding context, making context encoding a distinct open challenge.
  • Future Directions: AutoML can improve detection over baselines but may require more than 100 GPU cards to train a single model, motivating lower-computation frameworks.Future work may learn backbone, proposal-generation, and region-encoding structures or data-augmentation policies.
  • Future Directions: LVIS expands benchmark scope with 164000 images, 1000+ categories, and 2.2 million instance masks while simulating low-shot conditions.The survey presents LVIS as a benchmark for more challenging detection, segmentation, and low-shot learning tasks.
  • Future Directions: Low-shot detection remains underdeveloped because detectors are data-hungry and existing methods face narrow testing or overfitting limitations.RepMet was tested on datasets with similar animal concepts, while LSTD still suffers from overfitting.
  • Future Directions: Detection-aware backbone architectures are needed because classification and detection have conflicts, although most state-of-the-art detectors still use classification backbones.Only a few methods use alternative choices such as CornerNet with Hourglass Net.
  • Future Directions: Large-batch and incremental learning remain open issues, including insufficient study of batch size and catastrophic forgetting without initial training data.The survey calls for further attention to these fundamental problems.
  • Concluding Remarks: The survey synthesizes recent literature systematically across detector components, learning strategies, applications, and benchmark evaluations.Its stated aim is to provide a comprehensive understanding of deep-learning object detection and stimulate further research.
Loading 1908.03673v1…