PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Deep Learning-based Object Detection

Licheng Jiao, Fan Zhang, Fang Liu, Shuyuan Yang, Lingling Li, Zhixi Feng, Rong Qu

arXiv:1907.09408v2cs.CV

TL;DR

Object detection requires methods that locate semantic objects across widely used applications while balancing accuracy and efficiency. This survey systematically analyzes typical models, datasets, applications, and representative one-stage and two-stage methods, then discusses system architectures and research trends. It reports developments spanning faster region-proposal pipelines, multi-scale feature extraction, and accuracy–speed trade-offs.

  • Problem

    Object detection is an important and challenging computer-vision task used across applications, while rapidly evolving methods make its development status difficult to understand comprehensively.

  • Method

    The survey systematically reviews deep learning-based object detection methods, including one-stage and two-stage detectors, datasets, applications, representative branches, system architectures, and research trends.

  • Results

    The reviewed methods show progress toward effective and efficient detection through shared region-proposal computation, multi-scale feature extraction, and combinations of accuracy- and speed-oriented designs.

  • Takeaways & Limitations

    The field’s development emphasizes improving accuracy and efficiency together through new architectures, richer representations, faster processing, anchor-free methods, and improved post-processing.

Abstract

from arXiv · show

Object detection is one of the most important and challenging branches of computer vision, which has been widely applied in peoples life, such as monitoring security, autonomous driving and so on, with the purpose of locating instances of semantic objects of a certain class. With the rapid development of deep learning networks for detection tasks, the performance of object detectors has been greatly improved. In order to understand the main development status of object detection pipeline, thoroughly and deeply, in this survey, we first analyze the methods of existing typical detection models and describe the benchmark datasets. Afterwards and primarily, we provide a comprehensive overview of a variety of object detection methods in a systematic manner, covering the one-stage and two-stage detectors. Moreover, we list the traditional and new applications. Some representative branches of object detection are analyzed as well. Finally, we discuss the architecture of exploiting these object detection methods to build an effective and efficient system and point out a set of development trends to better follow the state-of-the-art algorithms and further research.

I. INTRODUCTION

This survey reviews deep learning-based object detection, covering its applications, detector architectures, datasets, methods, and research trends. It emphasizes systematic analysis of one-stage and two-stage detectors and recent solutions while retaining foundational context.

  • Object detection locates instances of semantic object classes and supports applications including security monitoring, autonomous driving, transportation surveillance, drone analysis, and robotic vision.
  • Two kinds of object detectors: Existing detectors are commonly divided into two-stage methods, which prioritize localization and recognition accuracy, and one-stage methods, which prioritize inference speed.Two-stage detectors use region proposals before classification and regression, whereas one-stage detectors predict bounding boxes directly from input images.
  • Contributions: The survey addresses a gap in prior reviews by systematically covering up-to-date deep learning detection solutions, significant research trends, and in-depth analyses across multiple aspects.It focuses on how different deep learning methods are used rather than providing a complete summary of every related paper.
  • Backbone networks: Backbone networks extract feature maps from images, with deeper architectures supporting accuracy-oriented requirements and lightweight architectures supporting mobile or speed-oriented deployment.Examples discussed include ResNet and ResNeXt for richer features, and MobileNet, ShuffleNet, and SqueezeNet for lightweight processing.
  • R-CNN: R-CNN combines region proposals, fixed-length region features, class-specific SVM classification, and bounding-box regression for object detection.It uses selective search, 4096-dimensional CNN features, and warped 227 × 227 proposal inputs.

2) Fast R-CNN:

The survey traces detector improvements from shared feature extraction and end-to-end training to faster proposals, multi-scale features, and reduced RoI misalignment. It also contrasts these two-stage advances with YOLO’s real-time one-stage design and its localization trade-off.

  • Fast R-CNN: Fast R-CNN extracts the whole image once, uses RoI pooling for fixed-size region features, and jointly trains classification and bounding-box regression.This avoids separate CNN passes for each proposal and replaces R-CNN’s multi-stage training with a one-stage multi-task objective.
  • Fast R-CNN: 66.9% mAP versus 66.0% for R-CNN on PASCAL VOC 2007, while Fast R-CNN training took 9.5 hours versus 84 hours.With truncated SVD, test time was 0.32 seconds per image versus 47 seconds for R-CNN on an Nvidia K40 GPU.
  • Faster R-CNN: Faster R-CNN replaces slow selective search with an RPN that shares convolutional features and predicts proposals across scales and aspect ratios.Multi-scale anchors provide references for generating differently sized proposals without multiple input-image or feature scales.
  • Faster R-CNN: 69.9% mAP versus 66.9% for Fast R-CNN, with 198 ms versus 1830 ms running time and 5 fps versus 0.5 fps on PASCAL VOC 2007.The comparison used the same VGG backbone and shared convolutional computations.
  • Mask R-CNN: Mask R-CNN extends Faster R-CNN for instance segmentation and uses a ResNet-FPN backbone combining bottom-up, top-down, and lateral feature pathways.FPN extracts RoI features from different pyramid levels according to object scale.
  • Mask R-CNN: Replacing RoI pooling with RoIAlign reduces quantization-related misalignment through bilinear interpolation, improving box AP by 1.1 points on MS COCO.The ResNet-FPN backbone contributed an additional 1.7-point box AP improvement.
  • YOLO: YOLO frames full-image detection as regression, predicts fewer than 100 boxes per image, and runs at 45 frames per second on a Titan X GPU.On PASCAL VOC, YOLO achieved 63.4% mAP at 45 fps, but localization error was its main prediction error component.

2) YOLOv2:

YOLOv2 improves the speed–precision trade-off of one-stage detection through architectural and training changes, including batch normalization, higher-resolution classification, clustered anchors, fine-grained features, and multi-scale training.

  • Design improvements: YOLOv2 adds batch normalization ahead of each convolutional layer to accelerate network convergence.Mini-batch activations are normalized to have mean zero and variance one.
  • Design improvements: Fine-tuning the classifier at 448 × 448 for 10 ImageNet epochs increases detection mAP by 4%.
  • Design improvements: K-means dimension clusters for anchor-box priors improve YOLOv2 by almost 5% over the preceding anchor-box version.
  • Design improvements: Concatenating higher-resolution features with low-resolution features provides a modest 1% performance increase for small-object localization.The higher-resolution features are stacked into different channels.
  • Design improvements: YOLOv2 uses multi-scale training to predict detections at different image resolutions with the same network.Every 10 batches, the training image dimension is randomly selected from 320 through 608.
  • Results: 78.6% mAP at 40fps makes YOLOv2 more accurate and faster than Faster R-CNN with ResNet and SSD500 on the reported comparison.The comparison reports Faster R-CNN at 76.4% mAP and 5fps, and SSD500 at 76.8% mAP and 19fps.

6) RetinaNet:

RetinaNet is a one-stage detector that uses focal loss and ResNet-FPN to address foreground–background imbalance while retaining fast dense prediction. The survey reports improved accuracy, especially for small and medium objects, relative to DSSD513.

  • Method: RetinaNet uses focal loss to down-weight well-classified examples and focus training on hard examples under extreme foreground–background imbalance.
  • Results: 39.1% AP with ResNet-101-FPN exceeds DSSD513's 33.2% AP on the MS COCO test-dev dataset.With ResNeXt-101-FPN, RetinaNet reaches 40.8% AP.
  • Results: RetinaNet improves detection precision on small and medium objects by a large margin.
  • Related methods: Deformable convolutions learn offsets from fixed convolution kernels to adapt receptive fields to geometric variation in targets.DCNv1 reports almost 4% higher accuracy than three plain ConvNets on COCO test-dev.
  • Architecture: RetinaNet uses ResNet-FPN, whose five feature levels P3–P7 correspond to C3–C7 for predicting different object sizes.

3) NAS-FPN:

NAS-FPN uses searched top-down and bottom-up feature-fusion connections to improve multi-scale object detection. On COCO, it outperformed the original FPN, while the survey situates it among benchmark datasets and detection evaluation methods.

  • 3) NAS-FPN:: NAS-FPN uses neural architecture search to construct top-down and bottom-up connections that fuse features across scales.Its architecture repeats FPN structures and concatenates them during the search process.
  • 3) NAS-FPN:: 2.9% higher mAP than the original FPN was achieved by NAS-FPN with a ResNet-50 backbone and 256 feature dimensions on COCO test-dev.
  • 3) NAS-FPN:: The best reported NAS-FPN configuration used AmoebaNet, seven stacked FPNs, and 384 feature dimensions, achieving 48.0% on COCO test-dev.
  • Metrics: Mean average precision summarizes precision-recall performance across object categories, while VOC evaluation penalizes missed, duplicate, and false-positive detections.
  • Datasets: COCO contains 91 common object categories and 2,500,000 labeled instances in 328,000 images, with varied viewpoints and natural environments.
  • Datasets: COCO and PASCAL VOC provide benchmark images spanning iconic objects, iconic scenes, non-iconic objects, and annotated object instances.

2) Metric:

Object-detection benchmarks use different dataset scales, annotation settings, and IoU criteria. VOC uses a single IoU threshold, whereas ImageNet and VisDrone address distinct object-size and drone-scene challenges.

  • Datasets: ILSVRC2014 contains 200 object classes, nearly 450k training images, 20k validation images, and 40k test images.
  • 2) Metric:: ImageNet relaxes the localization threshold for small objects because a few pixels of deviation can be unacceptable under the VOC criterion.The threshold depends on the ground-truth box width and height and permits annotation extension around the object.
  • Datasets: VisDrone2018 contains 263 video clips and 10,209 images captured across urban and suburban areas of 14 Chinese cities.
  • Datasets: VisDrone2018 is challenging because it contains many small objects, dense scenes, and dark night images; 82.4% of training images contain more than 20 objects.The training set averages 54 objects per image across 6,471 images.

E. Open Images V5

Open Images V5 is a large, richly annotated benchmark, and its evaluation modifies VOC-style scoring for hierarchical labels, unannotated classes, and group-of regions. The survey also reviews feature, semantic, attention, and deformable-representation methods.

  • E. Open Images V5: Open Images V5 contains 16M bounding boxes for 600 object classes across 1.9M images.The broader Open Images dataset has 9.2M images with image-level labels, boxes, masks, and visual relationships.
  • E. Open Images V5: Open Images evaluation ignores unannotated classes to avoid counting them as false negatives.
  • E. Open Images V5: Detectors must report each relevant class when an object belongs to both a class and its subclass; omitting one counts as a false negative.
  • E. Open Images V5: Group-of boxes represent multiple mutually occluding or touching instances without requiring individual object localization inside the region.
  • Detection methods: Feature pyramids, semantic relationships, attention, and deformable convolutions are reviewed as approaches for improving representations and detecting objects across scales or under occlusion.
  • Detection methods: The survey concludes that richer and more appropriate object representations can improve detection accuracy.

B. Increasing localization accuracy

Localization accuracy is improved through IoU-based objectives, imbalance-aware classification losses, and detector designs that balance speed with precision. Anchor-based detectors remain constrained by fixed, costly, and imbalanced proposals, motivating anchor-free methods.

  • B. Increasing localization accuracy: IoU-based loss functions measure predicted-box accuracy, and generalized IoU improves consistency over smooth L1 loss in the reviewed pipeline.IoU is the intersection area divided by the union area of predicted and ground-truth boxes.
  • B. Increasing localization accuracy: Focal loss down-weights well-classified examples so one-stage detectors focus on hard examples instead of abundant easy negatives.
  • B. Increasing localization accuracy: RefineDet combines advantages of one-stage and two-stage detectors, achieving better accuracy than two-stage detectors with comparable one-stage efficiency.
  • B. Increasing localization accuracy: Small objects remain difficult because low resolution and noisy representations hinder detection, motivating multi-scale features and super-resolution approaches.
  • B. Increasing localization accuracy: Occlusion-aware methods, repulsion loss, and related designs target recall and accuracy degradation in crowded or occluded scenes.
  • G. anchor-free: Pre-defined anchor boxes use fixed scales and aspect ratios that are dataset-sensitive and cannot adapt during training to objects of all sizes.
  • G. anchor-free: Dense anchor placement creates substantial computation and memory overhead, while the predominance of negative anchors causes severe training imbalance.
  • G. anchor-free: Anchor-free methods address anchor limitations by locating object centers, corners, or distances from centers to bounding-box sides.

H. Training from scratch

Training and detector design balance pretrained initialization, spatially detailed representations, inference speed, and accuracy across object scales. The survey also highlights lightweight and adaptive architectures for efficient detection.

  • H. Training from scratch: Most state-of-the-art detectors initialize from classification backbones pretrained on large-scale datasets before fine-tuning for detection.Training from scratch is also possible, using batch normalization to provide stable and predictable gradients.
  • H. Training from scratch: Detection architectures differ from classification networks because localization requires fine-grained representations while classification emphasizes high-level semantics.One proposed architecture maintains high spatial resolution in deeper layers and avoids pretraining on a large classification dataset.
  • H. Training from scratch: Two-stage detectors are generally slower because their heavy heads increase processing time, motivating light-head designs that preserve time efficiency.The survey specifically identifies Faster R-CNN and R-FCN as examples of two-stage networks with heavy heads.
  • H. Training from scratch: Limited computing and memory resources motivate efficient detector architectures for mobile devices, real-time systems, webcams, and autonomous driving.The survey identifies YOLO- and SSD-based families and their improved variants as typical real-time detectors.
  • H. Training from scratch: High accuracy and high efficiency are presented as the ultimate joint goal, pursued through approaches that combine or adapt fast and accurate detectors.Examples include adaptively selecting a detector according to image difficulty and strengthening lightweight features with receptive fields.
  • H. Training from scratch: Multi-scale detection uses scaled images, differently sized filters, or predefined reference boxes to address objects appearing at different scales.Anchor-based architectures require heuristics to assign anchor size levels to object scale ranges.

1) Security field:

Object detection supports security, military, transportation, and medical applications, each presenting domain-specific challenges. The survey describes representative tasks, datasets, and methods addressing these settings.

  • 1) Security field:: Security applications include face, pedestrian, fingerprint, fraud, and anomaly detection.Face detection remains difficult under extreme pose, illumination, and resolution variation.
  • 1) Security field:: Pedestrian detection targets pedestrians, cyclists, and other riders in natural urban traffic scenes, with datasets and real-time detectors developed for this setting.The EuroCity Persons dataset is cited as a benchmark containing these categories.
  • 1) Security field:: Military applications include remote sensing object detection, topographic surveying, and flyer detection.Remote sensing detection is challenged by large inputs, small targets, complex backgrounds, and false detections.
  • 1) Security field:: Remote sensing methods address orientation, scale, and rotation challenges using data fusion, rotation-invariant layers, robust structures, and rotatable proposal networks.Benchmarks include NWPU VHR-10, HRRSD, DOTA, DLR 3K Munich, and VEDAI.
  • 1) Security field:: Transportation applications include license plate recognition, autonomous-driving perception, and traffic-sign recognition.Autonomous-vehicle perception transforms sensory data into semantic information, while 3D detection provides detailed object size and location information.
  • 1) Security field:: Medical detection supports computer-aided diagnosis and related tasks including cancer, glaucoma, DNA modification, neuron, synapse, and skin-lesion detection.Medical datasets face individual differences, data scarcity, privacy constraints, and source-to-target distribution differences.

5) Life field:

Object detection extends across life-oriented applications and specialized branches, including retail, event, pattern, caption, rain, shadow, and species analysis. These tasks often require specialized data, context, or supervision.

  • 5) Life field:: Life-oriented applications include intelligent homes, commodity, event, pattern, image-caption, rain, shadow, and species detection.The survey frames these as representative applications in the life field.
  • 5) Life field:: Retail shelf displays are densely packed scenes, motivating precise detection methods and the SKU-110K dataset.The dataset is introduced specifically to meet the challenge of densely packed retail objects.
  • 5) Life field:: Event detection discovers real-world events from Internet data, with multi-domain frameworks and social-interaction features used to handle diverse social-media information.Examples include festivals, talks, protests, disasters, and elections.
  • 5) Life field:: Pattern detection must handle occlusion, pose variation, illumination changes, and sensor noise across 2D images and 3D point clouds.The survey notes strong baselines for repeated or periodic structures in both representations.
  • 5) Life field:: Weakly supervised detection reduces annotation demands by learning from images labeled with object classes but not bounding boxes.Bounding-box annotation is described as expensive, laborious, and impractical at large scale.
  • 5) Life field:: Salient-object detection predicts saliency scores and accurate saliency maps, commonly aggregating multi-level backbone features.One cited approach reports that discarding shallower features can improve speed while deeper features remain sufficient for precise maps.

4) Edge detection:

The survey covers edge and text detection alongside domain-general and video, 3D, and pose-related branches. These settings introduce challenges involving scale, orientation, temporal quality, domain variation, depth, and sparsity.

  • 4) Edge detection:: Edge detection extracts object boundaries and salient edges for higher-level tasks such as segmentation, detection, and recognition.It must combine object-level boundaries with useful local details across edge scales.
  • 4) Edge detection:: Text detection identifies text regions and supports tasks including classification and video analysis.Natural-scene text remains challenging under blur, uneven lighting, perspective distortion, and varied orientation.
  • 4) Edge detection:: Scene-text methods either treat text as a general object for box regression or directly segment text regions.The former struggles with large aspect ratios and arbitrary orientations, while the latter often requires complicated post-processing.
  • 4) Edge detection:: Corner-point localization and relative-region division can handle long oriented text with simple NMS post-processing.Rotation Region Proposal Networks provide another approach by generating inclined proposals with text orientation angles.
  • 4) Edge detection:: Universal detectors seek to work across image domains without prior knowledge of a newly encountered domain.Domain attention, domain-specific batch normalization, and adapter residual modules are cited as approaches for multi-domain representations.
  • 4) Edge detection:: Video detection uses temporal context to address motion blur, defocus, and unstable classifications across frames.Linking neighboring-frame detections can be difficult because object motion reduces overlap and predicted movements may be inaccurate.
  • 4) Edge detection:: LiDAR-based 3D detection provides depth for locating objects and characterizing shapes, while facing sparsity and variable point density.VoxelNet is described as learning point-cloud features and predicting accurate 3D bounding boxes end to end.
  • 4) Edge detection:: Human pose detection estimates 2D or 3D body-joint locations and pose classes, including simultaneous multi-person 2D and 3D prediction.The cited end-to-end architecture operates on natural images.

10) Fine-Grained Visual Recognition:

This section situates fine-grained recognition among related object-detection branches and highlights challenges from small inter-category differences, viewpoint variation, and complex scenes. It also identifies research directions involving richer representations, domain transfer, weak supervision, and improved detector design.

  • Fine-grained recognition: Fine-grained recognition distinguishes exact object categories, such as bird species or aircraft models, despite small visual differences.Pose, viewpoint, and object location can overwhelm category-specific visual cues.
  • Related detection branches: 3D detection uses monocular images or combined LiDAR and RGB inputs to predict oriented 3D bounding boxes in autonomous driving.The survey references an example result projected onto an image from the KITTI validation set.
  • Fine-grained recognition: 3D representations can improve viewpoint generalization by modeling both local feature appearance and object location.
  • Fine-grained recognition: Saliency-guided Faster R-CNN and weakly supervised discriminative localization are presented as approaches for fine-grained localization and classification.
  • Research directions: Future detector development targets the accuracy–efficiency trade-off, including architecture design, richer features, faster processing, anchor-free methods, and improved post-processing.
  • Research directions: Weakly supervised, multi-domain, and LiDAR-based detection address annotation cost, domain variation, and reliable 3D localization, respectively.The survey also identifies video complexity and combining one-stage with two-stage detectors as continuing challenges.

7) Salient object detection:

Salient object detection highlights salient regions to support object-level and video recognition, while related directions emphasize multitask, multimodal, edge, medical, remote-sensing, and synthetic-data systems. The survey presents these areas as active research directions requiring further study.

  • Salient object detection: Salient object detection highlights salient object regions, and those regions can assist accurate object detection in video frames.
  • Emerging directions: Unsupervised object detection seeks to reduce the need for expensive, laborious bounding-box annotation by using automatic annotation technology.
  • Emerging directions: Multitask learning aggregates detection with segmentation, edge detection, and other vision tasks to provide richer information.The survey describes this as a way to enhance separate task performance, while noting substantial challenges.
  • Emerging directions: Fusing images with textual descriptions is identified as an emerging direction enabled by accessible multi-source information.
  • Deployment applications: Lightweight networks and FPGA-based detection systems support more efficient terminal devices and real-time applications.
  • Deployment applications: Medical, remote-sensing, and synthetic-data applications extend object detection toward diagnosis, high-altitude analysis, and stronger detector robustness.The cited examples include IDx-DR, SoC-based remote sensing, and mixing real scenes with GAN-generated data.
  • Conclusion: The survey concludes that object detection still requires further study and anticipates future breakthroughs from deep learning methods.
Loading 1907.09408v2…