PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Modern Deep Learning based Object Detection Models

Syed Sahil Abbas Zaidi, Mohammad Samar Ansari, Asra Aslam, Nadia Kanwal, Mamoona Asghar, Brian Lee

arXiv:2104.11892v2cs.CVcs.LGeess.IV

TL;DR

Object detection requires classifying and localizing objects across increasingly demanding applications, while models must balance accuracy, speed, and deployment constraints. This survey synthesizes detector families, datasets, metrics, backbone architectures, and lightweight models, then compares reported performance across benchmark settings. It concludes that two-stage detectors are generally more accurate but slower, while newer one-stage detectors can be comparably accurate and much faster.

  • Problem

    Object detection must identify and localize objects across challenging settings, while practical systems increasingly require lightweight models for mobile and embedded deployment.

  • Method

    The paper surveys single- and two-stage detectors, benchmark datasets, evaluation metrics, backbone architectures, lightweight classification models, and reported comparative results.

  • Results

    Two-stage detectors are generally more accurate but slower, while one-stage detectors have become equally accurate and much faster according to the survey's comparison.

  • Takeaways & Limitations

    The survey highlights lightweight models and faster one-stage detectors as relevant to real-time applications and mobile or embedded deployment.

Abstract

from arXiv · show

Object Detection is the task of classification and localization of objects in an image or video. It has gained prominence in recent years due to its widespread applications. This article surveys recent developments in deep learning based object detectors. Concise overview of benchmark datasets and evaluation metrics used in detection is also provided along with some of the prominent backbone architectures used in recognition tasks. It also covers contemporary lightweight classification models used on edge devices. Lastly, we compare the performances of these architectures on multiple metrics.

I. INTRODUCTION

Object detection evolved from slow, handcrafted feature systems into deep learning-based models that classify and localize object instances. This survey reviews detector categories, backbone and lightweight architectures, datasets, metrics, and persistent deployment challenges.

  • Early handcrafted detectors were slow, inaccurate, and poor on unfamiliar datasets, whereas CNN-based deep learning changed visual perception research.
  • The survey comprehensively reviews single- and two-stage detectors, their historical evolution, backbone architectures, and lightweight classification models.
  • The paper systematically organizes object detection problems, challenges, benchmark datasets, evaluation metrics, backbone architectures, detector categories, and future trends.
  • Object detection identifies all instances of predefined classes and localizes them with axis-aligned bounding boxes.
  • Object detection remains challenging because real-world networks face intra-class variation, many object categories, and high computational requirements.

III. DATASETS AND EVALUATION METRICS

The survey presents commonly used object-detection datasets and explains how their scale, categories, viewpoints, and evaluation protocols differ. Pascal VOC, ILSVRC, and MS-COCO provide progressively varied benchmark settings and metrics.

  • Pascal VOC evolved from four object classes in 2005 to 20 categories, with VOC12 containing 11k training images and more than 27k labelled objects.
  • ILSVRC scaled beyond one million images across 1000 classification classes, using 200 selected classes and more than 500k images for detection.
  • MS-COCO evaluates detection using AP across IoU thresholds from 0.5 to 0.95 and separately reports performance for small, medium, and large objects.

4) Open Image:

Open Images is a large, diverse object-localization dataset, but its class frequencies are highly imbalanced. The survey also explains how IoU-based precision, recall, AP, and mAP evaluate detectors.

  • Open Images contains 9.2 million images and, for detection, 16 million bounding boxes across 600 categories on 1.9 million images.
  • Open Images is highly class-skewed, ranging from 378077 images for Man to only 3 for Paper Cutter.
  • Class-frequency skew is expected to bias training because frequent classes substantially outnumber rare classes across the discussed datasets.
  • IoU compares predicted and ground-truth boxes, with thresholded detections classified as true or false positives while missed objects are false negatives.
  • Mean average precision averages class-specific average precision values into a single final detector-evaluation metric.

IV. BACKBONE ARCHITECTURES

Backbone architectures extract image features for object detectors, and this survey reviews milestone designs from AlexNet through GoogLeNet while highlighting their accuracy, efficiency, and dataset context.

  • Backbone architectures extract features from input images for modern object detectors.
  • AlexNet: AlexNet used five convolutional and three fully connected learnable layers and achieved more than 26% higher accuracy than contemporary models.
  • The survey situates backbone analysis alongside comparisons of object-detection datasets, including PascalVOC, ImageNet, and MS-COCO.
  • VGG: VGG used small convolutional filters to construct networks of varying depths and later became a widely used classification and detection backbone.
  • GoogLeNet/Inception: GoogLeNet addressed resource use with locally sparse connections, stacked Inception modules, multiple filter sizes, and auxiliary intermediate classifiers.

D. ResNets

The survey reviews backbone designs that address degradation, complexity, redundant computation, and scaling trade-offs, while contrasting detector architectures and dataset annotation coverage.

  • ResNets: ResNet mitigated accuracy degradation in deep networks with parameter-free skip connections between stacked layers.
  • ResNets: A 16-layer VGG network had higher complexity and lower accuracy than considerably deeper 101- and 152-layer ResNet architectures.
  • ResNeXt: ResNeXt targeted diminishing gains from increasing depth or width with a simpler and more efficient architecture inspired by repeated blocks and split-transform-merge behavior.
  • CSPNet: CSPNet reduced duplicate gradient information by separating base-layer feature maps into different gradient-flow paths.
  • EfficientNet: EfficientNet used a compound coefficient to uniformly scale network depth, width, and resolution after studying their individual performance costs.
  • Object detectors: Single-stage detectors classify and localize objects in one shot, generally offering simpler design and better real-time performance than two-stage detectors.

A. Pioneer Work

The section traces object detection from hand-crafted feature methods to CNN-based two-stage architectures, emphasizing progressive gains in speed, accuracy, shared computation, and functionality.

  • Traditional detectors: Viola-Jones combined Haar-like features, integral images, AdaBoost, and cascading classifiers for face detection.
  • Traditional detectors: HOG extracted edge gradients and orientations from image-grid cells before classification with a linear SVM.
  • Traditional detectors: DPM improved on HOG by detecting individual object parts and modeling their probable arrangement as the final detection.
  • CNN-based detectors: R-CNN converted detection into classification and localization by generating about 2000 class-agnostic region proposals for CNN processing.
  • CNN-based detectors: 146x speed improvement over R-CNN made Fast R-CNN near real time, while its accuracy increase was supplementary.
  • CNN-based detectors: Faster R-CNN improved accuracy by more than 3%, reduced inference time by an order of magnitude, and reached 5 frames per second by learning region proposals.
  • Architectural advances: FPN supplied high-level semantic features at multiple scales, reducing detection error and becoming a standard component of later detectors.
  • Architectural advances: R-FCN shared almost all computations across proposals and was 2.5-20 times faster than its counterpart while addressing convolutional translation invariance.

C. Single Stage Detectors

Single-stage detectors predict objects in one pass, trading proposal-free speed for challenges in localization and class imbalance. The section traces improvements from YOLO and SSD through focal-loss, point-based, scalable, and transformer-based designs.

  • YOLO: YOLO reframed detection as regression by directly predicting object locations and bounding-box attributes from an S × S image grid.The grid cell containing an object’s center is responsible for detecting it.
  • YOLO: YOLO achieved major speed and accuracy gains but initially struggled with small or clustered objects and limited detections per cell.Later YOLO versions addressed these shortcomings.
  • SSD: SSD matched contemporary two-stage detector accuracy while maintaining real-time speed through multi-scale auxiliary convolutional layers.Earlier layers detect smaller objects, while deeper layers handle default-box offsets and aspect ratios.
  • SSD: SSD was significantly faster and more accurate than YOLO and Faster R-CNN but had difficulty detecting small objects.Later backbone improvements such as ResNet helped address this limitation.
  • RetinaNet: RetinaNet used focal loss to reduce easy-example contributions and address extreme foreground-background class imbalance.The survey reports better accuracy and runtime than two-stage detectors, alongside simpler training and faster convergence.
  • Later single-stage detectors: CenterNet represented objects as center points, eliminating the need for non-maximum suppression, while EfficientDet and YOLOv4 emphasized scalable efficiency and real-time performance.CenterNet required specialized backbones; EfficientDet was smaller and computationally cheaper, while YOLOv4 was twice as fast as EfficientDet with comparable performance.
  • Transformers: Swin Transformer achieved state-of-the-art performance on MS COCO but used comparatively more parameters than convolutional models.Transformers are presented as a paradigm shift from CNN-based vision architectures.

VI. LIGHTWEIGHT NETWORKS

Lightweight network research targets high-performing models for resource-constrained environments, especially IoT and edge deployments. The survey highlights efficient architecture design alongside compression and knowledge-transfer techniques.

  • Motivation: Many accurate real-time object detectors remain unsuitable for deployment because they require excessive computing resources.This motivates small and efficient networks for resource-constrained environments such as IoT deployments.
  • Approaches: Efficient model development uses pruning, quantization, hashing, and distillation to improve deep-learning efficiency.The survey also examines prominent efficient neural-network architectures for high performance.

A. SqueezeNet

SqueezeNet reduces model parameters while maintaining performance through compact convolutional design choices. Its fire modules combine squeeze and expand layers to limit channel growth.

  • Fire module: SqueezeNet’s fire module combines a 1×1-filter squeeze layer with an expand layer using 1×1 and 3×3 filters.The architecture stacks eight fire modules between convolutional layers.

B. MobileNets

MobileNet pursues compactness through efficient architecture rather than conventional model shrinking or compression. Its central operation is depthwise separable convolution.

  • Architecture: MobileNet uses depthwise separable convolution instead of shrinking, pruning, quantization, or compressing an existing network.This design factorizes standard convolution into depthwise and 1×1 pointwise convolution.

C. ShuffleNet

ShuffleNet targets mobile deployment by reducing computation and addressing information-flow limits in efficient convolutional networks. ShuffleNetv2 extends this focus by emphasizing direct platform speed and latency measurements.

  • ShuffleNet: ShuffleNet uses group convolution with channel shuffle to reduce the information-flow limitations associated with expensive 1x1 convolutions.The architecture was specifically designed for mobile devices and aims to remain computationally efficient when networks are scaled down.
  • Related lightweight architectures: MobileNetv2 introduced inverted residuals with linear bottlenecks, expanding, depthwise-filtering, and projecting representations to reduce computation while improving accuracy.This differs from common residual blocks, which compress, convolve, and then expand representations.
  • PeleeNet: PeleeNet uses conventional convolutions and computation-conserving design choices, including two-way dense layers and a stem block.Its two-way dense layers provide different receptive-field scales, while the stem block is intended to reduce information loss.
  • ShuffleNetv2: ShuffleNetv2 proposes four design principles centered on equal channel widths, platform-specific group convolution, and direct speed or latency measurement.The authors argue that speed and latency are more practical indicators of computational complexity than indirect metrics such as FLOPs.

G. MnasNet

MnasNet uses hardware-aware neural architecture search to balance accuracy and mobile latency, while MobileNetv3 and OFA extend automated design toward efficient, adaptable lightweight models.

  • G. MnasNet: MnasNet formulates architecture search as multi-object optimization of accuracy and mobile-device latency using an RNN-based reinforcement-learning controller.Sampled models are trained for accuracy and executed on real devices for latency before the controller is updated.
  • G. MnasNet: MnasNet was almost twice as fast as MobileNetv2 while achieving higher accuracy, but its reinforcement-learning search required astronomical computational resources.The result reflects the trade-off between efficient deployed models and expensive architecture-search procedures.
  • H. MobileNetv3: MobileNetv3 combines platform-aware neural architecture search with NetAdapt, channel trimming, fine-tuning, and cheaper hard-swish activations.It provides MobileNetv3-Large and MobileNetv3-Small for different resource-use cases.
  • I. Once-For-All (OFA): Once-for-All decouples model training from neural architecture search, allowing sub-networks to be distilled from one trained network for different requirements.The approach varies depth and width while preserving performance through channel sorting and selective layer use.
  • I. Once-For-All (OFA): 80% ImageNet top-1 accuracy was achieved by OFA, which also won the 4th Low Power Computer Vision Challenge while reducing GPU training hours by many orders of magnitude.The paper presents this as a paradigm for designing lightweight models across varied hardware requirements.
  • Evaluation: The survey compares object detectors on PASCAL VOC 2012 and MS COCO under similar input-image sizes, while lightweight models are compared using accuracy, latency, parameters, and MFLOPs.The comparison notes that input size, feature extractor, GPU architecture, proposals, training, and loss function affect detector performance.

VIII. FUTURE TRENDS

The survey identifies future challenges spanning efficiency, data annotation, domain transfer, video reasoning, 3D detection, and automated architecture search. It emphasizes that accurate, deployable detection remains unresolved despite substantial progress.

  • AutoML: Neural architecture search is promising for determining detector characteristics but remains nascent, complex, and resource intensive.The survey identifies search cost as a continuing barrier to broader use of NAS-designed detectors.
  • Lightweight detectors: Lightweight networks can match full-sized models on classification errors but still trail them in detection accuracy by more than 50%.The survey links this gap to the continuing need for small, efficient, and accurate on-device detectors.
  • Weakly supervised/few-shot detection: Training state-of-the-art detectors on millions of bounding-box annotations is unscalable because annotation requires substantial time and resources.Weakly supervised or few-shot detection using image-level labels is presented as a way to reduce these annotation costs.
  • Domain transfer: Domain transfer seeks to reuse models trained on labeled source tasks for related target tasks, reducing reliance on large target datasets.The passage frames transfer as reuse across related tasks rather than as a detector-specific architecture change.
  • 3D object detection: 3D detection remains critical for autonomous driving because below-human-level deployment performance raises safety concerns.The survey notes that high accuracy has been achieved, but safety remains the practical boundary.
  • Object detection in video: Video detection remains open because image-by-image detectors do not exploit spatial and temporal relationships between frames.The challenge is to incorporate cross-frame correlation into object recognition.
  • Conclusion: The survey concludes that detectors remain far from performance saturation, while real-world applications increase demand for lightweight mobile and embedded models.It reports that two-stage detectors are generally more accurate but slower, whereas recent one-stage detectors can be similarly accurate and faster; Swin Transformer is identified as the most accurate detector in Figure 10.
Loading 2104.11892v2…