PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Generic Object Detection: A Survey

Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, Matti Pietikäinen

arXiv:1809.02165v4cs.CV

TL;DR

Generic object detection has advanced rapidly through deep learning, but the field lacked a comprehensive recent survey. This paper synthesizes those advances across detection methods and concludes with performance patterns, challenges, and future directions.

  • Problem

    Rapid progress in deep-learning-based generic object detection created a need for a comprehensive survey of advances from the preceding five years.

  • Method

    The paper provides a structural survey covering detection frameworks, feature representation, proposal generation, context modeling, training strategies, datasets, evaluation criteria, and representative-method performance.

  • Results

    Two-stage detectors generally achieve higher accuracy, especially for small objects, while one-stage detectors are generally faster and competitive on large objects; multistage detection offers a prospective speed-accuracy trade-off.

  • Takeaways & Limitations

    The survey organizes recent achievements into a taxonomy and identifies backbone efficiency, robustness to variation, and multistage detection as important directions for future research.

  • Takeaways & Limitations

    Small-object detection accuracy remains substantially lower than large-object accuracy, making small objects a key unresolved challenge.

Abstract

from arXiv · show

Object detection, one of the most fundamental and challenging problems in computer vision, seeks to locate object instances from a large number of predefined categories in natural images. Deep learning techniques have emerged as a powerful strategy for learning feature representations directly from data and have led to remarkable breakthroughs in the field of generic object detection. Given this period of rapid evolution, the goal of this paper is to provide a comprehensive survey of the recent achievements in this field brought about by deep learning techniques. More than 300 research contributions are included in this survey, covering many aspects of generic object detection: detection frameworks, object feature representation, object proposal generation, context modeling, training strategies, and evaluation metrics. We finish the survey by identifying promising directions for future research.

1 Introduction

Generic object detection locates instances from predefined categories and supports many higher-level vision applications. This survey reviews rapid deep-learning-driven progress and organizes recent work into an accessible taxonomy.

  • Problem: Object detection determines whether category instances occur in an image and returns each instance’s spatial location and extent.It forms a basis for segmentation, scene understanding, tracking, captioning, and other vision tasks.
  • Deep-learning progress: Deep learning techniques learn feature representations automatically from data and have substantially improved object detection performance.The survey associates this progress with the emergence of deep learning and the post-2012 shift toward related deep networks.
  • Problem: Generic detection differs from specific-instance detection by generalizing to usually unseen instances from predefined object categories.Specific-instance detection is essentially a matching problem, whereas generic detection targets categories such as humans, cars, bicycles, and dogs.
  • Motivation: Existing surveys largely predate deep learning’s recent success, leaving a need for a comprehensive review of generic object detection.Earlier reviews mainly covered specific object detectors or work published before 2012.
  • Survey scope: The paper surveys recent techniques and proposes a taxonomy organized around datasets, evaluation metrics, context modeling, and detection proposal methods.Its organization is intended to clarify similarities and differences among strategies and help identify future research challenges.

2 Generic Object Detection

Generic object detection locates instances from broad predefined categories, requiring both accurate recognition and localization across substantial visual variation and efficient operation. Its challenges include intra-class diversity, inter-class similarity, computational demands, and scalability to growing categories and datasets.

  • Definition: Generic object detection identifies instances from many predefined categories and returns each instance’s spatial location and extent.The survey adopts bounding boxes as the most widely used representation for evaluating generic detection.
  • Goals: High-quality detection must combine distinctiveness between categories with robustness to appearance variation within categories.High efficiency additionally requires real-time operation with acceptable memory and storage demands.
  • Challenges: Detection accuracy is challenged by wide intra-class variation and the large number of object categories.Variations arise from object properties, poses, imaging conditions, occlusion, clutter, blur, and related factors.
  • Efficiency: Mobile and wearable devices make efficient detection important because their computational capabilities and storage space are limited.Complexity also grows with the number of categories, image locations, and scales.
  • Scalability: Scalable detectors must handle previously unseen objects, unknown situations, and high data rates as image and category counts grow.Increasing annotation demands motivate reliance on weakly supervised strategies.
  • Progress: Deep convolutional neural networks marked a major shift from handcrafted local descriptors toward learned representations for general object categories.Their success followed record-breaking image-classification results and transfer to object detection.

3 A Brief Introduction to Deep Learning

Deep learning uses multilayer models to learn hierarchical representations directly from data, with CNNs repeatedly applying convolution, nonlinearities, and pooling. These models have advanced visual recognition but depend on substantial data and computation and retain important practical deficiencies.

  • Survey context: Deep learning surveys have examined techniques from multiple perspectives and applications, including medical image analysis.
  • CNNs: CNNs learn representations at multiple abstraction levels through hierarchical layers exploiting translation invariance, local connectivity, and compositional structure.Deep CNNs are networks with a large number of layers.
  • CNN operations: A typical CNN repeatedly applies convolution, nonlinearities such as ReLU, and pooling to transform feature maps while reducing resolution.Pooling performs feature-map downsampling or upsampling, depending on the operation described.
  • Enablers: Large labeled datasets and highly capable GPUs have been key contributors to the success of deep convolutional neural networks.
  • Deficiencies: Deep convolutional neural networks require extensive labeled data and expensive computation, while their parameters and decisions remain difficult to interpret and vulnerable to degradation and attacks.

4 Datasets and Performance Evaluation

Object-detection datasets provide standardized grounds for training and comparing detectors, while evaluation combines speed, precision, and recall through metrics such as AP and mAP. Dataset design increasingly emphasizes realistic scenes, diverse scales, clutter, occlusion, and localization quality.

  • Datasets: PASCAL VOC, ImageNet, MS COCO, and Open Images form the backbone of major object-detection challenges.Each challenge provides images, ground-truth annotations, standardized evaluation software, and competitions or workshops.
  • Datasets: MS COCO addresses the atypical large, centered objects of ImageNet with complex everyday scenes, natural context, and fully segmented instance annotations.
  • Dataset properties: MS COCO includes objects at varied scales, clutter, heavy occlusion, and an evaluation metric that encourages more accurate localization.
  • Dataset properties: Just like ImageNet in its time, MS COCO has become the standard for object detection today.
  • Datasets: Open Images differs from earlier large-scale datasets through its expanded classes, images, annotations, and annotation process.
  • Evaluation metrics: Detection performance is evaluated using speed in Frames Per Second, precision, and recall, with Average Precision and mean Average Precision as common summary metrics.AP is computed per category from precision and recall, while mAP averages AP over categories.
  • Evaluation criteria: A detection is a True Positive when its category matches the ground truth and its Intersection over Union reaches the predefined threshold, typically 0.5.Confidence thresholds determine accepted predictions, and duplicate detections of one object are counted as false positives except for the highest-confidence detection.
  • Evaluation metrics: Average Precision is computed from ranked detections by determining true and false positives, then varying the confidence threshold to obtain precision and recall.

5 Detection Frameworks

Deep-learning detection frameworks evolved from proposal-based pipelines toward more integrated designs that share computation and reduce proposal bottlenecks. The survey organizes these approaches into two-stage and one-stage frameworks and highlights their differing speed, accuracy, and training trade-offs.

  • Framework taxonomy: Generic detectors are organized into two-stage frameworks with object proposals and one-stage frameworks without a separate proposal process.Region-based frameworks generate category-independent candidate regions before classification; one-stage frameworks combine detection into a single proposed method.
  • Region-based frameworks: RCNN combines selective-search region proposals with CNN features and category-specific classifiers in a multistage detection pipeline.Its training includes proposal computation, CNN fine-tuning, and subsequent classifier and bounding-box regressor training.
  • Region-based frameworks: RCNN’s multistage training, proposal-level feature extraction, storage demands, and slow testing motivated successive frameworks including SPPNet, Fast RCNN, and Faster RCNN.The cited drawbacks are especially problematic for large-scale detection and deep networks such as VGG16.
  • Region-based frameworks: Fast RCNN improves detection quality while enabling end-to-end training, typically making training 3 times faster and testing 10 times faster than RCNN or SPPNet.Its shared feature computation is followed by softmax classification and class-specific bounding-box regression for proposal refinement.
  • Region-based frameworks: Faster RCNN replaces external region proposals with a CNN-based proposal mechanism and reaches 5 FPS on VGG16 while achieving state-of-the-art PASCAL VOC 2007 accuracy using 300 proposals per image.This removes selective search and supports an integrated CNN-only detector.
  • Region-based frameworks: RFCN shares almost all computations across the image through a fully convolutional architecture, while position-sensitive score maps address localization concerns in shared convolutional features.The design avoids hidden fully connected layers and reduces the cost of applying a region-wise subnetwork separately to several hundred RoIs.
  • Unified frameworks: OverFeat gains a significant speed advantage by sharing convolutional computation across overlapping windows, but is less accurate than RCNN.Its classifier and regressors are trained sequentially, unlike later unified frameworks such as YOLO and SSD.

6 Object Representation

Object representation in generic detection has shifted from handcrafted descriptors to learned CNN features, with network architecture and multiscale design becoming central to accuracy. The survey covers deeper and more efficient backbones, feature fusion, scale-aware receptive fields, and geometric-variation challenges.

  • Feature representation: Deep CNNs learn multilevel feature representations directly from raw images, reducing dependence on domain-specific feature engineering and complex handcrafted procedures.This shifts the representation burden toward designing better network architectures and training procedures.
  • CNN architectures: CNN architecture plays a crucial role in detection accuracy, motivating improvements through backbones such as AlexNet, VGGNet, GoogLeNet, Inception, ResNet, DenseNet, and SENet.The survey summarizes these architectures and their use in generic object detection frameworks.
  • CNN architectures: Architecture evolution has generally increased depth while reducing parameter counts through designs such as Inception modules, skip connections, and avoidance of fully connected layers.ResNet demonstrated effective learning in networks with hundreds of layers, while newer architectures can be deep yet parameter-efficient.
  • Feature representation: Feature representations can be improved by combining information from different layers, including dense connectivity, channel-wise attention, and deep supervision.Dense connections encourage feature reuse and alleviate vanishing gradients, while Squeeze-and-Excitation blocks recalibrate channel-wise responses.
  • Training representations: Fine-tuning pretrained CNNs can significantly increase detection performance, and detection accuracy differs across features extracted from different network layers.For ImageNet-pretrained AlexNet, FC6, FC7, and Pool5 are reported in descending order of detection accuracy.
  • Handling object scale variations: Multiscale feature designs integrate deep, intermediate, and shallow features for proposals and prediction, improving descriptiveness and localization at increased computational complexity.FPN also shows significant improvement as a generic feature extractor in object detection and instance segmentation.
  • Handling object scale variations: Multibranch architectures use different kernels, resolutions, or receptive fields to enhance feature robustness and accommodate objects at different scales.RFB, MSCNN, and TridentNet represent this family of scale-aware designs.
  • Handling other intraclass variations: Generic detection research has limited focus on rotation invariance because common benchmarks such as PASCAL VOC, ImageNet, and COCO do not present rotated images.Rotation invariance is more prominent in applications including scene text, face detection, and aerial imagery.

7 Context Modeling

Context modeling uses scene-level, spatial, scale, and local relational cues to supplement object appearance, especially when appearance is weak. Deep learning detectors explore both explicit global context and local object interactions or enlarged contextual regions.

  • Context is important for detection when small size, occlusion, or poor image quality makes object appearance insufficient.
  • Semantic, spatial, and scale context capture scene likelihoods, positional relationships, and relative object sizes.
  • Although modern detectors exploit context implicitly through hierarchical DCNN representations, researchers also model contextual information explicitly.
  • Global Context: Global-context methods use image- or scene-level cues, including classification scores, spatial RNNs, Markov random fields, and semantic segmentation.
  • Local Context: Local-context methods model nearby-object relations or enlarge detection windows to extract surrounding features, including SMN, ORN, SIN, MRCNN, GBDNet, ACCNN, and CoupleNet.

8 Detection Proposal Methods

Detection proposals reduce the prohibitive cost of exhaustive sliding-window search by producing a limited set of likely object regions. Deep learning proposal methods include bounding-box and segment proposals, with RPN-based and multilayer-feature approaches becoming central to modern detectors.

  • Proposal Motivation: Object proposals seek high recall, accurate localization, and low computational cost while enabling faster detection with far fewer candidate regions.
  • Proposal Motivation: Since 2014, detection proposals have been a standard preprocessing step, appearing in all winning PASCAL VOC, ILSVRC, and MS COCO detection challenge entries.
  • Bounding Box Proposals: RPN slides a small network over shared convolutional features and predicts k proposals at each location using k anchor boxes.
  • Sliding Window Detection: Sliding-window detection can require 10^4-10^5 windows per image, increasing to 10^6-10^7 across scales and aspect ratios.
  • Multilayer Features: Multilayer-feature methods generate or refine proposals using cascades, aggregated features, or feature maps at different network depths to combine details and semantics.
  • Proposal Categories: DCNN proposal methods divide into bounding-box and object-segment approaches; segment proposals provide more information and move toward instance segmentation.

9 Other Issues

The survey reviews training and evaluation issues beyond core detector architectures, including augmentation, small-object scale variation, localization error, and class imbalance. Proposed solutions include contextual augmentation, multistage localization refinement, improved suppression, and imbalance-focused training.

  • Data Augmentation: Data augmentation perturbs images through transformations such as cropping, flipping, scaling, translation, color changes, and noise to reduce overfitting and improve generalization.
  • Novel Training Strategies: Higher image resolution increases the possibility of detecting small objects, making scaling a common component of augmentation strategies.
  • Reducing Localization Error: Localization errors arise from insufficient overlap or duplicate detections and are among the most influential detection errors besides confusion between similar objects.
  • Reducing Localization Error: Cascade RCNN uses sequential detectors with increasing IOU thresholds and achieves consistent gains of about 2 to 4 points at a marginal computational increase.
  • Class Imbalance Handling: Object detection suffers severe imbalance because numerous easy background examples can overwhelm training relative to labeled object instances.

10 Discussion and Conclusion

The survey synthesizes deep-learning advances in generic object detection, compares representative detectors, and identifies persistent challenges and future directions. Detection accuracy depends strongly on backbones, frameworks, and large-scale datasets, while robustness, efficiency, and real-world applicability remain open problems.

  • Scope: The survey organizes recent generic object detection research into a structural taxonomy, datasets, evaluation criteria, representative methods, state-of-the-art performance, and future directions.It covers more than 300 research contributions across detection frameworks and related sub-problems.
  • Evaluation: Detector comparisons can be misleading because frameworks, backbones, feature innovations, pretraining, and other contextual choices differ across methods.The survey advocates comparing representative public detectors on a common platform and unified setting.
  • State of the Art: 61.71% mAP on the public leaderboard and 58.66% mAP on the private leaderboard were achieved by combining several two-stage detectors in the Open Images challenge.The survey summarizes challenge results and identifies backbone networks, detection frameworks, and large-scale datasets as major accuracy factors.
  • Detector Design: Two-stage detectors generally achieve higher accuracy, whereas one-stage detectors are generally faster and remain competitive on large objects.One-stage methods typically perform much worse on small objects, while feature extraction is the most time-consuming step for both types.
  • Detector Design: Multistage detection is identified as a possible future framework for balancing speed and accuracy, based on recent cascade successes.The cited successes include object detection and instance segmentation on COCO and other challenges.
  • Efficiency: Deeper backbone networks generally improve detection but require greater computation, more training data, and substantial GPU resources.The survey also notes growing interest in compact networks, acceleration, and automated backbone architecture design.
  • Robustness: Small-object detection remains substantially less accurate than large-object detection, and object representations remain vulnerable to real-world variation, occlusion, and deformation.The survey discusses scale-dependent localization and spatial-transformer or deformable-part approaches for geometric variation.
  • Future Directions: Future directions include open-world adaptation, learning with weak or few annotations, detection in other modalities, and richer instance-level understanding.The survey concludes that generic object detection remains incomplete despite major progress and lists pixel-level instance segmentation among promising directions.
Loading 1809.02165v4…