PaperlayerView exact revision on arXiv

Source-linked AI summary

Backbones-Review: Feature Extraction Networks for Deep Learning and Deep Reinforcement Learning Approaches

Omar Elharrouss, Younes Akbari, Noor Almaadeed, Somaya Al-Maadeed

arXiv:2206.08016v1cs.CV

TL;DR

Selecting useful features and suitable extraction networks remains difficult across large-scale data and diverse computer vision tasks. This review describes widely used deep learning and deep reinforcement learning backbones, relates them to tasks, and compares reported performance. It finds task-dependent backbone patterns alongside strong reported performance in several evaluations, while noting that complex detection scenes still require improvement.

  • Problem

    Large-scale data and varied task scenarios make useful feature selection and suitable backbone choice difficult for computer vision models.

  • Method

    The paper reviews and describes established backbones, surveys their use across computer vision tasks, and compares reported results using task-specific datasets and metrics.

  • Results

    Across reviewed tasks, backbone usage is task-dependent, while face-recognition methods exceed 99% on LFW and 96% on YTF in the reported comparisons.

  • Takeaways & Limitations

    The review provides a consolidated reference for selecting and comparing feature-extraction backbones across deep learning and deep reinforcement learning applications.

  • Takeaways & Limitations

    Object-detection performance still needs improvement because scenes may contain small, occluded, and differently positioned objects.

Abstract

from arXiv · show

To understand the real world using various types of data, Artificial Intelligence (AI) is the most used technique nowadays. While finding the pattern within the analyzed data represents the main task. This is performed by extracting representative features step, which is proceeded using the statistical algorithms or using some specific filters. However, the selection of useful features from large-scale data represented a crucial challenge. Now, with the development of convolution neural networks (CNNs), the feature extraction operation has become more automatic and easier. CNNs allow to work on large-scale size of data, as well as cover different scenarios for a specific task. For computer vision tasks, convolutional networks are used to extract features also for the other parts of a deep learning model. The selection of a suitable network for feature extraction or the other parts of a DL model is not random work. So, the implementation of such a model can be related to the target task as well as the computational complexity of it. Many networks have been proposed and become the famous networks used for any DL models in any AI task. These networks are exploited for feature extraction or at the beginning of any DL model which is named backbones. A backbone is a known network trained in many other tasks before and demonstrates its effectiveness. In this paper, an overview of the existing backbones, e.g. VGGs, ResNets, DenseNet, etc, is given with a detailed description. Also, a couple of computer vision tasks are discussed by providing a review of each task regarding the backbones used. In addition, a comparison in terms of performance is also provided, based on the backbone used for each task.

1. Introduction

Large-scale data and task diversity make feature extraction and backbone selection difficult, while CNNs enable more automatic feature learning. This review catalogs backbones, relates them to computer vision tasks, and compares their use across tasks.

  • Motivation: CNNs make feature extraction from large-scale datasets more automatic and feasible across varied scenarios.Deep learning addresses limitations associated with statistical and traditional machine-learning approaches to large datasets and feature variation.
  • Motivation: Backbone selection is task- and computation-dependent because different computer vision tasks use different networks for feature extraction.The paper identifies backbone choice as costly and notes that a network suitable for one task may not suit another.
  • Scope and contributions: The review presents existing backbones for deep learning and deep reinforcement learning, including VGGs, ResNet, Inception variants, DenseNet, MobileNet, and EfficientNet.The paper provides descriptions of the networks and their use as feature-extraction backbones.
  • Scope and contributions: It surveys computer vision tasks according to the backbones used, covering applications such as classification, detection, recognition, segmentation, and action recognition.The contribution statements emphasize an overview of tasks and an evaluation-based comparison of backbone use.
  • Scope and contributions: The paper compares task-specific performance by backbone and discusses deep learning challenges and future directions.Its organization includes backbone descriptions, task reviews, comparisons, challenges, future directions, and conclusions.

2. Backbone families

Backbones are established CNN architectures reused for feature extraction and as starting points in deep learning models, with selection driven by task requirements and computational complexity. The reviewed families span classic, residual, densely connected, inception-based, efficient, and mobile-oriented designs.

  • Backbones are recognized CNN architectures used for feature extraction or at the beginning of deep learning models, selected according to the target task and model complexity.The paper frames backbone choice as a design decision rather than a random selection.
  • AlexNet: AlexNet introduced a relatively simple five-convolution architecture with ReLU activations and dropout, and became a backbone for detection and segmentation models.It was trained on ImageNet and has about 60 million parameters.
  • VGGs: VGG-16 and VGG-19 extend deep convolutional architectures and are widely reused as feature extractors in classification, detection, and related models.VGG-16 has 16 layers and 138 million parameters, while VGG-19 has 19 layers and 144 million parameters.
  • Inception and Dense connectivity: Inception and DenseNet families change connectivity or filtering to support multi-scale or feature-reuse behavior, while Inception-ResNet combines inception blocks with residual connections.Inception-ResNet-V2 has 164 layers and about 55 million parameters; DenseNet connects each layer to all preceding layers.
  • Efficient architectures: Efficiency-focused backbones address computational constraints through compact convolutions, reduced parameters, or scaling strategies for resource-limited deployment.ShuffleNet targets mobile devices using grouped point-wise convolution and channel shuffling; Inception-v2/v3 use fewer than 25 million parameters, while EfficientNet applies compound scaling.

3. Tasks related to Backbones

The review relates backbone choices to computer-vision tasks, emphasizing accuracy–complexity trade-offs and describing architectures used for classification and detection.

  • Image classification: ImageNet classification comparisons consider both accuracy and FLOPs, because higher-accuracy models generally require greater computational complexity.FLOPs measure model complexity in terms of speed and latency.
  • Object detection: YOLO-v3 detects multiple objects using one CNN that predicts classes and bounding boxes across image grid cells.Its Darknet-53 backbone has 53 convolutional layers, while the detection network totals 106 layers.
  • Object detection: 43.5% mAP at approximately 65 FPS on COCO is reported for YOLO-v4, which is described as faster and more accurate than YOLO-v3.YOLO-v4 uses a backbone, neck, and prediction heads, with Darknet-53 reported as its strongest tested backbone.
  • Object detection: YOLO-v4-tiny reduces network size and convolutional layers, using CSPDarknet and fewer anchor boxes to enable faster detection.
  • Object detection: Mask R-CNN with an FPN–R50 benchmark uses CNN-extracted features to generate region proposals and predict bounding boxes.The passage describes the architecture as modular and scalable.

3.3. Crowd counting

Crowd-counting methods increasingly use deep learning for accurate density estimation, while backbone selection balances effectiveness against computational cost.

  • Crowd counting: CNN-based crowd-counting methods are described as the most effective category for accuracy, particularly on large-scale datasets.Deep learning is also reported to improve computational cost relative to traditional sequential methods.
  • Crowd counting: VGG-16 is the most commonly used backbone for crowd counting, alongside VGG-19, ResNet-101, and other architectures.
  • Crowd counting: VGG-16 supports DENet, which combines a detection network and an estimation network for crowd counting.
  • Crowd counting: SCAR combines spatial-wise and channel-wise attention modules with VGG-16, while ADNet estimates density using adaptive dilated self-correction.
  • Crowd counting: A Bayesian-loss density-map method uses VGG-19, whereas SFCN uses ResNet-101 as its backbone.
  • Crowd counting: MobileNet-v2 is used to reduce parameters, network size, and FLOPs in a lightweight encoder–decoder crowd-counting model.

3.4. Video summarization

Video summarization targets efficient extraction of useful information from large videos, with deep and reinforcement-learning methods using diverse backbones and user-oriented rewards.

  • Video summarization: Video summarization is presented as a way to reduce event-search time and simplify analysis of large video collections.Scene type and preprocessing are among the factors considered when producing summaries.
  • Backbones used: Deep-learning video summarization methods use backbones including GoogleNet and ResNet for language-guided and multi-stage models.
  • Deep reinforcement learning: Deep reinforcement learning addresses differing user preferences by using a reward function to identify each user’s performance as a highlight candidate.The cited approach uses ResNet-50 as its backbone.
  • Backbones used: DRL summarization methods use AlexNet, GoogleNet, and Inception-v3 for feature extraction in approaches selecting distinguishable or representative key frames.Deep Summarization Network methods use rewards based on representativeness and diversity.
  • Backbones used: Table 2 catalogs methods and the backbones used for each task.

3.5. Action recognition

Action recognition is challenging across varied video conditions, motivating deep-learning and reinforcement-learning approaches that use specialized backbones and attention mechanisms.

  • Action recognition: Action recognition supports applications including surveillance, tracking, healthcare, and human–computer interaction, but requires powerful learning methods.
  • Deep learning methods: 3D ResNet is used in temporal-pyramid and Temporal Excitation and Aggregation networks to recognize actions across varying tempos.
  • Deep reinforcement learning: DRL-based action-recognition methods use ConvNet features and rewards for recognizability, predictability, and early recognition.
  • Deep reinforcement learning: A part-activated DRL approach uses VGG-16 to address noise from overall-frame modeling and lost human-body structural information.
  • Visual attention: A DRL visual-attention framework uses GoogleNet and an LSTM-based agent to select pertinent image regions at each timestamp.The attention mechanism is intended to reduce interference from irrelevant regions.

3.6. Face recognition

Face recognition research uses deep learning and DRL methods, with feature extraction backbones and loss functions supporting recognition and verification tasks.

  • Face recognition supports nonintrusive identification in applications including border control, airports, stations, companies, and offices.
  • Deep face recognition methods use Softmax and combined loss functions to improve classification results.
  • Attention-aware DRL formulates video face verification as a Markov decision process using both image and feature spaces.

3.7. COVID-19 detection

COVID-19 detection studies apply transfer learning, CNN variants, and multiple backbones to lung ultrasound and X-ray imagery.

  • COVID-19 detection methods use computer vision on lung ultrasound, X-ray, and related medical images to automate diagnosis.
  • Transfer-learning experiments evaluated VGG19, MobileNet-v2, Inception, Xception, and ResNet-v2 backbones for COVID-19 detection.
  • One X-ray study reported VGG-16 as the best-performing backbone among VGG-16, VGG19, Inception-ResNet-v22, InceptionV3, and Xception.
  • Other studies combined Xception with ResNet50-V2, tested AlexNet, GoogleNet, and ResNet-18, or developed Bayes-SqueezeNet-based COVIDiagnosis-Net.

3.8. Panoptic segmentation

Panoptic segmentation combines semantic and instance segmentation while retaining backbone features through the network for dense mask prediction.

  • Panoptic segmentation extends image segmentation by representing both things and stuff in unified segmentation outputs.
  • ResNet-50 and ResNet-101 are widely used backbones, including ResNet-50 in FPSNet, BANet, and video panoptic segmentation.
  • DetectoRS uses ResNeXt-101 with recursive feature pyramids and switchable atrous convolution at macro and micro levels.

3.9. Used backbones for each task

The review maps backbone usage across computer vision tasks and finds task-specific concentration, with particular networks repeatedly favored in several applications.

  • ResNet-50 is widely used for panoptic segmentation, while GoogleNet and ResNeXt are used for video summarization.
  • Crowd counting predominantly uses VGG-16, whereas VGG-19, MobileNet-v2, and ResNet-101 appear less often in that task.
  • Figure 9 summarizes backbone frequency by task, showing VGG prominence in crowd counting, ResNet prominence in face recognition, and ResNet prominence in action recognition.

4. Critical discussion

The review compares backbone performance across several computer-vision tasks, showing that outcomes depend on the task, dataset, architecture, and computational trade-offs.

  • 4.1. Image classification evaluation: Image classification shows a complexity–accuracy trade-off: EfficientNet-B7 achieves the lowest ImageNet errors, while lower-complexity models have higher error rates.EfficientNet-B7 uses 37G FLOPs; Inception-v3 is reported as 3% worse in Top-1 error and 1.2% worse in Top-5 error.
  • 4.2. Object detection evaluation: Object-detection results vary by method, with YOLO-V3-spp, YOLO-V4-P7, and DetectoRS reaching the best reported results, while MobileNet-v1 reaches 19.8 mAP.YOLO-family methods use DarkNet backbones, whereas DetectoRS uses ResNet and ResNeXt.
  • 4.3. Face recognition evaluation: Face-recognition accuracies are close across DL and DRL methods, with DL methods ArcFace and FairLoss reporting 99.83 and 99.75 on LFW and 98.02 and 96.2 on YTF.The reported accuracy differences do not exceed 0.5% on LFW and 5% on YTF; DRL methods exceed 99% on LFW and 96% on YTF.
  • 4.4. Action recognition evaluation: Backbone choice differentiates action-recognition performance: DL methods outperform DRL methods, and ResNet or ResNeXt generally outperform GoogleNet or VGG in the reported comparisons.For DRL methods, TSN-As reaches 94.6% on UCF-101 and is highest on HMDB-51, followed by another method with a 4.4-point difference.
  • 4.5. Panoptic segmentation evaluation: Panoptic-segmentation leaders differ by dataset and metric: Axial-DeepLab leads PQ on Cityscapes, while DR1Mask leads several COCO metrics with 46.1% PQ and 53.1% PQ_th.EfficientPS has the best Cityscapes SQ results and uses EfficientNet with Vistas pre-training, unlike most methods using ResNet-50.
  • 4.6. Crowd counting evaluation: Crowd-counting performance depends on dataset and backbone: ADSCNet leads three datasets, SCNet has the lowest MSE on UCF_CC_50, and VGG-16 methods are reported as most effective overall.ADSCNet reports an MAE of 55.4 on ShanTech_Part_A and is 2.3 points better than ASNet there.
  • 4.8. COVID-19 detection evaluation: COVID-19 X-ray methods generally achieve convincing results, with an Inception-v3 method reaching 100% sensitivity and specificity and 99.5% accuracy.SqueezeNet is reported second, within 1–2% of the Inception-v3 method; results also vary with data representation and preprocessing.

5. Challenges and future directions

The paper identifies data scale, context, annotation, and computational demands as continuing deep-learning challenges, while discussing augmentation and annotation as future directions.

  • Data challenges: Large-scale data availability and annotation remain central challenges because deep-learning performance depends on dataset size and many application datasets are unavailable or limited.The paper specifically notes constraints in industrial, medical, and narrowly scoped tasks.
  • Model understanding: Deep-learning models lack contextual understanding across related tasks, so transferring a trained model to another task requires retraining and processing.The paper characterizes models as learning only what they were trained on.
  • Data annotation: Computer-vision annotation is costly because segmentation and detection require labeling many objects and scenes, while annotation formats vary across methods.DetectronV2, YOLO, and EfficientDet are cited as using different annotation formats for object detection.
  • Future directions: The paper frames deep-learning development as addressing both data limitations and the practical burden of preparing annotations for computer-vision systems.These directions are discussed in the context of DL and DRL methods used for analyzing, deciding, and annotating data.
  • Future directions: Data augmentation is presented as a future direction for tasks lacking large datasets, including DRL-based generation of medical images before segmentation.The paper links augmentation particularly to medical imaging with limited disease data.

6. Conclusion

The paper reviews backbone networks, explains their use in computer-vision architectures, and compares task-specific experimental results according to the backbone used.

  • 6. Conclusion: The review describes backbone networks and their use for feature extraction across computer-vision tasks.It provides detailed network descriptions and discusses task-specific backbone usage.
  • 6. Conclusion: It collects and compares experimental results for each task according to the backbone used, providing a consolidated reference for researchers.The conclusion presents the review as a detailed summary of networks, tasks, and comparative results.
Loading 2206.08016v1…