PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of the Recent Architectures of Deep Convolutional Neural Networks

Asifullah Khan, Anabia Sohail, Umme Zahoora, Aqsa Saeed Qureshi

arXiv:1901.06032v7cs.CV

TL;DR

CNNs' learning ability is constrained by lost spatial information and their limited interpretability. This survey proposes a taxonomy of recent CNN architectures and reports that architectural innovations, especially layer blocks, have driven improvements in learning capacity and performance.

  • Problem

    Loss of spatial information limits CNNs' learning ability, while deep CNNs may be difficult to interpret and verify.

  • Method

    The survey proposes a taxonomy for recent CNN architectures based on the design patterns of their processing units.

  • Results

    Architectural innovations have significantly improved CNN learning capacity, with major performance boosts associated with replacing conventional layer structures with blocks.

  • Takeaways & Limitations

    Layer blocks are a prominent architectural pattern associated with recent improvements in CNN performance.

  • Takeaways & Limitations

    Deep CNNs are generally black boxes that may lack interpretation and explanation, making them difficult to verify.

Abstract

from arXiv · show

Deep Convolutional Neural Network (CNN) is a special type of Neural Networks, which has shown exemplary performance on several competitions related to Computer Vision and Image Processing. Some of the exciting application areas of CNN include Image Classification and Segmentation, Object Detection, Video Processing, Natural Language Processing, and Speech Recognition. The powerful learning ability of deep CNN is primarily due to the use of multiple feature extraction stages that can automatically learn representations from the data. The availability of a large amount of data and improvement in the hardware technology has accelerated the research in CNNs, and recently interesting deep CNN architectures have been reported. Several inspiring ideas to bring advancements in CNNs have been explored, such as the use of different activation and loss functions, parameter optimization, regularization, and architectural innovations. However, the significant improvement in the representational capacity of the deep CNN is achieved through architectural innovations. Notably, the ideas of exploiting spatial and channel information, depth and width of architecture, and multi-path information processing have gained substantial attention. Similarly, the idea of using a block of layers as a structural unit is also gaining popularity. This survey thus focuses on the intrinsic taxonomy present in the recently reported deep CNN architectures and, consequently, classifies the recent innovations in CNN architectures into seven different categories. These seven categories are based on spatial exploitation, depth, multi-path, width, feature-map exploitation, channel boosting, and attention. Additionally, the elementary understanding of CNN components, current challenges, and applications of CNN are also provided.

1 Introduction

Deep CNNs learn hierarchical representations through multiple convolutional, nonlinear, and subsampling stages, supporting strong performance across vision tasks. This survey reviews recent architectural innovations and organizes them into seven intrinsic categories.

  • CNN foundations: CNNs use convolutional kernels, nonlinear activation units, and subsampling layers to transform locally correlated data into increasingly abstract features.These stages support automatic feature extraction and reduce reliance on separate handcrafted feature extractors.
  • CNN foundations: Deep CNNs stack linear and nonlinear processing units to learn low-, mid-, and high-level representations for complex recognition problems.High-level features combine lower- and mid-level features, increasing representational capacity for tasks with many image categories.
  • CNN impact: Deep CNNs have substantially improved performance over conventional vision-based models and can learn invariant representations when given enough training data.The passages also report that deep CNNs may achieve human-level performance under sufficient training data.
  • Architectural evolution: Architectural research since AlexNet has introduced new processing units, connectivity patterns, and blocks, including VGG-style homogeneous designs and split-transform-merge inception blocks.These innovations contributed to applications in image segmentation, classification, detection, and retrieval.
  • Survey scope and taxonomy: The survey examines recent CNN architectures reported from 2012–2020 and classifies them into seven categories based on architectural information processing.The categories are spatial exploitation, depth, multi-path, width, feature-map exploitation, channel boosting, and attention-based CNNs.
  • Survey scope and taxonomy: In addition to its taxonomy, the survey introduces CNN fundamentals, historical development, and broader architectural insights for readers.The survey is also intended to support theoretical understanding of CNN architectures and developments.

2 Basic CNN components

CNN architectures combine convolution, pooling, activation, normalization, regularization, and classification layers to learn representations from grid-like data. Their component arrangement supports feature extraction, invariance, nonlinear processing, and classification.

  • CNNs learn representations from grid-like data and exploit their feature-generation and discrimination abilities for classification.
  • A typical CNN alternates convolution and pooling layers, followed by fully connected or global average pooling layers.
  • Convolution kernels scan local receptive fields with shared weights, extracting features while reducing parameters relative to fully connected networks.
  • Pooling combines neighborhood responses, reduces feature-map size, and increases tolerance to translation and small distortions while helping regulate complexity and overfitting.
  • Activation functions transform convolution outputs by adding nonlinearity; ReLU variants are preferred for helping overcome vanishing gradients, while MISH has shown better benchmark performance than ReLU in most recently proposed deep networks.
  • Batch normalization standardizes feature-map distributions, smooths gradient flow, and supports generalization, whereas dropout regularizes by randomly skipping units or connections.
  • Fully connected layers perform global analysis of preceding feature-extraction outputs and form nonlinear combinations for data classification.

3 Architectural evolution of deep CNNs

Deep CNNs evolved from early recognition systems into broadly applied architectures through advances in training, hardware, datasets, and architectural design. The section traces five eras of this development, from ConvNet and LeNet-5 through modern architectures.

  • Origin of CNN: Late 1980s-1999: CNN history begins with neurobiological experiments and progressed to ConvNet, whose supervised backpropagation training founded modern 2D CNNs.ConvNet achieved successful handwritten digit and zip code recognition results.
  • Origin of CNN: Late 1980s-1999: LeNet-5 extended CNN use to document character classification but remained limited on broader image-recognition classes.Its successful milestones included optical character recognition, while performance on other image-recognition tasks was poor.
  • Decline of CNN: Late 1990s-Early 2000s: Early CNN research slowed because networks were opaque, difficult and time-intensive to train, computational resources were limited, and datasets lacked diverse image categories.These constraints reduced attention to applications such as object detection and video surveillance.
  • Revival of CNN: 2006-2011: Greedy layer-wise pre-training, alternative activation functions, max-pooling, GPUs, and large datasets helped revive deep CNN research after 2006.Pre-training provided better initialization than random initialization, while GPUs accelerated training and ImageNet supplied millions of annotated images.
  • Rapid increase in architectural innovations and applications of CNN: 2012-Present: AlexNet marked a major performance breakthrough in 2012, reducing the ILSVRC error rate from 25.8 to 16.4 compared with conventional computer-vision techniques.Later architectures including VGG, ResNet, and ResNext also performed well on challenging tasks, while CNNs supported applications such as image captioning.
  • Rapid increase in architectural innovations and applications of CNN: 2015-Present: Recent CNN advances include channel shuffle, which reduces operations without affecting accuracy, and ANTBlock, which achieves good benchmark performance at low computational cost.These developments illustrate the continuing shift toward efficient architectural innovations.

4 Architectural innovations in CNN

The survey emphasizes that CNN performance improvements mainly came from restructuring processing units and designing new blocks. It organizes recent architectures into seven categories and summarizes their models, resources, strengths, and weaknesses.

  • Architectural innovations in CNN: Although CNN improvements include parameter optimization, regularization, and structural reformulation, the main performance gains came from restructuring processing units and designing new blocks.The survey distinguishes architectural innovation from other improvement strategies.
  • Taxonomy of deep CNN architectures: Recent CNN architectures are categorized into spatial exploitation, depth, multi-path, width, feature-map exploitation, channel boosting, and attention-based CNNs.The seven-category taxonomy is represented pictorially in Fig. 4.
  • Survey resources and comparisons: The survey summarizes models, parameters, and benchmark performance in Table 2, and identifies online resources and implementation platforms in Table 3.It also presents category-based strengths and weaknesses in Tables 5a–g.

4.1 Spatial Exploitation based CNNs

Spatial exploitation-based CNNs use convolutional filters and architectural adjustments to learn image features at different granularities. The section traces this progression from LeNet through AlexNet and later architectures such as VGG and GoogleNet.

  • Spatial filters: Convolution exploits neighboring-pixel correlations, using shared learnable parameters to extract similar features at multiple locations with fewer parameters.Small filters capture fine-grained information, whereas large filters capture coarse-grained information.
  • LeNet: LeNet reduced parameters and automatically learned features from raw pixels while classifying digits despite small distortions.Its architecture alternated convolutional and pooling layers before two fully connected layers.
  • Feature visualization: Feature visualization with DeconvNet revealed inactive neurons and aliasing artifacts, after which topology and parameter adjustments improved performance.The findings positioned visualization as a way to identify design shortcomings and adjust CNNs.
  • VGG: VGG used a simple, homogeneous, modular topology based on smaller filters and achieved second place in the 2014-ILSVRC competition.Its design principle helped motivate research using smaller filters in CNNs.

4.2 Depth based CNNs

Depth-based CNN innovations increase representational capacity through deeper nonlinear hierarchies, while addressing the optimization and computational difficulties caused by depth. VGG, Inception, Highway Networks, ResNet, and Inception-ResNet illustrate these directions.

  • Motivation: Deep networks can represent certain function classes more efficiently than shallow architectures, although shallow universal approximation may require exponentially many neurons.The section presents depth as a way to build richer feature hierarchies and improve approximation efficiency.
  • Highway Networks: Highway Networks exploit depth through cross-layer connectivity and gating, and 50-layer versions converged faster than thin but deep architectures.The architecture was categorized as both depth-based and multi-path-based.
  • Challenges: Increasing depth can improve learning capacity but also causes slow training, convergence difficulties, overfitting, and performance degradation.These limitations motivate architectural mechanisms that make deep networks easier to optimize.
  • ResNet: ResNet introduced residual learning and enabled substantially deeper networks, with 50-, 101-, and 152-layer models outperforming a 34-layer plain network.Its 152-layer model won the 2015-ILSVRC competition and showed lower computational complexity than earlier networks.
  • ResNet: ResNet’s performance on recognition and localization tasks supported representational depth as a central dimension of visual learning capacity.The passage reports a 28% improvement on the COCO benchmark.
  • Inception variants: Inception-ResNet combined residual learning with inception blocks, matching Inception-V4’s generalization while converging more quickly.Residual connections allowed increased depth and width and significantly accelerated Inception-network training.

4.3 Multi-Path based CNNs

Multi-path CNNs use shortcut or cross-layer connections to improve information and gradient flow through deep networks. Highway Networks, ResNet, and DenseNet implement this principle with gating, residual addition, or feature concatenation.

  • Motivation: Deep CNNs can suffer gradient vanishing, gradient explosion, and performance degradation as depth increases, including higher training and test error.Multi-path connectivity was proposed to address these training problems by allowing information to bypass intermediate layers.
  • Connectivity: Cross-layer connectivity partitions networks into blocks and provides specialized information paths that make gradients accessible to lower layers.Shortcut types include zero-padded, projection-based, dropout, skip, and 1x1 connections.
  • Training deep networks: Cross-layer connectivity enables networks exceeding 100 layers and, in one report, networks as deep as 900 layers trained with stochastic gradient descent.The passage also attributes a regularizing effect to cross-layer information flow.
  • Highway Networks: Highway Networks use transform and carry gates: an open transform gate sends transformed input forward, while a closed gate enables direct passage.This mechanism provides an information highway across layers without transformation.
  • ResNet: ResNet passes transformed signals through bypass pathways and adds the original input, making residual functions easier to optimize at increased depth.Its parameter-free identity shortcuts remain open and are reported to speed convergence and avoid gradient diminution.
  • DenseNet: DenseNet connects each preceding layer to subsequent layers by concatenating feature maps, creating direct access to gradients and preserved information.This connectivity can reduce overfitting on smaller training sets but becomes parametrically expensive as feature-map counts increase.

4.4 Width based Multi-Connection CNNs

Width-based CNNs address limitations of deep, narrow networks by increasing parallel processing capacity or widening residual structures. The surveyed designs include Wide ResNet, Pyramidal Net, and Xception, which modify width, feature-map depth, or convolutional organization.

  • Motivation: Increasing depth does not necessarily increase neural-network learning power, while sufficient width contributes to universal approximation.The cited discussion links width and depth as joint architectural factors, especially for ReLU networks.
  • Wide ResNet: Wide ResNet makes residual networks wider rather than deeper, exploiting residual units as the primary source of learning potential.It introduces a width factor k and places dropout between convolutional layers rather than inside residual blocks.
  • Wide ResNet: Wide ResNet used twice as many parameters as ResNet but was empirically easier to train, and widening could improve performance more effectively than additional depth.The survey presents this as an empirical comparison and architectural conclusion.
  • Pyramidal Net: Pyramidal Net gradually increases feature-map width across residual units, using additive or multiplicative widening while preserving changing spatial coverage.Its depth increase is regulated by a factor λ and implemented with zero-padded identity mappings that use fewer parameters than projection shortcuts.
  • Pyramidal Net: Pyramidal Net incurs a quadratic increase in space and time as width increases, limiting the efficiency of this strategy.The survey also frames drastic feature-map-depth growth combined with spatial-information loss as a limitation of CNN learning.

4.5 Feature-Map (ChannelFMap) Exploitation based CNNs

Feature-map exploitation methods select or recalibrate channels because some feature-maps contribute little to discrimination and excessive feature sets can add noise. The section surveys Squeeze-and-Excitation and related residual feature-map recalibration mechanisms.

  • Motivation: Some feature-maps contribute little to object discrimination, while enormous feature sets can introduce noise and over-fitting.This motivates selecting feature-maps in addition to engineering the overall network architecture.
  • Squeeze and Excitation Network: The SE-block suppresses less important feature-maps and gives higher weight to class-specifying feature-maps.The survey reports a record decrease in ImageNet error for SE-Network and describes the block as generically insertable into CNN architectures.
  • Squeeze and Excitation Network: SE-Network uses squeeze statistics and excitation gating to assign relevance-based weights that rescale feature-maps before the next layer.Squeeze suppresses spatial information through global average pooling, while excitation applies nonlinear transformations and a sigmoid gate.
  • Residual recalibration: In ResNet, standard SE weighting considers only residual information, which can reduce the impact of the SE-block and leave identity information underused.The related CMPE-SE design derives motifs from both residual and identity mappings and models their competition.
  • Residual recalibration: CMPE-SE applies squeeze to residual and identity mappings, combines their descriptors for excitation, and multiplies the resulting masks with residual information.Backpropagation optimizes competition between identity and residual feature-maps and relationships among residual-block feature-maps.

4.6 Channel(Input) Exploitation based CNNs

Channel-exploitation CNNs improve input representations by augmenting original channels with artificially generated channels from auxiliary learners. CB-CNN packages this channel-boosting process into a generic block for deep discriminative networks.

  • Motivation: CNN performance depends on input representation, and limited representation diversity or missing class-discerning information can restrict discrimination.The section motivates increasing the input-channel dimension rather than relying only on downstream feature learning.
  • Channel boosting: Channel boosting augments original input channels with artificial channels generated by auxiliary deep learners.The auxiliary channels are then supplied to a deep discriminative CNN.
  • CB-CNN: CB-CNN uses autoencoders as generative auxiliary learners and concatenates their channels with the original input channels.The resulting boosted input is convolved by the main CNN to produce feature-maps.
  • CB-CNN: CB-CNN uses inductive transfer learning to augment the original channel space with learned input distributions and inserts the boosting phase at the start of a deep network.The survey describes transfer learning as operating at both generation and discrimination stages.
  • CB-CNN: Multi-deep learners use generative models as auxiliary learners to enhance the representational capacity of the deep CNN discriminator.The architecture was evaluated on a medical image dataset and reported improved results in Table 4.
  • Scope: Channel boosting was evaluated only by inserting a boosting block at the network start, although the authors suggested extending auxiliary channels to other layers.This defines the reported scope of the evaluation rather than a demonstrated result of deeper insertion.

4.7 Attention based CNNs

Attention-based CNNs focus computation on context-relevant features and object-aware regions, supporting recognition in cluttered or complex scenes. The section covers residual attention, channel-spatial attention, and segmentation-oriented squeeze-and-excitation modules.

  • Motivation: Attention mechanisms help CNNs recognize objects from cluttered backgrounds and complex scenes by focusing on context-relevant features.The survey connects this focus to image localization and recognition.
  • Residual Attention Network: Residual Attention Network generates object-aware soft masks that recalibrate trunk-branch outputs as control gates for feature-maps.Its attention modules combine trunk and mask branches using bottom-up and top-down learning within a feed-forward CNN.
  • Residual Attention Network: Fixed attention modules in Transformation networks cannot adapt to changing circumstances, motivating adaptive attention designs such as RAN.The comparison identifies adaptability as the relevant architectural distinction.
  • Residual Attention Network: RAN stacks multiple hierarchical attention modules to adaptively weight feature-maps and recognize cluttered, complex, and noisy images.It incorporates mixed, channel, and spatial attention to capture object-aware features at different levels.
  • CBAM: CBAM sequentially applies channel attention and spatial attention, combining average- and max-pooling to refine feature-maps.The survey reports that max-pooling provides distinctive-object clues, whereas global average pooling alone can yield suboptimal feature-map attention.
  • CBAM: CBAM’s serial 3D attention-map formulation reduces parameters and computational cost while remaining easy to integrate into CNN architectures.The refined feature-maps are described as focusing on important regions and increasing representational power.
  • Segmentation attention: Segmentation-oriented cSE, sSE, and scSE modules separately or jointly exploit feature-map and spatial information to assign object-specific attention.The modules are inserted after encoder and decoder layers in an autoencoder-based convolutional network.

5 Applications of CNNs

CNNs are applied across vision, language, and speech tasks, with reported systems addressing recognition, detection, segmentation, and related challenges.

  • CNNs support computer-vision tasks including object detection, recognition, classification, regression, and segmentation.
  • Other applications: CNNs have been used for face recognition, pose estimation, medical-image diagnosis, and speech recognition.
  • Action recognition: Three-dimensional CNNs combined with LSTMs recognize actions from video frames and have outperformed other activity-recognition techniques.
  • Natural language processing: CNN applications in NLP include language modeling, sentence modeling, semantic analysis, and matching between sentences.
  • Object detection and segmentation: CNN-based systems have been developed for object detection and semantic or instance-based segmentation, including R-CNN, FCN, SegNet, Mask R-CNN, and U-Net.
  • Speech recognition: A speaker-independent CNN speech-recognition system achieved a ten percent reduction in error rate compared with earlier methods.

6 CNN challenges

Deep CNNs face challenges involving data, interpretability, robustness, tuning, hardware demands, and geometric reasoning in vision tasks.

  • Interpretability: Deep CNNs can behave like black boxes, making their interpretations and explanations difficult to verify.
  • Robustness: Small random perturbations to input images can cause CNNs to classify original and altered images differently.
  • Supervised deep CNNs require large annotated datasets, whereas humans can learn and generalize from few examples.
  • Optimization: Hyper-parameter selection strongly affects performance and remains a difficult, intuition-driven tuning task.
  • Resources: Efficient CNN training demands powerful hardware, while deep high-capacity architectures impose substantial memory and computational overhead.
  • Geometric reasoning: CNNs generally perform poorly when estimating an object's pose, orientation, and location in vision tasks.

7 Future directions

Future CNN research emphasizes richer representations, efficient deployment, automated design, distributed training, and broader use beyond conventional image processing.

  • Ensembles: Ensemble learning and combinations of diverse architectures may improve generalization and robustness across diverse image categories.
  • Generative learning at CNN feature-extraction stages may boost representational power, including in image segmentation.
  • Representation learning: Auxiliary learners can incorporate informative intermediate feature maps to enhance CNN learning capacity.
  • Attention: Attention mechanisms are expected to gain importance by preserving spatial relevance alongside discriminative features.
  • Hardware: Hardware accelerators and reduced-precision or ternary operations are proposed to reduce execution time, power consumption, chip area, and energy requirements.
  • Training infrastructure: Genetic algorithms, pipeline parallelism, and cloud platforms are identified as approaches for automating tuning, scaling training, and handling computationally intensive CNN applications.
  • Efficient deployment: Deep and wide CNNs remain difficult to execute on resource-limited devices because of computational cost and memory demands.

8 Conclusion

The survey organizes recent CNN architectural innovations into a taxonomy and connects block-based, information-aware designs with performance and modularity.

  • The survey reviews CNN architectural advances through processing-unit design patterns and proposes a taxonomy of recent architectures.
  • Taxonomy: The taxonomy covers spatial exploitation, depth, multi-path, width, feature-map exploitation, channel boosting, and attention-based CNNs.
  • Block architectures: Recent literature attributes the main CNN performance boost to replacing conventional layer structures with blocks.
  • Block architectures: Blocks can act as auxiliary learners that exploit spatial or feature-map information or boost input channels for problem-aware learning.
  • Modularity: Block-based CNNs encourage modular learning, making architectures simpler and more understandable.
  • Future emphasis: Attention and channel-information exploitation, alongside spatial information, are expected to gain importance in CNN research.
Loading 1901.06032v7…