PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects

Zewen Li, Wenjie Yang, Shouheng Peng, Fan Liu

arXiv:2004.02806v1cs.CVcs.LGeess.IV

TL;DR

Existing CNN reviews often focus on applications in specific scenarios and omit a general perspective and newer developments. This survey synthesizes CNN foundations, models, functions, applications, experiments, and prospects, concluding with practical guidance on architecture design and function selection.

  • Problem

    Existing CNN reviews emphasize scenario-specific applications rather than a general perspective and do not fully cover recent developments.

  • Method

    The paper surveys CNN building blocks, classic and advanced models, functions, optimizers, applications across dimensions, experimental analyses, and future directions.

  • Results

    Experiments and model analyses support guidance that nonlinearity is necessary in deep networks, residual connections enable extreme depth, and dimension reduction aids speed.

  • Takeaways & Limitations

    The survey provides guidelines for designing CNNs from accuracy and speed perspectives and rules of thumb for selecting functions and output activations.

  • Takeaways & Limitations

    CNNs may fail to recognize different patterns of the same object, while obtaining the diverse labeled data needed to compensate is difficult.

Abstract

from arXiv · show

Convolutional Neural Network (CNN) is one of the most significant networks in the deep learning field. Since CNN made impressive achievements in many areas, including but not limited to computer vision and natural language processing, it attracted much attention both of industry and academia in the past few years. The existing reviews mainly focus on the applications of CNN in different scenarios without considering CNN from a general perspective, and some novel ideas proposed recently are not covered. In this review, we aim to provide novel ideas and prospects in this fast-growing field as much as possible. Besides, not only two-dimensional convolution but also one-dimensional and multi-dimensional ones are involved. First, this review starts with a brief introduction to the history of CNN. Second, we provide an overview of CNN. Third, classic and advanced CNN models are introduced, especially those key points making them reach state-of-the-art results. Fourth, through experimental analysis, we draw some conclusions and provide several rules of thumb for function selection. Fifth, the applications of one-dimensional, two-dimensional, and multi-dimensional convolution are covered. Finally, some open issues and promising directions for CNN are discussed to serve as guidelines for future work.

I. INTRODUCTION · II. BRIEF OVERVIEW OF CNN

The paper reviews CNNs from their historical foundations and general architecture to address gaps in application-focused reviews. It explains core convolutional components and variants while motivating CNNs’ broad significance in deep learning.

  • I. INTRODUCTION: CNNs have enabled major advances in computer vision, including face recognition, autonomous vehicles, self-service supermarkets, and intelligent medical treatment.The paper presents CNNs as a representative deep-learning network whose achievements motivate a broader review.
  • I. INTRODUCTION: CNNs developed from ANN research through the MP neuron model, perceptrons, backpropagation networks, and TDNNs viewed as one-dimensional convolutional networks.Backpropagation addressed problems that single-layer perceptrons could not solve, while TDNNs were proposed for speech recognition.
  • I. INTRODUCTION: Earlier reviews emphasized chronological architectures or scenario-specific applications, leaving newer CNN ideas and a general perspective insufficiently covered.The paper specifically notes omissions involving newer-generation networks and broader analysis beyond image recognition or segmentation.
  • I. INTRODUCTION: This review covers CNN building blocks, classic models from LeNet-5 through MobileNet v3 and GhostNet, function-selection guidance, dimensional applications, and open issues.Its stated contributions emphasize architectural innovations and useful experience from representative models.
  • II. BRIEF OVERVIEW OF CNN: CNNs are feedforward networks that extract features through convolutional structures without manual feature extraction, with architecture inspired by visual perception.Artificial neurons correspond to biological neurons, while pooling uses image local correlation to down-sample data and retain useful information.
  • II. BRIEF OVERVIEW OF CNN: Convolution, padding, stride, and pooling form core CNN components for feature extraction, border-size adjustment, convolution density control, and data reduction.Convolution outputs feature maps; padding adds zero-valued borders, while larger stride controls convolving density.
  • II. BRIEF OVERVIEW OF CNN: Dilated convolution enlarges receptive fields without increasing valid kernel points, while deformable convolution adapts to irregular object shapes.A 2-dilated 3 × 3 kernel has a 7 × 7 receptive field, and deformable convolution focuses on relevant regions.
  • II. BRIEF OVERVIEW OF CNN: The overview also identifies separable, group, and multi-dimensional convolutions as important variants discussed later in the paper.These convolution types are covered in the sections on CNN models and dimensional applications.

III. CLASSIC CNN MODELS … C. VGGNets

The review surveys representative CNN architectures from LeNet-5 through AlexNet and VGGNets, emphasizing their structural designs and innovations. It highlights progression from foundational CNN components to deeper networks and smaller convolution kernels.

  • III. CLASSIC CNN MODELS: The section surveys representative CNN models and emphasizes their innovations to clarify core ideas for developing future architectures.It covers models that became deeper, wider, or lighter after AlexNet’s introduction.
  • A. LeNet-5: LeNet-5 is a seven-trainable-layer CNN for handwritten character recognition that combines convolution, pooling, and fully connected layers.Its architecture contains two convolutional layers, two pooling layers, and three fully-connected layers, using local receptive fields, shared weights, and subsampling.
  • B. AlexNet: AlexNet won the ImageNet 2012 competition and comprises eight layers, including five convolutional and three fully-connected layers.It applies CNN principles in a deep and wide network.
  • B. AlexNet: AlexNet introduced ReLU, dropout, LRN, overlapping max pooling, GPU acceleration, data augmentation, and group convolutions within CNN training.ReLU mitigates gradient vanishing, dropout reduces overfitting, and overlapping max pooling improves feature richness relative to average pooling.
  • B. AlexNet: LRN suppresses neurons with small values while relatively activating larger-valued neurons, thereby enhancing model generalization.The passage describes LRN as simulating lateral inhibition in the biological nervous system.
  • C. VGGNets: VGGNets include VGG-11, VGG-11-LRN, VGG-13, VGG-16, and VGG-19, and secured first place in ImageNet Challenge 2014 localization.Their authors show that increasing network depth can improve final performance to some extent.
  • C. VGGNets: VGGNets remove LRN because its effect was not obvious in deep CNNs and replace larger kernels with 3 × 3 convolutions.Several 3 × 3 kernels can match a 5 × 5 receptive field while providing more nonlinear variations.
  • C. VGGNets: 45% fewer parameters are achieved by using 3 × 3 kernels instead of one 5 × 5 kernel for the same receptive field.The passage also states that three small kernels provide three nonlinear variations.

D. GoogLeNet · E. ResNet

GoogLeNet develops Inception networks through multi-scale convolutions, normalization, factorization, and deeper ResNet extensions. ResNet addresses degradation in deep networks with shortcut-based residual blocks that enable deeper models and direct gradient flow.

  • D. GoogLeNet: GoogLeNet, the ILSVRC 2014 winner, introduced large-scale CNNs built by stacking Inception modules, which evolved through four versions.The versions are Inception v1, v2, v3, and v4.
  • D. GoogLeNet: Inception v2 applies batch normalization to normalize every layer’s output, improving robustness and enabling training with a relatively large learning rate.Batch normalization addresses internal covariate shift.
  • D. GoogLeNet: Inception v2 factorizes convolutions, replacing 5 × 5 kernels with two 3 × 3 kernels and n × n kernels with 1 × n and n × 1 kernels.Factorization is discouraged in early layers, favored on medium-sized feature maps, and applied to the last 3 × 3 convolution of each branch.
  • D. GoogLeNet: Inception v3 combines Inception v2 innovations with one-dimensional factorization, changes input size from 224 by 224 to 299 by 299, and uses RMSProp.The factorization uses one by seven and seven by one, and one by three and three by one kernels.
  • D. GoogLeNet: Inception-ResNet-v1 and Inception-ResNet-v2 extend Inception depth with ResNet structure and improve training speed and performance in experiments.These models harness residual structure within the Inception family.
  • E. ResNet: ResNet’s key contribution is a two-layer residual block with shortcut connections, while 50-layer, 101-layer, and 152-layer models use three-layer bottleneck blocks.The bottleneck’s narrower ends and 1 × 1 convolutions reduce parameters and improve nonlinearity.
  • E. ResNet: Experiments show shortcut connections mitigate gradient vanishing without deep-network degeneration because gradients can flow directly through them.This direct gradient path is the mechanism identified for ResNet’s optimization advantage.

G. MobileNets

MobileNets are lightweight CNNs designed for embedded devices, using depth-wise separable convolutions and related efficiency techniques. Across v1–v3, they progressively introduce channel and resolution scaling, inverted residuals with linear bottlenecks, platform-aware architecture search, attention, and efficient activations.

  • G. MobileNets: MobileNets comprise three versions—v1, v2, and v3—and target lightweight deployment on devices such as mobile phones.They use depth-wise separable convolutions and advanced techniques to construct thin deep neural networks.
  • MobileNet v1: MobileNet v1 adds width and resolution multipliers to reduce layer channels and input-image or feature-map resolution.Its depth-wise separable convolutions decompose standard convolution into depth-wise and pointwise operations.
  • MobileNet v2: MobileNet v2 introduces inverted residual blocks and linear bottlenecks, reversing residual-block channel flow from compression–convolution–expansion to expansion–depth-wise convolution–compression.The linear transformation replaces ReLU after channel compression because ReLU can destroy information in low-dimensional space.
  • MobileNet v3: MobileNet v3 combines platform-aware NAS for block-wise search with NetAdapt for layer-wise filter optimization.The RNN-based controller and hierarchical search space identify global network structures, while NetAdapt fine-tunes filters per layer.
  • MobileNet v3: MobileNet v3 adds squeeze-and-excitation channel reweighting and uses h-swish only in the model’s second half to reduce computation with little accuracy loss.The SE module reduces channels to 1/4 through a fully connected layer before recovering them to produce channel weights.

H. ShuffleNets

ShuffleNets target mobile-device computing constraints by combining pointwise group convolution, channel shuffle, and related architectural techniques to reduce computation with little accuracy loss. ShuffleNet v2 additionally uses experimental design guidelines that account for Memory Access Cost alongside FLOPs.

  • ShuffleNet overview: ShuffleNets reduce computational cost with little accuracy loss using pointwise group convolution, channel shuffle, and related techniques; the series includes ShuffleNet v1 and v2.The models were proposed by MEGVII for devices with insufficient computing power.
  • ShuffleNet v1: Pointwise group convolution lowers 1 × 1 convolution complexity, while channel shuffle enables information flow between otherwise isolated channel groups.The two operations address efficiency and inter-group communication, respectively.
  • ShuffleNet v1: ShuffleNet v1 further reduces parameters by combining group convolution and channel shuffle, removing a second ReLU, and using average pooling with concatenation in stride-two shortcuts.The final unit replaces element-wise addition with concatenation and uses 3 × 3 average pooling in shortcut paths.
  • ShuffleNet v2: ShuffleNet v2 evaluates speed using Memory Access Cost as well as FLOPs and derives four design guidelines from experiments.The guidelines concern channel balance, group count, network fragmentation, and elementwise operations.
  • ShuffleNet v2: ShuffleNet v2 findings favor equal input and output channels, fewer groups, less fragmentation, and fewer elementwise operations to improve computational efficiency.MAC is minimal when input and output channels match; increasing groups raises MAC and slows training, while fragmentation reduces parallel efficiency.
  • ShuffleNet v2: The ShuffleNet v2 unit implements these guidelines and introduces channel split, dividing channels into two branches with a modified unit for spatial downsampling.Channels are split into branches A and B before subsequent processing.

I. GhostNet · IV. DISCUSSION AND EXPERIMENTAL ANALYSIS

GhostNet reduces CNN computation by exploiting redundant feature maps through a two-part convolutional module. The experimental analysis compares activation functions and finds that nonlinear, especially ReLU-family, choices generally improve convergence, stability, and practical performance.

  • I. GhostNet: GhostNet targets redundant feature maps, called ghost features, to reduce computational cost while achieving state-of-the-art results.The approach addresses redundancy in traditional convolution layers for image cognition.
  • I. GhostNet: The Ghost module uses fewer convolution kernels for initial feature extraction, then applies linear transformations to generate multiple feature maps.This divides traditional convolution into direct extraction and subsequent linear transformation.
  • I. GhostNet: The Ghost module can be applied to other CNN models.
  • IV. DISCUSSION AND EXPERIMENTAL ANALYSIS: Without nonlinear activation, arbitrarily many neural-network layers remain a linear combination of the input, leaving hidden layers ineffective.The paper identifies this as the limited-learning-ability primitive perceptron case.
  • IV. DISCUSSION AND EXPERIMENTAL ANALYSIS: ELU provides the best accuracy, but only slightly exceeds ReLU, Leaky ReLU, and PReLU, while consuming more training time than ReLU and Leaky ReLU.
  • IV. DISCUSSION AND EXPERIMENTAL ANALYSIS: Leaky ReLU often offers better accuracy and training speed, while ReLU and Leaky ReLU show greater training stability than PReLU and ELU.Leaky ReLU can reduce neuron inactivation caused by ReLU’s zero gradient for negative inputs.
  • IV. DISCUSSION AND EXPERIMENTAL ANALYSIS: For hidden layers, ReLU or Leaky ReLU is generally recommended; Leaky ReLU’s negative slope can be set to 0.02 to speed up training.Sigmoid and tanh may suffer gradient vanishing, while Leaky ReLU, PReLU, and related variants can address inactive neurons.

B. Loss function

Loss functions measure prediction error and serve as optimization criteria for CNN regression and classification. Regression commonly uses MAE or MSE, while classification often uses cross entropy and its variants to improve class compactness or separation.

  • B. Loss function: Loss functions measure the distance between predicted and actual values and are minimized during CNN optimization for regression and classification.Common choices include Mean Absolute Error, Mean Square Error, and Cross Entropy.
  • Loss Function for Regression: MAE is more robust to outliers, whereas MSE is differentiable and provides controllable update rates during optimization.MAE is preferable when training data contain many harmful outliers; otherwise, MSE should be considered.
  • Loss Function for Classification: Cross entropy loss compares predicted class probabilities with actual outputs and is typically used with a softmax layer in CNN classification.AlexNet, Inception v1, and ResNet used cross-entropy loss in their original papers and achieved state-of-the-art results.
  • Loss Function for Classification: Because cross entropy does not enforce within-class compactness or between-class margins, contrastive, triplet, center, and large-margin softmax losses address these properties.Contrastive and triplet losses are commonly used in face recognition, while center and large-margin softmax losses add intra-class constraints or inter-class margins.
  • Rules of Thumb for Selection: Loss selection depends on task and application: L1 or L2 loss suits regression, other losses suit classification, and specialized variants suit class compactness or margins.For face recognition, contrastive and triplet losses are described as commonly used choices.

C. Optimizer

CNN training uses optimizers to minimize non-convex loss functions within acceptable time, with mini-batch gradient descent balancing update efficiency and convergence stability. Experiments show that optimizer behavior depends strongly on learning rate and may involve convergence or useful oscillation.

  • Optimizer overview: Optimizers minimize CNN loss functions to obtain network parameters within acceptable training time, including Momentum, RMSprop, and Adam.The review discusses gradient-descent methods and optimization algorithms for training CNNs.
  • Gradient descent methods: Mini-batch gradient descent combines efficient updates with reduced variance, making convergence more stable than batch or stochastic alternatives.It uses a small batch of samples for each update, trading off computational efficiency and stability.
  • Gradient descent methods: Mini-batch gradient descent is the most popular method among the three and trains classic CNNs including AlexNet, VGG, Inception v2, ResNet, DenseNet, FaceNet, DeepID, and DeepID2.These models used mini-batch gradient descent in their original or reported training procedures.
  • Gradient descent optimization algorithms: Momentum suppresses one-dimensional oscillations, NAG anticipates the next position, Adagrad adapts rates to parameter frequency, and Adam combines Momentum with RMSprop.Other discussed methods include Adadelta, RMSprop, AdaMax, Nadam, and AMSGrad, which modify learning-rate or gradient-update behavior.
  • Experimental evaluation: Almost all tested optimizers eventually converged, but mini-batch gradient descent had the slowest convergence rate.The experiment evaluated ten optimizers on VGG-16 using CIFAR-10.
  • Experimental evaluation: Nesterov, RMSprop, and Nadam were prone to oscillation, while learning rate strongly affected convergence and oscillation could help models escape local optima.The experiments also reported oscillation or non-convergence for Nesterov, Adagrad, RMSprop, Adamax, and Nadam.

V. APPLICATIONS OF CNN

CNN applications extend beyond two-dimensional image processing to one-dimensional and multi-dimensional scenarios, enabled by its ability to harness massive amounts of data for promising results.

  • Applications of CNN: CNN is applied not only to two-dimensional images but also to one-dimensional and multi-dimensional scenarios.Its broad application scope reflects CNN’s role as a crucial deep learning concept.

A. Applications of one-dimensional CNN

One-dimensional CNNs use one-dimensional convolutional kernels to process one-dimensional data. They are effective for extracting features from fixed-length segments when feature position is irrelevant, supporting applications such as time-series prediction and signal identification.

  • 1D CNNs process one-dimensional data with one-dimensional convolutional kernels.
  • They effectively extract features from fixed-length segments when the feature’s position in the dataset does not matter.
  • Example applications include time-series prediction and signal identification.

C. Applications of multi-dimensional CNN

Multi-dimensional CNNs can theoretically process data of any dimension, but applications above three dimensions are uncommon because high-dimensional data is difficult for humans to understand. The section therefore uses 3D CNNs as examples, including human action recognition in videos and detection of X-ray and CT images.

  • Applications of multi-dimensional CNN: CNNs can theoretically be applied to data with any dimension, although multi-dimensional CNNs above three dimensions are uncommon.The passage attributes this limited use to the difficulty humans have understanding high-dimensional data.
  • Applications of multi-dimensional CNN: 3D CNNs are presented as examples of multi-dimensional CNN applications without implying that higher-dimensional CNNs are infeasible.The review uses 3D CNNs to explain key points about multi-dimensional convolution.
  • Applications of multi-dimensional CNN: 3D CNNs support automatic human action recognition in videos and detection of high-dimensional images such as X-rays and CT images.The passage notes that practitioners are actively working on detecting X-rays and CT images with 3D CNNs.

VI. PROSPECTS FOR CNN … C. Network Architecture Search

The paper surveys CNN prospects spanning model compression, security, and network architecture search, addressing deployment constraints, vulnerabilities, and automation of architecture design. It highlights pruning and quantization trade-offs, data poisoning and adversarial attacks, and resource-efficient NAS methods.

  • A. Model Compression: High accuracy often requires deep and wide CNNs that are difficult to deploy on embedded devices, motivating model compression through low-rank approximation, pruning, and quantization.These approaches target the computational and deployment burden of large architectures.
  • A. Model Compression: Network pruning is a major practical compression method, while binary quantization greatly reduces model size at the cost of accuracy.Ternary and multivariate quantization are used to balance model size and accuracy.
  • B. Security of CNN: CNN security is important because these models support applications related to life and property security, including identification, medical imaging, traffic signs, and license plates.The paper frames model disruption or destruction in such applications as potentially severe.
  • B. Security of CNN: Data poisoning inserts difficult-to-detect noise into training data, potentially causing low accuracy or targeted misidentification during testing.The passage notes that poisoned data may be fine-tuned to make models identify certain targets incorrectly.
  • B. Security of CNN: Adversarial attacks add visually imperceptible noise that can make CNN models fail to recognize otherwise identical images.Goodfellow et al. attribute vulnerability to linear characteristics such as ReLU and Maxout, whose analytical disturbances can damage networks.
  • C. Network Architecture Search: Network Architecture Search automates CNN machine learning by searching design choices such as kernel counts and skip connections while selecting an optimizer to control the search.NAS is divided into approaches with agents and without agents, but its high computing-resource demand affects integrated models.
  • C. Network Architecture Search: Google’s reinforcement-learning search algorithm achieved similar precision and speed to similarly structured networks on CIFAR-10, but it was computationally expensive.ENAS reduces resource requirements by sharing parameters among sub-models.

D. Capsule Neural Network · VII. CONCLUSION

The paper surveys CNN foundations, architectures, functions, applications, and prospects while highlighting limitations in spatial relationships, generalization, equivariance, security, and crowded-scene performance. It presents Capsule Neural Networks as an architectural response to CNNs’ loss of whole–part relationships and spatial information.

  • D. Capsule Neural Network: CNNs can fail to recognize the same object under changes such as rotation, scaling, and symmetry.The paper illustrates this limitation using different views of the same cat.
  • D. Capsule Neural Network: Addressing CNNs’ architectural flaw may require massive diverse datasets, but labeled data is typically hard to obtain.Data augmentation can provide some improvement, but the passage indicates it does not eliminate the underlying challenge.
  • D. Capsule Neural Network: Capsule Neural Networks organize neurons around shared entities or attributes to preserve whole–part relationships and reduce spatial information loss.They are proposed as an alternative to pooling, which ignores relationships between wholes and parts.
  • VII. CONCLUSION: The survey covers CNN building blocks, classic networks, related functions, applications, and prospects, reflecting CNNs’ broad research and industrial deployment.The paper attributes this adoption to local connection, weight sharing, and down-sampling dimensionality reduction.
  • VII. CONCLUSION: The paper explains basic CNN building blocks and how to construct a CNN-based model from scratch.This is presented as the survey’s first contribution.
  • VII. CONCLUSION: For accuracy, deeper and wider structures learn better representations, while residual connections enable extremely deep networks for complex tasks.For speed, the paper identifies dimension reduction and low-rank approximation as relevant strategies.
  • VII. CONCLUSION: The survey analyzes activation functions, loss functions, and optimizers, deriving conclusions and rules of thumb for their selection.The conclusions are based on experimental analysis.
  • VII. CONCLUSION: CNN applications span two-dimensional image tasks as well as one-dimensional and multi-dimensional convolution scenarios, while open issues include model size, security, hyperparameter selection, generalization, equivariance, and crowded scenes.The paper argues that different dimensional convolutions should be designed for different problems and identifies several promising directions.
Loading 2004.02806v1…