PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Advances in Convolutional Neural Networks

Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Li Wang, Gang Wang, Jianfei Cai, Tsuhan Chen

arXiv:1512.07108v6cs.CVcs.LGcs.NE

TL;DR

Rapid CNN advances have increased architectural depth and training complexity, creating a need for a comprehensive synthesis. This paper surveys improvements and applications across major domains, providing an extensive overview of the field.

  • Problem

    Rapidly deepening CNN architectures increase optimization difficulty and overfitting risk, motivating a comprehensive review of recent advances.

  • Method

    The paper surveys CNN advances in layer design, activation, loss, regularization, optimization, fast computation, and applications across vision, speech, and language.

  • Results

    The survey provides an extensive account of CNN improvements and applications spanning image, video, speech, and text processing.

  • Takeaways & Limitations

    The survey supports broader understanding of CNNs and may facilitate future research and application development.

  • Takeaways & Limitations

    Deep CNNs require large labeled datasets and substantial computing power, while their memory and time demands hinder deployment on mobile devices.

Abstract

from arXiv · show

In the last few years, deep learning has led to very good performance on a variety of problems, such as visual recognition, speech recognition and natural language processing. Among different types of deep neural networks, convolutional neural networks have been most extensively studied. Leveraging on the rapid growth in the amount of the annotated data and the great improvements in the strengths of graphics processor units, the research on convolutional neural networks has been emerged swiftly and achieved state-of-the-art results on various tasks. In this paper, we provide a broad survey of the recent advances in convolutional neural networks. We detailize the improvements of CNN on different aspects, including layer design, activation function, loss function, regularization, optimization and fast computation. Besides, we also introduce various applications of convolutional neural networks in computer vision, speech and natural language processing.

1. Introduction

This section introduces CNNs as deep-learning architectures rooted in biological visual perception and surveys their rapid development, including architectural advances and broader improvements across CNN components and applications.

  • Origins: CNNs were inspired by biological visual perception, with the neocognitron proposed in 1980 as a predecessor to modern CNNs.The introduction traces CNN origins from receptive-field discoveries in animal visual cortex to Fukushima’s neocognitron.
  • Architectural development: Since 2006, researchers have developed methods to address difficulties in training deep CNNs, notably through AlexNet’s improved image-classification performance.AlexNet retained a LeNet-5-like overall architecture while using a deeper structure.
  • Architectural development: CNN architectures have generally become deeper, with ResNet about 20 times deeper than AlexNet and 8 times deeper than VGGNet.Greater depth can increase nonlinearity and improve feature representations, but also raises network complexity.
  • Survey scope: The survey organizes recent CNN advances by convolutional and pooling layers, activation and loss functions, regularization, optimization, and fast computation.It also covers CNN applications in computer vision, speech, and natural language processing.

2. Basic CNN Components

Basic CNNs combine convolutional, pooling, and fully connected layers to learn increasingly abstract representations, followed by an output layer for task-specific prediction. Convolution uses shared kernels, pooling reduces feature-map resolution, and training commonly minimizes a task-specific loss with stochastic gradient descent.

  • Basic architecture: LeNet-5 illustrates the three basic CNN layer types: convolutional, pooling, and fully-connected layers.Convolutional layers learn feature representations, pooling layers reduce feature-map resolution, and fully-connected layers support high-level reasoning.
  • Convolution: Shared convolution kernels reduce model complexity and make CNNs easier to train.Each kernel generates a feature map through a weight-sharing mechanism across spatial locations.
  • Pooling: Pooling achieves shift-invariance by reducing feature-map resolution, typically through average or max pooling between convolutional layers.Each pooling feature map connects to the corresponding preceding convolutional feature map.
  • Hierarchical features: Stacked convolutional and pooling layers progress from detecting low-level edges and curves to encoding more abstract features.The first convolutional layer detects low-level features, while higher layers encode increasingly abstract representations.
  • Output and training: CNNs commonly use softmax for classification, while stochastic gradient descent optimizes parameters by minimizing an appropriate task-specific loss function.Fully-connected layers are optional because a 1 × 1 convolution layer can replace them.

3. Improvements on CNNs

This section surveys major CNN improvements since AlexNet’s 2012 success, organizing them across six aspects of network design and training.

  • 3. Improvements on CNNs: The review covers improvements in convolutional layers, pooling layers, activation functions, loss functions, regularization, and optimization.These six aspects define the section’s organizational framework.

3.1. Convolutional Layer

The convolutional layer’s representation ability is enhanced through alternative connectivity, receptive-field designs, and micro-network or multi-branch architectures. These approaches target richer invariances, larger context, more abstract representations, or reduced computation.

  • Convolutional Layer: Transposed convolution reverses traditional convolution’s connectivity, mapping a single activation to multiple output activations.It is also called deconvolution or fractionally strided convolution.
  • Convolutional Layer: Dilated convolution inserts zeros between filter elements to enlarge receptive fields and cover more relevant information for predictions requiring broad context.Its dilation factor is an additional convolutional-layer hyper-parameter.
  • Convolutional Layer: Network In Network replaces linear convolutional filters with multilayer perceptron micro-networks to approximate more abstract latent-concept representations.Its mlpconv layer uses 1×1 convolutions for cross-channel parametric pooling followed by ReLU, then global average pooling.
  • Convolutional Layer: Inception modules combine pooling and multiple convolution sizes, using preceding 1×1 convolutions to reduce dimensions and expand network depth and width without increasing computational complexity.The module targets visual patterns at different sizes and approximates an optimal sparse structure.
  • Convolutional Layer: Inception modules reduce network parameters to 5 millions, while later designs balance filter count and depth and gradually decrease representation size from inputs to outputs.The later designs also perform spatial aggregation over lower-dimensional embeddings with relatively modest computation cost.

3.2. Pooling Layer

The section surveys pooling methods that reduce CNN computational burden by limiting connections between convolutional layers. It covers biologically inspired, stochastic, frequency-domain, spatial-pyramid, and multi-scale pooling approaches, including methods designed to improve generalization, reduce overfitting, or produce fixed-length representations.

  • Pooling Layer: Lp pooling is theoretically suggested to generalize better than max pooling, with average pooling and max pooling as its p = 1 and p = ∞ cases.Lp pooling is biologically inspired and modeled on complex cells.
  • Pooling Layer: Mixed pooling randomly selects average or max pooling and experimentally performs better than either method while better addressing overfitting.A recorded binary λ controls the pooling choice during forward propagation and backpropagation.
  • Pooling Layer: Stochastic pooling samples activations from a multinomial distribution rather than always selecting the maximum, allowing non-maximal activations to be used and avoiding overfitting.The sampling probabilities are obtained by normalizing activations within each pooling region.
  • Pooling Layer: Spectral pooling reduces dimensionality by cropping the central frequency representation after a discrete Fourier transform and mapping it back with an inverse transform.The retained frequency submatrix has the desired output dimensions h × w.
  • Pooling Layer: Spatial pyramid pooling generates fixed-length representations for variable-sized inputs, while multi-scale orderless pooling combines whole-image and local-patch activations to improve CNN invariance.SPP uses spatial bins proportional to image size, and MOP aggregates local activations with VLAD encoding.

3.3. Activation Function

The section surveys CNN activation functions, emphasizing ReLU’s computational and sparsity benefits, variants that address its zero-gradient limitation, and alternatives such as ELU, maxout, and probout. These functions modify negative responses, learn or randomize parameters, or replace maximum selection to improve training, robustness, or invariance.

  • 3.3. Activation Function: ReLU zeros negative inputs and retains positive ones, enabling faster computation than sigmoid or tanh while inducing sparse hidden representations.Its max operation supports efficient computation and sparse representations.
  • 3.3. Activation Function: Leaky ReLU compresses negative inputs to preserve a small non-zero gradient, addressing ReLU units’ zero-gradient problem when inactive.Inactive ReLU units may never activate because gradient-based optimization does not adjust their weights.
  • 3.3. Activation Function: PReLU adaptively learns rectifier parameters, adding only channel-count extra parameters with negligible computational cost and no extra overfitting risk.The learned parameters can be trained jointly with other parameters by backpropagation.
  • 3.3. Activation Function: RReLU randomizes negative-part parameters during training and fixes them during testing, while evaluation found non-zero negative slopes consistently improve performance.Its randomized nature can reduce overfitting.
  • 3.3. Activation Function: ELU uses an identity positive part and a saturating negative part to avoid vanishing gradients, support faster learning and higher classification accuracies, and improve noise robustness.The negative saturation decreases unit variation when units are deactivated.
  • 3.3. Activation Function: Maxout selects the maximum response across channels and inherits ReLU’s benefits, whereas probout samples among linear units to improve invariance but costs more computation during testing.Probout balances maxout’s desirable properties with improved invariance properties.

3.4. Loss Function

This section surveys hinge, softmax, contrastive, triplet, and divergence-based loss functions, emphasizing their formulations, applications, and proposed improvements. It highlights margin-based classification, similarity learning, metric learning, and generative modeling objectives.

  • Hinge loss: Hinge loss trains large-margin classifiers such as SVM, with L1 and differentiable L2 variants; on MNIST, L2-SVM outperformed softmax.L2-Loss imposes a larger penalty for margin violations than L1-Loss.
  • Softmax loss: Softmax loss combines multinomial logistic loss with softmax probabilities, while L-Softmax adds an angular margin and outperformed original softmax on MNIST, CIFAR-10, and CIFAR-100.When m = 1, L-Softmax reduces to original softmax; larger margins define a more difficult objective that can avoid overfitting.
  • Contrastive loss: Contrastive loss trains Siamese networks from matching and non-matching pairs, but fine-tuning on all pairs can sharply reduce retrieval performance because of matching-pair handling.Performance is better retained when fine-tuning only on non-matching pairs; two margin parameters can be set equal and learned from sampled-pair distributions.
  • Triplet loss: Triplet loss minimizes anchor-positive distance while maximizing anchor-negative distance, whereas Coupled Clusters loss replaces randomly selected anchors with cluster centers to address neglected or falsely judged triplets.Coupled Clusters loss clusters positive-set samples together and keeps negative-set samples relatively far away.
  • Divergence-based losses: Kullback-Leibler divergence measures the non-symmetric difference between probability distributions and serves as an information-loss term in autoencoder objectives, including VAE variants.In VAE, the KLD term enforces a prior distribution on the proposal distribution, alongside reconstruction cost.
  • Divergence-based losses: Jensen-Shannon divergence symmetrically measures distribution similarity and supports GAN objectives, where minimizing it can make generated and real distributions converge when models have sufficient capacity.GANs are explicitly optimized for generative tasks but are notoriously unstable to train in practice.

3.5. Regularization

Regularization reduces overfitting in deep CNNs by adding complexity penalties or randomly masking activations and weights. The section covers ℓp-norm regularization, Dropout, and DropConnect.

  • ℓp-norm Regularization: Regularization reduces overfitting in deep CNNs by adding terms that penalize model complexity.The regularized objective includes a regularization term R(θ) weighted by strength λ.
  • ℓp-norm Regularization: For p ≥1, ℓp-norm regularization is convex and easier to optimize, while p < 1 promotes weight sparsity but produces a non-convex function.For p = 2, it is commonly called weight decay; Tikhonov regularization instead rewards invariance to input noise.
  • Dropout: Dropout randomly masks fully-connected-layer outputs using independently sampled Bernoulli variables and is effective at reducing overfitting.Its output is y = r∗a(WT x), where each mask element satisfies ri ∼Bernoulli(p).
  • DropConnect: DropConnect extends Dropout by randomly setting elements of the weight matrix W to zero rather than randomly setting neuron outputs to zero.During training, biases are also masked, with output y = a((R ∗W)x) and Rij ∼Bernoulli(p).

3.6. Optimization

CNN optimization addresses difficult training caused by large parameter counts, non-convex loss functions, vanishing gradients, and degradation in deeper networks. Key approaches include careful initialization, gradient-based optimization, batch normalization, and architectural mechanisms such as highway and residual connections.

  • Network initialization: Careful network initialization helps accelerate convergence and avoid vanishing gradients in deep CNNs with non-convex loss functions.Biases can be initialized to zero, while weights should break symmetry; orthonormal initialization and layer-sequential unit-variance initialization are described as effective approaches.
  • Gradient-based optimization: Gradient descent and mini-batch SGD update CNN parameters using estimated gradients, with mini-batches reducing update variance and producing more stable convergence.Learning-rate selection remains challenging, and momentum, Nesterov momentum, parallelized SGD, and early stopping address additional optimization concerns.
  • Batch normalization: Batch normalization reduces internal covariate shift, improves gradient flow, permits higher learning rates without divergence, and regularizes the model.These effects reduce dependence on parameter scale or initialization and reduce the need for Dropout.
  • Deep-network optimization: Normalized initialization and batch normalization can prevent vanishing gradients but do not eliminate the degradation problem, in which deeper CNNs perform worse than shallower ones.Highway networks address this through transform gates that support efficient training of networks with tens or hundreds of layers.
  • Deep-network optimization: Residual networks use shortcut connections to directly propagate untransformed inputs with fewer parameters, while preactivation with BN + ReLU achieves higher accuracies than previous ResNets.Experiments find identity shortcut connections easiest for networks to learn and BN before addition considerably better than BN after addition.

4. Fast Processing of CNNs

The section surveys methods for accelerating CNN training, testing, and inference by transforming convolutions, reducing parameter complexity, lowering numerical precision, and exploiting redundancy. These approaches improve computational or memory efficiency, but may introduce additional memory costs or accuracy-performance tradeoffs.

  • Fourier-domain convolution: FFT-based convolution reuses Fourier-transformed filters and output gradients across minibatch operations and input channels to accelerate CNN processing.The approach performs convolution in the Fourier domain and reuses transformations during forward and backward computations.
  • Fourier-domain convolution: FFT convolution requires extra memory for Fourier-domain feature maps and becomes especially costly with strides greater than 1 and small convolutional filters.Filters must be padded to the input size, limiting the practical benefits of FFT-based acceleration in common network layers.
  • Efficient parameterizations: Low-rank factorization replaces an m×n rank-r matrix with factors A and B, reducing parameters when r < pmn/(m + n).The factorization uses an m×r matrix A and an r×n matrix B, with mr + rn < pmn required to reduce parameters by fraction p.
  • Efficient parameterizations: Adaptive Fastfood and circulant or ACDC transforms reduce fully connected layers from quadratic to near-linear complexity, reaching O(n) space and O(n log n) time.Adaptive Fastfood has O(n) space and O(n log n) time; circulant structures reduce O(n^2) space and time to O(n) and O(n log n), while ACDC has the same asymptotic complexities.
  • Binarization: Binary neural networks restrict network arithmetic to binary values, with XNOR-Net reporting top-1 accuracies up to 51.2% for full and 65.5% for partial binarization.A fully binarized MNIST network using XNOR and bit-count operations reports 98.7% accuracy.
  • Model compression and sparsity: CNN compression methods include vector quantization, pruning, hashing, and sparsification, which reduce parameters, operations, or storage while preserving useful performance.Basis-filter sparsification achieves 90% sparsifying, while LCNN learns from few training examples and reaches higher accuracy in fewer iterations than standard CNNs.

5. Applications of CNNs … 5.3. Object Tracking

CNNs have achieved state-of-the-art performance across image classification, object detection, object tracking, and other applications. The surveyed approaches improve recognition through joint feature-classifier learning, hierarchical or localized representations, efficient detection pipelines, and target-specific tracking models.

  • 5.1. Image Classification: CNNs improve large-scale image classification by jointly learning features and classifiers, with AlexNet achieving the best performance in ILSVRC 2012.The paper identifies the 2012 ImageNet breakthrough as a turning point for large-scale CNN classification.
  • 5.1. Image Classification: Hierarchical classifiers share information among related classes to improve performance when some classes have very few training examples.Tree-based priors and category hierarchies are used to transfer information across classes.
  • 5.1. Image Classification: Fine-grained classification benefits from localizing important object parts and representing their appearances discriminatively.Approaches use annotated or automatically generated parts, regions, proposals, alignment, co-segmentation, and attention.
  • 5.2. Object Detection: CNN-based object detection focuses on accurately and efficiently localizing objects in images or video frames, but progress was initially limited by scarce data and processing resources.The paper describes object detection as a long-standing computer-vision problem whose CNN progress accelerated after 2012.
  • 5.2. Object Detection: Object proposal methods first identify potential objects with fast generic measurements, then pass proposals to sophisticated detectors for classification.R-CNN is presented as a prominent proposal-based CNN detector.
  • 5.2. Object Detection: Feature-sharing methods reduce detection cost by computing CNN representations once and reusing them across overlapping windows or regions.OverFeat uses an image pyramid so computation can be shared between overlapping windows.
  • 5.2. Object Detection: YOLO and SSD enable single-pipeline detection by directly predicting class labels and bounding boxes, supporting end-to-end optimization.YOLO formulates detection as regression from the full image in one network evaluation, while SSD discretizes the output space.
  • 5.3. Object Tracking: CNN tracking methods address viewpoint changes, illumination changes, and occlusions through robust target representations, online updates, and target-specific discrimination.Methods include class-specific or target-specific CNNs, candidate pools of multiple CNNs, learned discriminative features, drift mitigation using initial-frame appearance, and online SVM layers.

5.4. Pose Estimation … 5.7. Action Recognition

The surveyed CNN advances span pose estimation, scene text understanding, visual saliency prediction, and action recognition in still images and videos. Across these tasks, methods learn structured representations from body parts, contextual regions, sequential characters, saliency contexts, and temporal dynamics.

  • 5.4. Pose Estimation: Pose estimation progressed from holistic CNN regression of body-joint coordinates to local body-part representations using jointly trained detectors, spatial models, and multi-resolution heat maps.DeepPose introduced a cascade of seven-layer CNNs, while later methods modeled local parts with convolutional priors and heat maps.
  • 5.4. Pose Estimation: Video pose estimation extends multi-resolution CNNs with RGB and motion features, using a sliding-window architecture whose input is a 3D tensor.The tensor combines an RGB image with corresponding motion features.
  • 5.5. Text Detection and Recognition: CNN-based scene text detection classifies text versus non-text patches or MSER candidates, then uses multiscale response maps, sliding windows, and Non-Maximal Suppression for localization.These approaches reduce search space with character candidates and split cluttered text components before suppression.
  • 5.5. Text Detection and Recognition: Scene text recognition uses multiple softmax classifiers or CRF-like CNNs to predict characters sequentially and jointly model character sequences and bigrams.These methods target recognition in multi-digit images and seek to avoid reliance on lexicons and dictionaries.
  • 5.5. Text Detection and Recognition: End-to-end text spotting applies CNNs across detection and recognition tasks, including case-sensitive and insensitive character classification and bigram classification.Feature sharing supports the four subtasks within an integrated text-spotting system.
  • 5.6. Visual Saliency Detection: Visual saliency methods combine CNN representations with local and global context, while other approaches rely on ensembles or jointly combine responses from every layer of a deeper CNN.Deep Gaze uses a pre-trained CNN with five convolutional layers to predict saliency values.
  • 5.7. Action Recognition: For still-image action recognition, CNN features describe full action images, human bounding boxes, and contextual regions, with part detection and contextual representation improving the description.Pre-trained CNN penultimate-layer outputs serve as visual descriptors, while contextual regions are selected from object proposals.
  • 5.7. Action Recognition: Video action recognition must model an additional temporal axis and larger signal sizes, motivating 3D convolutions, 2D feature-map fusion, and recurrent sequence learners such as LSTMs.Fusion policies include late, early, and slow fusion, while another framework feeds CNN features from individual frames to a sequence-learning module.

5.8. Scene Labeling · 5.9. Speech Processing · 5.10. Natural Language Processing

The surveyed applications show CNNs supporting dense scene labeling, speech recognition and synthesis, statistical language modeling, and text classification. Across these tasks, CNNs exploit local structure while enabling increasingly deep architectures for visual and language processing.

  • 5.8. Scene Labeling: CNNs model pixel-level class likelihoods from local image patches for scene labeling, learning features and classifiers that discriminate local visual subtleties.Scene labeling assigns a semantic class such as road, water, or sea to each input pixel.
  • 5.8. Scene Labeling: Pre-trained CNN features support semantic segmentation through zoom-out representations, while fully convolutional networks directly predict dense label maps.Zoom-out features concatenate local, proximal, distant, and global features from ConvNets and AlexNet.
  • 5.9. Speech Processing: Before CNN-based ASR, speech recognition was dominated by HMM and GMM-HMM methods that typically required handcrafted features such as MFCCs.ASR converts human speech into spoken words.
  • 5.9. Speech Processing: CNNs outperform GMM-HMMs and general DNNs in speech recognition by exploiting time-frequency correlations through local connectivity and capturing frequency shifts.CNNs have been applied to Mel filter bank features and, in some approaches, raw waveforms.
  • 5.9. Speech Processing: CNNs have also spread to statistical parametric speech synthesis, where deep learning is used to address the muffled quality of speech generated by shallow HMM networks.Speech synthesis generates speech sounds from text, possibly with additional information.
  • 5.10. Natural Language Processing: For statistical language modeling, CNN-based methods process incomplete word sequences using character-level CNN outputs, gated convolutional architectures, and ByteNet.genCNN replaces max-pooling with separate gating networks, while ByteNet is a CNN-based sequence-processing architecture.
  • 5.10. Natural Language Processing: CNNs achieve top performance in text classification by capturing local temporal or hierarchical relations, making architecture design important for complex sentence structures.The passage identifies text classification as a crucial NLP task and notes that sentence structures are sequential and hierarchical.
  • 5.10. Natural Language Processing: 29 convolutional layers were used in a deep text-classification architecture; shortcut connections improved results at 49 layers, but the model did not achieve state-of-the-art performance.The reported architecture was substantially deeper than the shallow CNNs previously mentioned for NLP.

6. Conclusions and Outlook

The paper surveys recent CNN advances across model design, training, computation, and applications. It highlights unresolved challenges involving data and compute demands, hyperparameter tuning, optimization, and the theoretical understanding of CNNs.

  • Survey scope: The survey covers CNN improvements in layer design, activation and loss functions, regularization, optimization, and fast computation, alongside applications across image, video, speech, and text.It presents an extensive survey of recent CNN advances and their applications to many tasks.
  • Data and computation: Deeper CNNs require large-scale datasets and massive computing power, motivating further investigation of unsupervised learning and faster training procedures.Manually collecting labeled data requires substantial human effort.
  • Optimization and tuning: Selecting CNN hyperparameters remains difficult because learning rates, filter kernel sizes, and layer counts have internal dependencies that make tuning expensive.The passage identifies substantial room for improving optimization techniques for learning deep CNN architectures.
  • Theory and design: A solid theory of CNNs is still lacking, creating a need to investigate their fundamental principles and leverage natural visual perception to improve CNN design.The passage notes that CNNs work well across applications, despite limited understanding of why and how they work essentially.
Loading 1512.07108v6…