PaperlayerView exact revision on arXiv

Source-linked AI summary

The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches

Md Zahangir Alom, Tarek M. Taha, Christopher Yakopcic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Brian C Van Esesn, Abdul A S. Awwal, Vijayan K. Asari

arXiv:1803.01164v2cs.CV

TL;DR

The survey addresses the need for a broad account of deep-learning development, advanced training techniques, generative models, and applications beyond earlier surveys. It synthesizes major learning architectures and variants, application domains, frameworks, SDKs, and benchmark datasets, reporting state-of-the-art performance across representative tasks and efficiency gains for binary networks.

  • Problem

    Earlier deep-learning surveys did not discuss individual advanced techniques for training large-scale models or recently developed generative methods.

  • Method

    The paper surveys supervised, unsupervised, semi-supervised, and reinforcement-learning approaches, their variants, applications, implementation frameworks, SDKs, and benchmark datasets.

  • Results

    Deep-learning approaches achieve state-of-the-art performance across application domains, while XNOR-based CNNs provide 58x faster convolution and 32x memory savings with 2.9% lower top-1 accuracy than full-precision AlexNet.

  • Takeaways & Limitations

    The survey presents deep learning as a broad family of methods spanning multiple learning categories, architectures, application domains, and implementation resources.

Abstract

from arXiv · show

Deep learning has demonstrated tremendous success in variety of application domains in the past few years. This new field of machine learning has been growing rapidly and applied in most of the application domains with some new modalities of applications, which helps to open new opportunity. There are different methods have been proposed on different category of learning approaches, which includes supervised, semi-supervised and un-supervised learning. The experimental results show state-of-the-art performance of deep learning over traditional machine learning approaches in the field of Image Processing, Computer Vision, Speech Recognition, Machine Translation, Art, Medical imaging, Medical information processing, Robotics and control, Bio-informatics, Natural Language Processing (NLP), Cyber security, and many more. This report presents a brief survey on development of DL approaches, including Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) including Long Short Term Memory (LSTM) and Gated Recurrent Units (GRU), Auto-Encoder (AE), Deep Belief Network (DBN), Generative Adversarial Network (GAN), and Deep Reinforcement Learning (DRL). In addition, we have included recent development of proposed advanced variant DL techniques based on the mentioned DL approaches. Furthermore, DL approaches have explored and evaluated in different application domains are also included in this survey. We have also comprised recently developed frameworks, SDKs, and benchmark datasets that are used for implementing and evaluating deep learning approaches. There are some surveys have published on Deep Learning in Neural Networks [1, 38] and a survey on RL [234]. However, those papers have not discussed the individual advanced techniques for training large scale deep learning models and the recently developed method of generative models [1].

I. INTRODUCTION

Deep learning emerged from neural networks as a rapidly growing machine-learning field, with supervised, semi-supervised, unsupervised, and reinforcement-learning approaches applied across many domains. Its learned hierarchical features and reported results support state-of-the-art performance in tasks including ImageNet classification and speech recognition.

  • I. INTRODUCTION: Deep learning is a neural-network subfield of machine learning developed largely from 2006 onward and applied across many application domains.The paper places deep learning within the taxonomy AI → ML → NN → DL.
  • I. INTRODUCTION: Supervised learning uses labeled input-output pairs and iteratively updates network parameters to approximate desired outputs.The agent predicts an output, receives a loss value, and modifies parameters during training.
  • I. INTRODUCTION: Unsupervised learning learns internal representations from unlabeled data to discover features, relationships, or structure through clustering, dimensionality reduction, and generation.The survey identifies autoencoders, restricted Boltzmann machines, and GANs as relevant deep-learning approaches.
  • I. INTRODUCTION: Deep reinforcement learning operates in unknown environments through interaction, where actions affect later states and no straightforward loss function is available.The survey contrasts this setting with supervised learning, which provides direct target outputs.
  • I. INTRODUCTION: Deep learning automatically learns hierarchical features rather than relying on handmade feature-extraction algorithms used in traditional machine learning.The paper identifies automatic hierarchical feature learning as a central distinction between deep and traditional machine learning.
  • I. INTRODUCTION: 3.57% error on ImageNet was reported for ResNet-152, compared with 5% human error, while deep-learning methods also outperformed earlier approaches on TIMIT speech recognition.The ImageNet figure is reported as an error rate; the TIMIT comparison is described using phone error rates over prior work.

E. Why deep Learning

Deep learning is presented as a broadly applicable machine-learning approach whose performance benefits from increasing data, while its development remains constrained by scalability, generation, energy, and multi-task challenges. The section also situates deep learning within neural-network history and highlights survey coverage of methods, frameworks, SDKs, and benchmarks.

  • Deep learning applies across many application domains and can automatically learn task-relevant features rather than requiring hand-designed features.
  • Key deep-learning challenges include big-data analytics, scalability, data generation, energy efficiency, and multi-task or transfer learning.
  • As data increases, traditional machine-learning performance becomes steady while deep-learning performance continues to increase.
  • The survey includes deep-learning frameworks, SDKs, and benchmark resources for implementation and evaluation.
  • Earlier neural-network milestones include perceptrons, their limitations, backpropagation, and later developments in deep learning.
  • The survey covers neural-network fundamentals, including neurons, multilayer perceptrons, and gradient-descent training.

C. Stochastic Gradient Descent (SGD)

Stochastic gradient descent trains neural networks by iteratively updating parameters using gradients, with momentum and learning-rate schedules used to improve training behavior.

  • Stochastic gradient descent updates network parameters using the loss, learning rate, dataset, and model predictions.
  • Backpropagation uses the chain rule on computation graphs to calculate gradients through multilayer networks.
  • Momentum accelerates SGD by using a moving average of gradients and is described as helping prevent convergence to a local minimum.
  • Learning-rate selection is sensitive: excessive values can cause divergence, whereas small values slow convergence.
  • Learning-rate decay can be constant, factored, or exponential, with η_t defined from η_0 and decay factor β.
  • A common practice is β=0.1, reducing the learning rate by a factor of 10 at each stage.

G. Weight decay

Weight decay is presented as L2 regularization for training deep models, while the surrounding discussion connects efficient learning to network design, optimization, and CNN feature extraction.

  • Weight decay applies L2 regularization during deep-model training to help prevent overfitting and support model generalization.
  • Deep networks can represent highly varying nonlinear functions and combine feature extraction with classification layers.
  • The survey identifies data preprocessing, augmentation, initialization, batch normalization, activation functions, dropout, and optimization as components of efficient training.
  • Before 2006, training deep supervised feed-forward networks often produced worse training and test errors than shallow networks.
  • CNN architecture: CNNs use convolution, max-pooling, and classification layers, with higher-level features derived from lower-level features.
  • CNN architecture: Convolutional layers combine input feature maps with learnable kernels and activation functions to produce output maps.
  • CNN architecture: Pooling downsamples feature maps while preserving their number, using average pooling or max-pooling to select summarized values.

3) Classification Layer

CNN classification pipelines transform images through convolution and pooling before a fully connected classifier, with architecture choices affecting computational cost and recognition performance. The survey reviews major architectures from LeNet and AlexNet through VGG and later alternatives.

  • Classification layer: The fully connected classification layer computes class scores from features extracted by preceding convolutional layers.
  • Classification layer: Fully connected layers can be computationally expensive, motivating global or average pooling alternatives that reduce network parameters.
  • Model complexity: CNN complexity is evaluated using computational parameter counts and total memory requirements.
  • Model complexity: For a convolutional layer, output dimensions depend on input size, filter size, and stride, while padding can preserve input and output dimensions.
  • Popular CNN architectures: Popular CNN architectures share convolutional and pooling components but differ in topology, including AlexNet, VGG, GoogLeNet, DenseNet, and FractalNet.
  • Popular CNN architectures: AlexNet used a deeper and wider CNN than LeNet and achieved state-of-the-art recognition accuracy in the 2012 ImageNet challenge.
  • Popular CNN architectures: ZFNet replaced AlexNet’s 11×11 kernels with 7x7 kernels, reducing weights and improving reported recognition accuracy.
  • Popular CNN architectures: VGG emphasized network depth, with VGG-11, VGG-16, and VGG-19 containing 8, 13, and 16 convolution layers, respectively.

6) GoogLeNet (2014)

The survey presents GoogLeNet's Inception design and subsequent architectures that improve efficiency, depth, feature reuse, or spatial representation. These approaches combine architectural changes such as dimensionality reduction, residual connections, dense connectivity, and capsules.

  • 6) GoogLeNet (2014): GoogLeNet used Inception layers with variable receptive fields to capture sparse correlation patterns while reducing computation complexity.Different kernel sizes created the variable receptive fields, and 1×1 convolutions reduced dimensions before expensive layers.
  • 6) GoogLeNet (2014): GoogLeNet achieved 7M network parameters, compared with 60M for AlexNet and 138M for VGG-19.
  • 7) Residual Network (ResNet in 2015): ResNet adds a residual connection, defining each layer output as x_l = ℱ(x_l−1) + x_l−1.The residual transformation may include convolution, batch normalization, and ReLU operations.
  • DenseNet: DenseNet connects each layer to all successor layers, supporting feature reuse and reducing network parameters.Dense blocks are separated by transition blocks that use 1×1 convolution, batch normalization, and average pooling.
  • FractalNet: FractalNet uses drop-path to build large models with nominal depth and shorter gradient-propagation paths, enforcing speed–accuracy tradeoffs.
  • CapsuleNet: CapsNet uses vector-valued capsules and encoding–decoding units, achieving state-of-the-art MNIST digit recognition while being more suitable for segmentation and detection.Its capsules can detect overlapped features by aggregating weighted outputs from preceding layers.

D. Comparison on different models

The survey compares deep learning architectures and reviews their applications, training components, and data-preparation methods. It emphasizes that efficient deep learning depends not only on model architecture but also on training techniques.

  • D. Comparison on different models: The survey compares recently proposed models using error, network parameters, and maximum numbers of connections.
  • Applications: Deep learning models are applied across image classification, detection, segmentation, localization, captioning, video classification, speech processing, and medical imaging.Examples include CNN-based speech enhancement, audio tagging, medical diagnosis, cardiac segmentation, and brain-tumor segmentation.
  • Training techniques: The survey identifies advanced training components including preprocessing, initialization, batch normalization, alternative convolutions, activation functions, pooling, and regularization.
  • Training techniques: Data preparation may include rescaling, mean subtraction, random cropping, flipping, color jittering, and PCA/ZCA whitening.
  • Training techniques: Initialization affects recognition accuracy, with Xavier, LSUV, and He initialization described as approaches for training deep networks.LSUV is described as data-driven and evaluated on benchmark datasets including ImageNet.
  • Training techniques: Batch normalization reduces internal covariance by transforming inputs toward zero mean and unit variance, supporting faster convergence and regularization.The supplied algorithm applies BN with scale and shift parameters γ and β to mini-batch values.

D. Alternative Convolutional methods

The survey reviews alternative convolutional, activation, pooling, and subsampling methods intended to improve computational efficiency, accuracy, or feature processing in deep networks.

  • D. Alternative Convolutional methods: Alternative convolutional techniques have been proposed that reduce multiplication costs by a factor of 2.5.
  • Activation functions: The survey presents sigmoid and hyperbolic tangent as traditional activation functions alongside newer rectified alternatives.
  • Activation functions: ReLU keeps positive values and sets negative values to zero, and is described as addressing the vanishing-gradient problem in deep learning.The survey notes that ReLU was first used in AlexNet.
  • Activation functions: Improved activation functions include PReLU, ELU, leaky ReLU, MELU, and S-shaped rectified linear units.The survey describes ELU as enabling a faster and more accurate DCNN structure.
  • Pooling methods: Pooling methods include average pooling, max-pooling, spatial pyramid pooling, and fractional max pooling.Average pooling aggregates values over patches, whereas max-pooling selects the highest value.

G. Regularization approaches for DL

The survey describes regularization and recurrent approaches for improving deep learning training and processing sequential information. It covers dropout variants, optimization methods, RNN formulations, LSTM gating, and GRU simplification.

  • G. Regularization approaches for DL: Dropout regularizes networks by setting a randomly selected subset of activations to zero within a layer.
  • G. Regularization approaches for DL: DropConnect instead sets a randomly selected subset of weights to zero, so each layer receives a subset of units from the preceding layer.
  • Optimization: Optimization methods discussed include SGD, Adagrad, AdaDelta, RMSprop, and Adam, with momentum and adaptive learning rates among their described features.
  • Recurrent neural networks: RNNs pass information between successive steps through recurrent loops, but their main problem is the vanishing-gradient problem.
  • Long Short-Term Memory: LSTMs use a cell state controlled by input, forget, and output gates to add or remove information.The survey gives equations for the gates, candidate cell state, cell-state update, and output.
  • Gated Recurrent Unit: GRUs combine the forget and input gates into an update gate and are lighter than LSTMs in topology, computation cost, and complexity.The survey reports no clear empirical winner: GRUs use fewer parameters, while LSTMs may perform better with sufficient data and computation.

D. Convolutional LSTM (ConvLSTM)

This section presents ConvLSTM for temporal video analysis and reviews recurrent architectures, attention-based applications, and auto-encoder representation learning. It also notes vanishing gradients as a limitation of deeper auto-encoders.

  • D. Convolutional LSTM (ConvLSTM): ConvLSTM combines convolutional operations with recurrent state updates for temporal data analysis on video datasets.The formulation uses convolution and Hadamard-product operations in its gates and cell-state updates.
  • E. Variant of architectures of RNN with respective to the applications: Word2Vec is a two-layer predictive neural network used to encode words or sentences for language understanding and related applications.The survey connects word embeddings with unsupervised word learning, relationship learning, sentence modeling, and language tasks.
  • E. Variant of architectures of RNN with respective to the applications: RNN architectures include one-to-one, many-to-one, one-to-many, and many-to-many mappings for tasks such as classification, sentiment analysis, and image captioning.Inputs, recurrent states, and outputs are organized differently according to the application.
  • E. Variant of architectures of RNN with respective to the applications: Attention-based RNNs support image description, time-series prediction, and visual question answering using image and textual inputs.The reviewed approaches include attention mechanisms with RNNs and GRUs, with CNNs used to encode images in visual question answering.
  • Auto Encoder: Auto-encoders learn lower-dimensional representations through encoding and reconstruct inputs through decoding for dimensionality reduction, compression, and feature learning.The encoder maps inputs into feature space, while the decoder regenerates the original features.
  • Auto Encoder: Vanishing gradients remain a major issue for deeper auto-encoder models because gradients become too small when propagated through many layers.The survey introduces advanced auto-encoder variants in response to this limitation.

B. Variational auto encoders (VAEs)

This section reviews variational and other auto-encoder-related approaches alongside restricted Boltzmann machines for unsupervised representation learning. It identifies basic GAN limitations that VAEs address through a Gaussian-structured latent space.

  • B. Variational auto encoders (VAEs): VAEs address basic GAN limitations by representing images with latent vectors that follow a unit Gaussian distribution.The cited limitations include difficulty controlling generated features and generating objects that satisfy specific visual constraints.
  • B. Variational auto encoders (VAEs): Auto-encoders support unsupervised feature extraction, clustering, and encoding or decoding in combination with CNN, DNN, RNN, and reinforcement learning.Applications cited include bio-informatics and cyber security, including feature extraction followed by Winner Take All clustering.
  • Restricted Boltzmann Machines: RBMs are undirected, energy-based generative models with visible and hidden stochastic units connected by symmetric weights.Their energy function uses visible-unit biases, hidden-unit biases, and weights between visible and hidden units.
  • B. Variational auto encoders (VAEs): Split-Brain auto-encoders divide the network into disjoint sub-networks that predict the feature representation of an entire image.The architecture is described as a modification of traditional auto-encoders for unsupervised representation learning.
  • Restricted Boltzmann Machines: RBM training samples hidden and visible states conditionally and uses alternating Gibbs sampling to reconstruct visible states.A simplified learning rule is used to pre-train neural networks in an unsupervised manner and initialize weights for DBNs.
  • Restricted Boltzmann Machines: RBM-derived learning rules provide unsupervised pre-training for neural networks and form the basis of Deep Belief Networks.The survey cites applications including data encoding, news clustering, and cyber security.

VII. GENERATIVE ADVERSARIAL NETWORKS (GAN)

This section reviews GANs as adversarial generative models for addressing data scarcity and producing or manipulating images. It covers the minimax training objective, training limitations, and examples of learned latent-space structure and unsupervised representation learning.

  • Review on GAN: GANs address the need for large labeled datasets by generating samples that can support data-driven deep-learning applications.The survey frames synthetic sample generation as an approach to the labeled-data demands of computer-vision tasks.
  • Review on GAN: GANs use competing generator and discriminator networks trained in a zero-sum game as an alternative to maximum likelihood estimation.The generator produces samples while the discriminator distinguishes generated outputs from real data.
  • Review on GAN: The original GAN objective can provide insufficient generator gradients early in training, and GAN training may be unstable or produce nonsensical outputs.The survey describes a non-saturating generator objective as providing better early-stage gradients, while noting convergence limitations.
  • Review on GAN: After five epochs, generated bedroom images show visual under-fitting through repeated noise textures across multiple samples.The repeated textures include baseboards on some beds.
  • Review on GAN: Latent-space interpolation produces smooth transitions between plausible bedroom images and supports arithmetic operations that modify semantic attributes.The examples include transforming rooms and combining facial attributes such as glasses and gender.
  • Review on GAN: A learned face-direction vector can transform pose by interpolating along an axis derived from faces looking left versus right.The direction, called a “turn” vector, is created from averaged samples.
  • Review on GAN: InfoGAN is reported as competitive with fully supervised representation learning while learning representations in an unsupervised manner.The survey also reviews extensions for interactive image manipulation, image translation, and multi-domain image distributions.

B. Applications of GAN

The survey describes GAN applications across image generation and restoration, speech, music, medical imaging, online learning, and other tasks. It also places DRL within broader deep-learning approaches for prediction, classification, generation, and interaction with environments.

  • GAN applications: GANs support image generation, super-resolution, semantic segmentation, text-conditioned synthesis, style transfer, and image de-raining.Applications include photo-realistic image generation, weakly supervised segmentation, TAC-GAN, MSG-Net, and single-image de-raining.
  • GAN applications: GANs have been applied to dialogue, speech enhancement, symbolic music generation, and audio waveform synthesis.SEGAN targets speech enhancement, while GAN-based symbolic generation is compared with Melody RNN and WaveNet generates audio waveforms.
  • GAN applications: Medical GAN applications include image denoising, brain-tumor segmentation, general medical-image segmentation, and MRI compressive sensing.The cited methods include Wasserstein and perceptual losses, conditional GANs, SegAN, and Deep GAN.
  • GAN applications: BC-GAN generates samples from deterministic inputs and addresses supervised, semi-supervised, and unsupervised learning problems.The Bayesian conditional formulation is presented as an approach for handling multiple learning settings.
  • GAN applications: MMD-GAN replaces the discriminator with a two-sample kernel-MMD approach and significantly outperforms the alternative GMMN technique.The comparison is framed around generative moment matching and statistical hypothesis testing.
  • DRL applications: DRL uses CNN, RNN, LSTM, or GRU representations to observe environments, select actions, and update behavior through rewards.The survey describes Q-learning, Bellman updates, discounted future rewards, and neural-network approximations for state-action utility.

C. Recent trends of DRL with applications

Recent DRL work spans algorithmic advances and applications in healthcare, policy learning, packing, and continuous control. The section also explains transfer learning as reuse and adaptation of pretrained representations, especially when labeled target data are limited.

  • DRL trends: Recent DRL research includes DQN, trust-region policy optimization, asynchronous advantage actor-critic, and visual-understanding approaches.The cited survey is presented alongside current DRL methods and applications.
  • Applications: DRL applications include information sharing and personalized interventions for mobile healthcare, using network cohesion and group-based policies.One approach groups people with K-means before sharing an RL policy for each group.
  • Applications: Option-Observation Initiation Sets support faster optimal-policy learning in challenging partially observable Markov decision processes than RNNs.The passage identifies OOIs as a strategy for difficult POMDP policy-learning tasks.
  • Applications: DRL is applied to 3D bin packing, where the objective is to place cuboid items while minimizing bin surface area.The application is presented as a recent DRL problem formulation.
  • DRL trends: Fuzzy RL targets continuous state and action spaces, while related work studies policy-gradient hyperparameters, algorithm variance, and baseline reporting.The discussion also mentions high-precision assembly tasks and continuous-control evaluation practices.
  • Transfer learning: Transfer learning adapts a network trained on a related labeled domain by replacing top layers and fine-tuning on the target domain.The survey describes ImageNet pretraining followed by target-domain supervised adaptation.
  • Transfer learning: Pretrained weights reduce the need for large labeled datasets and can speed convergence and improve network generalization.The survey also motivates reuse by citing the computation and time required to train large models from scratch.

X. ENERGY EFFICIENT APPROACHES AND HARDWIRES FOR DL

The section surveys approaches for reducing deep-learning computation, memory, power, and hardware costs while preserving classification accuracy. It covers efficient network structures, low-precision and binary operations, hardware architectures, mobile deployment, and implementation resources.

  • Motivation: Growing network size improves classification accuracy but increases storage consumption, memory bandwidth, and computational cost exponentially.
  • Efficient network design: Efficient deep-network design combines lower internal operational cost with low-precision operations or hardware-efficient architectures.
  • Efficient network design: Delayed down-sampling led to higher classification accuracy across four CNN architectures.
  • Binary and ternary networks: Binary neural networks replace most arithmetic with bit-wise operations, reducing memory use and improving power efficiency.Binary multiplication on GPUs was observed to be almost seven times faster than traditional matrix multiplication, while binarized CNN kernels can reduce dedicated-hardware complexity by around 60%.
  • Binary and ternary networks: XNOR-based CNNs achieved 58x faster convolution and 32x memory savings, with 2.9% lower top-1 classification accuracy than full-precision AlexNet on ImageNet.The reported efficiency makes CPU-based real-time implementation possible instead of requiring a GPU.
  • Hardware and resources: The section also notes specialized hardware, mobile deployment, frameworks, SDKs, and benchmark datasets used to implement and evaluate deep-learning approaches.

C. Language modeling

This section lists language-related corpora, evaluation datasets, publication venues, and a reinforcement-learning course. It also notes weak supervision through noisy or conflicting labeling functions and places these resources within the survey’s broader review of deep learning applications.

  • Language datasets: Language-modeling resources include free eBooks, the Brown and Stanford corpora, and the Google 1 Billion Word corpus.
  • Language datasets: The listed language and multimodal datasets include Flickr 8k, Flickr 30k, COCO, translation corpora, question-answering datasets, sentiment datasets, TIMIT, Voxforge, and IMDB.
  • Weak supervision: Data programming can label subsets using weak-supervision strategies or domain heuristics, even when labeling functions are noisy or conflict on samples.
  • Community resources: The survey identifies ArXiv, conferences, journals, and a Berkeley reinforcement-learning course as publication or learning resources for the deep-learning community.
  • Survey scope: The report provides an in-depth review of deep learning models, applications, frameworks, benchmark datasets, and relevant journals and conferences.
Loading 1803.01164v2…