PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review of Deep Learning with Special Emphasis on Architectures, Applications and Recent Trends

Saptarshi Sengupta, Sanchita Basak, Pallabi Saikia, Sayak Paul, Vasilios Tsalavoutis, Frederick Atiah, Vadlamani Ravi, Alan Peters

arXiv:1905.13294v3cs.LGstat.ML

TL;DR

Professionals need a practical entry point into deep learning’s rapidly expanding architectures, training and deployment methods, and application areas. This review surveys multilayer neural architectures, multi-agent optimization, reliability testing, and selected applications, concluding that deep learning offers broad pattern-recognition capacity while still facing data-scarcity and scope challenges.

  • Problem

    The expanding deep-learning literature makes it difficult for practitioners to choose, train, and deploy appropriate models across application areas.

  • Method

    The review synthesizes key multilayer architectures, multi-agent architecture optimization, neural-network testing and fault mitigation, and application-oriented research.

  • Results

    The review covers architectures, reliability methods, and applications spanning finance, prognostics, medical imaging, and power systems.

  • Takeaways & Limitations

    The paper serves as a reference for beginners and practitioners seeking to apply deep learning to statistical pattern recognition and emerging application areas.

Abstract

from arXiv · show

Deep learning has solved a problem that as little as five years ago was thought by many to be intractable - the automatic recognition of patterns in data; and it can do so with accuracy that often surpasses human beings. It has solved problems beyond the realm of traditional, hand-crafted machine learning algorithms and captured the imagination of practitioners trying to make sense out of the flood of data that now inundates our society. As public awareness of the efficacy of DL increases so does the desire to make use of it. But even for highly trained professionals it can be daunting to approach the rapidly increasing body of knowledge produced by experts in the field. Where does one start? How does one determine if a particular model is applicable to their problem? How does one train and deploy such a network? A primer on the subject can be a good place to start. With that in mind, we present an overview of some of the key multilayer ANNs that comprise DL. We also discuss some new automatic architecture optimization protocols that use multi-agent approaches. Further, since guaranteeing system uptime is becoming critical to many computer applications, we include a section on using neural networks for fault detection and subsequent mitigation. This is followed by an exploratory survey of several application areas where DL has emerged as a game-changing technology: anomalous behavior detection in financial applications or in financial time-series forecasting, predictive and prescriptive analytics, medical image processing and analysis and power systems research. The thrust of this review is to outline emerging areas of application-oriented research within the DL community as well as to provide a reference to researchers seeking to use it in their work for what it does best: statistical pattern recognition with unparalleled learning capacity with the ability to scale with information.

I. INTROUCTION

Artificial neural networks use layered, weighted neurons to learn mappings from inputs to outputs, with nonlinear activations enabling complex pattern recognition. Deep architectures extend this capability through multiple hidden layers.

  • ANNs use interconnected neurons to classify inputs or approximate functions mapping inputs to outputs.Input, hidden, and output layers receive, process, and relay information.
  • Each neuron computes weighted input sums, and a layer applies an m×n weight matrix to an n-dimensional input vector.The resulting output is an m-dimensional linear transformation.
  • Multiple layers of linear maps alone collapse into one linear map, so nonlinear activations are necessary for genuinely deeper nonlinear modeling.The product of the layer matrices is equivalent to a single matrix transformation.
  • Biases and nonlinear activation functions allow networks to classify nonlinearly and approximate nonlinear functions.The activation-function form is a design parameter learned alongside the network weights.
  • The universal approximation theorem states that sufficient neurons, layers, and an appropriate activation function can approximate arbitrary vector mappings with arbitrary accuracy.
  • Training optimizes weights over input-output pairs to minimize an error function and approximate the known mapping.The architecture, layer widths, activations, and error function define the optimization problem.

B. How do these networks learn?

The review explains how neural networks learn through weight updates and surveys the factors, architectures, research directions, and applications that have driven deep learning’s growth. It also identifies data scarcity and the review’s non-exhaustive scope as important boundaries.

  • How do these networks learn?: Learning adjusts neural-network weights through repeated data stimulation until approximation error falls below a practitioner-defined threshold.The process aggregates causal chains of neural computations across hidden layers to represent patterns.
  • Why are deep neural networks garnering so much attention now?: Deep learning’s recent rise reflects larger labeled datasets, stronger parallel computing, accessible software frameworks, regularization, and robust optimization algorithms.Examples include dropout, batch normalization, Adam, stochastic gradient descent, particle swarm optimization, and differential evolution.
  • Conclusions and Future Work: Deep learning remains data-hungry, while this article is a non-exhaustive collection of notable work across an evolving research landscape.The review discusses approaches such as capsules and unsupervised or meta-learning directions for smaller-data settings.
  • Review Methodology: The review introduces foundations and surveys architectures from feed-forward networks and RBMs through DBNs, autoencoders, CNNs, RNNs, and GANs.This architecture overview is intended as a baseline or refresher for further reading.
  • Review Methodology: It surveys multi-agent architecture generation and learning-rule optimization, alongside testing, troubleshooting, and robustness analysis for deep networks.These areas are presented as important for fault-tolerant, mission-critical applications.
  • Review Methodology: Application coverage includes financial anomaly detection, financial time-series forecasting, prognostics, medical imaging, and power systems.

II. DEEP ARCHITECTURES: WORKING MECHANISMS

Deep architectures use layered nonlinear transformations to model complex relationships, with architecture choice depending on the application and data. Training commonly uses backpropagation and gradient descent, while regularization and initialization techniques address major optimization difficulties.

  • Architecture Selection: Architecture selection depends on the application and data, with CNNs preferred for vision and recurrent networks for sequences and time series.
  • Deep Feed-forward Networks: Deep feed-forward networks stack multiple hidden layers to model complex nonlinear relationships more efficiently than shallow architectures.The review states that comparable performance can require fewer computational units than a similarly performing shallow network.
  • Training: Backpropagation with gradient descent trains deep feed-forward networks through repeated forward and backward passes that tune weights to minimize error.The forward pass propagates inputs through nonlinear hidden layers, while the backward pass uses error derivatives to update parameters.
  • Training Challenges: Naive deep-network training can encounter overfitting, local minima, and vanishing gradients, motivating pretraining, regularization, dropout, and batch normalization.

B. Restricted Boltzmann Machines

Restricted Boltzmann Machines are stochastic, undirected two-layer models whose restricted connectivity supports tractable conditional representations. They learn training-data distributions by updating parameters through contrastive-divergence-based energy minimization.

  • Role and Variants: RBMs can learn input probability distributions in supervised and unsupervised settings and have been applied across multiple task-specific variants.Examples include temporal, conditional, gated, convolutional, mean-covariance, and recurrent variants.
  • Architecture: An RBM is an undirected bipartite graph with visible and hidden layers and no intra-layer connections.The review describes the restriction as the defining difference from a general Boltzmann machine.
  • Conditional Structure: Restricted connectivity makes hidden variables conditionally independent given visible variables, and visible variables conditionally independent given hidden variables.This simplifies modeling by allowing variable distributions to be represented with conditional probabilities.
  • Training: Contrastive divergence trains RBMs by maximizing the expected probability of training samples and minimizing model energy through parameter updates.
  • Training: The RBM update uses the learning rate and the difference between data and model expectations.The cited passage defines the expectation notation used in the update rule.

C. Deep Belief Networks

Deep Belief Networks combine layered latent-variable modeling with RBM-based layerwise training, while autoencoders learn compact or robust representations by encoding and reconstructing inputs. These architectures support feature learning and applications including retrieval, speech, and computer vision.

  • Deep Belief Networks: A DBN is a generative graphical model with multiple latent-variable layers, an undirected top connection, and directed lower connections toward the input.
  • Deep Belief Networks: DBNs are trained by stacking RBMs so successive layers learn the input distribution and higher-order correlations among preceding hidden representations.This layerwise procedure was proposed as a fast way to train DBNs despite intractable inference.
  • Deep Belief Networks: Adding layers can improve training-data log-probability and increase the network’s representational power.
  • Autoencoders: An autoencoder reconstructs its input after encoding it into a typically smaller hidden representation, using deterministic units rather than RBM-style stochastic units.
  • Autoencoders: Deep autoencoders can be trained layer by layer as simple autoencoders to address the difficulty of training multiple hidden layers.The review cites document encoding, image retrieval, and efficient speech features as applications.
  • Autoencoder Variants: Denoising and sparse autoencoders improve representation robustness through noise or dropout and can support invariant CNN features for vision.The cited discussion links denoising and sparse pretraining with translation-, scaling-, and out-of-plane-rotation-invariant features.

E. Convolutional Neural Networks

CNNs are designed for image processing by learning spatially specific filters, while recurrent architectures model temporal dependencies and LSTMs mitigate gradient problems across long sequences.

  • Convolutional Neural Networks: AlexNet’s 2012 ILSVRC victory marked CNNs’ emergence as a powerful deep-learning architecture for computer vision.CNNs support image detection, segmentation, and classification, and later architectures sometimes surpassed human recognition performance.
  • Convolutional Neural Networks: CNNs process images through convolution and pooling layers that learn filters for specific image regions.Convolution preserves pixel spatial arrangement, while pooling summarizes pixel information.
  • Recurrent Neural Networks: RNNs span adjacent time steps by combining current inputs with hidden values that capture information from previous steps.This enables modeling of long-term dependencies that can make HMMs computationally unfeasible.
  • Recurrent Neural Networks: LSTM networks address vanishing and exploding gradients during backpropagation across multiple timesteps.Their gates control how much information is retained from previous time steps.
  • Recurrent Neural Networks: Peephole LSTMs update their three gates using cell-state information, while GRUs merge hidden and cell states through a single update gate.Distributed asynchronous SGD training has also been applied to a two-layer deep LSTM for acoustic modeling.

G. Generative Adversarial Networks

GANs jointly train generative and discriminative models in a minmax game, with applications spanning image synthesis, feature learning, video prediction, and text-conditioned generation.

  • Generative Adversarial Networks: GANs train a generative model to capture the data distribution alongside a discriminative model that estimates whether samples come from training data.The framework addresses difficulties in approximating unmanageable probabilistic measures through maximum likelihood estimation.
  • Generative Adversarial Networks: The GAN objective is a minmax game in which generated adversarial examples become indistinguishable from original data as both models improve.At the proposed solution, the generator recovers the training-data distribution and the discriminator assigns 50% probability to each sample.
  • Applications: GAN training procedures produced MNIST samples visually indistinguishable from original data and learned recognizable ImageNet features semi-supervised.The work also examined evaluation metrics for generative models and stable semi-supervised training.
  • Applications: 3DGAN generated high-quality three-dimensional objects from lower-dimensional probabilistic representations without reference images.Volumetric convolutional networks mapped probabilistic space into three-dimensional object space for sampling or exploration.
  • Applications: GAN-based models supported video recognition, action recognition, video generation, and prediction by separating foreground from background with spatio-temporal convolutions.The approach predicted future versions of static images while extracting meaningful video features with minimal supervision.
  • Applications: A DC-GAN generated images from detailed visual descriptions using encoded text features and a manifold interpolation regularizer.Generalizability was tested across varied objects and changing backgrounds.

III. SWARM INTELLIGENCE IN DEEP LEARNING

Swarm and evolutionary methods automate deep-network design and support scalable multi-agent reinforcement learning, while neural-network testing uses coverage criteria and adversarial inputs.

  • Automatic Architecture Optimization: Heuristic and meta-heuristic algorithms tune neural-network architectures, parameters, and hyperparameters to improve deep-learning performance.DENSER uses a two-level representation separating layer structure from layer-specific optimization.
  • Automatic Architecture Optimization: Variable-length Particle Swarm Optimization evolves CNN architectures using particle encodings, disabled layers, and partial datasets for faster evaluation.The disabled layer hides selected particle dimensions, allowing variable-length representations.
  • Deep Reinforcement Learning: End-to-end mean feature embeddings provide state information that helps swarm reinforcement learning scale with increasing agent numbers toward better policies and fast convergence.The embedding addresses changing dimensionality in each agent’s perceived information.
  • Testing Neural Networks: DNN test-generation methods adapt MC/DC-inspired coverage criteria and use linear programming to synthesize perturbed inputs with specified activation patterns.Inputs satisfying closeness definitions form adversarial examples when only one input is correctly labeled.
  • Testing Neural Networks: Across 10 DNNs, Sign-Sign, Distance-Sign, Sign-value, and Distance-Value methods achieved high coverage and identified a significant portion of adversarial examples.Adversarial examples were often found around middle layers, while deeper neuron-pair coverage became harder and could require larger datasets.
  • Verification: Reluplex and SMT-based verification methods test whether small perturbations can cause misclassification by exploring local adversarial manipulations.These approaches target adversarial robustness and neural-network safety.

A. Different Methods of Adversarial Test Generation

Adversarial test generation examines perturbations, attack settings, and defenses for neural networks, with safety-critical deployment requiring broader and more adaptable countermeasures.

  • Adversarial Test Generation: Adversarial robustness must be studied before neural-network architectures are applied in safety-critical systems.Scenario-based perturbations can be used to search for unexpected behavior in integrated controller and perception systems.
  • Adversarial Test Generation: Adversarial falsification includes false-positive and false-negative attacks, white-box and black-box access assumptions, and targeted or non-targeted objectives.Targeted attacks specify the desired output class, whereas non-targeted attacks allow an arbitrary class.
  • Adversarial Test Generation: FGSM constructs adversarial data by adding a perturbation η with magnitude ε to an input, and related methods extend the attack iteratively.BIM and ILLC extend FGSM, while DeepFool uses iterative linear approximation for multidimensional nonlinear cases.
  • Countermeasures: Countermeasures include reactive detection, input reconstruction, and verification, alongside proactive distillation, adversarial retraining, and classifier robustification.Network distillation uses high-temperature softmax activation, while adversarial retraining incorporates adversarial examples during training.
  • Countermeasures: Current defense strategies are mostly applicable to computer-vision tasks, leaving a need for real-time adversarial-input detection in safety-critical systems.The review identifies this as a practical boundary for the discussed countermeasures.
  • Countermeasures: No countermeasure is universally applicable to all adversaries, so reliable infrastructure requires robust defenses across different adversarial scenarios.DeepFense is described as an online unsupervised defense that reduces risk from integrated attacks.

V. APPLICATIONS

Deep learning fraud detection spans unsupervised, supervised, and one-class settings, with applications addressing novel fraud, imbalanced data, and domain-specific signals. The review identifies progress but substantial room for broader architectural contributions.

  • Fraud Detection Settings: Fraud detection can be formulated as unsupervised, supervised, or one-class classification.Unsupervised methods identify clusters or outliers, supervised methods use known labels, and one-class methods train primarily on genuine samples.
  • Deep Learning Applications: Deep autoencoders reconstruct normal transactions or entries to identify anomalous and novel fraud patterns.Reported applications include transaction monitoring and large-scale accounting fraud detection.
  • Deep Learning Applications: Deep networks combine textual and numeric claim features for automobile insurance fraud detection.Latent Dirichlet Allocation extracts features from accident descriptions before neural-network training.
  • Deep Learning Applications: A GAN-based telecom model outperformed traditional classifiers and helped two commercial banks reduce losses by about 10 million RMB in twelve weeks.The model estimated fraud probabilities for large transfers and used a threshold to support prevention.
  • Deep Learning Applications: GAN-generated minority examples address data imbalance in payment-card fraud detection, improving classifier effectiveness on augmented training data.The review notes drawbacks of conventional oversampling and reports better performance after augmentation.
  • Research Opportunities: The review concludes that fraud-detection progress remains limited to a few deep architectures, with potential for ResNet, gated recurrent units, and capsule networks.The stated opportunity includes detecting financial and cyber fraud.

B. Financial Time Series Forecasting

Deep learning financial forecasting uses market, technical, fundamental, and textual data across feedforward, convolutional, recurrent, and hybrid architectures. Reported studies generally favor tuned or hybrid models, while comparisons remain setting-dependent.

  • Motivation: Financial time series are difficult to predict, motivating deep-learning approaches for forecasting and automated trading.The review places these methods within stock and foreign-exchange markets.
  • Inputs: Deep-learning forecasting incorporates fundamental data such as news and reports alongside technical historical-market data.Fundamental analysis mines textual and economic information, whereas technical analysis uses historical price-related data.
  • Feedforward Models: Multilayer feedforward networks are widely used in financial markets, and Bayesian learning outperformed backpropagation for MFF on FOREX forecasting.The comparison is specific to the cited experimental analysis.
  • Recurrent Models: LSTM outperformed random forest, logistic regression, and DNN in one stock-prediction comparison, but random forest performed better during the 2008 financial crisis.The result demonstrates that model rankings vary across market conditions.
  • Hybrid Models: Hybrid architectures combining wavelets, autoencoders, LSTM, or GRU outperformed the corresponding standalone models in reported stock-prediction studies.The reviewed hybrids feed one model’s output into the next or combine recurrent architectures.
  • Prognostics and Health Management: Data-driven prognostics uses collected normal and degraded operating data, preprocessing, feature selection, and deep models for diagnosis and future-state prediction.The workflow is framed as prognostics and health management for systems under test.

D. Medical Image Processing

Deep learning is applied throughout medical image processing, including classification, detection, registration, generation, enhancement, and segmentation. The review reports broad success while emphasizing the need to incorporate prior knowledge in sensitive tasks.

  • Scope: Medical-image deep learning spans classification, detection, enhancement, generation, registration, and segmentation.The review describes a sharp rise in studies across these canonical tasks.
  • Classification: Image classification identifies disease presence, with architectures including stacked autoencoders, RBMs, and specialized CNNs.Examples include Alzheimer’s disease, lung CT, skin cancer, and malaria analysis.
  • Detection: Object or lesion detection combines localized-information identification with classification and faces class-imbalance challenges.A self-transfer-learning framework was tested for nodules in chest radiographs and lesions in mammography.
  • Segmentation: Segmentation supports fine-grained analysis of organs and substructures, especially in cardiac and brain images.The review illustrates brain-MRI segmentation with multiple segmented parts alongside the original slice.
  • Registration: Registration methods use coordinate transformation between image spaces and include autoencoder-based similarity measures and CNN regressors for transformation parameters.Reported similarity measures include normalized mutual information and local cross-correlation.
  • Caveat: Medical-image applications generally produce satisfactory results, but prior knowledge should constrain detection, recognition, and reconstruction to avoid implausible outputs.The caution is specific to the sensitivity of medical image processing.

E. Power Systems

Deep learning supports power-system forecasting and modeling, particularly where large, noisy, nonlinear, or nonstationary data challenge shallow methods. Applications reviewed include load, wind, and photovoltaic power forecasting.

  • Overview: Deep learning handles large power-sector datasets and can outperform traditional machine-learning methods in power-system problems.The review notes the rapid adoption of deep learning by power-system researchers.
  • Load Forecasting: Load forecasting supports spinning-reserve scheduling, utility interchanges, and system-security assessment.Even small reductions in forecasting error can substantially reduce operating costs.
  • Load Forecasting: A residual neural network forecasts hourly single-day load from historical load and temperature using fully connected layers and residual blocks.The model processes raw inputs into preliminary forecasts before residual refinement.
  • Renewable Energy: Renewable-energy forecasting benefits from deep architectures because renewable data are noisy and may contain complex nonlinear and nonstationary features.The review states that shallow models may be insufficient for these characteristics.
  • Photovoltaic Forecasting: Photovoltaic output is intermittent, volatile, and random because of chaotic weather, complicating real-time control and plant operation.Accurate PV forecasting is presented as important for planning and modeling PV plants.
  • Photovoltaic Forecasting: A PV forecasting architecture combines wavelet transform, deep CNN, and quantile regression for deterministic and probabilistic forecasts.Evaluation on historical data from two Belgian PV farms showed high forecasting stability and robustness.
  • Wind Forecasting: Wind-forecasting methods using deep models, clustering, ensembles, and transfer learning outperform cited benchmark approaches on reliability, sharpness, or overall skill.The reviewed evidence includes comparisons with BPANN, Morlet wavelet networks, and single-algorithm models.

VII. CONCLUSIONS AND FUTURE WORK

The review identifies data scarcity, unsupervised learning, cognitive neuroscience, and reinforcement learning as important future directions while surveying architectures, applications, and data resources.

  • VII. CONCLUSIONS AND FUTURE WORK: Data scarcity remains a central challenge because deep learning traditionally depends on large datasets, motivating augmentation, transfer learning, synthetic data, capsules, and one-shot learning.The review notes that these approaches aim to support learning from smaller available datasets.
  • VII. CONCLUSIONS AND FUTURE WORK: Combining deep learning with unsupervised learning and meta learning is presented as a future direction for reducing reliance on abundant labeled data.The review describes systems that set their own goals and automated model design as related research thrusts.
  • VII. CONCLUSIONS AND FUTURE WORK: Cognitive neuroscience and developmental psychology are informing applications in which artificial agents learn spatial navigation autonomously.
  • VII. CONCLUSIONS AND FUTURE WORK: Reinforcement-learning-based meta-modeling is being used to generate problem-specific neural-network architectures, including CNNs for image classification.The review cites MetaQNN as an example using Q-learning with ϵ-greedy exploration.
  • VII. CONCLUSIONS AND FUTURE WORK: The paper concludes by organizing architectures, applications, data repositories, swarm intelligence, fault tolerance, and open research challenges into a practitioner-oriented review.It explicitly aims to help beginners and practitioners make informed choices and provides category-wise public data repositories.
Loading 1905.13294v3…