PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU

Farhad Mortezapour Shiri, Thinagaran Perumal, Norwati Mustapha, Raihani Mohamed

arXiv:2305.17473v4cs.LGcs.AI

TL;DR

Because deep learning model selection is complicated by diverse real-world problems and limited labeled data in some domains, this paper surveys major architectures and compares them across datasets and tasks. Its experiments and synthesis indicate that model suitability depends on data type and task: CNN-based models perform strongly for images, recurrent models for time series, and Transformers particularly for text.

  • Problem

    Designing effective deep learning models is challenging because real-world problems vary and some specialized domains lack sufficient, affordable labeled data.

  • Method

    The paper surveys deep learning models and evaluates them comparatively on the IMDB, ARAS, and Fruit-360 public datasets.

  • Results

    CNN-based models such as DenseNet and MobileNet perform exceptionally well for image classification, while Transformers outperform classical RNN-based models particularly in text analysis.

  • Takeaways & Limitations

    Selecting a deep learning model according to the data type and task can support informed practical decision-making.

  • Takeaways & Limitations

    Simple RNNs have short-term memory limitations that restrict retaining information over long sequences.

Abstract

from arXiv · show

Deep learning (DL) has emerged as a powerful subset of machine learning (ML) and artificial intelligence (AI), outperforming traditional ML methods, especially in handling unstructured and large datasets. Its impact spans across various domains, including speech recognition, healthcare, autonomous vehicles, cybersecurity, predictive analytics, and more. However, the complexity and dynamic nature of real-world problems present challenges in designing effective deep learning models. Consequently, several deep learning models have been developed to address different problems and applications. In this article, we conduct a comprehensive survey of various deep learning models, including Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Temporal Convolutional Networks (TCN), Transformer, Kolmogorov-Arnold networks (KAN), Generative Models, Deep Reinforcement Learning (DRL), and Deep Transfer Learning. We examine the structure, applications, benefits, and limitations of each model. Furthermore, we perform an analysis using three publicly available datasets: IMDB, ARAS, and Fruit-360. We compared the performance of six renowned deep learning models: CNN, RNN, Long Short-Term Memory (LSTM), Bidirectional LSTM, Gated Recurrent Unit (GRU), and Bidirectional GRU alongside two newer models, TCN and Transformer, using the IMDB and ARAS datasets. Additionally, we evaluated the performance of eight CNN-based models, including VGG (Visual Geometry Group), Inception, ResNet (Residual Network), InceptionResNet, Xception (Extreme Inception), MobileNet, DenseNet (Dense Convolutional Network), and NASNet (Neural Architecture Search Network), for image classification tasks using the Fruit-360 dataset.

1 Introduction

Deep learning is presented as a machine-learning subset whose progress has enabled strong performance across tasks, motivating a broad review and comparative evaluation of model families.

  • Large datasets, hardware acceleration, and improved optimization and training methods have contributed to deep learning’s performance and faster training.
  • Deep learning is a subset of machine learning, while machine learning and deep learning are subfields of artificial intelligence.
  • The paper surveys supervised, generative, Transformer, deep reinforcement learning, transfer learning, hybrid, and application-oriented deep learning topics.
  • The study compares deep learning models across different applications using three publicly available datasets.

2 Deep Learning

Deep learning learns hierarchical data representations through multiple layers and can combine feature learning with classification, unlike the sequential feature-engineering pipeline typical of traditional machine learning.

  • Deep learning passes data through multiple hidden layers that progressively extract and combine features into hierarchical representations.
  • Traditional machine learning typically separates preprocessing, feature extraction, feature selection, learning, and classification.
  • Deep learning enables simultaneous feature learning and classification, reducing the need for separate feature-selection steps.
  • Deep learning models include discriminative supervised architectures such as CNNs and RNNs, alongside generative approaches such as GANs and autoencoders.

3 Supervised Deep Learning Models

Supervised deep learning models perform discriminative classification by learning class patterns from observable data. This section introduces MLPs, CNNs, and RNNs, emphasizing their architectures, feature-processing mechanisms, activation functions, and sequential modeling capabilities.

  • Supervised deep architectures model posterior class distributions from observable data for pattern classification, including MLP, CNN, RNN, TCN, and KAN variants.
  • 3.1 Multi Layers Perceptron (MLP): MLPs are supervised feedforward networks with input, output, and one or more fully connected hidden layers.The input layer receives data, hidden layers process signals, and the output layer produces predictions.
  • 3.2 Convolutional Neural Networks (CNN): CNNs combine convolution-based feature extraction with a classifier, enabling automatic feature learning and end-to-end training across tasks such as image classification and object detection.Convolutional layers learn increasingly abstract features, while the fully connected layer acts as the classifier.
  • 3.2 Convolutional Neural Networks (CNN): Pooling layers down-sample feature maps to reduce connections, computational burden, and overfitting while improving robustness to shape or viewpoint distortions.
  • 3.2 Convolutional Neural Networks (CNN): Activation functions introduce non-linearity, allowing CNNs to model complex patterns; ReLU is widely used because it helps address vanishing gradients.Sigmoid and Tanh can saturate, whereas ReLU suppresses negative inputs and preserves positive ones.
  • 3.3 Recurrent Neural Networks (RNN): RNNs maintain internal memory and reuse prior computations, making them suitable for sequential dependencies in language, video, and speech tasks.Their recurrent operation depends on both the current input and previous computations.

3.4 Temporal Convolutional Networks (TCN)

TCNs are convolutional architectures for sequential data that use causal and dilated convolutions to capture long-range dependencies efficiently, with residual connections supporting deeper networks.

  • TCN overview: TCNs extend one-dimensional CNNs for sequential data, including time series and action segmentation.They were introduced for video action segmentation and later generalized to other sequential data.
  • TCN architecture: TCNs use zero-padding to preserve sequence length and causal convolutions that use only current and earlier inputs.Causal convolutions prevent future information from contributing to the output at time t.
  • Long-range dependencies: Simple causal convolutions require large filters or deep networks for long histories, potentially causing vanishing gradients and degraded performance.This limitation motivates architectural mechanisms that expand the receptive field without proportional depth or computational growth.
  • Long-range dependencies: Dilated convolutions insert gaps between kernel weights, exponentially expanding the receptive field while maintaining computational efficiency.When d = 1, dilation is equivalent to regular convolution; larger d values incorporate broader input ranges.
  • Residual design: Residual connections let TCNs learn residual functions and facilitate data flow across stacked dilated causal convolutional layers.Typical residual blocks use two dilated causal convolution layers, nonlinear activation, weight normalization, and dropout.
  • KAN: KANs replace fixed node activations and linear weight matrices with learnable spline-parameterized functions on edges.KAN nodes sum incoming data, while variants target convolutional, temporal, multivariate, fractional, and wavelet applications.

4 Generative (Unsupervised) Deep Learning Models

Generative deep learning models learn representations or data distributions without direct labels, supporting reconstruction, synthesis, feature extraction, and related unsupervised tasks.

  • Overview: Generative models learn from unknown data-generating distributions to produce novel data and expand unsupervised learning possibilities.The section discusses autoencoders, GANs, RBMs, and DBNs as major generative model families.
  • Autoencoders: Autoencoders encode input x into a latent code z and decode z into a reconstruction r of the original input.Their encoder and decoder are represented as z = f(x) and r = g(z).
  • Autoencoders: Autoencoders support generative modeling, dimensionality reduction, feature extraction, anomaly detection, and denoising.Regularized autoencoders target representation learning, whereas variational autoencoders can function as generative models.
  • Variational autoencoders: VAEs use probabilistic latent-variable models with coupled recognition and generative components to learn representations and generate data.The recognition model estimates latent variables for the generative model, while the generative model supports recognition learning.
  • GANs: GANs combine a generator that produces data resembling real samples with a discriminator that distinguishes synthetic from real data.Training repeatedly improves the discriminator’s detection ability and the generator’s capacity to produce realistic samples.
  • DBNs: DBNs stack Restricted Boltzmann Machines trained layer by layer and then jointly fine-tuned to uncover patterns and features in unlabeled data.Their iterative forward and backward processes form joint probability distributions of activations and inputs.

5 Transformer Architecture

Transformers use self-attention in an encoder-decoder architecture, enabling parallel sequence processing and modeling of contextual and positional relationships.

  • Architecture: Transformers are self-attention encoder-decoder models introduced for machine translation and now widely used, especially in natural language processing.The encoder uses stacked layers containing multi-head self-attention and position-wise fully connected sublayers.
  • Architecture: Unlike sequential RNN training, Transformer training is parallel, allowing features to be learned concurrently and improving computational efficiency.This design avoids the long training times associated with iterative recurrent processing.
  • Attention: Multi-head attention lets each token aggregate information from other sequence elements to capture contextual dependencies.It combines multiple scaled dot-product attention operations applied to query, key, and value vectors.
  • Attention: Scaled dot-product attention extracts Q, K, and V vectors, normalizes Q·K scores by √d_k, applies softmax, and sums weighted values.The normalization is used for gradient stability.
  • Positional encoding: Because Transformers lack recurrence and convolution, positional encodings add sequence-order information to input embeddings.The encodings use sine and cosine functions whose wavelengths vary across dimensions and support relative-position learning.
  • Variants: Transformer variants span language, vision, multimodal analysis, efficient attention, long-context modeling, and specialized architectural designs.Examples include BERT, GPT, Transformer-XL, XLNet, ViT, Swin Transformer, MulT, and Fast Transformer.

6 Deep Reinforcement Learning

Deep reinforcement learning combines reinforcement-learning decision making with deep-learning perception, supporting sequential action selection across diverse application domains.

  • RL foundations: Reinforcement learning maps situations to actions while an agent learns through trial and error rather than explicit action instructions.The objective is to maximize associated reward over sequential decisions.
  • DRL framework: Deep reinforcement learning combines reinforcement-learning decision making with deep-learning perception of sensory inputs.Deep learning processes environmental inputs into current-state information, which reinforcement learning maps to actions and evaluates.
  • DRL models: Deep Q-learning Networks learn policies directly from high-dimensional inputs using convolutional neural networks.Double DQN, Dueling DQN, and Monte Carlo Tree Search are additional deep reinforcement-learning models.
  • Applications: DRL models are applied to video games, robotic manipulation, image segmentation, video analysis, and energy management.These applications involve perception and sequential decision-making across varied domains.

7 Deep Transfer Learning

Deep transfer learning addresses limited labeled data by transferring knowledge from source domains or pre-trained models to target tasks. The paper reviews four technique categories and describes model-based workflows and applications.

  • Limited labeled data makes supervised deep learning difficult because large annotated datasets are costly and time-consuming to construct.
  • Deep transfer learning is categorized into instances-based, mapping-based, network-based, and adversarial-based approaches.
  • Instances-based methods reweight selected source-domain instances, while mapping-based methods align source and target instances in a shared data space.
  • Network-based transfer reuses pre-trained architectures and parameters, commonly through pretraining, freezing, fine-tuning, and adding new layers.
  • Progressive Neural Networks freeze a pre-trained model and add new target-specific layers to leverage existing knowledge during new-task learning.
  • Deep transfer learning helps overcome limited training data, transfer knowledge across domains, and improve performance in image, speech, video, and signal-processing applications.

8 Hybrid Deep Learning Models

Hybrid deep learning architectures combine elements of different models to address real-world problems. Combining supervised models can extract more relevant and robust features.

  • Hybrid architectures combine different generative or discriminative models to improve performance on real-world problems.
  • Combinations such as CNN+LSTM and CNN+GRU integrate supervised models to extract more relevant and robust features.

9 Application of Deep Learning

Deep learning is applied across healthcare, vision, speech, language, agriculture, cybersecurity, recommendation, autonomous vehicles, manufacturing, and robotics. Different application goals require different tasks and model choices.

  • Deep learning applications span healthcare, computer vision, speech recognition, NLP, e-learning, smart environments, and other domains.
  • Five recurring application categories are classification, detection, localization, segmentation, and regression.
  • Natural Language Processing (NLP): Application-specific examples include transfer learning for genomic analysis, BERT for question answering, and attentional LSTM for text summarization.
  • Deep learning supports agriculture and disaster management through models including ViT+CNN, GRU+CNN, autoencoders, CNN+RNN, attention GRU, and LSTM.
  • Cybersecurity: Cybersecurity applications use CNN, LSTM, Bi-LSTM, CNN+Bi-LSTM, and deep autoencoders for intrusion, malware, phishing, and fraud detection.
  • Autonomous Vehicles: Autonomous-vehicle tasks include object and pedestrian detection, localization and mapping, lane detection, path planning, and predictive maintenance.

10 Deep Learning Challenges

Deep learning faces challenges involving data requirements, imbalance, overfitting, gradient instability, catastrophic forgetting, and underspecification. The paper pairs these challenges with transfer learning, augmentation, resampling, algorithmic methods, stress testing, and other remedies.

  • Data Requirements: Deep learning models often require large datasets, but sufficient data may be unavailable for effective training.
  • Data Requirements: Insufficient data can be addressed through transfer learning, data augmentation, or simulated data when the physical process is understood.
  • Class Imbalance: Class imbalance occurs when class distributions are unequal and is addressed through data-level, algorithm-level, or hybrid techniques.
  • Overfitting: Overfitting arises when models memorize training patterns and noise, especially with small datasets and complex classifiers.
  • Gradient Problems: Vanishing or exploding gradients can stall or destabilize training; ReLU activations and Xavier initialization help maintain gradient flow.
  • Catastrophic Forgetting: Catastrophic forgetting causes models trained on new tasks to lose performance on earlier tasks, with EWC and IMM proposed as mitigation strategies.
  • Underspecification: Underspecification allows multiple validation-performing models to behave unpredictably in deployment, motivating stress tests beyond standard metrics.

11 Analysis of Deep Learning Models

The study compares deep learning models across sentiment, activity-recognition, and image-classification datasets using standardized metrics and training procedures. Results indicate that the strongest model depends on data type, with Transformers prominent for sequence tasks and CNN variants for images.

  • Datasets and evaluation: The experiments use IMDB, ARAS, and Fruit-360 to compare models across sentiment analysis, human activity recognition, and fruit-image classification.IMDB contains labeled movie reviews, ARAS annotated sensor events, and Fruit-360 fruit images.
  • Experimental design: Eight models are evaluated on IMDB and ARAS, while eight CNN-based architectures are evaluated on Fruit-360.The sequence-model comparison includes CNN, RNN, LSTM, bidirectional LSTM, GRU, bidirectional GRU, TCN, and Transformer; the image comparison includes VGG through NASNet.
  • Datasets and evaluation: Accuracy, precision, recall, and F1-measure are used to assess classification performance.Accuracy measures overall correctness, precision the correctness of positive predictions, recall positive-instance identification, and F1 balances precision and recall.
  • IMDB dataset: Transformer and Bi-GRU achieve the best IMDB performance, while Transformer trains significantly faster than Bi-GRU.GRU also shows good accuracy and requires less training time than Bi-GRU.
  • ARAS dataset: Transformer and TCN outperform other models on ARAS, with Transformer strongest overall and stabilizing earlier alongside TCN and CNN.Transformer leads accuracy, recall, and F1-score; Bi-LSTM leads precision, and Transformer’s training time is second only to CNN.
  • Fruit-360 dataset: DenseNet and MobileNet perform best on Fruit-360, while MobileNet offers a favorable balance between accuracy and training efficiency.MobileNet trains significantly faster than DenseNet while maintaining performance close to DenseNet; Xception also shows good accuracy with less training time than DenseNet.

12 Research Directions and Future Aspects

The paper identifies generative, hybrid, and quantum deep learning as directions for further development. It emphasizes models that learn from unlabeled data, combine complementary architectures, or extend deep learning into new computational settings.

  • Generative models: Generative models are proposed as a future research avenue because they can create data representations and identify high-order correlations without labeled examples.The discussion includes Autoencoders, GANs, DBNs, and SOMs.
  • Generative models: GANs are highlighted for learning representations from unlabeled image data and training highly nonlinear mappings between latent and data spaces.The paper presents GANs as a pivotal area for future exploration.
  • Hybrid and ensemble modeling: Hybrid architectures combining CNNs and RNNs can capture spatial and temporal dependencies by leveraging the strengths of both model types.The paper also discusses combining generative and supervised learning for uncertainty handling in high-risk scenarios.
  • Quantum deep learning: Quantum RNNs, Quantum GANs, and other quantum deep learning architectures are identified as a cutting-edge research frontier.The paper also proposes exploring these models across various domains.
  • Overall direction: The proposed directions reflect the evolving nature of deep learning and the need for further advancement toward more powerful and efficient models.The conclusion frames these directions as responses to challenges identified through the review and analysis.

13 Conclusion

The paper concludes that deep learning model suitability varies with data type and task. Its experiments support CNN-based models for images, recurrent models for time series, and Transformers for text analysis, while emphasizing task-specific model selection.

  • Scope of the review: The article surveys neural networks, MLPs, CNNs, RNNs, TCNs, Transformers, generative models, DRL, and transfer learning.It also classifies models according to their applications and characteristics.
  • Model suitability: RNN variants including LSTM, Bi-LSTM, GRU, and Bi-GRU are particularly suited to time-series data because they capture temporal dependencies.CNN-based models are described as effective for image analysis through spatial-feature extraction.
  • Experimental conclusions: Experiments on IMDB, ARAS, and Fruit-360 reinforce that specific deep learning models suit different data types.The datasets cover text sentiment, sensor-based activity recognition, and image classification.
  • Experimental conclusions: DenseNet and MobileNet perform exceptionally well for image classification, while LSTM and GRU show strong performance in time-series analysis.The conclusion contrasts CNN-based image models with recurrent models for sequential data.
  • Experimental conclusions: Transformer outperforms classical RNN-based models particularly in text analysis, which the paper attributes to its attention mechanism.This finding is stated within the scope of the reported experiments and model comparisons.
  • Practical implication: The paper emphasizes selecting deep learning models according to the nature of the data and the task.The reported insights are intended to support informed decisions in practical applications.
Loading 2305.17473v4…