Source-linked AI summary
RMDL: Random Multimodel Deep Learning for Classification
Kamran Kowsari, Mojtaba Heidarysafa, Donald E. Brown, Kiana Jafari Meimandi, Laura E. Barnes
TL;DR
Complex classification tasks lack a general way to choose effective deep-learning architectures across data types. RMDL addresses this by combining randomly configured DNN, CNN, and RNN models in an ensemble, with evaluations reporting consistently higher accuracy than conventional approaches across several datasets.
Problem
Selecting an efficient deep-learning architecture and structure remains a dataset-specific trial-and-error problem for complex classification tasks.
Method
RMDL combines randomly generated DNN, CNN, and RNN models in a parallel ensemble with majority voting.
Results
Across WOS, Reuters, MNIST, CIFAR, IMDB, and 20NewsGroups, RMDL combinations consistently achieved higher accuracy than conventional approaches.
Takeaways & Limitations
RMDL provides a classification approach intended to handle multiple data types while improving ensemble robustness and accuracy.
Abstract
from arXiv · showhide
The continually increasing number of complex datasets each year necessitates ever improving machine learning methods for robust and accurate categorization of these data. This paper introduces Random Multimodel Deep Learning (RMDL): a new ensemble, deep learning approach for classification. Deep learning models have achieved state-of-the-art results across many domains. RMDL solves the problem of finding the best deep learning structure and architecture while simultaneously improving robustness and accuracy through ensembles of deep learning architectures. RDML can accept as input a variety data to include text, video, images, and symbolic. This paper describes RMDL and shows test results for image and text data including MNIST, CIFAR-10, WOS, Reuters, IMDB, and 20newsgroup. These test results show that RDML produces consistently better performance than standard methods over a broad range of data types and classification problems.
1 INTRODUCTION
RMDL addresses the difficulty of choosing effective deep-learning architectures across complex data types by combining randomly generated DNN, CNN, and RNN models. The approach uses varied feature extraction, optimization, dropout, and majority voting to improve classification robustness and accuracy.
- Complex-data classification requires more general methods because most deep architectures target specific data types or domains.
- Choosing an efficient architecture, structure, and number of layers or nodes remains a trial-and-error process for each dataset.
- RMDL ensembles DNN, CNN, and RNN architectures to address architecture-selection challenges across classification problems.
- RMDL uses architecture-specific feature spaces, including TF-IDF for DNN text inputs and 1D, 2D, or 3D CNNs for text, images, or video.
- The method randomly searches hidden-layer and node configurations, applies dropout and multiple optimization techniques, and combines models through majority voting.
- RMDL is designed to process text, images, and videos while improving ensemble robustness and accuracy.
2 RELATED WORK
Prior work develops feature extraction, conventional classifiers, and deep-learning architectures for complex data. RMDL builds on combinations of neural architectures while seeking a more general classification approach.
- Related work is organized around feature extraction, classification methods, and deep learning for classification.
- Feature Extraction: TF-IDF represents words using frequency and inverse document-frequency weighting, while Word2Vec represents contextual word relationships in vector space.
- Classification Methods and Techniques: Naïve Bayes is a fast text-classification baseline but its bag-of-words representation does not reflect sequence order.
- Deep Learning: Neural networks learn weighted combinations through nonlinear functions using iterative back-propagation and optimizers such as SGD.
- Deep Learning: DNNs use multiple hidden layers, whereas CNNs use convolutional layers with local and global pooling.
- Deep Learning: Prior methods combine CNNs and RNNs or organize multiple deep-learning techniques hierarchically to improve classification accuracy.
3 BASELINE
The paper evaluates RMDL against distinct text and image baselines because feature extraction and model structure differ across these modalities. Baselines include conventional classifiers, SVM variants, deep models, and image-specific methods.
- Text and image baselines are separated because their feature extraction methods and model structures differ.
- Text Classification Baselines: Text comparisons include DNN, CNN, RNN, SVM, naïve Bayes, and HDLTex.
- Text Classification Baselines: The text baseline uses deep learning without hierarchical levels, while HDLTex represents a hierarchical document-classification approach.
- SVM Baselines: SVM baselines support binary or multiclass classification through constructions such as One-Vs-One and all-at-once decision functions.
- SVM Baselines: String-kernel baselines map sequences into feature spaces using spectrum or mismatch representations, with mismatch kernels allowing k-mer mismatches.
- Image Classification Baselines: Image comparisons include Deep L2-SVM, Maxout Network, BinaryConnect, PCANet-1, and gcForest.
4 METHOD
RMDL's novelty is a parallel ensemble of randomly configured DNN, CNN, and RNN models for text and image classification, trained with multiple optimizers.
- RMDL trains multiple randomly configured DNN, CNN, and RNN models in parallel for text and image classification.
- The method applies multiple optimization techniques across different random models.
4.1 Feature Extraction and Data Pre-processing
RMDL extracts features differently for text, sequential data, images, and other structured or unstructured inputs, using vectorization and modality-specific representations.
- Text and sequential datasets are treated as unstructured, whereas image datasets have structured feature spaces.
- Image features use height, width, and color dimensions, while grayscale images omit the color dimension.
- Text feature extraction includes word embeddings such as GloVe and Word2vec, TF-IDF, and N-gram representations.
- For CNN and RNN models, text is represented with 200-dimensional GloVe vector-space models.
- In the document vector-space model, l_j denotes document length and w_i,j denotes the GloVe vectorization of word i in document j.
4.2 Random Multimodel Deep Learning
RMDL combines randomly generated DNN, CNN, and RNN models into an ensemble whose final classification uses majority voting.
- RMDL builds ensembles of DNN, CNN, and RNN classifiers with randomly generated layer and node counts.A cited example uses 9 unique models: 3 CNNs, 3 RNNs, and 3 DNNs.
- Each model produces a prediction for a data point, and the ensemble aggregates these predictions across its n random models.
- The ensemble contains d random DNN models, c CNN models, and r RNN models, with r + c + d = n.
- The final RMDL prediction is calculated using majority voting among the trained models.
4.3 Deep Learning in RMDL
RMDL uses three deep-learning architectures in parallel: DNNs, RNNs, and CNNs, with specialized structures for feedforward, sequential, and convolutional processing.
- The RMDL structure contains d random DNNs, r RNNs, and c CNNs operating as three basic architectures in parallel.
- DNNs: RMDL DNNs randomly assign the number of hidden layers and nodes and use back-propagation with sigmoid, ReLU, and Softmax activations.
- RNNs: RNNs emphasize previous sequence data and are used primarily for text and sequential classification, while RMDL also applies them to images.
- RNNs: LSTM addresses vanishing gradients through gated control of information, while GRU is a simplified LSTM variant with two gates and no internal memory.
- CNNs: CNNs support document and image classification, use pooling to reduce output size, and adjust both weights and feature-detector filters during back propagation.
- CNNs: For text, CNN channel dimensionality can be very high because the feature-space size may reach 50K, compared with 3 RGB channels for images.
4.4 Optimization
RMDL uses stochastic optimizers and combines models trained with different optimization techniques so majority voting can reduce the influence of inefficient fits.
- The implementation uses stochastic gradient optimizers including RMSProp and Adam.
- SGD uses momentum on a rescaled gradient, while RMSProp does not perform bias correction and can face sparse-gradient problems.
- Adam uses the first two gradient moments, handles non-stationary objectives, and addresses the sparse-gradient issue described for RMSProp.
- RMDL can use models with different optimizers so majority voting may ignore inefficient models when n > k.
5 EXPERIMENTAL RESULTS
The experiments evaluate RMDL on text and image datasets using accuracy, Micro F1-Score, and error rate, with comparisons against baseline methods. The tested datasets include WOS, Reuters, IMDB, 20NewsGroups, MNIST, and CIFAR-10.
- Evaluation: The reported evaluation measures include accuracy, Micro F1-Score, and error rate.The paper states that accuracy and Micro F1-Score are reported, while image results are presented as error rates.
- Datasets: RMDL is evaluated on text and image data, although the model is theoretically intended for video, text, and images.The experiments use four text datasets and two image datasets.
- Text datasets: The text experiments use WOS, Reuters-21578, IMDB, and 20NewsGroups datasets.WOS includes corpora with 5,736, 11,967, and 46,985 documents; Reuters contains 10,788 documents and 90 classes.
- Image datasets: The image experiments use MNIST handwritten digits and CIFAR-10 images.MNIST uses 28 × 28 × 1 inputs, while CIFAR uses 32×32×3 images assigned to 10 classes.
5.3 Hardware
The experiments run on CPU and GPU hardware using Python, CUDA, TensorFlow, and Keras. Results report lower image error rates and higher text accuracy across multiple RMDL ensemble sizes, while Figure 5 examines individual-model accuracy and loss.
- Hardware: The experiments use an Intel Xeon E5-2640 CPU with 12 cores and 64 GB memory, plus two GTX 1080 Ti GPUs and one Tesla K20c.RMDL can be implemented using CPU, GPU, or both.
- Software: The implementation uses Python, CUDA, TensorFlow, and Keras to create the neural networks.
- Image results: 0.21 is the reported MNIST error rate for RMDL with 15 random models, while CIFAR-10 reaches 8.74 with 15 RDLs.
- Text results: 93.57 is the reported accuracy for WOS-5,736 with 30 RDLs, while IMDB reaches 90.79 with 15 RDLs.
- Training behavior: Figure 5 tracks individual RDL accuracy and loss across epochs for text and image classification.The figure uses 9 RDLs for text classification and 15 RDLs for image classification.
6 DISCUSSION AND CONCLUSION
RMDL combines DNNs, RNNs, and CNNs in a parallel ensemble that consistently outperforms conventional approaches across multiple datasets and data types.
- RMDL combines DNNs, RNNs, and CNNs with parallel learning and majority voting for classification.
- Across WOS, Reuters, MNIST, CIFAR, IMDB, and 20NewsGroups, RMDL achieves consistently higher accuracy than naïve Bayes, SVM, and single deep learning models.
- The approach provides flexibility for classifying datasets across a wide range of data types and applications.