Source-linked AI summary
Random Vector Functional Link Neural Network based Ensemble Deep Learning
Rakesh Katuwal, P. N. Suganthan, M. Tanveer
TL;DR
Ensembling deep neural networks can be computationally costly. This paper introduces stacked and ensemble deep RVFL networks with fixed random hidden parameters, reporting better generalization than pertinent randomized neural networks.
Problem
Ensembling deep neural networks can incur substantial computational cost, motivating more efficient randomized-network approaches.
Method
The paper proposes dRVFL and edRVFL, using stacked RVFL layers with randomly generated fixed hidden parameters and an ensemble derived from one trained network.
Results
The proposed deep learning RVFL networks achieve better generalization and superior performance than pertinent randomized neural networks.
Takeaways & Limitations
Deep RVFL networks provide a randomized neural-network framework with reported performance advantages over relevant neural-network alternatives.
Takeaways & Limitations
The paper identifies computational cost as a trade-off for ensembling deep neural networks.
Abstract
from arXiv · showhide
In this paper, we propose a deep learning framework based on randomized neural network. In particular, inspired by the principles of Random Vector Functional Link (RVFL) network, we present a deep RVFL network (dRVFL) with stacked layers. The parameters of the hidden layers of the dRVFL are randomly generated within a suitable range and kept fixed while the output weights are computed using the closed form solution as in a standard RVFL network. We also propose an ensemble deep network (edRVFL) that can be regarded as a marriage of ensemble learning with deep learning. Unlike traditional ensembling approaches that require training several models independently from scratch, edRVFL is obtained by training a single dRVFL network once. Both dRVFL and edRVFL frameworks are generic and can be used with any RVFL variant. To illustrate this, we integrate the deep learning networks with a recently proposed sparse-pretrained RVFL (SP-RVFL). Extensive experiments on benchmark datasets from diverse domains show the superior performance of our proposed deep RVFL networks.
1. Introduction
The paper addresses the limitations and computational cost of back-propagation-trained deep networks by developing deep and ensemble deep architectures based on randomized RVFL networks. The proposed dRVFL and edRVFL models retain RVFL’s efficiency and applicability across variants while achieving superior performance on real-world classification datasets.
- Motivation and background: The proposed frameworks target limitations of conventional deep learning, including time-consuming optimization, possible non-global convergence, large data requirements, and weak performance on limited-data domains.Traditional ensembling can improve robustness and accuracy but incurs substantial computational cost for deep networks.
- Motivation and background: Randomized neural networks with closed-form solutions avoid pitfalls of back-propagation networks by training faster, while RVFL further uses direct input-output links and fixed random hidden parameters.RVFL’s direct links reuse original features, regularize the randomization, and help keep the model thinner and simpler.
- Proposed models: dRVFL extends RVFL into stacked hidden layers with randomly generated, fixed hidden parameters and analytically computed output weights.This preserves the randomized-network training paradigm while enabling deep representational learning.
- Proposed models: edRVFL implicitly ensembles the hidden-layer models of one dRVFL network instead of independently training L neural networks from scratch.Higher-layer models combine direct original features with nonlinear features from preceding layers, making the framework simultaneously ensemble- and deep-learning based.
- Scope and evaluation: dRVFL and edRVFL are generic across RVFL variants, and experiments on real-world classification datasets show superior performance compared with other relevant neural networks.The authors instantiate the frameworks with standard RVFL and sparse-pretrained RVFL (SP-RVFL).
2. Related Works
This section reviews RVFL and its ELM variant, then describes SP-RVFL’s sparse pretraining and HELM’s randomized multilayer architecture. These methods motivate the paper’s deep and pretrained RVFL developments.
- ELM: ELM is an RVFL variant without direct links or a bias term, unlike the standard RVFL architecture.Figure 1 contrasts the networks’ direct links, bias term, and randomly generated fixed hidden-layer weights.
- SP-RVFL: SP-RVFL addresses random hidden-parameter limitations by using an l1-regularized autoencoder to learn hidden-layer weights before RVFL training.The sparse autoencoder is optimized with FISTA, and its pretrained weights replace the standard RVFL’s randomly generated hidden weights.
- HELM: HELM is a randomized multilayer network based on ELM that stacks hidden feature-extraction layers and uses an ELM classifier for final decisions.Its feature encoder uses a sparse autoencoder, with multiple hidden layers stacked for feature extraction.
3. Deep RVFL for representational learning
This section introduces dRVFL, a stacked RVFL architecture with fixed random hidden parameters, direct links to original features, and closed-form output weights, then extends it to edRVFL. edRVFL reuses one dRVFL training run to form multiple intermediate-feature classifiers whose outputs are combined, reducing the cost of independently trained ensembles.
- 3.1 Deep RVFL network: dRVFL stacks hidden layers whose randomly generated weights and biases remain fixed, while only the output weights are computed using RVFL closed-form solutions.Each layer transforms the preceding layer’s representation, and the output combines nonlinear features from all hidden layers with the original input features.
- 3.2 Ensemble deep RVFL network: edRVFL uses intermediate representations for final decisions and decomposes output-weight computation into several independently solved models combined by voting or averaging.Each model receives preceding nonlinear features together with the original input through direct links.
- 3.2 Ensemble deep RVFL network: Training one dRVFL once produces the edRVFL ensemble at slightly higher cost than one dRVFL but lower cost than training several independent dRVFL models.The framework is generic, allowing any RVFL variant to be used within edRVFL.
4. Experiments
Across 13 real-world classification datasets, edRVFL achieves the strongest standard-RVFL results, while dSP-RVFL and edSP-RVFL outperform SP-RVFL and the proposed deep frameworks obtain the best overall ranks. Experiments use fixed randomized deep networks, 10-fold cross-validation, and comparisons spanning shallow, deep, and ensemble RVFL variants.
- Datasets: The experiments cover 13 publicly available real-world classification datasets spanning biomedical, face-image, handwritten-digit, object-recognition, and text domains.The datasets include Carcinom, Lung, ORL, Yale, BA, Gisette, MNIST, USPS, COIL20, COIL100, BASEHOCK, RCV1, and TDT2.
- Standard RVFL-based methods: 12 of 13 datasets achieve their best accuracy with edRVFL, which outperforms dRVFL on all remaining datasets and matches it on three.Table 2 reports accuracy (%) for standard RVFL-based methods; edRVFL is also the top-ranked classifier.
- SP-RVFL-based methods: dSP-RVFL and edSP-RVFL outperform SP-RVFL, while edSP-RVFL performs at least as well as dSP-RVFL and both are significantly better than SP-RVFL.Table 3 provides the SP-RVFL-based comparison, and the Friedman analysis rejects the null hypothesis at α = 0.05 with critical value F(2,24) = 3.4.
5. Conclusion
The paper introduces generic deep and ensemble RVFL frameworks that use fixed hidden parameters and analytically computed output weights, with edRVFL training once instead of independently training multiple models. Experiments show improved generalization over pertinent randomized neural networks, while future work targets additional learning tasks.
- 5. Conclusion: dRVFL uses randomly generated, fixed hidden-layer parameters and computes output weights analytically while extracting representations and weighting features from hidden layers and direct links.This combines deep feature extraction with RVFL’s closed-form output-weight solution.
- 5. Conclusion: edRVFL combines ensemble and deep learning by training one deep network once, with slightly higher cost than dRVFL but significantly lower cost than traditional ensembles.Unlike traditional ensembles, edRVFL avoids independently training several models.
- 5. Conclusion: Both frameworks are generic across RVFL variants, demonstrated through dSP-RVFL and edSP-RVFL based on sparse-autoencoder-learned rather than randomly generated hidden parameters.The SP-RVFL extension replaces random hidden-layer parameter generation with sparse-autoencoder learning.
- 5. Conclusion: Extensive classification experiments show that the proposed deep RVFL networks achieve better generalization than pertinent randomized neural networks.The conclusion reports this result across several classification datasets.
- 5. Conclusion: Future work will examine regression, time-series forecasting, semi-supervised learning, and incremental learning applications.These tasks extend evaluation beyond the classification datasets considered here.