PaperlayerView exact revision on arXiv

Source-linked AI summary

Random vector functional link network: recent developments, applications, and future directions

A. K. Malik, Ruobin Gao, M. A. Ganaie, M. Tanveer, P. N. Suganthan

arXiv:2203.11316v2cs.NEcs.LGcs.RO

TL;DR

Neural-network training commonly relies on back propagation, which can involve local minima, learning-rate sensitivity, and slow convergence. This review synthesizes RVFL developments, variants, applications, and optimization approaches, concluding that RVFL-based models have achieved significant success while identifying architectural and application gaps for future work.

  • Problem

    Iterative back-propagation training can encounter local minima, learning-rate sensitivity, and slow convergence, motivating alternatives based on randomized neural networks.

  • Method

    The paper reviews RVFL evolution from shallow and ensemble models to deep and ensemble deep architectures, including applications, hyperparameter optimization, and future directions.

  • Results

    RVFL-based models have achieved significant success across various domains, with ensemble methods boosting single RVFL performance and some models achieving state-of-the-art time-series performance.

  • Takeaways & Limitations

    RVFL combines randomized nonlinear features, preserved linear patterns through direct links, and closed-form output-layer training as a basis for continued model development.

  • Takeaways & Limitations

    Spatial-temporal time-series research remains immature, with only one RVFL-related paper identified as addressing this problem to the authors’ knowledge.

Abstract

from arXiv · show

Neural networks have been successfully employed in various domains such as classification, regression and clustering, etc. Generally, the back propagation (BP) based iterative approaches are used to train the neural networks, however, it results in the issues of local minima, sensitivity to learning rate and slow convergence. To overcome these issues, randomization based neural networks such as random vector functional link (RVFL) network have been proposed. RVFL model has several characteristics such as fast training speed, direct links, simple architecture, and universal approximation capability, that make it a viable randomized neural network. This article presents the first comprehensive review of the evolution of RVFL model, which can serve as the extensive summary for the beginners as well as practitioners. We discuss the shallow RVFLs, ensemble RVFLs, deep RVFLs and ensemble deep RVFL models. The variations, improvements and applications of RVFL models are discussed in detail. Moreover, we discuss the different hyperparameter optimization techniques followed in the literature to improve the generalization performance of the RVFL model. Finally, we give potential future research directions/opportunities that can inspire the researchers to improve the RVFL's architecture and learning algorithm further.

1. Introduction

The introduction motivates RVFL as a randomized alternative to iterative neural-network training and reviews its evolution, variants, applications, and future directions.

  • Artificial neural networks support diverse applications but are typically trained iteratively, creating issues including local minima, learning-rate sensitivity, and slow convergence.
  • Randomization-based networks address these training issues while offering fast convergence and universal approximation properties.
  • RVFL uses random hidden-layer parameters, analytically computed output weights, and direct input-output links.
  • Random feature mappings can make RVFL unstable, motivating ensemble methods that combine diverse base models for greater stability and robustness.
  • The paper surveys shallow, deep, and ensemble RVFL developments, applications, and future research directions.

2. The standard RVFL architecture and it’s mathematical foundation

The standard RVFL is a single-hidden-layer randomized network that preserves original features through direct links while analytically learning output weights.

  • RVFL has input, hidden, and output layers; hidden-layer weights and biases are randomly generated and fixed, while output weights are computed analytically.
  • The model combines original input features with hidden-layer outputs through direct links to form the output representation.
  • The hidden representation applies an activation function to the inner product between each random hidden-node weight vector and an input sample, plus a bias.
  • Training minimizes a quadratic objective involving output-weight fitting, regularization, and sample error under linear constraints.
  • The closed-form solution uses the Moore-Penrose generalized inverse, with alternative expressions selected according to feature and sample dimensions.

3. Research methodology

The review identifies and organizes RVFL research through a targeted literature-search and screening process focused on model developments, applications, and future research questions.

  • Studies were collected from Google Scholar and Scopus using the keywords random vector functional link and deep RVFL.
  • Articles were screened first by title and abstract and then by full text.
  • The review covers shallow RVFL, ensemble RVFL, deep RVFL, ensemble deep RVFL, and their applications.
  • Its research questions address initialization, robustness, hidden-node selection, ensemble learning, deep architectures, and semi-supervised, unsupervised, and regression settings.

4. Developments and applications of RVFL model

RVFL developments address sensitivity to randomization, data geometry, noise, outliers, and hyperparameter selection while extending the model across applications. The reviewed variants include manifold, kernel, robust, feature-selection, and optimization-based approaches.

  • Core properties: Direct links, randomization ranges, activation functions, and output-weight computation have been systematically examined as factors affecting RVFL performance.A comprehensive evaluation covered 121 UCI classification datasets.
  • Core properties: RVFL’s random feature mapping enables fast training and few tunable parameters but can make the model unstable.Performance depends significantly on the quality of randomly generated weights and hidden biases.
  • Optimization and initialization: Optimization and learned initialization methods tune randomization ranges or generate weights and biases using data-informed procedures to improve generalization.Reported approaches include SVM-based initialization, random-subspace Fisher discriminant weighting, and Jaya optimization.
  • Manifold and kernel methods: Manifold-learning variants preserve or exploit data geometry to improve discriminability, within-class compactness, and robustness over standard RVFL.Reported applications include rolling-bearing fault diagnosis and hemodialysis dry-weight assessment.
  • Manifold and kernel methods: Kernel RVFL variants improve robustness, stability, and generalization, while kernelization can avoid selecting hidden-layer nodes and activation functions.Kernel-based models are not applicable to large-scale datasets when the number of samples is large enough.
  • Robust and efficient variants: Robust and feature-selection RVFL variants reduce the influence of noisy samples, outliers, or unnecessary hidden nodes in regression and industrial applications.Examples use augmented-Lagrangian feature selection, kernel-density weighting, or Cauchy-distribution weighting.

4.7. Imbalance learning based on RVFL model

RVFL variants have been developed for imbalanced classification, where one class has far fewer samples than another. IF-RVFL combines SMOTE with a fuzziness-based RVFL model.

  • Imbalance learning: IF-RVFL combines synthetic minority over-sampling with fuzziness-based RVFL to address class-imbalanced data.The experiments used a real-life liver disease dataset.
  • Imbalance learning: Experiments on the liver disease dataset found IF-RVFL performed better than standard RVFL and F-RVFL.
  • Imbalance learning: Standard RVFL is described as incapable of handling imbalanced data without a specialized variant.

4.8. Multi-label classification based on RVFL model

RVFL research has expanded from conventional classification and regression to multi-label classification and numerous shallow-model variants. The reviewed literature spans alternative activations, optimization methods, kernels, feature selection, and application-specific models.

  • Multi-label classification: Multi-label RVFL, ML-KRVFL, ML-BLS, and ML-FBLS were proposed for samples associated with more than one target label.ML-KRVFL performed better than the other compared models.
  • Shallow RVFL variants: Shallow RVFL variants modify initialization, kernels, manifold learning, fuzzy theory, regularization, online learning, and optimization procedures.Table 2 summarizes variants, activation functions, hyperparameter methods or solutions, and applications.
  • Applications: The summarized applications include classification, regression, forecasting, signal analysis, industrial modeling, medical diagnosis, and power-system assessment.
  • Learning procedures: RVFL studies use both closed-form least-squares solutions and iterative methods across different model variants.

5. Semi-supervised methods based on RVFL model

Semi-supervised RVFL research addresses settings where many samples are unlabeled and only a small number have labels. The review identifies this area as relatively underexplored and summarizes proposed models.

  • Motivation: Semi-supervised RVFL methods target datasets in which only a small number of samples are labeled.
  • Research gap: RVFL variants have been used broadly for classification and regression, but comparatively little research has addressed semi-supervised learning.
  • Methods: The review identifies JOSRVFL as a joint-optimized semi-supervised RVFL model with a novel objective-function optimization approach.

6. Clustering methods based on RVFL model

RVFL-based clustering methods use unsupervised discriminative learning to group similar samples while incorporating local data structure.

  • UDRVFL is an unsupervised discriminative RVFL model proposed for clustering.
  • Local manifold learning is used to capture local information within the data.

7. Ensemble frameworks based on RVFL model

Ensemble RVFL methods address instability from randomized hidden features by combining multiple base learners, including decomposition-based and heterogeneous model pools.

  • Ensemble RVFLs improve stability and performance by combining models with different hidden features.
  • Ensemble construction first builds base learners in parallel or sequentially, then combines their outputs for decisions.
  • Bagging: Bagging trains RVFL base learners on randomly sampled training subsets drawn with replacement.
  • Boosting: Boosting incrementally emphasizes samples that previous base RVFLs learned incorrectly.
  • Stacking: Stacking trains a meta-learner, such as a meta-RVFL, on outputs from the base learners.
  • Decomposition-based ensembles: Decomposition-based ensembles model frequency-specific time-series sub-series with individual RVFLs and aggregate their forecasts.
  • Weighting: Ensemble weights are often equal, although evolutionary algorithms and other methods can learn unequal weights.
  • Diverse model pools: Heterogeneous pools combine RVFL with models such as ELM and Schmidt neural networks to increase diversity.

8. Deep architectures based on RVFL model

Deep RVFL architectures stack randomized enhancement layers and differ mainly in their direct-link designs, with stacked, hybrid, and ensemble categories reviewed.

  • Deep RVFLs use multiple enhancement layers to extract multi-scale features from raw data.
  • Their main architectural distinction is how direct links connect inputs, hidden layers, and outputs.
  • Deep RVFL architectures are classified as stacked, hybrid, or ensemble deep RVFL models.
  • Stacked deep RVFL: Stacked deep RVFLs use multiple enhancement layers, while variants differ in whether direct links connect only inputs and outputs or also hidden layers.
  • Ensemble deep RVFL: The dRVFL uses features from all hidden layers, but this requires inversion of a super large matrix.
  • Ensemble deep RVFL: The edRVFL assigns an individual output layer to each enhancement layer and balances computational efficiency with feature utilization.
  • Hybrid deep RVFL: Hybrid deep RVFLs use external feature extraction methods, such as ResNet-50 or convolutional sparse coding, before an RVFL decision block.

9. Hyper-parameters optimization and experimental setup

RVFL studies tune architectural and learning hyperparameters through grid, evolutionary, stage-based, and layer-wise strategies, with evaluation protocols emphasizing fair comparison and reproducibility.

  • Hyperparameter optimization: RVFL hyperparameters include input scaling, hidden-node count, activation functions, regularization strength, and random-weight distributions.
  • Hyperparameter optimization: Grid search is common, while evolutionary algorithms search configurations across generations by selecting better-performing candidates.
  • Hyperparameter optimization: Grid search depends on manually defined candidate ranges, whereas evolutionary optimization can use a larger search region.
  • Deep-model tuning: For deep RVFLs, the number of hyperparameters grows exponentially and the number of layers may be fixed during tuning.
  • Deep-model tuning: Stage-based and layer-wise tuning reduce computational burden while allowing configurations to vary across layers.
  • Experimental setup: Classification studies commonly use k-fold cross-validation for tuning and evaluate models on a remaining test set.
  • Experimental setup: Time-series evaluations place the test set at the end, using earlier observations for training and hyperparameter tuning.
  • Evaluation: Evaluation combines task-specific metrics with group-wise or pair-wise statistical tests, including Nemenyi and Wilcoxon tests.

10. Time series forecasting and other applications

RVFL and its variants have been applied broadly to forecasting, often combined with signal decomposition, optimization, ensemble, or incremental strategies. Coverage spans energy, financial, and other time-series tasks, but spatial-temporal forecasting remains sparsely studied.

  • 10. Time series forecasting and other applications: RVFL variants have demonstrated performance across electricity load, solar power, wind power, financial, and other forecasting tasks.Applications also include temperature, landslide displacement, COVID-19 cases, travel time, molten iron quality, and energy consumption.
  • 10. Time series forecasting and other applications: Signal decomposition is widely used to split time series into sub-series, after which RVFL models forecast and aggregate the component predictions.Reported schemes include EWT, DWT, CEEMD, EEMD, and bivariate EMD, with aggregation sometimes performed by another RVFL.
  • 10.2. Solar power: Solar forecasting studies combine RVFL with direct-link comparisons, decomposition, and meta-heuristic optimization of kernel or network hyperparameters.Examples use maximum overlap DWT, water cycle algorithm, and Chimp Optimization Algorithm.
  • 10.4. Financial time series: Financial forecasting studies mainly target crude oil prices, combining decomposition with separate RVFL predictors and summation or incremental RVFL aggregation.Reported methods include EEMD, CEEMD, improved CEEMD with adaptive noise, bivariate EMD, and sine cosine optimization.
  • 10. Time series forecasting and other applications: Heuristic optimization, ensemble weighting, feature selection, and incremental structure updates are recurring strategies for adapting RVFL models to forecasting tasks.Incremental RVFLs update structure or weights when new observations arrive, while ensemble methods combine models trained on bootstrap samples or decomposed modes.
  • 10. Time series forecasting and other applications: Research on spatial-temporal time series forecasting is not mature, with only one RVFL-related paper identified by the authors.That study uses a kernel RVFL to predict temporal dynamics decomposed via the Karhunen–Loève method.

11. Comparison with other state of the art machine learning techniques

RVFL addresses several training limitations of back-propagation networks through randomized hidden-layer parameters, analytically computed output weights, and direct input-output links. Comparisons report strong results and faster training against several state-of-the-art models, while RVFL lacks CNN-style feature extraction.

  • 11. Comparison with other state of the art machine learning techniques: Back-propagation networks are sensitive to learning-rate settings, converge slowly, and can become trapped in local minima.RVFL instead randomizes input-to-hidden weights and computes hidden-to-output weights using a closed-form solution.
  • 11. Comparison with other state of the art machine learning techniques: Direct links distinguish RVFL from other randomized networks and play an important role in classification and regression.The direct input-output connection is a defining architectural feature of RVFL.
  • 11. Comparison with other state of the art machine learning techniques: Deep RVFL has been reported to outperform ResNet on tabular classification, while advanced RVFLs outperform LSTM and TCN in forecasting literature.The cited comparisons also report deep stacked RVFL outperforming stacked denoising auto-encoders on two benchmark MR brain datasets.
  • 11. Comparison with other state of the art machine learning techniques: RVFL training is reported as faster than back-propagation-based ResNet, LSTM, and TCN training in the cited comparisons.The comparison attributes the difference to the greater time required to optimize weights and biases with back propagation.
  • 11. Comparison with other state of the art machine learning techniques: RVFL lacks CNN-type feature-extraction layers for image or sequence data, but can use CNN-learned features as inputs to its decision module.This positions RVFL as a possible downstream decision component for features produced by gradient-based deep networks.

12. Conclusions and future directions

The review consolidates RVFL’s theoretical foundations, variants, applications, and performance-oriented developments, while identifying open directions for broader architectures, learning settings, and feature handling.

  • Conclusions: RVFL combines randomized nonlinear features, direct links preserving linear patterns, frozen hidden parameters, and closed-form output training for fast computation.The review presents these properties as central to RVFL’s architecture and computational appeal.
  • Conclusions: RVFL-based models have achieved strong results across domains, including state-of-the-art forecasting performance for wind speed, solar energy, and electricity load.Ensemble methods and signal decomposition are reported as especially successful for forecasting tasks.
  • Limitations and robustness: Robustness and generalization remain constrained by noisy or outlier-contaminated data, unsuitable randomization ranges, high-dimensional direct links, and disrupted topological structure.The review discusses kernelized, fuzzy, ensemble, regularization, and feature-treatment strategies as possible responses, with some approaches limited for large-scale data.
  • Open learning settings: Important scope gaps include limited RVFL research for imbalanced data, semi-supervised learning, multi-label tasks, spatial-temporal graph signals, and online streaming data.The review identifies these settings as opportunities for developing new RVFL variants.
  • Future architectures: Future work should combine RVFL with advanced architectures such as Transformers and graph convolutional networks to retain performance while reducing training time.The review specifically contrasts these possibilities with the conventional feedforward basis of most deep RVFL networks.
  • Model improvement: Dynamic weighting, advanced optimization, and intelligent random-feature selection are proposed to address static aggregation, tuning sensitivity, and redundant or inferior features.The suggested directions include evolving ensemble weights, evolutionary optimization, and feature-selection methods beyond current linear selection and pruning.
Loading 2203.11316v2…