Source-linked AI summary
Stochastic Configuration Networks: Fundamentals and Algorithms
Dianhui Wang, Ming Li
TL;DR
The paper addresses limitations in randomized and constructive neural-network learning, particularly architecture selection, costly deterministic optimization, and conditional approximation guarantees. It proposes incrementally built Stochastic Configuration Networks with supervised random parameter selection and analytical output-weight evaluation, and reports fast learning, sound generalization, and strong performance across modelling tasks. The authors identify universal approximation guarantees for the proposed framework as a central theoretical objective.
Problem
Existing constructive methods can be computationally costly, while incremental randomized networks may lack universal approximation guarantees under unsuitable configurations.
Method
SCNs incrementally configure hidden-node input weights and biases under supervisory constraints, then compute output weights constructively or through optimization.
Results
SC-III exhibits the best convergence-rate performance, while SC-I still outperforms IRVFL; SC-I, SC-II, and SC-III outperform other methods on learning and generalization results.
Takeaways & Limitations
SCNs provide a randomized learner-model framework with adaptive parameter scopes, incremental construction, fast data modelling, and reduced human intervention in network-size setting.
Takeaways & Limitations
Prior derivative-based constructive optimization can stall on error-surface plateaus, and the paper’s RVFL discussion states approximation guarantees are conditional under some random configurations.
Abstract
from arXiv · showhide
This paper contributes to a development of randomized methods for neural networks. The proposed learner model is generated incrementally by stochastic configuration (SC) algorithms, termed as Stochastic Configuration Networks (SCNs). In contrast to the existing randomised learning algorithms for single layer feed-forward neural networks (SLFNNs), we randomly assign the input weights and biases of the hidden nodes in the light of a supervisory mechanism, and the output weights are analytically evaluated in either constructive or selective manner. As fundamentals of SCN-based data modelling techniques, we establish some theoretical results on the universal approximation property. Three versions of SC algorithms are presented for regression problems (applicable for classification problems as well) in this work. Simulation results concerning both function approximation and real world data regression indicate some remarkable merits of our proposed SCNs in terms of less human intervention on the network size setting, the scope adaptation of random parameters, fast learning and sound generalization.
1 Introduction
Neural networks offer universal approximation and sample-based learning, but choosing an architecture that generalizes well is difficult. The paper addresses this through supervised randomization and incremental SCN construction.
- Neural networks support nonlinear data modelling through universal approximation and learning from training samples.
- Constructive approaches start with a small network and incrementally add hidden nodes and output weights until a termination criterion is met.
- The proposed SCNs assign random parameters under an inequality constraint while adaptively selecting their scope to preserve universal approximation.
- SC-I, SC-II, and SC-III share supervised random-parameter configuration but differ in how output weights are computed.
2 Related Work
Prior constructive methods provide approximation guarantees but can be computationally expensive or rely on difficult optimization, while randomized incremental networks may lose universal approximation under unsuitable configurations. These limitations motivate supervisory control of random parameters.
- Constructive Neural Networks: Deterministic Methods: Constructive neural networks add hidden nodes incrementally, avoiding prior knowledge of task complexity when selecting the architecture size.
- Constructive Neural Networks: Deterministic Methods: Barron’s greedy framework applies when the target lies in the closure of the convex hull of the basis set, so convergence to all L2 functions is not guaranteed.
- Constructive Neural Networks: Deterministic Methods: Kwok and Yeung established universal approximation when each new basis function maximizes its squared residual correlation normalized by its norm.
- Constructive Neural Networks: Deterministic Methods: Gradient-ascent construction can encounter local minima and plateaus where hidden-parameter updates become excessively slow and residual reduction nearly stops.
- Constructive Neural Networks: Deterministic Methods: Deterministic construction is often impractical because searching for suitable basis functions is time consuming despite its approximation guarantees.
- Constructive Random Basis Approximators: RVFL networks randomize and fix hidden parameters, then solve output weights analytically, but incremental RVFL may lack universal approximation guarantees.
- Constructive Random Basis Approximators: Theoretical results show that IRVFL can fail to approximate universally under specified residual-error conditions, including when global least squares computes output weights.
- Constructive Random Basis Approximators: The paper attributes conditional RVFL guarantees to factors such as parameter-scope setting or improper randomization and proposes supervisory configuration as a solution.
3 Stochastic Configuration Networks
SCNs incrementally construct single-layer networks by supervisory, data-dependent random configuration of hidden parameters and analytical output-weight evaluation. The resulting algorithms are designed to retain universal approximation while supporting constructive, recalculated, and selective learning schemes.
- Constructive formulation: SCNs incrementally add random basis functions until the residual error reaches a predefined tolerance, while analytically evaluating output weights.The model updates as fL = fL−1 + βLgL, with hidden parameters configured for each new node.
- Universal approximation: Under dense-basis assumptions and bounded nonzero basis functions, the constructive scheme yields monotonically decreasing residuals converging to zero.The proof uses conditions involving r, µL, and δL,q to establish universal approximation.
- Supervisory configuration: The supervisory mechanism constrains random hidden weights and biases using training-data-dependent inequalities rather than unconstrained random assignment.This mechanism is presented as a bridge between demanding global nonlinear optimization and freely random hidden-parameter selection.
- Output-weight evaluation: Recalculating all output weights by global least squares can improve convergence, whereas selective window-based recalculation trades optimization scope for large-scale feasibility.The selective scheme keeps earlier weights unchanged and renews only a window once the hidden-node count exceeds K.
- Algorithms: The three SC algorithms combine constrained random hidden-parameter generation with constructive or selective output-weight determination, and repeated candidate sampling can help build compact networks.SC-I, SC-II, and SC-III correspond to the constructive, selective, and globally recalculated schemes described by the paper’s theorems.
4 Performance Evaluation
The evaluation compares SC-I, SC-II, and SC-III with MQ and IRVFL across function approximation and real-world regression tasks. SC methods generally provide stronger learning, generalization, and efficiency, while SC-II shows robustness across reasonable window sizes.
- Experimental setup: The study evaluates learning, generalization, and efficiency on one function-approximation task and three real-world regression datasets.The experiments compare SC algorithms with Modified QuickProp and IRVFL, using averages over 100 independent trials.
- Accuracy: SC-I, SC-II, and SC-III outperform MQ and IRVFL on training and test performance across DB1–DB4, with SC-III performing best among the SC variants.Table results report average values and standard deviations over 100 independent trials.
- Efficiency: SC-II and SC-III reach the training tolerance ε = 0.05 with few hidden nodes, whereas MQ, IRVFL, and SC-I fail within an acceptable time on DB1 or DB2.The proposed SC-II and SC-III methods also use fewer hidden nodes and show better generalization than MQ in these comparisons.
- DB1 results: On DB1, SC algorithms reduce training error faster than MQ and IRVFL, while SC-II and SC-III also keep reducing test error as hidden nodes are added.SC-II and SC-III outperform SC-I, and their decreasing trends are consistent for L ≤ 15 because SC-II uses window size K = 15.
- DB2 results: On DB2, MQ, SC-II, and SC-III converge faster than the other methods, while SC-III’s error curves remain lowest overall.For L > 50, the curves of SC-II, SC-III, and MQ differ negligibly, although SC-III remains lowest overall.
- Window-size robustness: For SC-II, K = 15, 20, and 25 require around 0.2s and 16 hidden nodes to achieve ε = 0.05 on DB2, indicating comparable results across these settings.The reported robustness is favorable when K is chosen from a reasonable range; on DB1, most remaining settings use about 20 nodes and 0.26s.
- Methodological limitations: MQ may be hindered by slow or stalled optimization, while freely random hidden parameters and constructive output-weight evaluation can produce slow residual-error reduction.The paper identifies plateau behavior, slow parameter updating, and inflexible construction as obstacles for MQ and related approaches.
- Methodological comparison: SC-I is reported as more efficient and effective than IRVFL because IRVFL’s random assignment of w and b from [-1,1] is described as unworkable.This comparison is stated across activation-function choices in the cited discussion.
5 Conclusions
SCNs provide an alternative randomized framework for incrementally building learner models with supervisory configuration of hidden-node parameters and analytical output-weight evaluation. The framework supports theoretical convergence guarantees, while simulations identify SC-III as fastest and SC-II as a balance between efficiency and scalability; the study focuses on sigmoidal SLFNNs.
- Framework: SCNs incrementally configure hidden-node input weights and biases stochastically, then determine output weights through constructive evaluation or linear optimization.The framework uses supervisory mechanisms during randomized learner-model construction.
- Framework: The SC algorithms theoretically ensure convergence when newly generated nodes continue to be added to the model.
- Simulation findings: SC-III exhibits the best convergence rate in the reported simulations, while SC-I converges slowest but still outperforms IRVFL.
- Simulation findings: SC-II offers a trade-off between efficiency and scalability by updating output weights using a given window size.
- Scope: The work focuses on single-layer feed-forward networks with sigmoidal hidden-node activation functions, with extensions to RVFL and Echo State Networks described as immediate.