Source-linked AI summary
Stochastic Configuration Networks Ensemble for Large-Scale Data Analytics
Dianhui Wang, Caihao Cui
TL;DR
Large-scale neural-network ensembles face computational and hardware constraints, and heterogeneous features call for a more general framework. The paper combines SCN base models with NCL and iterative output-weight methods; across two large-scale datasets, SCNE outperforms DNNE, and the authors report good potential for large-scale analytics.
Problem
Existing ensemble methods are challenging to apply to large datasets because of computational complexity, least-squares scalability, hardware constraints, and the need to accommodate heterogeneous features.
Source: §1 · pp. 1–2Method
The paper builds an NCL ensemble from SCN base models and uses block Jacobi and block Gauss-Seidel iterations to evaluate output weights.
Source: §5 · p. 19Results
Across two large-scale benchmark datasets, SCNE always outperforms DNNE; in the Twitter experiments, iterative methods peak below 5.4GB of memory versus about 7.9GB for the pseudo-inverse method.
Source: §4.4 · p. 18, §5 · p. 19Takeaways & Limitations
The authors conclude that robustness to the NCL regularizing factor gives the proposed ensemble good potential for large-scale data analytics.
Source: §5 · p. 19The full-rank condition is hard to meet for all base models, so convergence and uniqueness of iterative solutions cannot be ensured; the impact of hidden-node number remains unclear.
Source: §19 · pp. 10–11
Abstract
from arXiv · showhide
This paper presents a fast decorrelated neuro-ensemble with heterogeneous features for large-scale data analytics, where stochastic configuration networks (SCNs) are employed as base learner models and the well-known negative correlation learning (NCL) strategy is adopted to evaluate the output weights. By feeding a large number of samples into the SCN base models, we obtain a huge sized linear equation system which is difficult to be solved by means of computing a pseudo-inverse used in the least squares method. Based on the group of heterogeneous features, the block Jacobi and Gauss-Seidel methods are employed to iteratively evaluate the output weights, and a convergence analysis is given with a demonstration on the uniqueness of these iterative solutions. Experiments with comparisons on two large-scale datasets are carried out, and the system robustness with respect to the regularizing factor used in NCL is given. Results indicate that the proposed ensemble learning techniques have good potential for resolving large-scale data modelling problems.
1 Introduction
The paper addresses the computational and hardware limits of existing neural-network ensembles on large datasets and the need to handle heterogeneous features. It introduces an SCN-based, NCL ensemble framework and reviews the supporting SCN, generalization-error, and NCL concepts.
1 Introduction
- Existing neural-network ensembles face large-scale data limits from computational complexity, least-squares scalability, and hardware constraints, while heterogeneous data sources motivate a more general framework.The paper identifies data from different sensors, localized sources, or feature-selection runs as examples of heterogeneous inputs.Source: §1 · pp. 1–2
- The proposed framework feeds separate feature inputs to SCN base models and uses two iterative methods to evaluate ensemble output weights.The paper also analyzes convergence and correlations between iterative solutions and the pseudo-inverse solution.Source: §1 · p. 2
2.1 Revisit of stochastic configuration networks
- SCNs incrementally select random input weights and biases from a data-dependent stochastic configuration support, and this constructive approach guarantees universal approximation for a given nonlinear map.This contrasts with the default scope setting used by RVFL networks discussed in the paper.Source: §2.1 · p. 2
2.2 Generalization error of ensembles with heterogeneous features
- The heterogeneous-feature generalization-error analysis identifies covariance among base models as a term to manage; validation error can represent generalization error under the stated sampling setup.The validation samples are described as i.i.d. with the training data.Source: §2.2 · pp. 4–5
2.3 Negative correlation learning
- Negative correlation learning reduces covariance among base models while aiming not to increase the ensemble's variance and bias terms.The scheme minimizes each base model's cost over its training data.Source: §2.3 · p. 5
3 Stochastic configuration networks ensemble
SCNE builds an ensemble from SCN learners with heterogeneous features, using NCL to evaluate output weights. Its iterative solvers address the large linear system that makes a pseudo-inverse impractical for large-scale data.
3 Stochastic configuration networks ensemble
- SCNE uses heterogeneous-feature SCN learners and block Jacobi or Gauss-Seidel solvers to address pseudo-inverse limits in large-scale modeling.The earlier RVFL ensemble used shared inputs and identical architectures.Source: §3 · p. 5
3.2 SCNE construction
- After fixing the base models’ input weights and biases, NCL determines the ensemble output weights.Source: §3 · p. 6, §3.2 · p. 6
3.1 SCN base model generation
- Each SCN base model is built independently on its feature-specific dataset using SC-III, with validation pruning or trial and error used to choose its hidden-node count.Adding too many hidden nodes can cause overfitting.Source: §3.1 · p. 6, §3 · p. 6
3.2.1 Analytical solution
- NCL output-weight optimization yields a huge linear system for the analytical solution, and full column rank is not assured for every base model.The analytical approach uses a pseudo-inverse when the full-rank condition cannot be ensured.Source: §3.2.1 · p. 7
18 end
The iterative approach avoids computing all output weights at once by updating each base model’s weights gradually. Block Jacobi and Gauss-Seidel updates can be stopped using an error tolerance or an iteration cap.
3.2.2 Iterative solutions
- The huge-system pseudo-inverse can be impractical or time-consuming on large datasets, so the iterative approach updates each base model’s weights gradually to reduce computing time and memory cost.Source: §3.2.2 · p. 8
- At each iteration, the local output weights are obtained by solving an equation, with block Jacobi and Gauss-Seidel defining the two update schemes.Source: §3.2.2 · p. 8
- The algorithms stop when ensemble error Eens falls below tolerance τ or the iteration count reaches kmax.Source: §3.2.2 · pp. 8–9
8 end
The convergence analysis shows that block Jacobi and block Gauss-Seidel can reach the pseudo-inverse solution under stated rank and spectral-radius conditions. The analysis expresses the system using matrix components to assess those conditions.
19 end
- Both iterative schemes converge to the pseudo-inverse solution if every Hm has full column rank and λ makes the iteration matrix’s spectral radius less than 1.Source: §19 · pp. 9–10
- The analysis decomposes H as D + R and L + U, where D is diagonal, R is the remainder, L is lower triangular, and U is strictly upper triangular.Source: §19 · p. 10
- The Gershgorin circle theorem constrains eigenvalues of D^-1R, providing a theoretical estimate for λ.Source: §19 · p. 10
8 end
In a synthetic demonstration, iterative methods scale more favorably in construction time than the pseudo-inverse method as hidden-node count grows. Regularization brings the three methods’ output weights into near agreement, while rank and regularization-performance caveats remain.
19 end
- Full rank is hard to ensure for every base model, so iterative-solution convergence and uniqueness are not guaranteed; regularization makes the system matrix invertible.Source: §19 · pp. 10–11
- The effect of regularization r on modeling performance remains unresolved, especially with many hidden nodes.Source: §19 · p. 11
3.3 Demonstration
- The demonstration uses 5,000 synthetic samples and omits training and test results; the authors infer that a similar consequence holds for large-scale datasets.Source: §3.3 · p. 11
- For L > 2000 in the demonstration, pseudo-inverse construction is clearly inefficient and infeasible, while iterative-method computing time grows linearly rather than exponentially.The iterative methods may take more time when the hidden-node count is small.Source: §3.3 · p. 11, Figure 2 · p. 12
- With r = 0.1, output-weight correlations across all three methods are almost one; without regularization, A1–A2 and A1–A3 correlations decrease as L grows while A2–A3 remains consistent.Source: §3.3 · pp. 11–12, Figure 2 · p. 12
4 Performance evaluation
The evaluation tests SCNE on Twitter Buzz Prediction and Year Prediction MSD, using dataset-specific heterogeneous feature groups and a training-validation-testing workflow. SCNE achieves lower training and testing RMSE than DNNE on both datasets, while its testing performance changes only slightly across the tested NCL regularization factors.
4.1 Large-scale datasets
- The Twitter and Year datasets support prediction of discussion popularity and song release year, respectively, with 77 and 90 features.Twitter samples are described by 77 features; Year includes 12 timbre averages and 78 timbre covariance features.Source: §4.1 · p. 12, Table 1 · p. 13
4.2 Heterogeneous feature generation
- Twitter’s 77 features are partitioned into 11 groups, while Year’s 90 features form seven groups, including six 13-feature covariance groups.The Year groups comprise one 12-feature timbre-average group and six smaller timbre-covariance groups.Source: §4.2 · p. 13, Table 2 · p. 13, Table 3 · p. 13
4.3 Experimental setup
- The experiments normalize data to [0, 1] assuming no outliers, then tune on validation data, retrain using training plus validation data, and test the final ensemble.Twitter uses a 70-15-15 split; Year uses 70-20-10, with its testing set specified in advance.Source: §4.3 · p. 14, Figure 5 · p. 15
4.4 Results and discussion
- SCN hidden-node counts are selected using repeated down-sampled training and validation sets, while DNNE matches the SCNE architecture and tunes each RVFL range α.SCN selection uses 70,000 training and 30,000 validation samples, repeats the procedure 10 times, and chooses the median hidden-node count; RVFL α is selected by minimum validation error.Source: §4.4 · p. 15, Table 4 · p. 16, Table 5 · p. 16
- SCNE has lower training and testing RMSE than DNNE on both datasets; pseudo-inverse and iterative schemes also have lower errors than the naive algorithm.Results average 10 independent repeats; the comparison uses λ = 0.10 and kmax = 10.Source: §4.4 · p. 16, Table 8 · p. 17, Table 9 · p. 17
4.5 Robustness analysis
- Across λ values from 0.08 to 0.12, the NCL regularizing factor affects SCNE testing performance only slightly and not significantly on either dataset.The analysis uses 10 testing results per setting, with hidden-node counts and data partitions held fixed.Source: §4.5 · p. 18, Figure 7 · p. 18
5 Conclusion
The proposed SCNE uses stochastic configuration networks, heterogeneous features, and negative correlation learning, with block iterative schemes to evaluate output weights. Across two large-scale benchmark datasets, it outperforms DNNE and performs robustly with respect to the NCL regularizing factor.
5 Conclusion
- Ensemble learning can improve base-learner generalization, but efficiency and scalability remain challenges for large-scale modelling.Source: §5 · pp. 18–19
- Block Jacobi and block Gauss-Seidel schemes evaluate SCNE output weights without requiring a pseudo-inverse of the huge linear system.The paper reports that pseudo-inverse-based solution speediness holds only for smaller datasets; computational cost can be very high or infeasible at large scale.Source: §5 · p. 19
- Ridge regression is necessary when building SCN base models for iterative SCNEs to match non-iterative models in output-weight correlations.Source: §5 · p. 19
- Across two large-scale benchmark datasets, the proposed SCNE outperforms DNNE and performs robustly with respect to the NCL regularizing factor.The authors report good potential for large-scale data analytics.Source: §5 · p. 19