Source-linked AI summary
A study on effectiveness of extreme learning machine
Yuguang Wang, Feilong Cao, Yubo Yuan
TL;DR
ELM's random input weights and biases can make its hidden-layer output matrix rank-deficient, reducing effectiveness. The paper proposes EELM, which selects those parameters before computing output weights; the authors report faster learning than ELM while avoiding singular or non-full-column-rank H and retaining acceptable prediction accuracy.
Problem
ELM's random input weights and biases can produce a singular or non-full-column-rank hidden-layer output matrix H, limiting prediction accuracy and efficient inverse computation.
Method
EELM selects input weights and biases using a strictly diagonally dominant matrix criterion before calculating output weights.
Results
EELM avoids the risk of a singular or non-full-column-rank H, permits a faster inverse method, and generally learns faster than ELM with acceptable prediction accuracy.
Takeaways & Limitations
EELM provides a simple SLFN learning algorithm that combines the ELM framework with selected hidden-layer parameters and improved effectiveness.
Takeaways & Limitations
The EELM construction excludes sigmoidal activation functions and requires an activation function satisfying the theorem's assumptions.
Abstract
from arXiv · showhide
Extreme learning machine (ELM), proposed by Huang et al., has been shown a promising learning algorithm for single-hidden layer feedforward neural networks (SLFNs). Nevertheless, because of the random choice of input weights and biases, the ELM algorithm sometimes makes the hidden layer output matrix H of SLFN not full column rank, which lowers the effectiveness of ELM. This paper discusses the effectiveness of ELM and proposes an improved algorithm called EELM that makes a proper selection of the input weights and bias before calculating the output weights, which ensures the full column rank of H in theory. This improves to some extend the learning rate (testing accuracy, prediction accuracy, learning time) and the robustness property of the networks. The experimental results based on both the benchmark function approximation and real-world problems including classification and regression applications show the good performances of EELM.
1 Introduction
ELM offers fast, simple training for SLFNs, but random input weights and biases can make its hidden-layer matrix rank-deficient. The paper proposes EELM, which selects these parameters to improve effectiveness while retaining speed.
- ELM background: ELM is faster than conventional SLFN learning methods and usually requires no parameter tuning beyond the predefined network architecture.Most training can be completed in seconds, or minutes for large-scale complex applications.
- Problem: Randomly chosen input weights and biases can produce a hidden-layer output matrix that is not full column rank.This can make the output-weight linear system unsolvable and lower prediction accuracy.
- Motivation: The resulting learning method should overcome rank deficiency while approximating as fast as ELM.The motivation also reflects the reported relationship between smaller weight norms, training error, and better generalization performance.
- EELM proposal: EELM selects input weights and biases before computing output weights, using a strictly diagonally dominant criterion to ensure full column rank.Its first phase sorts samples by affine transformation and uses a Gaussian radial basis-type activation function.
- Reported properties: The proposed algorithm retains ELM's easy implementation and generalization performance while reducing time spent in its input-parameter selection phase.The paper reports that EELM is faster than ELM and can use orthogonal projection after establishing full column rank.
- Paper scope: The paper develops theoretical results for the two input-parameter selection steps and evaluates EELM's performance experimentally.The evaluation covers benchmark function approximation and real-world classification and regression applications.
Neural Network
This section formalizes how affine sorting and selected Gaussian radial basis parameters produce a full-column-rank hidden-layer output matrix. The construction makes H strictly diagonally dominant and therefore nonsingular under stated assumptions.
- Hidden-layer matrix: The hidden-layer output matrix H contains activation responses g(W_i · X_j + b_i) for samples and hidden nodes.H is the matrix used to represent hidden-layer outputs before output weights are calculated.
- Sample ordering: Inverse lexicographical ordering and an affine transformation construct a vector W that orders distinct high-dimensional samples.For d ≥ 2, the theorem establishes W · X_1 < W · X_2 < · · · < W · X_n.
- Parameter selection: Theorem 2.3 selects input weights and biases so each diagonal activation reaches the maximum M while off-diagonal activations remain smaller.The construction uses W_i = k_iW and b_i = x_0 − W_i · X_i under assumptions on the activation function.
- Rank guarantee: The constructed matrix H is strictly diagonally dominant and therefore nonsingular.This is the theorem's stated conclusion for the selected weights and biases.
- Rectangular H: When H has more rows than columns, the weights and biases are calculated from a square matrix formed by the forward rows of H.The construction is therefore adapted to the rectangular case.
3 Extreme learning machine using iterative method
The ELM baseline randomly assigns hidden-layer input weights and biases, which can yield a singular H and force slower computation. EELM adds a constructive selection phase, then computes output weights using the resulting full-column-rank matrix.
- ELM algorithm: Standard ELM randomly assigns input weights and biases, calculates H, and obtains output weights with the Moore-Penrose generalized inverse.Its baseline procedure does not first enforce a rank condition on H.
- ELM limitation: A singular or rank-deficient H can increase training error, lower prediction accuracy, and prevent ELM from using the faster orthogonal projection method.ELM instead uses singular value decomposition to calculate the generalized inverse when H is singular.
- Theoretical guarantee: EELM uses a Gaussian radial basis activation function and chooses parameters so that H is full column rank for distinct samples under the theorem's assumptions.The paper states that this makes H^T H nonsingular and enables fast orthogonal projection for the Moore-Penrose inverse.
- Square case: If the number of hidden nodes equals the number of samples, H is an invertible square matrix.The paper separately identifies this square-matrix case after establishing full column rank.
- Implementation detail: The input-parameter selection phase can use randomly chosen training samples from the original set to avoid random error.The selected samples are sorted as required by the construction.
4 Complexity and performance
EELM selects input weights and biases before computing output weights, adding a small training overhead while targeting a full-column-rank hidden-layer matrix. Across benchmark approximation and real-world tasks, EELM improves robustness and regression steadiness, although ELM can achieve higher testing rates on some classification problems.
- Complexity: The proposed EELM spends more time selecting input weights and biases, with extra cost O(n0d), while output-weight calculation remains the second algorithm phase.The paper treats this added selection time as comparatively limited against the output-weight calculation phase.
- Benchmark approximation: In the SinC benchmark, EELM reached testing accuracy 0.1595 with zero training error in 0.0624s, while ELM used 0.0870s and had error 5.6642.The reported ELM training error was 3.1431 × 10−6.
- Benchmark approximation: For SinC, EELM approximated training data with zero error and maintained long-term prediction accuracy without consuming more time than ELM.ELM performed well on [−10, 10] but had unsatisfactory long-term prediction accuracy.
5 Discussions and conclusions
The paper proposes EELM, which properly selects input weights and biases to improve the effectiveness of ELM for single-hidden layer feedforward neural networks. Compared with ELM, EELM is generally faster, more robust, and maintains acceptable prediction accuracy for longer, subject to activation-function and high-dimensional-data limitations.
- Performance: EELM's learning speed is generally faster than ELM, while selecting weights and biases consumes little time compared with output-weight training.The paper identifies the input-weight and bias selection as the main algorithmic difference and reports that this added selection remains short relative to output-weight training.
- Proposed algorithm: EELM properly selects input weights and biases before calculating output weights, distinguishing it from ELM's random selection.The selection is intended to avoid a singular or non-full-column-rank hidden-layer output matrix H.
- Proposed algorithm: EELM avoids the risk of a singular or non-full-column-rank H, enabling a faster Moore-Penrose generalized-inverse calculation.The paper links this matrix property to a faster method for calculating the generalized inverse.
- Performance: EELM has a longer prediction period with acceptable accuracy and better robustness than ELM, especially where ELM regression performance is sometimes poor.The paper reports that EELM remains steady and has good performance in that regression setting.
- Limitations and open questions: For high-dimensional data with very large inter-sample deviations, the affine-transformation sorting procedure can produce excessively large weights, motivating large-number-operation algorithms.The paper also leaves open whether better methods can sort high-dimensional data effectively and simply by affine transformation.
- Limitations and open questions: EELM's effectiveness depends on activation functions satisfying Theorem 2.3 assumptions; Gaussian radial basis functions qualify, whereas sigmoidal functions do not.The paper identifies algorithms for other activation functions that are as effective and fast as EELM as an open problem.