Random Vector Functional Link Networks for Function Approximation on Manifolds

1 INTRODUCTION

• RVFL networks randomize input-to-hidden weights and biases, leaving hidden-to-output weights to be learned from data, which converts supervised learning into a linear problem.The output weights can be obtained through a Moore-Penrose pseudoinverse after fixing the random parameters.Source: §1 · pp. 2–3
• RVFL networks address training difficulties associated with back-propagation while retaining practical use across prediction, recognition, tracking, classification, regression, and forecasting applications.Traditional single-layer RVFL networks are reported to perform comparably to multilayer variants with lower training costs.Source: §1 · pp. 2–3

1.2 Main results

• The paper corrects flaws in the Igelnik and Pao theorem while proving universal approximation guarantees for continuous compactly supported functions.The corrected result is presented as a technically weaker version of the original claim.Source: §1.2 · pp. 3–4
• The asymptotic construction guarantees ε-closeness rather than exact convergence, while exact convergence requires parameter sequences that may have no guaranteed convergence rate and increasingly broad weight ranges.The latter behavior may create practical problems as the network grows.Source: §1.2 · p. 4
• The non-asymptotic theorem gives high-probability approximation when the node count is sufficiently large, at the cost of requiring a Lipschitz activation function.The stated bound has the form n ≥ C(N, f, ρ) ε^-1 log(η^-1/ε), with probability at least 1 − η.Source: §1.2 · p. 5
• The manifold construction is motivated by settings where high-dimensional observations occupy lower-dimensional structures, including cellular and neural manifolds.The paper also notes that approximating the manifold atlas has computational complexity O(CdNm log(m)).Source: §1.2 · p. 5
• For smooth compact manifolds, the proposed RVFL-like architecture replaces ambient-dimension dependence with dependence on intrinsic dimension and atlas complexity.The approximation combines several RVFL networks in local manifold coordinates rather than forming one single RVFL network.Source: §1.2 · p. 6

1.3 Organization

• The paper’s proofs use limit-integral representations and Monte Carlo approximation, followed by numerical experiments illustrating the manifold theorem.The organization also includes theoretical preliminaries on concentration bounds and smooth compact manifolds.Source: §1.3 · p. 6

2.1 A concentration bound for classic Monte-Carlo integration

• Monte Carlo integration estimates a uniform integral using equal-weight averages of independent samples, with expectation equal to the target integral.Its mean-square error scales with the variance and 1/n.Source: §2.1 · p. 7
• The non-asymptotic analysis uses probability bounds for deviations between the Monte Carlo estimate and the target integral under a boundedness condition.The relevant probability is P(|I_n(f,S) − I(f,S)| ≥ t).Source: §2.1 · p. 7

2.2 Smooth, compact manifolds in Euclidean space

• A chart provides Euclidean coordinates on a manifold, while an atlas is a collection of charts covering the manifold.The paper works with smooth, compact d-dimensional manifolds embedded in R^N.Source: §2.2 · pp. 7–8
• A partition of unity consists of nonnegative smooth functions summing to one, with compact supports available on compact manifolds.This construction supports decomposing continuous manifold functions into chart-local pieces.Source: §2.2 · p. 8
• The manifold representation combines chart-local terms and Jacobian factors through a change of variables for integration.The differential of each chart map is invertible, so its determinant is nonzero.Source: §2.2 · p. 8

3 RESULTS

• The section contains proofs of the main theoretical results and numerical simulations illustrating RVFL approximation performance in a low-dimensional submanifold setting.Source: §3 · p. 8

3.1 Proof of Theorem 1

• The proof splits into cases where ρ or ρ′ belongs to L1(R) ∩ L∞(R), with the derivative case handled using integration by parts.Source: §3.1 · p. 8, §3.1.1 · p. 11

3.1.1 Proof of Theorem 1 when ρ ∈L1(R) ∩L∞(R)

• The construction first approximates f by a convolution, then represents the convolution as a multidimensional integral over RVFL parameters.Source: §3.1.1 · p. 9
• The integral representation uses random variables for hidden-layer parameters and Monte Carlo sampling to form the finite RVFL network.Source: §3.1.1 · pp. 9–10
• The approximation error is decomposed into convolution and sampling terms, with the sampling contribution controlled asymptotically as n grows.Source: §3.1.1 · pp. 10–11

• The theorem assumes that the activation function is κ-Lipschitz, while the proof remarks that this condition may potentially be removed.Source: §3.2 · pp. 11, 15
• The proof combines deterministic continuity bounds with concentration over a finite δ-net and a union bound to obtain uniform control on the compact set.Source: §3.2 · pp. 12–14
• For any target accuracy and confidence level, sufficiently many network nodes yield an RVFL approximation that is ε-close to f with probability at least 1 −η.Source: §3.2 · pp. 12, 15
• For fixed ambient dimension, a coarse node requirement is n ≳ ε^-1 log(η^-1/ε), whereas variable dimension can produce superexponential dependence on N.Source: §3.2 · p. 15

3.3 Results on submanifolds of Euclidean space

• High ambient dimension makes the original RVFL guarantees costly, motivating approximation methods that exploit lower-dimensional manifold structure.The original node-count bounds depend superexponentially on ambient dimension N.Source: §3.3 · pp. 15–16

3.3.1 Adapting RVFL networks to d-manifolds

• For each chart, the method approximates the coordinate representation of f in R^d and sums the chartwise RVFL networks using a partition of unity.This converts manifold approximation into finitely many low-dimensional approximation problems.Source: §3.3.1 · pp. 16–17

3.3.2 Main results on d-manifolds

• The manifold construction preserves the asymptotic mean-square error behavior of the Euclidean RVFL result while using d-dimensional random input weights and biases.The guarantee holds for continuous functions on smooth, compact manifolds under the stated activation-function assumptions.Source: §3.3.2 · pp. 17–19
• The non-asymptotic theorem provides a high-probability approximation guarantee for sufficiently many independently sampled chartwise random variables and suitable hidden-to-output weights.The construction applies the Euclidean non-asymptotic result separately on each chart.Source: §3.3.2 · pp. 19–20
• Manifold structure replaces superexponential dependence on ambient dimension N with dependence on intrinsic dimension d and atlas complexity.The resulting random variables are d-dimensional, though the number of charts |J| also enters the bounds.Source: §3.3 · p. 16, §3.3.2 · p. 20
• The bounds on chart covering radii and hidden nodes are not optimal and may improve using local curvature, reach, and other geometric structure.In particular, the dependence on |J| may improve when the manifold is locally well-behaved.Source: §3.3.2 · p. 20

3.4 Numerical Simulations

• The algorithm builds and trains one RVFL network for each GMRA affine patch, then assigns each query point to its nearest center and evaluates the corresponding network.GMRA patches provide approximate tangent spaces, while local coordinates reduce approximation to networks on R^d.Source: §3.4 · pp. 21–22
• For sufficiently many local nodes, Theorem 8 gives small local approximation error with high probability, and the global construction inherits this guarantee when the GMRA resolution and node counts are adequate.The analysis combines local RVFL approximation with the projection error controlled by GMRA resolution.Source: §3.4 · p. 22
• The manifold approximation requires f to be continuously defined on a region containing both M and its scale-j GMRA approximation.This permits evaluating f on the affine subspaces generated by the GMRA.Source: §3.4 · p. 22
• In the S^2 experiment embedded in R^20, the networks use a level-15 GMRA built from 20,000 samples, constant local width n, and fixed α and Ω parameters.The test set contains 200 randomly selected manifold points.Source: §3.4 · p. 22
• As n increases, mean relative error decreases until a floor set by GMRA approximation error, consistent with the need to choose j so its inherent error, scaling like 2^-j, is below ε.The experiment uses f(x)=exp(sum_k x^(k)) and the hyperbolic secant activation.Source: §3.4 · p. 23
• With fixed αj and Ωj, increasing n eventually stagnates because the GMRA resolution remains the limiting error term, while constructing the GMRA has complexity O(CdNm log(m)).The fixed parameters also induce a theorem-level error floor, but the plotted solid and dashed curves appear limited primarily by GMRA error.Source: §3.4 · p. 23

3.5.1 Proof of Lemma 3

• The proof of Lemma 3 shows that scaled kernels act as approximate identities, so convolution with hΩ converges uniformly to f for continuous functions vanishing at infinity.The argument controls the tails using decay of f and uses uniform continuity on a compact region.Source: §3.5.1 · pp. 23, 24, 26

3.5.2 Proof of 4: The limit-integral representation

• The representation's approximation rate is not fully quantified by Lemma 7, and the proof cannot freely interchange the limits in α and Ω because cosΩ is not in C0(R) when Ω is infinite.These remarks identify technical limitations in the rate analysis and limit ordering.Source: §3.5.1 · p. 26, §3.5.2 · p. 28
• The limit-integral representation constructs RVFL-compatible integrands by approximating cosine terms with scaled activations and using Fourier inversion for an integrable, even kernel.Uniform convergence on compact domains permits exchanging the activation limit with integration under the stated boundedness conditions.Source: §3.5.2 · pp. 27–28

3.5.3 Proof of Lemma 5: Monte-Carlo integral approximation

• The Monte Carlo step samples uniformly over the representation domain, bounds the variance using the boundedness of ϕ and f, and rewrites the sampled terms into RVFL form.The resulting random network matches the Monte Carlo estimator pointwise on the compact domain.Source: §3.5.3 · pp. 29–30
• For activation derivatives in L1(R) ∩ L∞(R), the proof modifies the approximate-identity argument and then reuses the remainder of the original theorem proof.Integration by parts supplies the corresponding cosine and sine representations.Source: §3.5.3 · p. 31

3.5.5 Proof of Theorem 8

• The manifold theorem applies the Euclidean result independently on each chart, allocates local errors and failure probabilities, and combines them into a uniform high-probability bound on M.Choosing ηj=η/|J| distributes the total failure probability across the atlas.Source: §3.5.5 · pp. 32–33

4 DISCUSSION

• RVFL training learns only output weights while fixing randomly selected input weights and biases, making the architecture computationally efficient for repeated retraining.The paper also motivates the analysis because random weights are commonly used to initialize neural-network training.Source: §4 · p. 33
• The paper corrects technical errors in Igelnik and Pao and proves a slightly weaker asymptotic result, then adds explicit high-probability node-count guarantees and manifold extensions.These results revise the theoretical basis for practical RVFL use.Source: §4 · pp. 33–34
• The required node count has superexponential dependence on ambient dimension N, which the authors identify as likely an artifact of their proof methods.They suggest refined limit-integral constructions and restricted function classes as directions for improving this dependence.Source: §4 · p. 34
• The paper does not analyze learning output weights or robustness to noise in training data, leaving provable robustness of RVFL training as future work.This omission limits the current treatment of practical noisy-training behavior.Source: §4 · p. 34