PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual-Guided Randomized Neural Networks

Mushir Akhtar, M. Tanveer, Mohd. Arshad

arXiv:2608.28267v1cs.LG

TL;DR

Randomized neural networks rely on a single task-uninformed draw of hidden features, which can misalign representations with supervised residuals and use capacity inefficiently. The proposed residual-guided framework incrementally selects random candidate features by closed-form residual reduction, refits output weights analytically, and preserves model agnosticism and monotonic objective decrease. Across 71 binary and multiclass benchmark datasets, residual-guided variants improve accuracy, stability, and overall ranking performance over their baselines.

  • Problem

    Randomized neural networks' one-shot, task-uninformed hidden-feature construction can produce misaligned or uninformative features, limiting efficient model-capacity use and adaptive selection.

  • Method

    The framework incrementally evaluates randomly generated candidate features by their closed-form residual reduction, retains the most effective features, and preserves closed-form readout training across randomized neural-network architectures.

  • Results

    Across 71 benchmark datasets covering binary and multiclass classification, residual-guided RVFL, ELM, and BLS models consistently improve accuracy, stability, and overall ranking performance over corresponding baselines.

  • Takeaways & Limitations

    Residual-guided construction provides adaptive hidden-layer expansion while retaining the computational efficiency, analytical tractability, and broad applicability of randomized neural networks.

  • Takeaways & Limitations

    The framework addresses the limitations of one-shot random features, whose task-uninformed units may be weakly correlated with supervised residuals and inefficiently use model capacity.

Abstract

from arXiv · show

Randomized neural networks enable fast and analytically tractable training by fixing the input to hidden layer parameters at random and learning the output weights in closed form; however, their performance critically depends on a single uninformed draw of hidden units. This one shot and task uninformed feature construction often leads to redundant representations and suboptimal utilization of model capacity. To address this limitation, we propose a simple and broadly applicable residual guided procedure that greedily constructs the hidden layer using a closed form residual decrease criterion. At each stage, we (i) generate a pool of random candidate units, (ii) score each candidate by the exact reduction it induces in the ridge regularized objective, (iii) select the top k units, and (iv) refit the readout in closed form using the standard design with direct input links. This procedure yields a progressive training process with a guaranteed monotonic decrease of the training objective. The method is model agnostic: only the candidate generation is architecture specific, while the scoring selection refitting loop is shared across models. Extensive experiments on 71 benchmark datasets from the UCI repository, covering both binary and multiclass classification tasks, demonstrate that the proposed residual-guided models consistently outperform their baseline counterparts in terms of accuracy, stability, and overall ranking performance.

I. INTRODUCTION

Randomized neural networks offer efficient closed-form training, but fixed random feature construction can waste capacity and adapt poorly to supervised tasks. The proposed residual-guided framework selects hidden features data-dependently while preserving model flexibility and closed-form learning.

  • Randomized neural networks fix hidden-layer parameters randomly and learn output weights in closed form, avoiding gradient-based optimization.
  • RVFL combines nonlinear random features with direct input-to-output links, while ELM removes those links and BLS expands representations horizontally.
  • Existing extensions still rely on fixed random mappings, so adding hidden units can introduce redundancy and fail to reduce supervised residual error meaningfully.
  • The framework incrementally constructs hidden representations by evaluating random candidates through closed-form residual reduction and retaining only effective features.
  • Across 71 binary and multiclass benchmark datasets, residual-guided models consistently outperform their corresponding baselines.

A. Notation

RVFL combines raw inputs with randomized hidden activations in an augmented design matrix and learns the output readout through ridge-regularized least squares. The resulting solution is available in closed form, with the architecture retaining both linear and nonlinear representations.

  • The hidden matrix G contains activations generated from random affine transformations of input data followed by an elementwise nonlinearity.
  • The augmented design A = [ U G ] concatenates the input matrix U and hidden-feature matrix G for the output readout.
  • RVFL uses direct input-to-output links alongside hidden features, jointly representing linear input structure and nonlinear randomized features.
  • The output weights are learned by solving a ridge-regularized least-squares problem with a closed-form primal or dual solution.

III. MOTIVATION & PROBLEM STATEMENT

The problem is to construct hidden features that improve the ridge objective without abandoning randomized networks’ closed-form training. One-shot random sampling can misalign features with the supervised task, motivating stagewise residual-guided selection.

  • Core Limitations: Randomized networks’ performance depends on a single task-uninformed draw of hidden features despite their efficient closed-form training.
  • Core Limitations: One-shot random mappings can produce weakly task-aligned features, inefficient capacity use, limited adaptive selection, and sensitivity to initialization.
  • Problem Formulation: The formal design problem selects h admissible hidden features that minimize the ridge-regularized objective under a hidden-unit budget.
  • Problem Formulation: Because the admissible feature set is typically continuous, exact optimization becomes a computationally intractable combinatorial feature-selection problem.
  • Our Approach: The proposed alternative samples candidate pools, scores features by closed-form objective reduction, selects effective units, and exactly refits the readout.

IV. METHOD

The method replaces one-shot hidden-feature sampling with supervised, stagewise selection guided by the current residual while retaining RVFL’s augmented design and closed-form readout fitting. Its objective is to construct features that progressively reduce the training criterion.

  • Residual-guided construction replaces a single task-uninformed feature draw with supervised stagewise selection based on current residual error.
  • The selected hidden features are concatenated with direct input links in A = [ U G ], and the readout is learned by ridge-regularized least squares.
  • For a fixed hidden-feature matrix, the training objective is evaluated after fitting the readout, providing the criterion used to assess feature constructions.
  • Learning therefore focuses on constructing a hidden-feature matrix under a prescribed budget that yields a small ridge objective.

B. Residual-Guided Stagewise Construction

The method incrementally expands the hidden-feature matrix instead of generating all features at once. Each stage fits the current model, generates temporary random candidates, evaluates them against the residual, and permanently retains selected features.

  • The hidden-feature matrix G is constructed incrementally in a stagewise manner rather than generated in one step.
  • At each stage, the augmented representation combines the input matrix U with the current hidden-feature matrix Gt, after which the model is fitted by ridge regression.
  • The residual Rt measures the supervised signal not captured by the current representation, and reducing its Frobenius norm decreases the training objective.
  • Each stage generates a temporary pool of M random candidate features, evaluates their residual-reduction ability, selects the most effective subset, and refits the readout.

2) Residual-based scoring:

Residual-based scoring turns candidate selection into a greedy optimization step: candidates are scored by their exact objective decrease, the top-k are selected, and the readout is refitted on the expanded representation.

  • 2) Residual-based scoring:: Each candidate is scored by the exact decrease in the ridge-regularized objective when its coefficient is optimized with existing readout parameters fixed.
  • 2) Residual-based scoring:: The score Δ(g) is a supervised, scale-aware measure of the immediate objective reduction obtained by adding candidate g alone.
  • 3) Selection and refitting:: The algorithm selects the top-k candidates with the largest scores after evaluating the temporary candidate pool.
  • 3) Selection and refitting:: The readout is refitted exactly after selection, and the process repeats until the hidden-unit budget h or a validation-based stopping criterion is reached.
  • 3) Selection and refitting:: Selected features are appended to Gt, producing Gt+1 and the corresponding augmented representation At+1.

C. Monotonicity Guarantee

The stagewise procedure has a monotonicity guarantee: adding selected hidden features cannot increase the minimum ridge-regularized training objective.

  • The ridge-regularized training objective is non-increasing across stages as additional hidden features are incorporated.
  • The formal guarantee is J(Gt+1) ≤ J(Gt) at every stage.
  • Because At is a column submatrix of At+1, the expanded representation can retain the previous solution and assign zero coefficients to newly added columns.
  • The previous solution therefore remains attainable on the expanded representation, while re-optimization can only maintain or lower the objective value.

D. Computational Complexity

The method combines candidate-generation, scoring, and readout-refitting costs, while incremental updates exploit the small number of features added per stage. For fixed pool size M and block size k, total cost grows approximately linearly with h.

  • Generating M candidate features costs O(npM) operations per stage, while scoring them adds O(nMq) and O(nM) costs.
  • Naive readout refitting costs O(n(p + ht)^2) in the primal formulation or O(n^2(p + ht)) in the dual formulation.
  • Incremental readout updates exploit the low-rank modification caused by adding a small block of k hidden features, reducing refitting cost.
  • For T ≈⌈h/k⌉ stages, the overall computational cost grows approximately linearly with the hidden-unit budget h when M and k are fixed.

V. EXPERIMENTAL RESULTS

The study evaluates residual-guided versions of RVFL, ELM, and BLS against their baseline counterparts on 71 UCI classification datasets using accuracy, average rank, and standard deviation.

  • 71 UCI datasets covering binary and multiclass classification tasks are used to compare residual-guided models with their baselines.The evaluated models are RG-RVFL, RG-ELM, and RG-BLS.

A. Performance Analysis

Across 32 binary and 39 multiclass datasets, residual-guided RVFL, ELM, and BLS variants consistently improve accuracy, stability, and ranking performance over their baselines.

  • Residual-guided RVFL, ELM, and BLS consistently improve results across binary and multiclass datasets.Table I reports average accuracy, standard deviation, and average rank for all baseline and residual-guided models.
  • 80.92% to 82.92%, 80.82% to 82.76%, and 81.65% to 82.78% are the binary accuracy gains for RVFL, ELM, and BLS, respectively.The corresponding multiclass averages increase from 73.33% to 75.94%, 73.34% to 75.35%, and 74.96% to 76.20%.
  • 8.91 to 7.88, 8.36 to 7.76, and 6.24 to 5.75 are the binary standard-deviation reductions for RVFL, ELM, and BLS, respectively.The passage also reports reduced variability in multiclass settings.
  • The proposed framework produces systematic improvements in accuracy, stability, and overall ranking performance across all three randomized architectures.The gains are reported for both binary and multiclass classification tasks.
  • Stagewise analysis empirically validates the monotonicity property established for the residual-guided framework.The detailed analysis is provided in the supplementary material.

S.I. EXPERIMENTAL SETUP AND HYPERPARAMETER CONFIGURATION

The experiments use MATLAB with 5-fold cross-validation and grid-based hyperparameter search, while supplementary analyses track stagewise objective and residual behavior and provide dataset-wise tables.

  • All models are evaluated with 5-fold cross-validation combined with grid-based hyperparameter search.Experiments use MATLAB R2023a on an 11th-generation Intel Core i7-11700 system with 16 GB RAM and Windows 11.
  • RVFL and ELM tune λ from {10^-5, 10^-4, . . . , 10^5} and hidden nodes over 5:10:205.BLS separately tunes regularization, feature windows, feature nodes per window, and enhancement nodes.
  • The breast cancer analysis tracks the ridge-regularized objective J(G_t), selected hidden features h_t, and residual energy across stages.These quantities are illustrated in Fig. S.1.
  • The training objective is strictly non-increasing as hidden features are incrementally selected, while residual energy decreases steadily across stages.The observed behavior agrees with the theoretical guarantee and indicates that added features capture previously unexplained supervised-signal components.
  • Dataset-wise results are provided for 32 binary and 39 multiclass datasets in Tables S.I and S.II.The supplementary tables list RVFL, RG-RVFL, ELM, RG-ELM, BLS, and RG-BLS.
Loading 2608.28267v1…