PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual-Guided Randomized Neural Networks

Mushir Akhtar, M. Tanveer, Mohd. Arshad

arXiv:2608.28267v1cs.LG

TL;DR

Randomized neural networks rely on a single task-uninformed draw of hidden features, which can misalign representations with supervised residuals and use capacity inefficiently. The proposed residual-guided framework incrementally selects random candidate features by closed-form residual reduction, refits output weights analytically, and preserves model agnosticism and monotonic objective decrease. Across 71 binary and multiclass benchmark datasets, residual-guided variants improve accuracy, stability, and overall ranking performance over their baselines.

  • Problem

    Randomized neural networks' one-shot, task-uninformed hidden-feature construction can produce misaligned or uninformative features, limiting efficient model-capacity use and adaptive selection.

    Source: §III · p. 2, A. Core Limitations · p. 3

  • Method

    The framework incrementally evaluates randomly generated candidate features by their closed-form residual reduction, retains the most effective features, and preserves closed-form readout training across randomized neural-network architectures.

    Source: §I · pp. 1–2, B. Residual-Guided Stagewise… · p. 4

  • Results

    Across 71 benchmark datasets covering binary and multiclass classification, residual-guided RVFL, ELM, and BLS models consistently improve accuracy, stability, and overall ranking performance over corresponding baselines.

    Source: A. Performance Analysis · p. 5

  • Takeaways & Limitations

    Residual-guided construction provides adaptive hidden-layer expansion while retaining the computational efficiency, analytical tractability, and broad applicability of randomized neural networks.

    Source: §I · p. 2, §VI · p. 6

    The framework addresses the limitations of one-shot random features, whose task-uninformed units may be weakly correlated with supervised residuals and inefficiently use model capacity.

    Source: A. Core Limitations · p. 3

Abstract

from arXiv · show

Randomized neural networks enable fast and analytically tractable training by fixing the input to hidden layer parameters at random and learning the output weights in closed form; however, their performance critically depends on a single uninformed draw of hidden units. This one shot and task uninformed feature construction often leads to redundant representations and suboptimal utilization of model capacity. To address this limitation, we propose a simple and broadly applicable residual guided procedure that greedily constructs the hidden layer using a closed form residual decrease criterion. At each stage, we (i) generate a pool of random candidate units, (ii) score each candidate by the exact reduction it induces in the ridge regularized objective, (iii) select the top k units, and (iv) refit the readout in closed form using the standard design with direct input links. This procedure yields a progressive training process with a guaranteed monotonic decrease of the training objective. The method is model agnostic: only the candidate generation is architecture specific, while the scoring selection refitting loop is shared across models. Extensive experiments on 71 benchmark datasets from the UCI repository, covering both binary and multiclass classification tasks, demonstrate that the proposed residual-guided models consistently outperform their baseline counterparts in terms of accuracy, stability, and overall ranking performance.

I. INTRODUCTION

Randomized neural networks offer efficient closed-form training, but fixed random feature construction can waste capacity and adapt poorly to supervised tasks. The proposed residual-guided framework selects hidden features data-dependently while preserving model flexibility and closed-form learning.

  • Randomized neural networks fix hidden-layer parameters randomly and learn output weights in closed form, avoiding gradient-based optimization.Source: §I · p. 1
  • RVFL combines nonlinear random features with direct input-to-output links, while ELM removes those links and BLS expands representations horizontally.Source: §I · p. 1
  • Existing extensions still rely on fixed random mappings, so adding hidden units can introduce redundancy and fail to reduce supervised residual error meaningfully.Source: §I · p. 1
  • The framework incrementally constructs hidden representations by evaluating random candidates through closed-form residual reduction and retaining only effective features.Source: §I · pp. 1–2
  • Across 71 binary and multiclass benchmark datasets, residual-guided models consistently outperform their corresponding baselines.Source: §I · p. 2

A. Notation

RVFL combines raw inputs with randomized hidden activations in an augmented design matrix and learns the output readout through ridge-regularized least squares. The resulting solution is available in closed form, with the architecture retaining both linear and nonlinear representations.

III. MOTIVATION & PROBLEM STATEMENT

The problem is to construct hidden features that improve the ridge objective without abandoning randomized networks’ closed-form training. One-shot random sampling can misalign features with the supervised task, motivating stagewise residual-guided selection.

Core Limitations

  • Randomized networks’ performance depends on a single task-uninformed draw of hidden features despite their efficient closed-form training.Source: §III · p. 2
  • One-shot random mappings can produce weakly task-aligned features, inefficient capacity use, limited adaptive selection, and sensitivity to initialization.Source: A. Core Limitations · p. 3

Problem Formulation

  • The formal design problem selects h admissible hidden features that minimize the ridge-regularized objective under a hidden-unit budget.Source: B. Problem Statement · p. 3
  • Because the admissible feature set is typically continuous, exact optimization becomes a computationally intractable combinatorial feature-selection problem.Source: B. Problem Statement · p. 3

Our Approach

  • The proposed alternative samples candidate pools, scores features by closed-form objective reduction, selects effective units, and exactly refits the readout.Source: B. Problem Statement · p. 3

IV. METHOD

The method replaces one-shot hidden-feature sampling with supervised, stagewise selection guided by the current residual while retaining RVFL’s augmented design and closed-form readout fitting. Its objective is to construct features that progressively reduce the training criterion.

  • Residual-guided construction replaces a single task-uninformed feature draw with supervised stagewise selection based on current residual error.Source: §IV · p. 3
  • The selected hidden features are concatenated with direct input links in A = [ U G ], and the readout is learned by ridge-regularized least squares.Source: A. Training Objective · p. 3
  • For a fixed hidden-feature matrix, the training objective is evaluated after fitting the readout, providing the criterion used to assess feature constructions.Source: A. Training Objective · p. 3
  • Learning therefore focuses on constructing a hidden-feature matrix under a prescribed budget that yields a small ridge objective.Source: A. Training Objective · p. 3

B. Residual-Guided Stagewise Construction

The method incrementally expands the hidden-feature matrix instead of generating all features at once. Each stage fits the current model, generates temporary random candidates, evaluates them against the residual, and permanently retains selected features.

2) Residual-based scoring:

Residual-based scoring turns candidate selection into a greedy optimization step: candidates are scored by their exact objective decrease, the top-k are selected, and the readout is refitted on the expanded representation.

2) Residual-based scoring:

  • Each candidate is scored by the exact decrease in the ridge-regularized objective when its coefficient is optimized with existing readout parameters fixed.Source: 2) Residual-based scoring · p. 4
  • The score Δ(g) is a supervised, scale-aware measure of the immediate objective reduction obtained by adding candidate g alone.Source: 2) Residual-based scoring · p. 4

3) Selection and refitting:

C. Monotonicity Guarantee

The stagewise procedure has a monotonicity guarantee: adding selected hidden features cannot increase the minimum ridge-regularized training objective.

D. Computational Complexity

The method combines candidate-generation, scoring, and readout-refitting costs, while incremental updates exploit the small number of features added per stage. For fixed pool size M and block size k, total cost grows approximately linearly with h.

V. EXPERIMENTAL RESULTS

The study evaluates residual-guided versions of RVFL, ELM, and BLS against their baseline counterparts on 71 UCI classification datasets using accuracy, average rank, and standard deviation.

  • 71 UCI datasets covering binary and multiclass classification tasks are used to compare residual-guided models with their baselines.The evaluated models are RG-RVFL, RG-ELM, and RG-BLS.Source: §V · p. 5

A. Performance Analysis

Across 32 binary and 39 multiclass datasets, residual-guided RVFL, ELM, and BLS variants consistently improve accuracy, stability, and ranking performance over their baselines.

  • Residual-guided RVFL, ELM, and BLS consistently improve results across binary and multiclass datasets.Table I reports average accuracy, standard deviation, and average rank for all baseline and residual-guided models.Source: A. Performance Analysis · p. 5, Table I · p. 5
  • 80.92% to 82.92%, 80.82% to 82.76%, and 81.65% to 82.78% are the binary accuracy gains for RVFL, ELM, and BLS, respectively.The corresponding multiclass averages increase from 73.33% to 75.94%, 73.34% to 75.35%, and 74.96% to 76.20%.Source: A. Performance Analysis · p. 5
  • 8.91 to 7.88, 8.36 to 7.76, and 6.24 to 5.75 are the binary standard-deviation reductions for RVFL, ELM, and BLS, respectively.The passage also reports reduced variability in multiclass settings.Source: A. Performance Analysis · p. 5
  • The proposed framework produces systematic improvements in accuracy, stability, and overall ranking performance across all three randomized architectures.The gains are reported for both binary and multiclass classification tasks.Source: A. Performance Analysis · p. 5
  • Stagewise analysis empirically validates the monotonicity property established for the residual-guided framework.The detailed analysis is provided in the supplementary material.Source: A. Performance Analysis · p. 5, §VI · p. 6

S.I. EXPERIMENTAL SETUP AND HYPERPARAMETER CONFIGURATION

The experiments use MATLAB with 5-fold cross-validation and grid-based hyperparameter search, while supplementary analyses track stagewise objective and residual behavior and provide dataset-wise tables.

  • All models are evaluated with 5-fold cross-validation combined with grid-based hyperparameter search.Experiments use MATLAB R2023a on an 11th-generation Intel Core i7-11700 system with 16 GB RAM and Windows 11.Source: S.I. EXPERIMENTAL SETUP AND… · p. 8
  • RVFL and ELM tune λ from {10^-5, 10^-4, . . . , 10^5} and hidden nodes over 5:10:205.BLS separately tunes regularization, feature windows, feature nodes per window, and enhancement nodes.Source: S.I. EXPERIMENTAL SETUP AND… · p. 8
  • The breast cancer analysis tracks the ridge-regularized objective J(G_t), selected hidden features h_t, and residual energy across stages.These quantities are illustrated in Fig. S.1.Source: S.II. EMPIRICAL ILLUSTRATION OF… · p. 8, Figure S.1 · p. 11
  • The training objective is strictly non-increasing as hidden features are incrementally selected, while residual energy decreases steadily across stages.The observed behavior agrees with the theoretical guarantee and indicates that added features capture previously unexplained supervised-signal components.Source: S.II. EMPIRICAL ILLUSTRATION OF… · p. 8, Figure S.1 · p. 11
  • Dataset-wise results are provided for 32 binary and 39 multiclass datasets in Tables S.I and S.II.The supplementary tables list RVFL, RG-RVFL, ELM, RG-ELM, BLS, and RG-BLS.Source: S.II. EMPIRICAL ILLUSTRATION OF… · pp. 9–10
Loading 2608.28267v1…