PaperlayerView exact revision on arXiv

Source-linked AI summary

LAIR-Net: Leaky Alignment-Impulse Residual Networks for Tabular Regression

Rahul Goswami, Aryan Bhambu, Bittu Karmakar

arXiv:2610.11538v1stat.MLcs.LG

TL;DR

Deep randomized networks typically stack fixed random transformations without target-aware control of hidden-state evolution. LAIR-Net mixes a learned anchor into hidden states through leaky residual transitions; across 23 benchmark datasets it achieves the best average ranks among randomized and conventional models.

  • Problem

    Deep randomized networks add depth through fixed random transformations without target-aware control of how hidden states evolve.

    Source: §1 · p. 1

  • Method

    LAIR-Net combines a separately learned target-aware anchor with leaky residual transitions and fixed random transformations, with closed-form layerwise readouts.

    Source: §1 · p. 1

  • Results

    Across 23 benchmark datasets, LAIR-Net has the best average ranks among eight randomized models (1.96) and twelve conventional models (3.26).

    Source: §7 · p. 8

  • Takeaways & Limitations

    Controlled experiments attribute most improvement to the anchor, with benefits depending on learnable nonlinear structure and diminishing for nearly linear targets or dominant noise.

    Source: §7 · p. 8

    The homogeneous state-perturbation corollary concerns a comparison not encountered in the implemented algorithm, which fixes the initial hidden state at zero.

    Source: §4 · p. 6

Abstract

from arXiv · show

Deep randomized models fix hidden-layer parameters through random initialization and learn only closed-form readouts, typically adding depth by stacking random trans formations without target-aware control of hidden-state evolution. We propose LAIR Net, the Leaky Alignment-Impulse Residual Network, which mixes a shallow learned anchor into each hidden state through a leaky residual transition. We derive a depth uniform bound on input-perturbation sensitivity and use controlled simulations to attribute gains over a randomized baseline to the anchor rather than recursion or added capacity. Benefits emerge when a nonlinear target structure is learnable at the available noise level and diminish for nearly linear targets or dominant noise. Across 23 benchmark datasets, LAIR-Net achieves the best average rank among eight randomized networks and twelve conventional models, with relative performance associated with the same nonlinear-structure and noise quantities identified in simulation.

1 INTRODUCTION

LAIR-Net adds a target-aware anchor and leaky transitions to randomized networks, alongside conditional stability analysis. It is evaluated against 18 competing methods on 23 regression datasets.

1 INTRODUCTION

  • LAIR-Net combines a target-aware anchor from a separately trained shallow network with leaky residual updates and closed-form layerwise ridge readouts.The architecture retains fixed hidden width across depth.Source: §1 · pp. 1–2
  • Under bounded inputs, anchors, and realized weights, a Lipschitz activation, and convex mixing coefficients, the analysis establishes finite-depth boundedness and conditional depth-uniform stability.When the effective propagation factor is below one, initial-state differences attenuate geometrically; the analysis also bounds anchor deviation and establishes readout uniqueness for positive ridge regularization.Source: §1 · p. 1
  • Across 23 regression datasets, LAIR-Net achieves the lowest average MSE rank (4.22), compared with 6.70 for the next-best multilayer perceptron.All pairwise Wilcoxon comparisons are significant after Holm adjustment (p < 0.05).Source: §1 · pp. 1–2
  • The study evaluates LAIR-Net against 18 baselines on 23 regression datasets and uses controlled perturbation experiments to examine its sensitivity and gains.The introduction lists the architecture, stability analysis, controlled experiments, and predictive evaluation as its main contributions.Source: §1 · p. 2

2 RELATED WORK

Prior deep randomized networks emphasize constructing and reusing representations across depth, while leaky updates regulate state change. LAIR-Net combines controlled transitions with repeated guidance from a target-aware representation.

2 RELATED WORK

  • Deep RVFL and ensemble variants build representations across layers and aggregate depth-specific readouts, while skip-connected edRVFL reuses earlier representations.Skip connections expand the information supplied to later layers but do not directly constrain transitions between consecutive states.Source: §2 · p. 2
  • Residual pathways preserve information between successive transformations, but do not determine how strongly a new transformation replaces the previous representation.In randomized networks, hidden transformations remain fixed, so state evolution depends on the realized random mappings.Source: §2 · p. 2
  • Leaky-integrator updates regulate state change by balancing newly generated and retained states, but do not provide a supervised reference derived from the target.Source: §2 · p. 2
  • LAIR-Net combines leaky state control with a target-aware anchor at every depth, modifying the hidden-state update rather than merely expanding the representations supplied to later layers.The leaky update controls the new random representation's contribution relative to the preceding state; the alignment impulse adds the supervised anchor.Source: §2 · p. 2

3 PROPOSED METHOD

LAIR-Net constructs hidden states using a learned target-aware anchor, leaky residual transitions, and fixed random transformations. It fits a ridge readout at each depth and aggregates the resulting predictions.

3.1 Building the hidden state

  • The method uses a shallow learned alignment network to provide a target-aware reference for hidden states, rather than as the final predictor.Only its hidden representation is retained as the anchor supplied across the randomized hierarchy.Source: §3.1 · p. 3
  • Each randomized layer receives the original covariates and only the immediately preceding hidden state, keeping its effective input dimension fixed as depth increases.The original inputs therefore remain directly available to every layer.Source: §3.1 · p. 3
  • At each depth, a fixed random transformation creates a new nonlinear representation, and a leaky update balances it against the preceding hidden state.The leakage coefficient γ controls the contributions of the new response and the previous state.Source: §3.1 · p. 3
  • An alignment impulse then mixes the leaky state with the same learned anchor at every depth, with α controlling the anchor's contribution.At α = 0 the recursion is leaky randomized; at α = 1 every hidden state equals the anchor.Source: §3.1 · p. 4

3.2 From hidden states to a prediction

  • A separate ridge readout at each depth combines the original covariates with that layer's hidden state, preserving direct input access with fixed readout width.The random hidden weights are not optimized; layerwise output weights are obtained by closed-form ridge regression.Source: §3.2 · p. 4
  • LAIR-Net aggregates predictions from its depth-specific readouts to produce the final output.The architecture applies the same anchor throughout the hierarchy and combines layerwise predictions using an aggregation function.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

4 THEORETICAL ANALYSIS

The analysis bounds how perturbations propagate through LAIR-Net’s hidden states, conditional on fixed random weights. It also establishes state boundedness and readout uniqueness, while clarifying the assumptions and limits of the stability and aggregation results.

Assumptions

  • The sensitivity bound is conditional on fixed random weights and assumptions including a Lipschitz activation and convex mixing coefficients.The bounded-state result additionally requires bounded inputs and anchors; that input-and-anchor assumption is not needed for the perturbation-difference bound.Source: §4 · p. 5

Input-perturbation sensitivity

  • When the effective propagation factor ρ is below one, input-perturbation sensitivity is bounded uniformly over network depth, with contributions from both the direct input path and the alignment anchor.If the alignment mapping is Lipschitz, the bound is proportional to the input difference.Source: §4 · p. 6
  • With no alignment and no leak, ρ = c1; if this factor exceeds one, the bound grows with depth rather than remaining uniform.This describes the scope of the bound, not the behavior of any particular trajectory.Source: §4 · p. 6

Incremental stability

  • When ρ < 1, differences between trajectories with the same input, anchor, and weights but different initial states decay geometrically with depth.The implemented algorithm instead starts every input from the zero hidden state, making the input-perturbation bound the directly relevant result.Source: §4 · p. 6

Bounded hidden state

  • The hidden state is uniformly bounded over layers under the bounded-state theorem’s assumptions.An unbounded state could make the readout ill-conditioned, even if perturbations remain controlled.Source: §4 · pp. 6–7

Aggregation

  • Mean aggregation has variance σ²/L only when layerwise errors are pairwise uncorrelated with common variance; LAIR-Net’s shared anchor correlates errors, reducing that practical variance-reduction benefit.For median aggregation, if more than half of layerwise errors have magnitude at most τ, the aggregate error also has magnitude at most τ.Source: §4 · p. 7

5 EXPERIMENTS

The experiments compare LAIR-Net with randomized networks and conventional models. Against randomized networks, it has the best reported average rank, though Nemenyi comparisons do not separate it from every baseline.

5 EXPERIMENTS

  • The experiments address two comparisons: whether the anchor improves the randomized recursion and whether LAIR-Net is competitive with conventional models outside that family.Protocol, dataset details, and extended results are reported in Appendix C.Source: §5 · p. 7

5.1 Comparison with randomized networks

  • LAIR-Net attains an average rank of 1.96 among eight randomized models and the lowest mean error on 13 of 23 datasets; Holm-corrected tests reject every baseline, with largest adjusted p-value 0.0045.The comparison covers RVFL, dRVFL, edRVFL, edRVFL-SC, Fuzzy RVFL, and bagged and boosted RVFL.Source: §5.1 · p. 7
  • Nemenyi comparisons separate LAIR-Net from five randomized baselines, but not from edRVFL-SC or boosted RVFL.Figure 4 shows mean ranks and intervals across 23 datasets; lower ranks are better.Source: §5.1 · p. 7, Figure 4 · p. 7

6 SIMULATION STUDY

Controlled simulations test when the alignment anchor improves on a randomized baseline, while benchmark analyses examine whether related signal and noise measures track relative performance. The reported gains are strongest when nonlinear structure is learnable at the available noise level.

Controlled perturbation experiments

  • At α = 0.5, LAIR-Net overtakes edRVFL in low noise once nonlinear structure is present, but not within the tested nonlinear-share grid at the highest noise level.The edRVFL-to-LAIR-Net test-error ratio rises with stronger nonlinear signal and falls as noise increases.Source: §6 · p. 7
  • Removing the supervised anchor leaves error within 3.7% of edRVFL across 15 tested cells; capacity and untrained-anchor controls do not close the anchor-associated gap in the high-signal regime.Validation-based selection of α brings test error within 1.8% of the grid-best value in every cell and reduces losses to edRVFL from nine of fifteen cells to one of fifteen.Source: §6 · pp. 7–8
  • At 250 training samples, the direction of the simulation pattern persists, but effect sizes shrink and only low-noise gains remain clearly separated.The best setting remains ahead of edRVFL in every cell.Source: §6 · p. 8

Benchmark associations

  • Across 23 benchmark datasets, the nonlinearity proxy correlates with LAIR-Net’s log error ratio against the best randomized baseline at −0.598 (p = 0.003), while the noise proxy correlates at +0.425 (p = 0.043).These are observational associations, not causal evidence.Source: §6 · p. 8

7 CONCLUSION

LAIR-Net combines a target-aware alignment anchor with leaky residual transitions and ranks best across the evaluated randomized and conventional models. Controlled simulations attribute its advantage mainly to the anchor, with benefits depending on nonlinear structure and noise.

7 CONCLUSION

  • LAIR-Net achieves the best average ranks among eight randomized models (1.96) and twelve conventional models (3.26).Holm-adjusted tests favor it over every baseline; Nemenyi tests separate it from five randomized and six conventional baselines.Source: §7 · p. 8
  • The conclusion flags smaller gains with limited samples, unresolved high-dimensional mixed-feature behavior, vulnerability to out-of-range inputs, and several-fold higher training cost than edRVFL.Theorem 4.4 applies only under ρ < 1 and characterizes propagation across depth rather than explaining empirical performance.Source: §7 · p. 8

A SIMULATION STUDY

  • Simulations independently vary nonlinear signal share and noise while matching model settings, isolating the components associated with differences from randomized baselines.Each comparison uses 1,000 training and 1,000 test samples per seed, ten seeds, and fixed rather than tuned hyperparameters.Source: A SIMULATION STUDY · p. 11

A.1 Where the advantage appears

  • At α = 0.5, LAIR-Net overtakes edRVFL for κ ∈(0, 0.25] at σ = 0.1 and κ ∈(0.5, 1.0] at σ = 0.5, but not within κ ≤1 at σ = 1.0.The median edRVFL-to-LAIR-Net test-error ratio rises with κ and falls with σ; Table 2 and Figure 5 show the comparisons and crossovers.Source: §A.1 · p. 11, Table 2 · p. 11, Figure 5 · p. 12

A.2 Which component produces the difference

  • Across fifteen cells, the recursion without an anchor differs from edRVFL by at most 3.7%, while the anchor term reaches 2.889 at κ = 1.0, σ = 0.1.The authors attribute essentially all of the difference to the alignment anchor, with little contribution from randomized recursion alone.Source: §A.2 · p. 12, Figure 6 · p. 13
  • Giving randomized baselines up to eight times LAIR-Net’s hidden width still leaves the best baseline at 3.111 times its error in the high-signal cell.An untrained anchor of identical width and distribution buys nothing at tested sample sizes, supporting target-aware information rather than generic shrinkage as the relevant distinction.Source: §A.2 · p. 12
  • Validation selection places error at 1.000–1.018 times the best grid setting; it reduces losses from nine of fifteen cells with fixed α = 0.5 to one selected-cell loss.The remaining loss is 1.1% on the purely linear high-signal problem; selected and best α have rank correlation 0.838.Source: §A.2 · pp. 12–13
  • At N = 250, effect sizes shrink and only the low-noise advantage remains clearly separated; across three capacity settings, the qualitative pattern holds but crossover locations shift.The σ = 0.5 crossover appears within κ ∈(0.25, 0.5] at the wider setting and not at the narrower one.Source: §A.2 · p. 13

A.3 Does the account transfer to real data

Across benchmark datasets, relative LAIR-Net performance is associated with proxies for nonlinear reward and noise in the directions predicted by simulation. The authors caution that these observational associations do not establish causation and note an out-of-support dataset case.

A.3 Does the account transfer to real data

  • Rank correlation between bκ and log error ratio is −0.598 (p = 0.003), while correlation with bσ is +0.425 (p = 0.043).Above the median bκ, LAIR-Net improves on the best randomized baseline on ten of eleven datasets, compared with three of twelve below it.Source: §A.3 · p. 13
  • These are observational associations, not causal findings; the proxies reuse outcome error values and dataset properties covary with nonlinearity.The authors interpret the results as consistency between the simulation account and benchmark behavior, not proof that nonlinear structure causes improvement.Source: §A.3 · p. 13
  • Gas is an out-of-support case: a test value for one feature lies far outside its training range, affecting models fitted on standardized inputs.Excluding gas strengthens the reported correlations, but the authors retain the with-gas correlations as headline values.Source: §A.3 · pp. 13–14

B PROOFS

The proofs establish conditions for bounded hidden states, controlled perturbation propagation, and unique ridge readouts, then describe the benchmark protocol and dataset provenance.

Proof of Theorem 4.4

  • When ρ < 1, the Theorem 4.4 perturbation bound is uniform over depth, and Corollary 4.5 gives geometric decay for trajectories sharing the same input and anchor.The bound depends on input and anchor differences; the corollary sets both to zero.Source: Proof of Theorem 4.4 · p. 15, Proof of Corollary 4.5 · p. 15

Proofs: associated figures

  • Figures 8–10 show selected α versus best available α and its error cost, alignment at N = 250 with smaller effects and clearest low-noise separation, and LAIR-Net relative MSE versus bκ.In Figure 10, points below zero favour LAIR-Net; gas is marked off-scale.Source: Figure 8 · p. 15, Figure 9 · p. 16, Figure 10 · p. 17

Proof of Theorem 4.6

  • When ρ < 1, Theorem 4.6 bounds hidden-state norms uniformly over layers under its stated boundedness and Lipschitz assumptions.The proof obtains a layerwise recursion with a bounded additive term.Source: Proof of Theorem 4.6 · p. 16

Proof of Proposition 4.7

  • A positive ridge parameter yields a strictly convex readout objective and a unique stationary point, whose normal equations give Equation (8).The proof states that the coefficient matrix is invertible.Source: Proof of Proposition 4.7 · p. 17

Proof of Proposition 4.8

  • With pairwise uncorrelated errors of common variance, the proof simplifies the variance of their average; it also bounds the median error by τ when more than half the errors lie within ±τ.The supplied variance expression is incomplete, so no formula is reported.Source: Proof of Proposition 4.8 · pp. 17–18

C.1 Protocol

  • The benchmark evaluates 23 public regression datasets over ten folds, with preprocessing fitted on training data and test MSE computed after predictions return to the original target scale.Hyperparameters are selected per dataset and fold by training-split cross-validation; deep randomized models use L = 20 and median aggregation.Source: §C.1 · p. 18
  • The comparison protocol uses within-family ranks and paired Wilcoxon tests with Holm correction; pooling all nineteen models raises the critical difference to 5.84 and separates LAIR-Net from nine baselines in the weaker half.The pooled test separates none of the nine models immediately below LAIR-Net in rank.Source: §C.1 · p. 18

C.2 Datasets

  • Dataset provenance distinguishes package-defined benchmark files from original data sources, and the gas-sensor benchmark uses a preprocessed regression subset of a raw classification drift dataset.The Boston Housing data is retained as a legacy dataset; the pendulum row remains tied to benchmark lineage and package source unless an authoritative generator is identified.Source: §C.2 · p. 18

C.3 Comparison with conventional models

LAIR-Net ranks first among the conventional models, while the variant study distinguishes the supervised anchor from alternative impulses and tests other configuration changes that do not isolate single mechanisms.

C.3 Comparison with conventional models

  • LAIR-Net ranks first among twelve conventional models at 3.26, ahead of the multilayer perceptron at 3.87 and random forest at 4.96; Holm-adjusted tests reject every baseline.The largest adjusted p-value is 0.0389; Figure 11’s Nemenyi comparison separates LAIR-Net from six of eleven baselines, while five remain unseparated despite lower ranks for LAIR-Net.Source: §C.3 · p. 18, Figure 11 · p. 19

D ABLATION AND MECHANISM-VARIANT STUDY

  • The variant study compares complete LAIR-Net with six controlled variants, but two alter multiple mechanisms and cannot support attribution to a single component.The study changes alignment, initialization, random-weight construction, and readout or aggregation mechanisms.Source: D ABLATION AND MECHANISM-VARIANT STUDY · p. 19

D.1 Reference configuration: LAIR-Net

  • The reference LAIR-Net starts at zero, injects the target-aware SLAN anchor at every depth, fixes sampled hidden weights, and uses layerwise ridge readouts with median prediction aggregation.The comparison variants retain input recursion, layerwise readouts, and median aggregation unless stated otherwise.Source: §D.1 · pp. 19–20

D.2 Alignment ablations

  • Alignment ablations remove the supervised anchor, replace it with a target-independent random impulse, or reuse a running-best hidden state, distinguishing target-aware reference effects from persistence alone.The fixed random impulse is recomputed from each sample’s covariates but does not change across layers; the running-best state is selected by training LOOCV and replayed at prediction without test targets.Source: §D.2 · p. 20

D.3 Initial-state and orthogonal-weight variants

  • LAIR-SLANInit changes only the initial condition while retaining repeated alignment, whereas LAIR-Ortho changes both initialization and random-weight geometry, so its comparison does not isolate orthogonality.LAIR-OrthoBMA additionally changes ridge penalties and replaces median aggregation with LOOCV-weighted aggregation, jointly changing two mechanisms.Source: §D.3 · pp. 20–21

D.4 Results within the variant family

Within the variant family, zero-initialized LAIR-Net has the strongest average-rank consistency, while SLAN initialization wins on more individual datasets; the accompanying search protocol uses training-fold validation.

D.4 Results within the variant family

  • LAIR-Net has the lowest average rank, 2.565, while LAIR-SLANInit records the most dataset wins and the second-lowest average rank.The comparison indicates stronger full-suite consistency for zero initialization despite more individual wins for SLAN initialization.Source: §D.4 · p. 21
  • The gas-sensor dataset is an exception: LAIR-Net ranks second among variants but has much larger MSE than LAIR-SLANInit, even though both repeatedly inject the same SLAN anchor.The passage frames this as evidence that initial condition can matter strongly for this dataset, not as evidence against repeated alignment.Source: §D.4 · p. 21

E BAYESIAN OPTIMISATION AND SEARCH SPACE

  • The search uses a tree-structured Parzen estimator for mixed continuous, integer, and categorical hyperparameters, minimizing five-fold training-split cross-validated MSE before refitting on the full training split.The held-out test split is not used in hyperparameter search.Source: E BAYESIAN OPTIMISATION AND SEARCH SPACE · pp. 21–22

E.1 Searched hyperparameters

The searched spaces and fixed settings define how the benchmark comparisons were tuned, while several baseline constraints and optimization details qualify exact reproduction and interpretation.

E.1 Searched hyperparameters

  • Conventional models use the same estimator, number of trials, and nested protocol as the randomized models, while ridge and lasso use internal cross-validation rather than the outer search.The search-space tables list ranges for the conventional and randomized model families.Source: §E.1 · p. 22, Table 7 · p. 25, Table 8
  • Deep randomized models use L = 20 layers and median aggregation, with shared within-fold random seeds; the alignment network uses L-BFGS capped at 2,000 iterations.Its activation is matched to that of the randomized layers rather than searched separately.Source: §E.1 · p. 22
  • The conventional comparison is constrained by 20 estimators for three tree ensembles, a 1,000-iteration SVR cap, and internally cross-validated ridge and lasso; the authors expect tree ensembles to improve with more estimators.These settings are held fixed rather than searched.Source: §E.1 · p. 22
  • Exact reproduction may be affected by ridge values near 10^-8, which can make readouts sensitive to linear-algebra implementation, and by alignment runs that hit the L-BFGS iteration cap.In those cases, the reported anchor uses the parameter vector at the cap rather than a converged optimum.Source: §E.1 · p. 22
Loading 2610.11538v1…