PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse Federated Echo State Networks for Time-Series Classification via Smooth-L1 Regularization

Gengcheng Lyu, Manuel Röder, Frank-Michael Schleif

arXiv:2610.21742v1cs.LG

TL;DR

Dense ESN readouts increase model storage and inference costs, while L1-based sparse optimization does not provide the smooth curvature needed for the paper’s federated second-order update. The paper uses a twice-differentiable Smooth-L1 readout penalty for centralized training and federated Newton aggregation. Across seven classification benchmarks, it reports sparse readouts with accuracy close to dense baselines and convergence in a few federated rounds.

  • Problem

    Dense ESN readouts raise storage and inference costs, while L1’s nondifferentiability complicates smooth, second-order federated optimization.

    Source: §1 · p. 2

  • Method

    The paper trains ESN readouts with a twice-differentiable Smooth-L1 penalty and uses client gradients and Gauss-Newton Hessians for second-order server updates.

    Source: §1 · p. 3, §2.2 · p. 5, §4 · p. 8

  • Results

    77–95% readout sparsity centrally and 75–95% federally, with accuracy close to dense baselines under one fixed shrinkage threshold across seven classification tasks.

    Source: §1 · p. 3, §7 · p. 19

  • Takeaways & Limitations

    The sparse readouts reduce model storage, inference cost, and server-to-client broadcast, while the federated Newton update converges in a few communication rounds.

    Source: §7 · p. 19

    Federated experiments use uniform IID client partitions; the paper does not claim the reported behavior transfers to non-IID data.

    Source: §6.4 · p. 19

Abstract

from arXiv · show

Echo State Networks (ESNs), as lightweight implementations of reservoir computing, provide efficient solutions for temporal data processing in resource-constrained environments. While standard ESNs offer computational efficiency, their dense readout layers inflate the stored model and the per-sample inference cost. This paper introduces a Smooth-L1 regularized ESN (SL1ESN) that promotes sparsity in the readout layer while preserving differentiability for gradient-based optimization. The method employs a twice-differentiable Smooth-L1 approximation compatible with Newton's method, which we exploit both for centralized training and for a second-order (Newton-step) federated aggregation scheme. Under a single fixed shrinkage threshold shared by every dataset and both settings, experiments on seven time-series classification benchmarks show that SL1ESN learns readouts that are $77$--$95\%$ sparse in the centralized setting and $75$--$95\%$ sparse in the federated one, while staying within $0.7$ accuracy points of the strongest dense baseline on five of the seven benchmarks in the centralized setting and within $1.6$ points on the same five in the federated one, and the second-order federated update converges in only a few rounds. The resulting sparsity compresses the stored model, the per-sample inference cost, and the server-to-client broadcast by a comparable factor. These results demonstrate the practical value of sparse readouts for edge computing and IoT deployments.

1 Introduction

Dense ESN readouts raise storage and inference costs, while L1-based sparse solvers lack the smooth, aggregatable updates needed for the paper’s federated setting. The paper introduces Smooth-L1 readout training and a second-order federated extension, reporting high sparsity with accuracy close to dense baselines.

1 Introduction

  • Dense ESN readouts increase memory and inference costs, motivating compact models for resource-constrained and federated deployments.ESNs train only the readout while keeping the recurrent reservoir fixed, but standard readouts connect every reservoir neuron to every output unit.Source: §1 · p. 2
  • L1 readout sparsity is difficult to aggregate federally because LARS and coordinate descent use sequential updates and discrete supports, while L1 has no full-objective Hessian at zero.Averaging models with different supports can lose sparsity, and the nonsmooth penalty provides no client-computable Hessian for second-order aggregation.Source: §1 · pp. 2–3
  • Smooth-L1 approximates L1 with a twice-differentiable penalty, enabling gradient-based Newton optimization and integration into federated learning.The differentiability is the stated basis for using second-order optimization.Source: §1 · p. 3
  • FedSL1ESN has clients return local gradients and Gauss-Newton Hessians for damped server Newton steps, converging in a few rounds and producing a sparse readout.The sparse global readout reduces storage, inference cost, and server-to-client broadcast.Source: §1 · p. 3
  • 77–95% readout sparsity centrally and 75–95% federated, with accuracy close to dense baselines under one fixed shrinkage threshold.The evaluation covers seven time-series classification tasks from the UCR and UCI archives.Source: §1 · p. 3

2 Related Work

Related work spans fixed-reservoir ESNs, sparsification methods, sparse recovery, and federated reservoir computing. The paper positions Smooth-L1 readout sparsification as a route to second-order federated aggregation, rather than as a claim of superior centralized accuracy.

2.1 Reservoir Computing and Echo State Networks

  • Classic ESNs pair a fixed, randomly initialized recurrent reservoir with a trainable readout, and prior work explores reservoir construction, sparse connections, and broader reservoir-computing architectures.The paper focuses on classic ESNs while framing sparse learning as potentially relevant to the wider reservoir-computing family.Source: §2.1 · pp. 3–4

2.2 Sparsity in Neural Networks

  • Prior neural-network sparsification includes L1 regularization, pruning, and structured sparsity; sparse ESN readouts offer a direct route to compact models, but existing sparse solvers can fit poorly with federated optimization.The paper adopts smoothing because client gradients and Gauss-Newton Hessians of the full objective can be summed for a damped server Newton step; it does not claim better accuracy or convergence than alternatives.Source: §2.2 · pp. 4–5

2.3 Sparsity in Signal Analysis and Inverse Problems

  • Compressed-sensing guarantees rely on incoherence that reservoir states violate, while correlated features make selected supports unstable and the paper’s goal is compression, not support recovery.Discrete supports and the lack of additive gradients or Hessians also make L0 and greedy methods unsuitable for the second-order aggregation used here.Source: §2.3 · p. 5

2.4 Federated Reservoir Computing

  • Federated reservoir-computing methods include shared sufficient statistics, unsupervised reservoir adaptation, and incremental anomaly detection; the cited approaches lack explicit readout sparsification for communication efficiency.The paper applies smooth sparsification to the ESN readout to address that gap.Source: §2.4 · pp. 5–6

3 Methodology

The method keeps the ESN reservoir fixed and trains its readout using a smooth approximation to L1. A continuation schedule and Newton optimization support sparsity induction while maintaining a differentiable objective.

3.1 Echo State Network Fundamentals

  • An ESN separates a fixed randomly generated reservoir from a trainable readout, with input projected into reservoir states that evolve through recurrent dynamics.The spectral radius scales the sampled recurrent matrix, while the input scaling, leak rate, and reservoir dimension shape the state representation.Source: §3.1 · pp. 6–7
  • The method tunes spectral radius, leak rate, input scaling, and readout regularization while fixing reservoir dimension at 500 and connectivity at 0.1.The readout matrix is the only trained component; the reservoir weights remain fixed.Source: §3.1 · p. 7

3.2 Smooth-L1 Regularization for ESN Readout

  • Smooth-L1 regularization approximates L1 with a smooth, convex function, addressing L1’s nondifferentiability and L2’s failure to produce exact sparsity.The approximation approaches the absolute-value penalty as α increases.Source: §3.2 · p. 7
  • The regularized objectives use λ for penalty strength and are minimized with Newton’s method and backtracking line search because the loss is twice differentiable.The stated objective uses one-hot targets and softmax class probabilities.Source: §3.2 · p. 7
  • Continuation starts at α0 = 1, multiplies α by κ = 2 after each Newton iteration, and ends at αmax = 5×10^6 to approach L1 while supporting optimization stability.The paper evaluates classification only in the main experimental discussion; regression results are reported in Appendix C.Source: §3.2 · p. 7

4 Proposed Method

SL1ESN replaces closed-form readout estimation with iterative optimization using a twice-differentiable sparsity regularizer, supporting both centralized and federated training. Centralized training uses Newton optimization, while FedSL1ESN aggregates client gradients and curvature for server-side Newton updates.

4 Proposed Method

  • SL1ESN replaces closed-form readout estimation with iterative optimization using a twice-differentiable sparsity regularizer for centralized and federated training.The smooth formulation supports gradient-based optimization and its extension to federated learning.Source: §4 · p. 8, §4.1 · p. 8
  • Smooth-L1 continuation pulls redundant coordinates toward zero, while fixed τ = 0.1 thresholding converts near-zeros to exact zeros after accuracy-only model selection.The same threshold is used across datasets and both settings; sparsity is not part of the model-selection objective.Source: §4 · p. 8

4.1 Centralized Setting

  • Centralized training extracts fixed-reservoir states, then optimizes readout weights with damped Newton steps, Armijo line search, and soft-thresholding.The smoothing parameter advances along a continuation schedule.Source: §4.1 · p. 8
  • Training stops after α reaches αmax and nonzero support changes by less than 0.2 percentage points over P = 8 iterations.This support-stagnation rule terminates training within the fixed iteration budget at unchanged accuracy.Source: §4.1 · p. 9

4.2 Distributed Setting

  • In FedSL1ESN, clients compute local gradients and per-class Hessian blocks without updating weights; reservoir states are cached after the first round.The server uses this local curvature information to update a global readout without sharing raw data.Source: §4.2 · pp. 9–10
  • The server sums client gradients and Hessians, solves damped per-class Newton systems, then thresholds and broadcasts the sparse readout in CSR form as α advances.An Armijo line search selects the step size on the aggregated loss.Source: §4.2 · p. 10

5 Experiments

Experiments compare sparse readouts across seven time-series classification datasets in centralized and federated settings, using a fixed shrinkage threshold. The evaluation also examines accuracy–sparsity behavior, convergence, and hyperparameter sensitivity.

5.1 Datasets

  • The seven UCI and UCR datasets span univariate and multivariate sequences, binary and multiclass tasks, and varied training sizes and sequence lengths.Character uses a fixed 80/20 split because its repository provides no train/test partition.Source: §5.1 · p. 11, Table 1 · p. 11

5.2 Centralized setup

  • Centralized comparisons used a fixed reservoir size NR = 500 and the same damped Gauss–Newton solver across unregularized, L2, and SL1ESN readouts.Hyperparameters were selected by validation accuracy alone, and reported results average ten random seeds.Source: §5.2 · p. 11

5.3 Distributed setup

  • Federated runs used five IID clients for 50 rounds; clients returned local gradients and Gauss–Newton Hessians, and the server applied damped Newton steps with line search and shrinkage.Non-IID partitions were not implemented; the global reservoir was initialized once and held fixed.Source: §5.3 · pp. 11–12

5.4.1 Centralized setting

  • On five of seven datasets, centralized SL1ESN was within 0.7 points of the best dense readout while discarding 77–95% of readout weights.It was most accurate on UCI HAR; Yoga and Strawberry trailed the unregularized readout by 5.8 and 5.7 points.Source: §5.4.1 · pp. 12–13, Table 3 · p. 13

5.4.1 Centralized setting; 5.4.2 Distributed setting

  • 77–95% centralized and 75–95% federated readout sparsity came with accuracy within 0.7 and 1.6 points of the strongest dense baseline, respectively, on five of seven benchmarks.The same fixed threshold τ = 0.1 was used for every dataset and both settings.Source: §5.4.1 · p. 12, Table 3 · p. 13, Table 4 · p. 15

5.4.2 Distributed setting

  • Federated FedSL1ESN was within 1.6 points of the best dense readout on five of seven benchmarks while producing 75–95% sparse global readouts.It was most accurate on Distal.; Yoga and Strawberry trailed the best baseline by 4.5 and 3.6 points.Source: §5.4.2 · p. 15, Table 4 · p. 15
  • Accuracy reached near-peak within a few communication rounds on most datasets, while sparsity largely stabilized only by round 50.Strawberry accuracy climbed until about round 20; round-50 sparsity standard deviations were below 3 points on every dataset.Source: Figure 3 · p. 14, §5.4.2 · p. 15

5.4.3 Ablation and Sensitivity Analysis

  • On UCI HAR, threshold τ kept validation accuracy near 97% through about 80% sparsity, whereas λ = 1 cost about 1.5 accuracy points at matched ∼65% sparsity.Accuracy rose from about 88% at NR = 50 to about 98% at NR = 800, saturating beyond NR ≈ 500; κ left accuracy essentially invariant.Source: §5.4.3 · p. 16, Figure 5 · p. 16

6 Discussion

SL1ESN removes substantial readout redundancy while keeping accuracy close to dense readouts on most classification benchmarks. Federated sparsity reduces model, inference, and broadcast costs, but leaves client uploads unchanged; evidence is bounded by the study’s IID setup and other stated limitations.

6.1 Sparsity–Accuracy Trade-off

  • Readouts were 77–95% sparse centrally and 75–95% sparse federated, while accuracy stayed within 0.7 and 1.6 points of the strongest dense baseline on five of seven benchmarks, respectively.JapaneseVowels remained within 0.6 points centrally and 1.4 points federated of the best baseline.Source: §6.1 · p. 17
  • Yoga and Strawberry trailed the best baseline by 5.8 and 5.7 points centrally and 4.5 and 3.6 points federated; only the federated shortfall partly arose from sparsification.Centrally, the accuracy gap was already present at the dense end; federated accuracy fell by 1.7 points on Yoga and 3.8 on Strawberry at τ = 0.1.Source: §6.1 · p. 17

6.2 Communication Efficiency

  • The sparse global readout reduces stored-model, inference, and server-to-client broadcast costs, but not the dominant client-to-server upload, which carries gradients and Hessians.At NR = 500, the Newton upload is about 250 times a FedAvg weight upload; CSR broadcast savings are 2–10×.Source: §6.2 · pp. 17–18

6.3 Second-Order Optimization Compatibility

  • Clients send local gradients and Gauss-Newton Hessians for the server’s damped Newton update, which reaches near-peak accuracy within one to two rounds on several datasets.The transmitted Hessian is a label-weighted second moment of local reservoir states; when a client has fewer samples than NR, it reveals their spanned subspace to the server.Source: §6.3 · p. 18

6.4 Limitations

  • The study reports sparsity rather than identifying stable reservoir features, since correlated reservoir features allow many equally sparse and accurate supports.The authors leave stable, interpretable support extraction for future work.Source: §6.4 · p. 18
  • Federated experiments use uniform IID partitions; non-IID transfer, larger client populations, system effects, and implemented privacy protections remain untested.The main-text setup uses five clients and 50 rounds; neither secure aggregation nor a differentially private variant is implemented in the study.Source: §6.4 · p. 19

7 Conclusion

The paper presents SL1ESN, a Smooth-L1-regularized ESN readout for classification. It reports highly sparse readouts with accuracy close to dense baselines, while the federated Newton update converges in a few rounds; the claims are limited to classification.

7 Conclusion

  • SL1ESN applies Smooth-L1 regularization to an ESN readout, keeping it amenable to second-order optimization while driving redundant weights toward zero.The method uses a twice-differentiable approximation of the L1 norm.Source: §7 · p. 19
  • Under one shared threshold, readouts were 77–95% sparse centrally and 75–95% sparse federated, with accuracy close to dense unregularized and L2 baselines.The evaluation covers time-series classification.Source: §7 · p. 19
  • In federation, the second-order server update converged in a few rounds, and sparse readouts reduced model storage, inference cost, and server-to-client broadcast.The paper confines its claims to classification; supplementary chaotic-regression tests tolerated far less pruning before rollout accuracy degraded.Source: §7 · p. 19

A Notation

Table 5 provides a notation summary.

A Notation

  • Table 5 is identified as a notation summary.The supplied passage does not give the table’s individual entries.Source: Table 5 · p. 26

B.1 Hyperparameter Settings

Classification runs use a shared shrinkage threshold and fixed continuation and early-stop settings, while hyperparameters are selected by validation-based random search. Newton damping and line-search settings are specified separately.

B.1 Hyperparameter Settings

  • The classification shrinkage threshold is fixed at τ = 0.1 across datasets and centralized and federated settings; the continuation schedule and early-stop parameters are also fixed uniformly.The continuation schedule is α0 = 1, κ = 2, αmax = 5 × 10^6, with P = 8 and εs = 0.2 percentage points.Source: §B.1 · p. 26
  • Centralized model selection uses stratified 80/20 fitting and validation splits, leaves test data untouched, and scores uniform random-search configurations by validation accuracy.All three penalties receive the same per-dataset search budget.Source: §B.1 · p. 26
  • Federated search inherits centralized reservoir hyperparameters and redraws only λ, using 200 configurations for FedSL1ESN and 100 for each baseline per dataset.Selected λ values are listed by dataset in the passage.Source: §B.1 · pp. 26–27
  • Newton damping is increased by factors of ten until the system is solvable and the step is bounded; Armijo backtracking uses c1 = 10^-4 and factor 0.5.The centralized budget is 200 Newton iterations and the federated budget is 50 rounds.Source: §B.1 · p. 27
  • All runs use a reservoir size of NR = 500.The value is stated in the Table 6 caption.Source: Table 6 · p. 27

C Regression on Chaotic Benchmarks

On two chaotic-system regression benchmarks, SL1ESN is competitive in prediction quality, but accuracy-selected sparsity varies sharply by system. Forcing greater sparsity can sacrifice the long-horizon prediction that matters for chaotic rollouts.

C Regression on Chaotic Benchmarks

  • The appendix applies the centralized damped-Newton Smooth-L1 readout to Mackey-Glass and Lorenz63, selecting hyperparameters by valid prediction time across five seeds.The shrinkage threshold is set to τ = 10^-5, so reported sparsity comes from the Smooth-L1 penalty alone.Source: C Regression on Chaotic Benchmarks · p. 27
  • Mackey-Glass SL1ESN reaches RMSE 0.118 and VPT 377, while Lorenz reaches RMSE 10.14 and VPT 148, the best RMSE among regularized readouts.Mackey-Glass VPT trails only the unregularized readout; Lorenz VPT trails the L2 and unregularized readouts.Source: C Regression on Chaotic Benchmarks · p. 27, Figure 6 · p. 28, Table 7 · p. 28
  • The accuracy-selected readout is 46% sparse on Mackey-Glass but only 0.2% sparse on Lorenz; beyond roughly 70% sparsity, long-horizon predictability collapses.Mackey-Glass reaches 65% sparsity at VPT 107, and Lorenz reaches 36% at VPT 69; above roughly 70%, Mackey-Glass VPT falls below 10 and Lorenz VPT is at most 5.Source: C Regression on Chaotic Benchmarks · p. 28
  • L1 reaches 91–93% sparsity on these systems, but its valid prediction time is below SL1ESN on both; the paper confines its sparsity claims to classification-style tasks.The proposed explanation is that chaotic prediction depends on many weakly weighted, cross-dimensional reservoir couplings that magnitude pruning can remove.Source: C Regression on Chaotic Benchmarks · pp. 28–29

D Sensitivity to the Number of Federated Clients

Across client counts from 2 to 50, federated test accuracy and readout sparsity remain broadly stable under the tested IID protocol. The reported sweep uses 50 rounds and the shared threshold τ = 0.1.

D Sensitivity to the Number of Federated Clients

  • The sweep fixes 50 communication rounds, IID partitioning, τ = 0.1, and ten random seeds while varying K across 2, 3, 5, 10, 20, and 50.The main-text federated experiments use K = 5.Source: D Sensitivity to the Number of… · pp. 29–30
  • Across K = 2 to 50, every dataset's accuracy varies by at most 2.6 points, while accuracy is within about one point on five datasets.At K = 50, each JapaneseVowels client has about five training sequences, yet the global readout reaches 94.8%, indistinguishable from K = 5.Source: Figure 8 · p. 29, D Sensitivity to the Number of… · p. 30
  • Under IID partitioning, summed local gradients and Gauss-Newton Hessians make the data-fidelity contribution independent of sample division, leaving K to affect regularization and random variation.Each client's local loss includes its own copy of the Smooth-L1 term, implicitly scaling the aggregated penalty with K.Source: D Sensitivity to the Number of… · p. 30
  • Readout sparsity stays in the K = 5 band with no systematic client-count trend, including 73.9–80.2% on UCI HAR and 94.3–98.0% on JapaneseVowels.The tested sweep retains IID partitioning; behavior under heterogeneous client distributions remains future work.Source: Figure 8 · p. 29, D Sensitivity to the Number of… · p. 30
Loading 2610.21742v1…