PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep networks for system identification: a Survey

Gianluigi Pillonetto, Aleksandr Aravkin, Daniel Gedon, Lennart Ljung, Antônio H. Ribeiro, Thomas B. Schön

arXiv:2301.12832v1cs.LGcs.SYeess.SY

TL;DR

System identification seeks to predict dynamic-system behavior from input-output data, including when physical or structural knowledge is unavailable. This survey reviews deep-learning and kernel-based modeling approaches and their training and theoretical foundations. It concludes that deep networks can offer more parameter-efficient representations for compositional problems, while also examining overparameterization and generalization.

  • Problem

    System identification must model complex dynamic processes from input-output data, often without structural or physical knowledge, and high-dimensional data pose additional challenges.

    Source: §1 · p. 1

  • Method

    The survey reviews deep architectures for system identification, their optimization and regularization, and connections between neural networks and kernel-based models.

    Source: §1 · p. 3, §8 · p. 35

  • Results

    For compositional problems, deep networks can reduce required parameter scaling from exponential to linear, and the survey reviews overparameterization and double-descent behavior.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Deep networks can serve as prediction-model structures for system identification, while concatenated kernels provide another route to regularized methods.

    Source: §8 · p. 35

    The conventional asymptotic analysis discussed assumes N → ∞ and m < N, so it does not cover parameterizations with m > N.

    Source: §2.2 · p. 5

Abstract

from arXiv · show

Deep learning is a topic of considerable current interest. The availability of massive data collections and powerful software resources has led to an impressive amount of results in many application areas that reveal essential but hidden properties of the observations. System identification learns mathematical descriptions of dynamic systems from input-output data and can thus benefit from the advances of deep neural networks to enrich the possible range of models to choose from. For this reason, we provide a survey of deep learning from a system identification perspective. We cover a wide spectrum of topics to enable researchers to understand the methods, providing rigorous practical and theoretical insights into the benefits and challenges of using them. The main aim of the identified model is to predict new data from previous observations. This can be achieved with different deep learning based modelling techniques and we discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks. Their parameters have to be estimated from past data trying to optimize the prediction performance. For this purpose, we discuss a specific set of first-order optimization tools that is emerged as efficient. The survey then draws connections to the well-studied area of kernel-based methods. They control the data fit by regularization terms that penalize models not in line with prior assumptions. We illustrate how to cast them in deep architectures to obtain deep kernel-based methods. The success of deep learning also resulted in surprising empirical observations, like the counter-intuitive behaviour of models with many parameters. We discuss the role of overparameterized models, including their connection to kernels, as well as implicit regularization mechanisms which affect generalization, specifically the interesting phenomena of benign overfitting ...

1 Introduction

System identification learns dynamic-system descriptions from input-output data, and this survey examines deep learning as a modeling approach for prediction. It connects system-identification practice with deep architectures, optimization and regularization, kernel methods, and theoretical insights about model complexity and generalization.

1 Introduction

  • System identification models dynamics from input-output measurements, and this survey builds a bridge between deep learning and regression-focused identification.It aims to present deep-learning methods pedagogically for researchers with a system-identification background.Source: §1 · pp. 1, 3
  • Deep neural networks expand high-dimensional model search beyond shallow kernel estimators, whose RKHS estimates are finite sums of kernel basis functions.The survey presents deep networks as another way to define the high-dimensional search space.Source: §1 · p. 2
  • The survey covers feedforward NFIR and NARX models [108], fading-memory concepts, and temporal-convolutional, kernel-compositional, recurrent, state-space, and latent-state approaches.It also discusses probabilistic energy-based models [138] and deep variational autoencoders.Source: §1 · p. 3
  • Deep structures require high-dimensional non-convex optimization; the survey discusses first-order stochastic methods and complexity controls such as normalization, dropout, and early stopping.It also covers vanishing and exploding gradients, variance reduction, adaptive scaling, and automatic differentiation.Source: §1 · p. 3
  • For compositional problems, deep networks can reduce required parameter scaling from exponential to linear; the survey also reviews overparameterization, double descent, and implicit regularization.Related topics include links to Gaussian processes and neural tangent kernels, and the role of parameter count in model flexibility and capacity.Source: §1 · p. 4

2 Modeling of dynamical systems

System identification combines a model family, parameter estimation, and validation, with model structure and complexity central to prediction on new data. The section contrasts the classical bias–variance account with deep learning’s potential to achieve greater flexibility with fewer parameters.

2.1 Three main players

  • Identification defines a model family mapping past input-output data to a prediction, estimates its parameters against observed outputs, and validates the resulting model.An unaccepted model may prompt revision of earlier choices, especially its structure.Source: §2.1 · p. 4

2.2 Number of parameters – a blessing and a curse (bias and variance)

  • As parameter count grows, training fit improves but variance can hurt generalization, yielding a typically U-shaped model-error curve and a bias–variance trade-off.The section relates model variance to the number of parameters and observations through m/N.Source: §2.2 · p. 5

2.1 Three main players

  • Residual analysis checks prediction-error independence from itself and the input, while cross-validation evaluates fit on an independent validation dataset.The best validation fit can guide selection among candidate models.Source: §2.2 · p. 5

2.2 Number of parameters – a blessing and a curse (bias and variance)

  • The conventional asymptotic result assumes N → ∞ and m < N, so it does not cover very large parameterizations with m > N, where double descent is possible.This qualifies the traditional conclusion that more parameters necessarily harm generalization.Source: §2.2 · p. 5

2.3 Model flexibility and number of parameters

  • Deep multi-layer structures can create many more piecewise-linear breakpoints with few parameters, offering flexible black-box models with compact parameterizations.A one-layer piecewise-linear model requires m parameters for m breakpoints.Source: §2.3 · p. 6

3 Deep neural networks architectures

This section surveys deep neural network architectures for representing dynamic input-output behavior in system identification, including feedforward, convolutional, recurrent, autoencoder-based latent-variable, and deep regression models. It describes how architectural structures relate to prediction, stability, and training, and how recurrent and regression formulations address temporal dependencies and output uncertainty beyond a fixed Gaussian mean.

3 Deep neural networks architectures

  • Neural networks compose layers to map inputs to predicted outputs; the section develops fully connected, skip/direct, convolutional, and recurrent designs for system identification.A network is called deep when it has a large number of layers.Source: §3 · p. 6

3.1 Fully connected neural networks

  • For fully connected networks, past inputs define FIR models, while adding past measured outputs gives ARX models whose long-term predictions recursively feed estimated outputs back into the model.Output-error networks also use feedback during training; they may better handle output-error noise but can make optimization more challenging.Source: §3.1 · pp. 6–7
  • Stability is not guaranteed by a stable one-step predictor; proposed approaches include fading-memory regularization and structural conditions such as contraction or incremental stability.The section notes that nonlinear stability is multifaceted and that ISS and δISS are difficult to incorporate as priors in system identification.Source: §3.1 · p. 7

3.2 Skip and direct connections

  • Skip connections add a layer’s input to its transformed output, letting deep networks learn deviations from identity and change information gradually; direct connections concatenate input and transformed values.Skip connections can also be viewed as discretized ODEs, motivating neural ODEs for dynamical systems [71] [263].Source: §3.2 · p. 8

3.3 Convolutional neural networks

  • Convolutional layers act as learnable filter banks and, without downsampling, can be equivalent to FIR or ARX fully connected networks; practical CNNs use downsampling to encode longer dependencies.Dilations support sequence-to-sequence prediction at the input sampling frequency [13] [25] [92] [329], while downsampling can cause aliasing; anti-aliasing filters have been explored to improve transformation invariance [330].Source: §3.3 · pp. 8–10

3.4 Recurrent neural networks

  • Recurrent networks recursively compute predictions from past states, but exploding or vanishing gradients can hinder optimization and long-term dependency learning.Proposed modifications include gating, gradient clipping, non-saturating activations [68], altered gradient paths, and orthogonal RNNs [336].Source: §3.4 · p. 10
  • LSTMs and GRUs [81] [84] use gates to control information flow and support long-term dependencies, though these may be unnecessary for exponentially stable or Markovian systems.Gated units have been evaluated for nonlinear system identification, and modifications for that setting have been proposed.Source: §3.4 · p. 11
  • Echo State Networks use randomly generated reservoir dynamics while training only the output readout, enabling efficient fitting and applications to chaotic systems.The reservoir weights are fixed, and the readout can be fit by least squares.Source: §3.4 · p. 11
  • Probabilistic RNNs [109] model an output distribution, such as a normal distribution whose mean and log standard deviation are predicted by the readout, and can be trained by maximum likelihood.The log standard deviation parameterization ensures non-negative variance.Source: §3.4 · p. 11

3.5 Latent variable models

  • Autoencoders encode inputs into a lower-dimensional latent variable and decode a reconstruction; dynamic versions include past inputs or inputs and outputs and can propagate latent states over time, [38].The basic autoencoder is trained by minimizing reconstruction error; the dynamic version reconstructs system outputs.Source: §3.5 · p. 12
  • Variational autoencoders [166] use deep nonlinear networks and an approximate posterior, maximizing an ELBO with reconstruction and regularization terms via the reparameterization trick [166].Sampling from the latent prior enables generation of new data points.Source: §3.5 · p. 12
  • Dynamic VAEs model system outputs from inputs with causally ordered latent states, using temporal networks such as RNNs, temporal convolutions, or fully connected networks; they can be viewed as deep probabilistic state-space models [118].Examples include the Kalman VAE [111] and Variational RNN [85].Source: §3.5 · pp. 12–13

3.6 Formulating and solving regression problems using deep learning

  • Conditional energy-based models use a neural network to score input-output pairs and can represent asymmetric, heavy-tailed, or multimodal output densities, but their normalization constant is intractable.Approximate inference can address the intractable normalization; an initial nonlinear system-identification application is described in [139].Source: §3.6 · pp. 13–14
  • Deep regression approaches include direct and probabilistic regression, confidence-based regression, and regression by classification, whose limitations include fixed distribution forms, difficult design choices, or loss of neighborhood structure [83],, [66].Classification-based predictions remain bounded by the interval of the discretized classes.Source: §3.6 · p. 13

4 Optimization and regularization

The section reviews optimization and regularization methods for deep learning, with the aim of supporting strong out-of-sample prediction amid nonconvex objectives, large datasets, and many parameters. It covers stochastic and proximal methods, variance reduction, adaptive scaling and momentum, and explicit and implicit regularization, including dropout and its ensemble interpretation.

4.1 Optimization Formulation for Deep Learning

  • Deep-learning objectives are non-convex, so general optimization guarantees stationary points rather than local or global minima; evaluation emphasizes generalization and computational efficiency.The nested nonlinearities make the objective non-convex, and ReLU networks yield continuous but nonsmooth objectives.Source: §4.1 · p. 14

4.2 Overview

  • Large datasets and parameter counts make progress per data pass central, favoring stochastic methods that use subsampled data and explicit or implicit regularization.Batch normalization normalizes activations and can act as an architecture-adapted preconditioner to stabilize gradients across scales.Source: §4.1 · p. 15, §4.2 · p. 15

4.3 Steepest Gradient, Prox Gradient, and Stochastic Gradient

  • Steepest and prox-gradient methods guarantee descent, but require full data passes; stochastic gradient uses batches and needs decreasing step sizes for convergence.Without stronger assumptions, steepest descent's minimum observed gradient norm vanishes no slower than ν^-1/2; SGD step-size schedules must decrease without shrinking too quickly.Source: §4.3 · pp. 15–16

4.4 Variance Reduction

  • Variance reduction reuses historical gradient information to reduce stochastic noise, allowing fixed step sizes and faster theoretical convergence, but periodically requires full-gradient evaluations.Proximal variants apply when R is lower semi-continuous with a computable prox; convergence step sizes are problem-dependent but need not decrease.Source: §4.4 · p. 17

4.5 Adaptive scaling and momentum

  • With differently scaled network weights, one uniform step size can suppress information from smaller-scale features in favor of larger-scale ones; ADAM [165] combines momentum and adaptive scaling to speed stochastic gradient descent.Because neural nets are nonconvex, ADAM settings cannot guarantee optimal performance; it can struggle with standard step-size schemes and may require warm-up regimes that start with small step sizes, increase them, then decrease them again [188].Source: §4.5 · pp. 17–18

4.6 Dropout

  • Explicit regularizers impose structure on output weights, with sparsity a flagship example, while dropout is identified as an implicit regularization method.The supplied passage introduces the contrast with explicit regularization but provides no further dropout mechanism or findings.Source: §4.2 · p. 15, §4.6 · p. 18
  • As an implicit regularizer, dropout addresses overfitting and generalization but does not promote a particular type of structure.The survey briefly reviews dropout as the section’s closing topic.Source: §4.6 · p. 18
  • Dropout randomly removes entries of θ during training; in the classic scheme, each weight θ_i is retained with probability p_i and scaled to p_iθ_i at prediction.The prediction-time value is the expected weight.Source: §4.6 · p. 18
  • Dropout can be viewed as continuously training an ensemble of randomly obtained networks, and it can be combined with any structure, regularizer, or training algorithm [311].Analyzing such combinations is challenging; with other strategies, particularly max-norm regularization, dropout can decrease test error across applications and estimate model uncertainty [117].Source: §4.6 · p. 18

5 Deep kernel-based learning

This section explores kernel-based methods in deep architectures, using kernels to encode system properties and extend model expressivity. It covers latent-space, vector-valued, and prediction-network approaches, their estimation and limitations, shared-kernel recursive prediction, and Gaussian-process kernels induced by infinite-width networks for convex kernel estimation.

5.1 Reproducing kernel Hilbert spaces

  • RKHS kernels encode expected system properties through function-space smoothness, with the stable spline/TC kernel penalizing FIR coefficients that do not decay exponentially.The decay-rate parameter α is a kernel hyperparameter estimated from data.Source: §5.1 · p. 19

5.2 Kernel ridge and Gaussian regression

  • Kernel ridge regression has a finite expansion over observed kernel sections, and its estimate matches Gaussian-process posterior estimation when γ = σ2/λ.The representer theorem turns the RKHS variational problem into a linear system involving the kernel matrix; γ balances data fit and regularization.Source: §5.2 · pp. 19–20

5.3 Deep learning using kernels over an intermediate latent space

  • Deep networks can map inputs to a latent space where a kernel models the remaining function, addressing cases where smoothness-oriented kernels struggle with oscillatory or discontinuous dynamics.For f = sin(ex/2), the transformation G(x) = ex/2 yields the smooth function sin(x) in the latent space.Source: §5.3 · pp. 20–21
  • Latent maps can be learned from inputs alone or estimated with deep networks from input-output data, with empirical Bayes tuning network weights and kernel hyperparameters via marginal likelihood.The likelihood’s Occam factor may not prevent overfitting in high-dimensional non-convex optimization; full Bayesian inference can account for weight and hyperparameter uncertainty, but may require costly, delicately tuned MCMC.Source: §5.3 · p. 21

5.4 Deep learning using vector-valued RKHSs

  • Vector-valued RKHS kernels define every layer of a deep architecture, and the concatenated representer theorem reduces estimation to a finite-dimensional but non-convex optimization problem without a closed-form solution.Each layer estimate lies in a subspace of dimension Ndℓ.Source: §5.4 · p. 22

5.5 Deep prediction networks

  • Deep prediction networks recursively apply kernel ridge regression across prediction horizons, propagating earlier predicted outputs as inputs to later layers.The survey reports that this architecture can much improve long-term prediction performance over using the kernel estimator alone.Source: §5.5 · p. 23
  • Deep prediction networks progressively replace observed outputs with predictions up to the desired depth, using a shared kernel to express similarities across layers.Hyperparameters can be estimated by marginal likelihood optimization.Source: §5.5 · p. 23

5.6 Building compositional kernels using limits of deep neural networks

  • A 500-layer deep prediction network using a degree-3 polynomial kernel avoids the instability seen with the same kernel in a single-step model and achieves accurate long-term predictions.The single-step prediction diverges after a few time instants; both methods use the same regularization parameter.Source: Figure 10 · p. 24
  • Infinite-width neural networks induce Gaussian-process kernels; for ReLU activations, the resulting arccosine kernel has a closed form and can be used with convex kernel ridge regression.The construction places priors on network weights and lets layer widths grow to infinity.Source: §5.6 · pp. 23–24
  • The ReLU arccosine kernel favors nonnegative and sparse data representations, distinguishing it from Gaussian and polynomial kernels.Its deep-network-derived features can be implemented with convex estimators such as kernel ridge regression.Source: §5.6 · p. 24
  • Estimates from a large-width neural network trained only in its last layer are quite similar to Gaussian regression using the corresponding infinite-width network kernel.Source: §5.6 · p. 24

6 Theoretical development

This section develops theoretical accounts of overparameterization, including its effects on optimization, generalization, and cases where it may be harmful. It also examines benign overfitting, regularization and robustness to changed inputs, and approximation properties of deep networks, including ReLU networks and functions with compositional structure.

6.1 Infinitely-wide neural networks

  • Infinite-width gradient-flow training is governed by a deterministic neural tangent kernel and yields a kernel-regression-like solution with initialization offsets [151].The mean prediction matches kernel ridge regression for that kernel, while random initialization adds covariance terms.Source: §6.1 · p. 26
  • Infinite width makes optimization more tractable, but may not explain neural-network properties fully; finite ReLU networks can fit in bounded-variation spaces beyond RKHS [236] [238].The cited comparison suggests infinite-width kernel models can struggle with spatially inhomogeneous functions that finite, adaptive models fit better [237].Source: §6.1 · pp. 26–27

6.2 Implicit regularization

  • Factorized gradient descent in matrix completion achieves lower test reconstruction error than the non-factorized formulation despite zero training error, implicitly inducing a low nuclear norm [128].This occurs with d = n, without an explicit rank constraint.Source: §6.2 · p. 27
  • One-hidden-layer networks trained with SGD can gain capacity beyond perfect training fit without harming generalization, suggesting gradient descent regularizes beyond parameter count.The authors associate this behavior with convergence to a minimum-norm solution.Source: §6.2 · p. 27

6.3 Interpolation and double-descent curves

  • In the illustrated double-descent example, test error rises near interpolation but can fall again when parameters exceed data points; the lowest test risk instead occurs at an intermediate parameter norm.In the overparameterized region, the example selects a minimum-norm solution; increasing parameter count reduces its norm.Source: Figure 12 · p. 28, §6.3 · pp. 28–29
  • Theory studies growing feature and sample counts through high-dimensional asymptotics and random-matrix methods, or finite-sample non-asymptotic bounds; covariance structure can qualify overparameterized performance.An isotropic-feature example has better performance in the underparameterized region, while a spiked covariance with sufficiently decaying remaining eigenvalues is associated with good overparameterized performance.Source: §6.3 · pp. 29–30
  • Benign overfitting depends on covariance eigenvalue decay: covariates must align with dominant directions while retaining small components in directions where label noise can be hidden; many DNN interpolators may be tempered.Bartlett et al. [33] characterize conditions for benign overfitting in linear regression, while Mallinar et al. distinguish tempered overfitting from benign and catastrophic overfitting.Source: §6.3 · p. 30

6.4 The risk of overfitting in linear system identification

  • With white-noise validation inputs, carefully tuned regularization often generalizes better across changed input statistics, while pseudo-inverse performance is comparable under low-pass validation inputs.The comparison comes from a 100-run Monte Carlo experiment; the figure displays prediction-fit boxplots for both validation conditions.Source: Figure 14 · p. 31, §6.4 · p. 31

6.5 Approximation properties of deep neural networks

  • Shallow networks are universal approximators but require O(ε^-m) trainable parameters; for compositional functions, deep networks can achieve the same resolution with O((m-1)ε^-2) parameters.The reduction can change complexity from exponential to linear in the input dimension m when the function's compositional structure is known.Source: §6.5 · p. 32
  • For scalar inputs and outputs, ReLU networks can gain breakpoints rapidly with depth without a matching parameter increase, and deep ReLU networks match free-knot linear-spline approximation power with comparable parameters.For special weight and bias choices, the maximum breakpoint count scales as d^L when all L hidden layers have width d.Source: §6.5 · p. 32

7 Applications

The applications show how MATLAB and PyTorch support deep-network system identification, then demonstrate model validation on two nonlinear system examples. In the Silverbox example, the deep cascadeforwardnet achieves a higher validation fit than the Box–Jenkins model; on coupled electronic drives, deep models perform well against a linear baseline despite having many more parameters than training samples.

7.1 Software issues

  • The survey illustrates MATLAB integration through its Deep Learning and System Identification Toolboxes, and PyTorch implementation using neural-network components and automatic differentiation.It presents examples in MATLAB and PyTorch; PyTorch also provides GPU acceleration.Source: §7.1 · pp. 32–33

7.2 An example with real data: forced duffing oscillator

  • Fit=99.18 for the deep cascadeforwardnet, versus Fit=29.71 % for the Box-Jenkins model on Silverbox validation data.The deep model's validation simulation is described as remarkably good, while the Box-Jenkins simulation is not very good.Source: §7.2 · pp. 33–34, Figure 17 · p. 34, Figure 16 · p. 34
  • The Silverbox deep model has insignificant correlation between residuals and past inputs at positive lags, though some residual autocorrelation remains.Validation data have a different character from estimation data, and the simulation uses no measured validation output.Source: §7.2 · p. 34

7.3 PYTORCH example: coupled electronic drives

  • The coupled-drives example compares FC, LSTM, deep SSM, and linear ARX models, training on 60% of uniformly excited data and validating on the remaining 40%.The deep models use stochastic gradient descent with momentum; the ARX baseline uses least squares.Source: §7.3 · p. 34
  • Deep models show good performance compared with the linear baseline, notably with 300 time samples for training despite overparameterization: FC has 184210 parameters, LSTM 169801, and deep SSM 111902.Source: §7.3 · p. 35

8 Conclusions

The survey frames deep networks and concatenated kernels as model structures and regularized methods for system identification. It highlights overparameterization and computational demands, while identifying theory for dynamically generated data and robustness as open concerns.

8 Conclusions

  • Deep networks fit predictions of future observations to data in the conventional prediction-error approach, while concatenated kernels define regularized methods.The survey presents deep learning as a rich set of issues and techniques for expanding understanding of system identification.Source: §8 · p. 35
  • Deep estimation uses many parameters and layered compositions, creating numerical and algorithmic demands while also exhibiting unexpectedly good generalization when overparameterized.The survey relates this behavior to its discussion of double descent and benign overfitting.Source: §8 · p. 35
  • Attention models, transformers, flow-based models, GANs, and graph neural networks are identified as architectures not yet extensively used for nonlinear system identification.The survey notes their use or design for related time-series, generative, or graph-based tasks.Source: §8 · p. 35
  • Theoretical models for double descent with dynamically generated data and uncertainty bounds for deep dynamic-system models remain future challenges.The discussed double-descent studies have so far focused on static systems.Source: §8 · p. 36

Related papers

Loading 2301.12832v1…