On kernel design for regularized LTI system identification

2.1 Impulse Response Estimation

• For LTI systems, identification is equivalent to estimating the impulse response, an ill-conditioned problem that requires balancing data fit and model complexity.The impulse response uniquely characterizes the LTI system.Source: §1 · p. 1

1 Introduction

• The paper proposes systematic kernel design methods for two prior-knowledge perspectives: machine learning and system theory.The distinction reflects different ways of representing impulse-response knowledge, such as decay and variation rate versus system properties.Source: §1 · p. 2
• AMLS kernels include the stable spline and diagonal-correlated kernels, and combine a rank-1 kernel with a stationary kernel.The design assigns decay to the rank-1 kernel and variation rate to the stationary kernel.Source: §1 · p. 2
• The SI-kernel approach uses a nominal model and stable uncertainty in a multiplicative-uncertainty configuration, then analyzes stability, entropy, and Markov properties.The paper gives conditions for stability, a suitably defined entropy-maximization property, and a Gaussian Markov process; the Markov property yields a banded inverse kernel matrix that can support more stable and efficient KRM implementation.Source: §1 · p. 2

2.2 Kernel-based Regularization Method

• KRM regularizes impulse-response estimation with a kernel-defined quadratic penalty, while empirical Bayes commonly estimates kernel hyperparameters by marginal likelihood.The regularized least-squares criterion trades off data fit and the RKHS norm; the kernel’s hyperparameters govern model complexity.Source: §2.2 · p. 3

2.3 Existing Single Kernels

• The stable spline (SS) and diagonal/correlated (DC) are the two mostly widely used single kernels; TC is a special case of DC.For CT with ρ < 0 and noninteger |t−s|, ρ^|t−s| is complex, so kDC(t, s; θ) is not well defined.Source: §2.3 · pp. 3–4

2.4 Optimal Kernel and Stable Kernel

• The MSE-optimal kernel depends on the true impulse response, so kernel design should mimic its behavior while using prior knowledge; stability requires the RKHS to lie in ℓ1 or L1.For a stable system, the kernel’s diagonal should tend to zero as time tends to infinity.Source: §2.4 · p. 4

2.5 Kernel Design

• Kernel design should embed the given impulse-response prior, with the appropriate method depending on the type of prior knowledge.The paper considers machine-learning and system-theory perspectives.Source: §2.5 · p. 4

3.1 Amplitude Modulated Locally Stationary Kernels

• Both SS and DC kernels are AMLS kernels, whose rank-1 factor modulates amplitude while the stationary factor represents local variation.The AMLS kernel is introduced for modeling impulse responses; its stationary factor is also the correlation coefficient of the associated zero-mean process.Source: §3.1 · p. 5
• The SS and DC realizations decay toward zero, unlike the stationary-kernel realizations, because the rank-1 factor supplies amplitude decay.The stationary factor controls how quickly realizations vary; for the DC kernel, negative ρ is defined only in discrete time.Source: §3.1 · pp. 5–6, Figure 1 · p. 6

3.2 Construct AMLS Kernels for Regularized Impulse Response Estimation

• For impulse-response design, choose the rank-1 factor for decay and the stationary factor for varying rate; stationary choices can include damped oscillation.The framework includes isotropic stationary kernels and kernels that need not decay monotonically and can take negative values.Source: §3.2 · p. 6, §3.2.1 · p. 6

3.2.2 Rank-1 Kernel kd(t, s)

• An AMLS kernel is stable if b(t) is summable—b(t) ∈ ℓ1 in discrete time or L1 in continuous time; this establishes stability of SS and DC.For a stable AMLS kernel under the stated series-expansion condition, b(t) cannot be bounded below by a positive constant.Source: §3.2.2 · pp. 6–7
• Requiring b(t) > 0 encourages independent roles for kd and kc, easing kernel design and analysis; relaxing it permits broader choices but can obscure those roles.The broader choices include arbitrary real-valued b(t) and an exponentially convex kd; then kd does not describe decay and kc is not the Gaussian process’s correlation coefficient.Source: §3.2.2 · p. 7

4 A System Theory Perspective

• The SI-kernel approach treats the impulse response as that of a stable LTI system and designs a stochastic-process model to encode this system-level prior knowledge.The paper’s motivating scope includes systems that may be overdamped or underdamped and may have multiple time constants or resonant frequencies.Source: §4 · p. 7, §4.1 · p. 7

4.2 Simulation Induced Kernel

• The simulation-induced kernel is obtained by simulating a nominal model and stable multiplicative uncertainty with an impulsive input; under stated conditions, AMLS kernels including SS and DC can also be represented as SI kernels.In this interpretation, the SS kernel’s nominal model is second order and the DC kernel’s is first order; their pole patterns correspond to different impulse-response behaviors.Source: §4.1 · pp. 7–8, §4.2 · p. 9
• When SI kernels lack closed-form expressions, hyperparameter and impulse-response estimation becomes more demanding, and particle filtering is not directly applicable because outputs depend on past states as well as the current state.The authors defer further details to an independent paper.Source: §4.2 · p. 8

4.3 Stability of SI kernels

• For DT and CT models, Proposition 4.3 gives sufficient SI-kernel stability conditions, with the accompanying guideline that b(t) decay faster than the slowest mode of G0(q).The paper states that these conditions are sufficient but not necessary.Source: §4.3 · pp. 9–10

4.4 Maximum Entropy Property of SI Kernels

• SI-kernel Gaussian processes solve a maximum-entropy problem under specified mean, initial-covariance, and variance constraints; the DC kernel also has a maximum-entropy characterization.The paper does not discuss the CT case, stating that entropy is not well defined for CT stochastic processes.Source: §4.4 · p. 10

4.5 Markov Property of SI Kernels

• For an n-order DT nominal model, the SI process is Markov of order n and its kernel matrix has an n-band inverse; the DC process is Markov of order 1.The paper notes that this banded structure can support more numerically stable and efficient implementations; it does not discuss whether CT sampling preserves the property.Source: §4.5 · p. 11

5.1 The Machine Learning Perspective

• AMLS kernels can model damped oscillation by placing oscillation in either the stationary kernel or the rank-1 amplitude-modulating function.Figure 5 illustrates AMLS realizations with damped oscillation.Source: §5.1 · p. 12, Figure 5 · p. 12

5.2 The System Theory Perspective

• The SI design uses a second-order LTI nominal model with complex-conjugate poles to encode prior knowledge of damped oscillation.The specified kernel uses Q = I2 and an exponentially decaying b(t).Source: §5.2 · p. 12

5.3 Numerical Simulation

• The numerical simulation tests whether AMLS kernels (43) and SI kernel (45) model systems with strong oscillation.The tested systems are generated as strongly oscillatory DT systems.Source: §5.3 · p. 12, §5.3.1 · p. 12

5.3.1 Test Data-bank

• The data bank contains 200 DT systems, each paired with 210 observations generated using Gaussian input and output noise.Noise standard deviation is set to a random factor from 0.5 to 1 times the noise-free output's standard deviation.Source: §5.3.1 · pp. 12–13

5.3.2 Simulation Results

• The simulation compares AMLS and SI kernels with TC, DC, SS-based SSp, and summarizes model fits with averages and distributions.Figure 6 omits negative fits: 5 for TC, 3 for DC, 3 for SSp, 3 for AMLS-2Os, 4 for AMLS-2Od, and 3 for SI-2Od.Source: §5.3.2 · p. 13, Figure 6 · p. 13

5.3.3 Findings

• The SI kernel (45) is best among tested kernels in average accuracy and robustness; AMLS (43b) beats TC but is slightly worse than DC in average accuracy.AMLS (43a) has better average accuracy than TC but is described as not working very well; the authors suggest SI performs best because its second-order complex-pole nominal model is closest to the truth.Source: Figure 6 · p. 13, §5.3.3 · p. 13
• The authors propose a conic combination of fixed kernels to represent prior knowledge such as multiple distinct time constants and resonant frequencies.The fixed kernels can include SS, TC, DC, and SI instances evaluated over hyperparameter grids.Source: §5.3.3 · p. 13

6 Conclusion

• The analysis relates SS and DC kernels to both AMLS and simulation-induced kernels, and identifies maximum-entropy properties.These results clarify properties of existing kernels and inform kernel design.Source: §6 · p. 14
• Kernel design should follow the type of prior knowledge: AMLS kernels encode function properties, while SI kernels encode knowledge about the generating LTI system.For AMLS, rank-1 and stationary kernels address decay and varying rate; SI uses a nominal model and stable uncertainty.Source: §6 · p. 14
• A suitable kernel structure is only one part of kernel-based regularization; hyperparameter tuning remains a separate topic.The paper points to and for discussion and for related asymptotic results.Source: §6 · p. 14

Proof of Proposition 3.1

• For any finite set of time points, the matrix defined by kd(ti, tj) equals the outer product b̄b̄T.Source: Proof of Proposition 3.1 · p. 15
• The outer-product form makes Kd positive semidefinite and gives it rank 1.Source: Proof of Proposition 3.1 · p. 15
• The proof states that identity (14) and parts (b) and (c) follow directly.Source: Proof of Proposition 3.1 · p. 15

Proof of Proposition 3.2

• For the continuous-time case, the proof reduces the result to showing that the kernels in (16) and (17) are stationary.Source: Proof of Proposition 3.2 · p. 15
• The exponential kernel e^-β|t-s| with β > 0 is a Matérn covariance function of order 1/2.Source: Proof of Proposition 3.2 · p. 15
• The proof identifies kc(t-s) in (16) as an isotropic stationary kernel, then concludes it is a kernel.The discrete-time result follows because its kernel is the continuous-time kernel restricted to N.Source: Proof of Proposition 3.2 · pp. 15–16

• The continuous-time proof uses the domain X = {t|t ≥0}; the discrete-time proof is stated to follow similarly.Source: Proof of Proposition 3.3 · p. 16
• The proof uses |kc(t −s)| ≤1 for t, s ≥0 together with an integrability condition on b(t).Source: Proof of Proposition 3.3 · p. 16
• By Corollary 2.1, the AMLS kernel (13) is stable.Source: Proof of Proposition 3.3 · p. 16

• The proof begins by assuming that the AMLS kernel (13) is stable, meaning Hk ⊂ L1.Source: Part (b) · p. 16
• Lemma A.1 applies to positive semidefinite kernels with an absolutely and uniformly convergent expansion using positive coefficients and linearly independent functions.The stated convergence is uniform on products of compact subsets of X.Source: Part (b) · p. 16
• The proof applies Lemma A.1 after rewriting the AMLS kernel (13) using (19).Source: Part (b) · p. 16
• Because kc is not stable, the proof selects h ∈ Hkc with h ∉ L1; if b(t) is bounded below by a positive constant, this leads to a contradiction with (A.6).Source: Part (b) · p. 16

• For rational power spectral density, Lemma A.2 provides spectral factors G with pole and zero locations constrained for the discrete-time and continuous-time cases.The DT poles lie inside the unit circle and zeros inside or on it; the CT poles lie in the left half plane and zeros in the left half plane or on the imaginary axis.Source: Proof of Proposition 4.1 · pp. 16–17
• A state-space realization of G yields a state-space representation of the stationary Gaussian process, with Q defined through a Lyapunov equation.Source: Proof of Proposition 4.1 · p. 17
• The proof attributes the factorization and representation steps to spectral factorization and representation theorems [42, Theorem 3.1 in Chapter 4] [42, Theorem 3.2 in Chapter 4] [40, Theorem 5.3 and Equation (5.92)] for DT, and corresponding CT theorems [42, Theorem 5.1 in Chapter 4] [42, Theorem 5.2 in Chapter 4] [40, Theorem 5.3].Source: Proof of Proposition 4.1 · p. 17
• For the specified AMLS kernel with kd(t, s) = cλt+s, the state-space parameters use Q = cQ̄ and b(t) = c times a decaying exponential, with the realization depending on the stated parameterization.The snippets give one DT mapping and one CT mapping, including λ = e^−β in the latter.Source: Proof of Proposition 4.1 · p. 17

• The proof sketches the SS-kernel case and states that the DC-kernel proof follows similarly.Source: Proof of Proposition 4.2 · p. 17
• For both DT and CT, the proof starts from the IS kernel's power spectral density and uses realization theory to derive the corresponding state-space models and results (27) and (29).Source: Proof of Proposition 4.2 · p. 17

Proof of Proposition 4.3

• The SI kernel is stable in both the discrete-time and continuous-time cases.The proof invokes Corollary 2.1 for each case.Source: Proof of Proposition 4.3 · pp. 17–18

Proof of Proposition 4.4

• The maximum-entropy solution has independent Gaussian initial state and unit-variance Gaussian inputs.The initial state has distribution N(0, Q), while each f(t) has distribution N(0, 1).Source: Proof of Proposition 4.4 · p. 18

Proof of Corollary 4.1

• The DC kernel is a simulation-induced kernel with the parameters in (28), so Theorem 4.4 holds for it.The proof verifies the required relation for the nonzero-parameter case after noting the result also holds when either parameter is zero.Source: Proof of Corollary 4.1 · pp. 18–19

Proof of Proposition 4.5

• The DC kernel’s Markov-process representation yields a kernel matrix with a 1-band inverse.The proof identifies the relevant process as first-order Markov and applies the lemma relating Markov structure to covariance-matrix banding.Source: Proof of Proposition 4.5 · p. 19

Proof of Proposition 4.6

• The discrete-time SI kernel defines an order-n Gaussian Markov process, so its kernel matrix has an n-band inverse.In controllable canonical form, the state coordinates correspond to lagged values of g(t).Source: Proof of Proposition 4.6 · p. 19