Automatic Differentiation to Simultaneously Identify Nonlinear Dynamics and Extract Noise Probability Distributions from Data

1 Introduction

• Noisy measurements compromise model-discovery accuracy and robustness, while many optimization frameworks assume Gaussian noise despite its limited real-world fit.The paper also identifies derivative approximation from noisy time-series as a challenging step, especially for nonzero-mean or asymmetric noise.Source: §1 · pp. 1–2
• SINDy uses sparse regression to select a parsimonious set of terms for interpretable governing equations.The paper frames interpretability and generalizability as important for understanding, designing, and controlling complex systems.Source: §1 · pp. 1–2
• Automatic differentiation lets modified SINDy differentiate library functions rather than noisy measurements while jointly denoising data and identifying sparse nonlinear models.The method also explicitly characterizes the underlying noise distribution, a capability the paper says current state-of-the-art methods lack.Source: §1 · p. 2
• The paper compares modified SINDy with Rudy et al.'s neural-network approach and presents numerical examples, noise-distribution extraction, and discrepancy modeling.The introduction previews these topics across Sections 3–5.Source: §1 · pp. 2–3

2.1 Sparse Identification of Nonlinear Dynamics

• SINDy formulates dynamics discovery as sparse regression over candidate functions, selecting a small set of terms to describe the observed dynamics.The coefficient matrix Ξ selects candidate terms, and λ controls the ℓ0-norm sparsity penalty.Source: §2.1 · p. 3
• Noise corrupts both derivative estimates and the nonlinear library matrix, motivating simultaneous noise estimation and model identification from denoised data.The paper notes that derivative noise is magnified by O(1/dt).Source: §2.1 · p. 4

2.2 Simultaneously Denoising and Learning System Model

• Modified SINDy treats estimated noise as an optimization parameter and forms estimated clean states as ˆX = Y − ˆN.Minimizing derivative error alone admits trivial solutions, so the method adds a time-stepping constraint based on the estimated vector field.Source: §2.2 · p. 4
• The time-stepping constraint compares estimated states with forward or backward numerical simulations over q steps, using fourth-order Runge–Kutta in this paper.Simulation errors farther from the starting point are penalized less through weights ω_i = c|i|^-1, with c = 0.9 used throughout.Source: §2.2 · pp. 4–5
• Automatic differentiation supplies gradients for the joint loss over ˆN and Ξ, while repeated optimization loops set coefficients below λ to zero.Each loop initializes from the previous noise estimate and updates the estimated states used to calculate Ξ.Source: Figure 1 · p. 5, §2.2 · p. 6

3 Performance Comparison with Neural Network Denoising Approach

• The section compares modified SINDy with Rudy et al.’s NN denoising approach and considers robustness to noise and data amount.Source: §3 · p. 6

3.1 Performance Criteria

• The comparison uses vector-field error, noise-identification error, and prediction error, with parameter error and success rate additionally used for comparison with Weak-SINDy.Success rate is the percentage of trials identifying the correct model structure.Source: §3.1 · pp. 6–7

3.2 Robustness to Noise

• Modified SINDy correctly identified the Lorenz model at every tested noise level, showed better forward-simulation performance than the NN approach, and had better noise-identification error; it was about 2 times more robust than original SINDy.Figure 2 also shows that neither approach recovers zero noise when none is applied, attributed to a small difference between learned and true dynamics.Source: §3.2 · pp. 7–8, Figure 2 · p. 8

3.3 Robustness to Data Length

• The data-length experiment varies Lorenz training-series duration with 10% Gaussian noise, and omits prediction error because low-data-limit simulations of identified models are unstable.The passage says Figure 4 suggests modified SINDy outperforms the NN approach when the correct parameters and library are used, but its comparison condition is cut off.Source: §3.3 · p. 8

4 Results

• 3.5 seconds of training data yielded 100% system-model identification accuracy; at 5 seconds, a fixed threshold caused a small success-rate drop that a larger threshold reversed.The comparison also tracks vector-field and noise-identification errors as training-data length increases.Source: Figure 4 · p. 10

4.1 Van der Pol Oscillator

• Modified SINDy correctly identified the true noise distribution in the Van der Pol oscillator example.The example tests three noise levels.Source: §4.1 · pp. 10–11

4.2 Rössler Attractor

• For the Rössler attractor, modified SINDy denoised measurements while learning a model that simulated forward with high accuracy.The example compares denoised signals with trajectories simulated from the identified model.Source: §4.2 · p. 11, Figure 6 · p. 12

4 Results

• Across canonical dynamical systems, modified SINDy correctly identified system models and noise distributions regardless of noise magnitude.The examples used zero-mean Gaussian noise.Source: §4.3 · p. 12, Figure 8 · p. 14

4.4 Identification of Noise Distributions

• The method handled Gaussian, uniform, and Gamma noise distributions, but learning non-zero-mean noise was more difficult than learning zero-mean noise.The Van der Pol example used 10% noise; the paper notes iterative learning as a possible route to better non-zero-mean results.Source: §4.4 · pp. 12–13, Figure 9 · p. 15

4.3 Lorenz 96 Model

• Modified SINDy denoised Lorenz 96 measurements while learning the model; the figure displays only the first three states.The system has six states.Source: §4.3 · p. 12, Figure 7 · p. 13

4.5 Discrepancy Modeling

• In discrepancy modeling, modified SINDy identified missing dynamics when part of the system dynamics was already known.Known dynamics can be constrained in the library and their parameters used as an initial guess, leaving only the missing dynamics to learn.Source: §4.5 · pp. 13–15

5 Conclusion and Future Work

• The method jointly denoises time series, parametrizes noise distributions, and identifies the parsimonious system generating the data.The conclusion also reports improved noise robustness with less training data than Rudy et al.’s NN denoising approach.Source: §5 · p. 15
• The method can be integrated with discrepancy modeling, using known dynamics to help identify missing dynamics.Figure 10 depicts this use alongside discrepancy learning and measurement denoising.Source: §5 · p. 15, Figure 10 · p. 16
• Modified SINDy learns Gaussian, uniform, and non-zero-mean noise distributions such as Gamma.The authors describe it as having practical potential for highly noisy data or settings with partial model information.Source: §5 · p. 16
• The authors identify control inputs, faster optimization, and rational or implicit dynamics as directions for extending the method.They note that Adam is slower than standard sequential least-square thresholding, and that simulation error is difficult to calculate for rational dynamics.Source: §5 · p. 16
• Modified SINDy is currently not robust with large libraries because the optimization is non-convex, and reducing the optimizer’s learning rate may stabilize it.The authors suggest variable-time-step simulation as a possible alternative to the fixed-step scheme used in the paper.Source: §5 · p. 16

A Algorithm for Simultaneously Denoising and Learning System Model

• The algorithm initializes estimated noise from a smoothed residual or zeros, then iteratively optimizes the model and noise while thresholding small coefficients.It updates the estimated clean state as the measured data minus estimated noise.Source: A Algorithm for Simultaneously… · p. 21

B Effect of Thresholding Parameter λ

• Too-small λ can leave the sparsity constraint too weak and trap the model and noise estimates in a local minimum.The illustrated setting uses Lorenz data with 10% Gaussian noise, 8 optimization loops, and 10 runs per λ.Source: B Effect of Thresholding Parameter λ · p. 21, Figure 11 · p. 22
• Too-large λ can eliminate necessary terms, producing an incorrect model structure and a large difference between identified and true noise.Figure 11 describes these opposing failure modes and indicates λ needs proper tuning.Source: B Effect of Thresholding Parameter λ · p. 21, Figure 11 · p. 22
• The authors note stepwise sparse regression as a possible way to avoid sweeping λ values, but do not evaluate its performance here..Source: B Effect of Thresholding Parameter λ · p. 21

C Effect of Prediction Step q

• The Figure 12 comparison uses Lorenz trajectories with x0 = [−5, 5, 25] and dt = 0.01, simulated for 3 seconds.Its left panels show error distributions across q, the middle compares true and identified noise, and the right compares simulated and true trajectories.Source: Figure 12 · p. 23
• Modified SINDy’s performance is not greatly affected by prediction step q, unlike the NN approach, which has an optimal q value.The comparison uses 10% noise on a Lorenz attractor and reports medians over 10 runs.Source: C Effect of Prediction Step q · pp. 22–23, Figure 12 · p. 23
• Computational time increases linearly with q for both approaches, so modified SINDy can use a small q to save time without sacrificing much performance.Figure 12 compares noise-identification, prediction, and vector-field errors, along with identified noise and trajectories.Source: C Effect of Prediction Step q · p. 23, Figure 12 · p. 23

D Effect of Optimization Iteration Nloop

• As Nloop increases, noise-identification error decreases toward an optimum and vector-field error converges, while identified noise approaches the true noise.Figure 13 reports distributions across 10 runs and compares identified and true noise for four optimization-loop choices.Source: D Effect of Optimization Iteration Nloop · pp. 23–24, Figure 13 · p. 24
• About 2 times more robust than original SINDy, modified SINDy can handle a higher noise level while generating the correct model structure.The comparison uses Tikhonov regularization to pre-smooth the noisy data.Source: Figure 14 · p. 24
• Table 1 lists modified SINDy parameter settings for Figure 14 under the maximum tolerable noise levels.It notes that the constant term is included for Rössler and Lorenz 96 libraries, but not the other examples.Source: Table 1 · p. 25

• The robustness test compares SINDy and modified SINDy on Van der Pol, Lorenz, and Rössler systems by the maximum noise percentage tolerated while recovering the correct model structure.Five noisy datasets are tested at each noise level; any failure among them counts as not robust at that level.Source: E Noise Robustness Comparison with SINDy · p. 25
• The reported modified-SINDy noise threshold is approximate because parameter tuning could raise it, but broad parameter sweeps are computationally heavy.The figure uses q = 1 and Nloop = 8 for modified SINDy.Source: E Noise Robustness Comparison with SINDy · p. 25
• At dt = 0.001, Weak-SINDy and modified SINDy have almost the same accuracy; at dt = 0.01, modified SINDy has a slightly higher success identification rate.The comparison uses Lorenz attractor data.Source: Figure 15 · p. 26

F Noise Robustness Comparison with Weak-SINDy

• The Lorenz comparison evaluates parameter error and success rate for Weak-SINDy and modified SINDy at dt = 0.01 and dt = 0.001; the final comparison is shown in Fig. 15.The approaches use different test-function settings, and modified SINDy uses pre-smoothing only at dt = 0.01.Source: F Noise Robustness Comparison with… · p. 26

G Parameters Used in Fig. 8

• Fig. 8 simulations use Duffing parameters p1 = 0.2, p2 = 0.1, p3 = 1; Cubic parameters p1 = −0.1, p2 = 2, p3 = −2, p4 = 0.1; and Lotka-Volterra parameters p1 = 1, p2 = 0.5.Across the examples, Nloop = 5 and dt = 0.01.Source: G Parameters Used in Fig. 8 · p. 27

H Tips on Learning Non-Zero Mean Noise

• For non-zero-mean noise, modified SINDy is applied, the learned noise mean is subtracted from measurements, and the learning repeats until convergence and correct model recovery.In a Van der Pol example with 20% Gamma noise, the authors caution that large bias may defeat the approach and careful tuning is needed; soft start can also help denoising.Source: H Tips on Learning Non-Zero Mean Noise · p. 27

• To illustrate identifying the noise distribution, the Van der Pol oscillator shown in Eq. (20) is simulated, and Table 3 compares the identified noise distribution with the true distribution.For the Gamma distribution, k represents shape and θ represents scale.Source: I Identifying Noise Distribution Type · p. 27, Table 3 · p. 29
• With 10% noise, modified SINDy correctly identified the model across the tested cases, then compared the identified noise against seven candidate distributions using sum of squared errors.The candidates were normal, uniform, Gamma, Dweibull, Rayleigh, Cauchy, and Beta; the lowest-error fit is selected.Source: I Identifying Noise Distribution Type · p. 28
• When data are insufficient, another distribution may be misidentified as the true noise distribution, and the number of data points needed for correct identification is outside the study’s scope.The final results are summarized in Table 3.Source: I Identifying Noise Distribution Type · p. 29

• If the library omits terms in the true dynamics, modified SINDy fails to recover the correct noise and system model; libraries should be broad enough for the dynamics but not excessively large.The authors advise incorporating prior information when possible and do not expect the method to work with hundreds or thousands of terms.Source: J Caveats of the Approach · p. 29
• For the Lorenz example above 20% noise, the largest reported manageable library order is 4, about 32 terms; higher-order terms can cause NaN cost and optimizer failure.Suggested adjustments include lowering the learning rate, pre-smoothing, improving the initial estimate of Ξ, reducing library size, or using float64.Source: J Caveats of the Approach · p. 29
• Whether to include a constant basis term is case-specific; with non-zero-mean noise or significant outliers and no true constant term, including it can make convergence to a local minimum easier.The authors advise trying libraries both with and without the constant basis.Source: J Caveats of the Approach · p. 29
• A good initial estimate of noise and Ξ can improve optimization conditioning; pre-smoothing may initialize both, while zero noise is an option when no other information is available.Source: J Caveats of the Approach · pp. 29–30