Source-linked AI summary
A unified sparse optimization framework to learn parsimonious physics-informed models from data
Kathleen Champion, Peng Zheng, Aleksandr Y. Aravkin, Steven L. Brunton, J. Nathan Kutz
TL;DR
Physical-science ML needs models that are interpretable, generalizable, and effective for extrapolation, but standard approaches often struggle with these goals. The paper introduces a unified sparse optimization framework that selects governing terms and incorporates non-convex regularization, robustness, parametrization, and constraints. Across example problems, SINDy-SR3 discovers parsimonious dynamical models and is presented as a viable framework for engineering sciences.
Problem
Physical-science ML needs interpretable, generalizable models that can extrapolate, while neural networks often struggle with extrapolation, interpretation, and enforcing known physics.
Method
The framework uses sparse regression with non-convex regularization to select governing terms while integrating corrupt-data trimming, physical constraints, and parametric dependencies.
Results
SINDy-SR3 discovers parsimonious dynamical models across example problems and demonstrates accuracy and efficiency as a framework for engineering sciences.
Takeaways & Limitations
The unified framework can be adapted to physical-system data containing outliers, parametrizations, forcing, and partially known physics.
Takeaways & Limitations
Parameterized SINDy works best with only a small number of library parameters because initialization scans scale combinatorially and the joint problem has many possible local minima.
Abstract
from arXiv · showhide
Machine learning (ML) is redefining what is possible in data-intensive fields of science and engineering. However, applying ML to problems in the physical sciences comes with a unique set of challenges: scientists want physically interpretable models that can (i) generalize to predict previously unobserved behaviors, (ii) provide effective forecasting predictions (extrapolation), and (iii) be certifiable. Autonomous systems will necessarily interact with changing and uncertain environments, motivating the need for models that can accurately extrapolate based on physical principles (e.g. Newton's universal second law for classical mechanics, $F=ma$). Standard ML approaches have shown impressive performance for predicting dynamics in an interpolatory regime, but the resulting models often lack interpretability and fail to generalize. We introduce a unified sparse optimization framework that learns governing dynamical systems models from data, selecting relevant terms in the dynamics from a library of possible functions. The resulting models are parsimonious, have physical interpretations, and can generalize to new parameter regimes. Our framework allows the use of non-convex sparsity promoting regularization functions and can be adapted to address key challenges in scientific problems and data sets, including outliers, parametric dependencies, and physical constraints. We show that the approach discovers parsimonious dynamical models on several example systems. This flexible approach can be tailored to the unique challenges associated with a wide range of applications and data sets, providing a powerful ML-based framework for learning governing models for physical systems from data.
1 Introduction
The paper motivates parsimonious, interpretable dynamical models because standard ML often struggles with extrapolation, interpretation, and known physical constraints. It develops a unified sparse optimization framework that selects governing terms while accommodating robustness, parametrization, and physics-informed structure.
- Motivation: Scientific ML requires parsimonious models that support interpretation, generalization, and extrapolation rather than only interpolatory prediction.The motivation contrasts these goals with highly parameterized neural networks that typically lack interpretability or generalizability.
- Unified framework: The framework unifies model discovery with corrupt-data trimming, physical constraints, and parametric dependencies.These extensions are presented as adaptations of a common sparse optimization formulation.
- SINDy formulation: SINDy identifies the fewest candidate-library terms needed to model nonlinear dynamics through sparse regression.The formulation assumes that the governing dynamics contain only a few active terms.
- SINDy formulation: The basic sparse-regression problem builds a library of candidate functions and estimates sparse coefficient vectors for the dynamics.The formulation uses snapshot data, a candidate library Θ(X), and sparse loading vectors Ξ.
- Optimization motivation: LASSO uses convex ℓ1 regularization but often retains many small nonzero coefficients, motivating non-convex regularization such as the ℓ0 norm.STLSQ can produce sparse fits but is less readily adapted to extensions beyond its least-squares formulation.
- Unified framework: SR3 provides a flexible sparse-regression framework for selecting active governing terms and adapting SINDy to additional physical-data challenges.The overview specifically identifies outlier identification, parameterized library functions, and forcing as adaptable features.
2 Formulation and approach
SR3 reformulates sparse dynamical-system identification so non-convex penalties can produce truly sparse models while retaining an extensible optimization structure. In Lorenz-system comparisons, SR3 matches STLSQ’s correct sparsity pattern, whereas strongly regularized LASSO produces poor dynamics fits.
- 2 Formulation and approach: SR3 extends the SINDy optimization to incorporate additional structure, outlier robustness, and nonlinear parameter estimation through relaxation and partial minimization.The formulation introduces an auxiliary variable W and generalizes the original sparse-regression problem.
- 2 Formulation and approach: Algorithm 1 solves the relaxed problem by alternating updates that require least-squares solves and proximal operators.As ν ↓0, the resulting solution approximates the original optimization problem.
- 2 Formulation and approach: With the ℓ0 penalty, SR3 uses hard thresholding and remains adaptable to new SINDy problems beyond thresholded least squares.The method is similar to, but not equivalent to, thresholded least squares.
- 2.1 Performance of SR3 for SINDy: SR3 identifies truly sparse models because its framework supports non-convex regularizers such as the ℓ0 norm.STLSQ can work well but lacks a standalone optimization cost, while LASSO commonly leaves small nonzero coefficients.
- 2.1 Performance of SR3 for SINDy: In Lorenz-system tests, STLSQ and SR3 recover the same correct coefficient sparsity pattern and both track test trajectories around the attractor.The comparison uses polynomial libraries through order 3 and simulations from 20 initial conditions.
- 2.1 Performance of SR3 for SINDy: Low-penalty LASSO has similar simulation performance but is not truly sparse, while high-penalty LASSO removes true coefficients and fits the dynamics poorly.Increasing LASSO regularization does not reliably eliminate only unimportant terms.
3 Simultaneous Sparse Inference and Data Trimming
SINDy SR3 can jointly fit sparse dynamical models while trimming corrupted samples, improving identification when outliers contaminate measurements or derivative estimates.
- Robust sparse inference: Outliers corrupt derivative estimates, motivating a trimmed SINDy formulation that simultaneously removes anomalous samples and fits the model.The trimming formulation uses an estimate h of the number of inliers among m candidate rows.
- Lorenz example: With 10% of Lorenz samples corrupted, trimming identifies the Lorenz model despite removing some off-attractor initial-condition data.The trimmed method consistently recovers the model, sometimes with only 1–2 extra coefficients; standard SINDy SR3 without trimming fails.
- Rossler example: For the Rossler system, trimming 5% of the data consistently identifies or nearly identifies the system when standard SINDy SR3 does not.The near-identifications contain only 1–2 incorrect coefficients in repeated trials.
- Rossler example: Rossler excursions into the x3 dimension can initially appear outlying, but convergence typically recognizes most of these points as part of the true dynamics.This behavior reflects the attractor’s concentration in the x1, x2 plane rather than evidence that the excursions are corrupted.
4 Incorporating physical constraints
The constrained SINDy SR3 formulation incorporates physical structure as coefficient constraints, preserving gradient relationships that unconstrained fitting can lose as noise increases.
- Physical constraints: Physical principles such as conservation laws can be imposed as constraints on the SINDy coefficient matrix.For Hamiltonian and gradient systems, the equations must be partial derivatives of the same potential or Hamiltonian function.
- 2D Duffing example: In the 2D Duffing example, gradient structure creates linear relationships among coefficients that can be written as Cξ = d after vectorization.The coefficient constraints encode the interdependence between the equations’ polynomial terms.
- Constrained optimization: SR3 solves the constrained coefficient update as a quadratic problem with affine equality constraints, using an augmented system and Kronecker-product identities.The resulting large linear system is reduced to a smaller problem whose dimension matches the number of constraints.
- Results: As noise increases, unconstrained Duffing coefficients drift away from the gradient constraint, whereas constrained coefficients retain the desired form.At very low noise, both constrained and unconstrained algorithms identify coefficient matrices satisfying the constraints.
5 Parameterized library functions
Parameterized library functions let SINDy jointly estimate nonlinear forcing parameters and sparse model coefficients, but the resulting nonconvex optimization is sensitive to initialization and parameter count.
- Motivation: Parameterized basis functions and external forcing introduce nonlinear dependencies that polynomial libraries alone may not represent.The framework therefore includes parameterized library terms and estimates their parameters jointly with the model.
- Optimization: The SR3 parameter-estimation problem combines nonlinear least squares, sparsity regularization, and a relaxation penalty over model coefficients.The parameter α is updated with a true Newton step using algorithmic differentiation for the gradient and Hessian.
- Limitation: Parameterized SINDy has potentially many local minima and works best with only a small number of library parameters.Scanning initializations becomes combinatorial as additional parameters are added, so model-selection criteria are needed.
- Lorenz with forcing: With parameterized hyperbolic-tangent forcing, joint optimization recovers the forcing parameters and sparsity pattern when initialized sufficiently close to the correct parameters.The identified system tracks the true behavior for several trips around the attractor.
- Lorenz with forcing: Without the forcing term, the model activates several polynomial terms and fails to reproduce the correct simulated behavior.This comparison shows why the forcing term must be included in the candidate library for this example.
6 Discussion
The framework targets interpretable, generalizable model discovery by extending sparse optimization to corrupt data, constraints, and parametrizations. Its accuracy and efficiency are demonstrated across example problems, with code and data available publicly.
- SINDy-SR3 extends sparse model discovery to handle corrupt data, physical constraints, and parametric dependencies.
- The framework is demonstrated as accurate and efficient on several example problems.
- The authors provide code and data through a public GitHub repository.
S1 Choice of parameters for SR3
SR3 balances fidelity to the relaxed coefficient matrix against sparsity through ν and λ. With λ tied to a desired coefficient threshold, ν becomes the main parameter controlling the regularization–least-squares trade-off.
- ν controls how closely the relaxed coefficient matrix W matches Ξ, with smaller values enforcing closer agreement.
- λ determines regularization strength and, with the ℓ0 norm, sets coefficient thresholding through ν and λ.
- When λ is defined from a known threshold η, decreasing ν emphasizes regularization while increasing ν emphasizes least-squares fit.
S2 Simulation details
The simulations evaluate SR3 across sparse identification, outlier trimming, physical constraints, and parameter estimation. Results show correct sparsity or constraints in several settings, while forcing-aware parameter estimation outperforms a library without the forcing term.
- S2.1 Performance of SR3 for SINDy: SR3 and STLSQ identify the correct Lorenz sparsity pattern, unlike the tested LASSO models.
- S2.1 Performance of SR3 for SINDy: 0.84 is the off-attractor R2 score for both STLSQ and SR3, compared with 0.82 for low-regularization LASSO.
- S2.3 Data trimming: Rossler: Trimming 5% of Rossler data with 1% outliers correctly identifies the system in most repeated trials.
- S2.4 Physical constraints: 2D Duffing: Under stronger noise, constrained Duffing models satisfy the gradient constraint while unconstrained models do not, despite nearly identical R2 scores.
- S2.5 Parameter estimation: Joint estimation recovers the forcing parameters and sparsity pattern, whereas omitting the forcing term produces many active terms and poor simulation.
S3 Convergence results
The convergence analysis connects the algorithms to proximal gradient and PALM methods. It establishes sub-linear convergence for Algorithm 1 and convergence to a stationary point for Algorithm 2, while leaving guarantees for Algorithm 4 open.
- S3 Convergence results: Theoretical guarantees for Algorithm 4 remain future work because its Newton step makes analysis more difficult.
- S3.1 Convergence of Algorithm 1: Algorithm 1 is analyzed as a proximal gradient method through variable projection, with convergence results stated for its iterates.
- S3.1 Convergence of Algorithm 1: Algorithm 1 has a sub-linear convergence rate for all prox-bounded regularizers R.
- S3.2 Convergence of Algorithm 2: Algorithm 2 is equivalent to alternating proximal gradient updates and converges to a stationary point of p.