Source-linked AI summary
Sparse Orthogonal Regression Technique: A Spectral Framework for Equation Discovery, Approximation, and Integration
Sabin Roman, Ljupco Todorovski, Saso Dzeroski
TL;DR
Learning governing equations, integrals, and approximations from finite noisy or irregular samples requires a stable, reusable functional representation. SORT learns sparse orthonormal-basis expansions directly from observations and reuses their coefficients across tasks; experiments show competitive or more robust behavior when bases are suitable, while basis mismatch and dimensional growth remain important limits.
Problem
Finite, noisy, and irregularly sampled data make stable and reusable reconstruction of functions, dynamical systems, and integrals a central challenge.
Method
SORT estimates sparse coefficients in prescribed orthonormal bases from observations using L1-regularized regression, then reuses the learned expansion for reconstruction, integration, and approximation.
Results
SORT remains competitive with SINDy-style identification at fine sampling, degrades less abruptly under coarse sampling, supports coefficient-readout integration, and exhibits order-consistent coefficient persistence.
Takeaways & Limitations
SORT provides a reusable intermediate representation that makes basis design an explicit part of modelling and can precede searches for simpler analytic expressions.
Takeaways & Limitations
SORT does not remove the curse of dimensionality, can degrade under severe basis mismatch, and may be less suitable than symbolic or library-based methods when compact human-readable formulas are required.
Abstract
from arXiv · showhide
We develop the Sparse Orthogonal Regression Technique (SORT), a sparse spectral framework for learning orthonormal-basis expansions from noisy and irregularly sampled data. SORT estimates expansion coefficients directly from observations using L1-regularized regression, avoiding explicit quadrature or analytic inner-product evaluation. The central application is data-driven discovery of ordinary differential equations: vector fields are represented in chosen orthogonal bases and learned as sparse coefficient expansions. This provides a complementary route to symbolic regression, grammar-based discovery, and SINDy-style sparse identification by first recovering a compact spectral representation, which can later guide searches for simpler analytic forms. Across the dynamical-system experiments, SORT matches or improves upon library-based sparse-regression baselines when the basis is well adapted to the problem, and shows more stable degradation under sparse sampling, noisy derivative estimates, and representation mismatch. Specific examples illustrate why this representation is useful: if a finite library misses the problem-specific nonlinearity, the resulting model can fail. SORT is not immune to mismatch, but it shifts the problem away from brittle selection among generic terms to basis design adapted to the problem domain. The experiments also show that dominant low-order coefficients persist as model order increases, supporting order-consistent model growth. Beyond equation discovery, the same learned expansion supports nonlinear approximation and estimation of complex, high-dimensional integrals by coefficient readout. Overall, SORT provides a reusable intermediate representation for system identification, approximation, and integration, while making basis design an explicit part of the scientific modeling problem.
1 Introduction
The paper presents SORT as a reusable framework for reconstructing functions from finite, noisy, and irregular samples. It applies sparse orthonormal expansions to dynamical-system reconstruction, integration, approximation, and order-consistent model growth.
- Motivation: The shared problem across surrogate modelling, integration, and equation discovery is stable reconstruction of functional relationships from finite, noisy, irregularly sampled data.
- Framework: SORT estimates sparse expansions in prescribed orthonormal bases from sampled observations using sparsity-promoting regression.The paper frames this as an application-level and representational contribution rather than a new sparse-regression algorithm.
- Dynamical-system reconstruction: For dynamical systems, SORT represents vector-field components in fixed orthonormal bases, complementing library-based sparse identification when candidate nonlinearities are missing or unsuitable.
- Numerical integration: For integration, SORT estimates coefficients from sampled observations and reads out the coefficient associated with an integration functional instead of using explicit quadrature or analytic inner products.
- Order-consistent growth: Increasing truncation order preserves the meaning of lower-order basis functions, allowing coefficient persistence to diagnose stable learning across model growth.
- Evaluation: Experiments evaluate SORT across system reconstruction, integration, and nonlinear approximation against dense, kernel-based, and library-based baselines under noise and subsampling.
2 Background and related work
SORT recombines established spectral approximation, sparse recovery, polynomial-chaos, and sparse-regression tools into a task-oriented workflow. Its distinctive emphasis is learning reusable sparse orthonormal coefficients directly from data, with basis choice determining applicability and scalability.
- Spectral foundations: Classical spectral methods use global orthogonal bases and typically obtain coefficients through projection, quadrature, interpolation, or structured transforms.
- Sparse recovery: Sparse recovery theory establishes that compressible expansions in orthogonal polynomial systems can be recovered from limited samples using L1 minimization.
- Related frameworks: Sparse polynomial chaos combines orthogonal expansions with sparse regression or adaptive truncation primarily for surrogate modelling and uncertainty quantification.
- Algorithmic distinction: Unlike finite-library methods that orthogonalize or regularize candidate terms, SORT begins with a prescribed orthonormal coordinate system and grows truncations within that representation.
- Workflow: SORT places sparse orthonormal coefficients at the center of a common workflow for vector-field modelling, integration, nonlinear approximation, and model comparison.
- Data-driven estimation: SORT estimates coefficients from noisy samples with L1-regularized regression, but irregular finite sampling makes empirical orthogonality approximate and coefficient stability requires numerical assessment.
- Integral estimation: Including a normalized Riesz representer as a basis direction makes the corresponding coefficient proportional to the target integral, which SORT recovers by sparse-regression readout.
- Scaling and limitations: For N samples and K basis functions, design-matrix construction costs O(NK), while dimensionality and truncation growth remain the main computational bottlenecks.
3 Applications
SORT applies sparse regression in prescribed orthonormal bases to equation discovery, integration, approximation, and stable model growth from limited or noisy data. Its experiments show that basis choice affects robustness, while learned coefficients support prediction, integral readout, and order-consistent refinement.
- SORT evaluates sparse orthogonal regression for system reconstruction, numerical integration, and nonlinear approximation under noise and subsampling.The workflow estimates orthonormal coefficients from data and reuses them for prediction, integration, vector-field reconstruction, and model refinement.
- Equation discovery: In the Bessel-driven system, SINDy degrades with lower training fractions or higher noise, whereas SORT remains more robust under representation mismatch.The Bessel coupling is not sparse in the selected low-degree polynomial or augmented trigonometric libraries, making SINDy coefficient selection brittle.
- Equation discovery: Under sparse temporal sampling, SORT degrades more gradually than a SINDy-style library model using the same sampled states and finite-difference derivatives.The comparison includes systems where SINDy rollout failure becomes abrupt as sampling becomes coarse, while the orthogonal sparse model remains bounded.
- Integral estimation: For data-driven integration, SORT estimates coefficients from pointwise samples and obtains integrals by reading the coefficient associated with the integration functional.The construction maps bounded physical domains to a canonical domain and uses a basis element representing the integration functional.
- Integral estimation: Integration matches closed-form values for one-dimensional oscillatory and Gaussian examples, while higher-dimensional accuracy decreases gradually with dimension.The higher-dimensional decline is consistent with increased basis size and sample complexity for sparse coefficient recovery.
- Approximation: SORT preserves dominant low-order spectral structure and a stable coefficient hierarchy as model order increases, unlike several dense or monomial representations.The approximation experiments use noisy training data and compare SORT with dense least-squares, kernel, and random-feature baselines.
4 Conclusion
SORT provides a sparse orthonormal representation that supports dynamical-system reconstruction, integration, and nonlinear approximation. Its experiments emphasize robustness under degraded data, order-consistent refinement, and the importance of basis design.
- SORT turns learned sparse coefficient vectors into a reusable representation for integration, dynamical-system reconstruction, and nonlinear approximation.
- Dynamical-system reconstruction: SORT remains competitive with SINDy-style identification under fine sampling and degrades less abruptly when sampling becomes coarse.
- Dynamical-system reconstruction: Basis design is part of the modeling problem because performance depends on derivative quality and how well the chosen basis captures the dynamics.
- Numerical integration: Data-driven quadrature estimates expansion coefficients from samples and obtains integrals by reading the coefficient associated with the integration functional.
- Nonlinear approximation: Coefficient persistence across truncation levels provides a practical diagnostic for stable learning as model order increases.
- SORT can serve as an intermediate representation between raw samples and later symbolic or analytic descriptions.
Code Availability
Code for reproducing the experiments and figures is available online.
- Reproducibility code and figure-generation materials are available at https://doi.org/10.5281/zenodo.21707070.
Disclaimer
The work was co-funded by the European Union, but the authors retain responsibility for the expressed views and opinions.
- The authors state that the views and opinions expressed do not necessarily reflect those of the European Union or the European Research Executive Agency.