PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Symbollic Regression Using Deep Learning and Sparse Modelling

Ravi Kumar U, Sumitra S

arXiv:2609.01102v1cs.LGcs.NEcs.SC

TL;DR

Symbolic regression seeks interpretable equations, while prior approaches are limited by coupled optimization, scalability, noise sensitivity, or fixed feature libraries. NSR uses a decoupled pipeline in which a neural network approximates the target over nonlinear interaction features before LASSO extracts a sparse expression. Experiments on Nguyen report lower RMSE, stronger symbolic recovery, and greater noise and out-of-distribution robustness than SINDy and unoptimized neural baselines, although feature-library growth constrains scalability.

  • Problem

    Existing symbolic-regression approaches face coupled optimization, scalability, noise sensitivity, or dependence on manually designed feature libraries, limiting robust interpretable equation discovery.

  • Method

    NSR builds an interaction-aware nonlinear feature library, trains a neural approximator as a functional preconditioner, and applies LASSO for sparse symbolic extraction with automated tuning.

  • Results

    NSR reliably exceeds SINDy and unoptimized neural baselines on the Nguyen suite, with reduced RMSE, improved symbolic recovery accuracy, and greater noise resilience.

  • Takeaways & Limitations

    NSR connects neural approximation with interpretable mathematical formulas and shows robust out-of-distribution performance for scientific machine-learning applications.

  • Takeaways & Limitations

    NSR scalability is constrained by combinatorial feature-library growth, direct feature-matrix storage, and redundancy from a fixed library.

Abstract

from arXiv · show

Symbolic Regression (SR) seeks to find succinct mathematical expressions that represent the fundamental relationships within data, providing interpretability and scientific understanding that exceeds that of black-box models. Nevertheless, traditional methods like Genetic Programming face challenges with scalability and are highly sensitive to noise, while sparse regression techniques such as SINDy rely significantly on predetermined feature libraries. In this work, we present a Neural Symbolic Regression (NSR) framework that treats neural networks as functional preconditioners for symbolic discovery. Our approach uses a decoupled pipeline: a neural network first learns a smooth, noise-robust approximation of the target function in an interaction- aware nonlinear feature space. LASSO is then applied to extract sparse, interpretable closed-form expressions. To improve predictive accuracy and symbolic fidelity by integrating distributed hyperparameter optimization with Ray Tune and ASHA scheduling. Experiments on the Nguyen benchmark suite show that our approach consistently outperforms SINDy and non-tuned neural baselines in RMSE, noise robustness, and out-of-distribution generalization. Ablation studies confirm the significance of feature interactions, neural depth, and tuning strategies. In general, this study presents a scalable and understandable neural-symbolic framework, creating a solid link between neural approximation and the discovery of sparse equations for scientific machine learning.

I. INTRODUCTION

Symbolic regression seeks interpretable closed-form equations, but existing approaches face scalability, noise, feature-library, and optimization challenges. NSR addresses these issues through a decoupled neural approximation and sparse symbolic-recovery framework with interaction-aware features and automated tuning.

  • Motivation: Symbolic regression discovers closed-form formulas that expose underlying relationships more clearly than black-box predictive models.This interpretability supports scientific and engineering analysis, extrapolation, and analytical reasoning.
  • Limitations of Existing Methods: Genetic Programming offers expressive symbolic search but suffers from slow convergence, high computational cost, code bloat, and limited scalability on noisy or high-dimensional data.
  • Limitations of Existing Methods: SINDy avoids combinatorial search with LASSO over a predefined nonlinear feature library, but its effectiveness depends on manually designed library expressiveness.Rigid libraries may fail to represent intricate nonlinear interactions.
  • Limitations of Existing Methods: Neural symbolic methods improve flexibility but often jointly optimize approximation and symbolic discovery, which may increase instability, computational complexity, and interpretability challenges.
  • NSR Framework: NSR uses neural networks as smooth, noise-resistant functional preconditioners before an independent sparse-regression phase for symbolic extraction.The separation is intended to enhance noise robustness, stabilize equation recovery, and improve symbolic identifiability.
  • NSR Framework: The framework combines interaction-aware nonlinear feature construction, GPU-boosted neural approximation, LASSO extraction, and Ray Tune with ASHA scheduling.The Nguyen benchmark suite evaluates polynomial, transcendental, and mixed functional types.
  • Evaluation: Ablation studies assess noise robustness, out-of-distribution generalization, and the effects of architecture and feature-design choices.

A. Problem Formulation

The methodology formulates symbolic regression as interpretable error minimization, then constructs an interaction-aware nonlinear library and trains a regularized MLP over its features.

  • A. Problem Formulation: Symbolic regression seeks an analytical expression that minimizes prediction error while enforcing interpretability through a complexity penalty.NSR approximates this objective with a hybrid neural–sparse learning pipeline.
  • B. Nonlinear Feature Library Construction: The nonlinear basis includes identity, trigonometric, exponential, logarithmic, polynomial, and hyperbolic-tangent transformations.The listed transformations are F = {id, sin, cos, exp, log(1 + x), x2, x3, tanh}.
  • B. Nonlinear Feature Library Construction: NSR adds pairwise products of transformed features to capture higher-order interactions.
  • B. Nonlinear Feature Library Construction: The complete feature library combines the basis and interaction terms, with dimensionality determined by the resulting feature count.
  • B. Nonlinear Feature Library Construction: Joblib parallelizes feature-library construction to improve efficiency.
  • C. Neural Approximation Network: A multilayer perceptron parameterized by θ models the target from the constructed feature representation.
  • C. Neural Approximation Network: The MLP uses L hidden layers of width H, dropout and weight decay regularization, and the AdamW optimizer.
  • C. Neural Approximation Network: Training uses GPU acceleration through CUDA or Apple MPS backends.

D. Sparse Symbolic Extraction Using LASSO

NSR recovers symbolic structure after neural approximation by applying LASSO to the feature representation, with Ray Tune and ASHA optimizing model and sparse-regression parameters.

  • Sparse Symbolic Extraction: Following neural training, LASSO performs sparse symbolic extraction from the learned representation.
  • Sparse Symbolic Extraction: Nonzero LASSO coefficients identify active symbolic terms, which SymPy converts into an analytical expression.
  • Sparse Symbolic Extraction: Exact coefficient recovery is motivated by compressed-sensing theory when the full feature matrix satisfies the RIP condition.
  • Hyperparameter Optimization: Ray Tune automates selection of neural and sparse-model parameters by searching configurations including regularization strength and batch size.The stated search values are λ ∈ {10^-4, 10^-3, 10^-2} and batch sizes 32, 64, and 128.
  • Hyperparameter Optimization: Ray Tune evaluates configurations using validation RMSE, while ASHA aggressively prunes poor-performing trials.
  • NSR Pipeline: Algorithmically, NSR constructs nonlinear and interaction features, trains a neural network, extracts coefficients with LASSO, and outputs a SymPy expression.

A. Notation

The NSR framework combines nonlinear feature construction, neural approximation, and one-time LASSO-based symbolic extraction. Its computational profile is dominated by feature-library size, neural training, and sparse regression costs.

  • A. Notation: N denotes samples, d input dimensionality, F base nonlinear functions, k expanded features, H hidden width, L hidden layers, and E training epochs.
  • Feature construction: Feature construction evaluates all k features across N samples, requiring O(Nk) time and O(Nk) space.The passage identifies this phase as the main memory limitation.
  • Neural approximation: Neural approximation uses a multilayer perceptron in the expanded feature space, with parameter memory O(kH+(L−1)H2) and activation memory O(NLH).
  • Sparse extraction: LASSO symbolic extraction scales as O(Nk2) time and O(Nk+k) memory, and runs once after neural training rather than through repeated symbolic search.
  • Overall complexity: GPU acceleration substantially reduces neural-training wall-clock time for medium-sized feature libraries, affecting total computation despite the framework’s multiple stages.

G. Limitations and Scalability Considerations

NSR’s scalability is constrained by feature-library construction and storage, while its experiments use Nguyen benchmarks, regularized MLPs, and distributed tuning. GPU acceleration makes the added neural and tuning costs feasible while supporting robustness and symbolic recovery.

  • G. Limitations and Scalability Considerations: Feature-library creation grows combinatorially with input dimensionality and interaction order, increasing memory use and producing quadratic sparse-regression complexity.Direct storage of Φ(X) can become unfeasible for large-scale or high-dimensional datasets.
  • G. Limitations and Scalability Considerations: Future scalability directions include adaptive library pruning, learned feature selection, and low-rank or streaming representations while maintaining interpretability.
  • Benchmark setup: The evaluation uses Nguyen-1 through Nguyen-7, with polynomial, trigonometric, and transcendental targets sampled from [−1, 1]d.Each task uses n = 1000 uniformly distributed input samples.
  • Feature library: The nonlinear library applies functions including identity, sin, cos, exp, log(1 + x), powers, and tanh, then adds second-order pairwise interactions.
  • Neural model: The neural approximator is a regularized MLP with depth L ∈{1, 2, 3} and width H ∈{32, 64, 128}, selected through hyperparameter optimization.Layers use GELU or ReLU, dropout, and weight decay.
  • Runtime: GPU acceleration makes NSR’s neural approximation and tuning costs feasible while improving robustness and symbolic recovery performance.Runtime assessment covers feature creation, model training, and symbolic extraction.
  • Hyperparameter optimization: Ray Tune runs 20 parallel trials with ASHA scheduling across learning rate, depth, width, batch size, and LASSO regularization.The search includes λ ∈{10−4, 10−3, 10−2}.

E. Baselines

The evaluation compares NSR with SINDy and a non-tuned neural baseline using RMSE, symbolic accuracy, sparsity, and out-of-distribution generalization. Across Nguyen benchmarks, optimized NSR generally provides reliable predictive performance, while tuned training improves convergence and symbolic recovery.

  • Baselines: NSR is compared with SINDy and a non-tuned neural baseline on RMSE, symbolic accuracy, sparsity, and OOD generalization.The experiments use Nguyen benchmarks and average runtime values over five independent runs on identical hardware configurations.
  • Baselines: NSR attains the minimum RMSE on Nguyen-1, Nguyen-2, and Nguyen-3, while PySR excels on Nguyen-4.The comparison covers PySR, SINDy, and optimized NSR across multiple Nguyen benchmarks.
  • Baselines: PySR’s Nguyen-4 advantage is attributed to evolutionary exploration of compact trigonometric forms, whereas NSR behaves more reliably on noisy polynomial benchmarks.The passage contrasts PySR’s task-specific strength with NSR’s cross-task reliability.
  • Baselines: Tuned NSR converges faster, reaches lower minimum loss, and shows more consistent optimization than baseline NSR.These training improvements are attributed to ASHA-driven hyperparameter selection.
  • Baselines: NSR’s extracted equations closely align with ground truth, differing only by minor numerical amounts.This result supports the interpretability of LASSO-based symbolic extraction on the Nguyen-3 example.

D. Prediction Quality

Prediction-quality analyses examine approximation accuracy, noise robustness, OOD behavior, and the contributions of NSR’s design components. Tuned NSR remains accurate under noise and outside the training interval, while ablations show that feature expansion, interactions, depth, and tuning materially affect performance.

  • D. Prediction Quality: Predicted and true outputs align closely along the diagonal for the Nguyen-5 example, indicating high-accuracy functional approximation.The comparison is shown as a scatter plot of predicted versus actual values.
  • E. Noise Robustness: Tuned NSR consistently surpasses SINDy as Gaussian noise increases across σ ∈ {0, 0.01, 0.05, 0.1}.Its performance degrades more gradually, supporting neural smoothing before sparse extraction.
  • F. Out-of-Distribution Generalization: NSR shows considerably more consistent predictions than SINDy when trained on [-1,1] and tested on [-2,2] for Nguyen-4.SINDy shows marked divergence beyond the training interval.
  • Ablation Study: Removing nonlinear transformations greatly increases RMSE and reduces symbolic fidelity, demonstrating the importance of a varied feature library.The ablation compares the complete nonlinear library with polynomial-only and trigonometric-only alternatives.
  • Ablation Study: Removing second-order interactions increases approximation error and often omits cross-dimensional relationships, particularly on Nguyen-6 and Nguyen-7.The result highlights the role of interaction-aware features in multivariate benchmarks.
  • Ablation Study: Three-layer networks balance nonlinear representation and symbolic extraction better than shallow models, while greater depth yields diminishing returns and may increase variance.The study evaluates L = 1, L = 2, and L = 3 networks.
  • Ablation Study: Ray Tune lowers validation RMSE and improves symbolic accuracy by selecting learning rates, batch sizes, depths, widths, and regularization strengths.The tuned model also converges more quickly and discovers simpler, more accurate expressions.
  • Ablation Study: The ablation study finds that feature expansion, interaction modeling, neural depth, and hyperparameter tuning are all essential to accuracy or expression interpretability.Removing any listed component reduces accuracy or produces less understandable expressions.

VIII. DISCUSSION

The discussion finds that NSR combines neural approximation with sparse modeling to improve symbolic recovery, robustness, interpretability, and generalization. It also identifies computational, smoothness, coefficient-bias, and training-cost limitations.

  • Performance and robustness: NSR models intricate nonlinear and transcendental functions before applying LASSO, producing more consistent and accurate symbolic recovery than SINDy on such benchmarks.SINDy performs well for low-complexity polynomials but struggles with substantial nonlinearities or transcendental elements.
  • Performance and robustness: Ray Tune and ASHA improve convergence and architecture selection, enabling optimized NSR to surpass SINDy and untuned NSR in almost all benchmarks.The reported improvements include reduced RMSE and enhanced symbolic coefficients.
  • Performance and robustness: LASSO expressions closely match ground-truth structure and coefficients under noise because neural approximation smooths the underlying function before symbolic extraction.This stability contrasts with direct genetic-programming approaches, which are more affected by noise and initial conditions.
  • Performance and robustness: NSR generalizes better than SINDy in out-of-distribution scenarios, although very deep networks can overfit.The authors attribute extrapolation to a continuous latent mapping learned across an extensive feature library.
  • Limitations: Interaction terms can expand the feature library combinatorially, increasing computational expense and limiting scalability in high-dimensional settings.Feature overlap among interaction terms is also identified as a high-dimensional difficulty.
  • Limitations: NSR assumes smooth target functions, while LASSO may shrink coefficients and neural training adds runtime compared with purely sparse methods.Highly discontinuous or piecewise functions may be difficult to approximate accurately.
  • Implications: The framework is presented as applicable to system identification, physics discovery, engineering optimization, and broader scientific machine-learning tasks.The authors describe a balance among accuracy, interpretability, and computational practicality.

X. FUTURE WORK

Future work focuses on reducing feature-library growth, improving adaptive symbolic discovery, mitigating training dynamics, and validating NSR on real scientific data. The broader aim is to extend the method from controlled benchmarks to complex systems.

  • Scalability and adaptability: Adaptive library learning could build or remove functions according to relevance, reducing library size and computational expense.Suggested directions include neural-guided basis selection, evolutionary library creation, and dictionary learning.
  • Scalability and adaptability: Transformer-guided symbolic reasoning could complement LASSO's predetermined feature set and extend discovery beyond linear combinations of basis functions.The proposed combination targets more extensive and versatile symbolic discoveries.
  • High-dimensional systems: The exponential growth of interaction terms restricts high-dimensional NSR, motivating sparse interactions, hierarchical or low-rank features, and attention-based variable selection.These approaches are proposed for multivariate systems such as PDEs, biological networks, and control systems.
  • Training dynamics: Future optimization strategies include educational progression for symbolic activities, meta-learning across benchmarks, and loss functions designed for symbolic sparsity.These directions address the continuing influence of neural training dynamics despite Ray Tune's convergence benefits.
  • Scientific integration: Integrating NSR with physics engines or numerical solvers could identify governing equations directly from trajectory data.This would connect data-driven learning with mechanistic modeling.
  • Scientific validation: Applying NSR to climate, biological, and engineering sensor datasets would test effectiveness beyond synthetic benchmarks.The authors identify real-world noisy data as a route toward broader scientific relevance.

APPENDIX A FUNDAMENTAL IMPLEMENTATION ASPECTS

The implementation constructs a safeguarded nonlinear feature space, fits an MLP smooth approximation with training safeguards and hardware acceleration, and supports reproducible execution.

  • Reproducibility: The appendix presents selected implementation components, while the full repository contains additional utilities, ablation scripts, and plotting code.The source is referenced as the location of the complete implementation.
  • Feature construction: The feature library transforms inputs into nonlinear basic functions and interaction components while replacing NaN and infinite values with safe numerical values.Feature names are generated for each function-input combination.
  • Neural approximation: An MLP learns a smooth approximation over the nonlinear feature space using configurable input dimension, hidden width, depth, linear layers, and GELU activations.The model output is squeezed to produce the prediction vector.
  • Model training: Training uses early stopping, gradient clipping, and GPU acceleration to control optimization and improve execution efficiency.The loop evaluates validation loss and tracks patience before stopping.

D. Sparse Symbolic Equation Extraction

Sparse equation extraction uses LASSO to identify active terms from the feature representation, while Ray Tune searches neural hyperparameters with early stopping and evaluates noise and OOD behavior.

  • Sparse extraction: LASSO regression promotes sparsity in the coefficient vector during symbolic expression recovery.The implementation fits LASSO to the feature matrix and target values.
  • Sparse extraction: Coefficients above the activity threshold are converted into an analytical expression, with nonzero terms identifying the selected symbolic components.SymPy reconstructs the final expression from active coefficients.
  • Hyperparameter optimization: Ray Tune performs distributed hyperparameter search with early stopping over hidden width, network depth, learning rate, and batch size.The listed search space includes widths 64, 128, and 256; depths 1, 2, and 3; learning rates from 10^-4 to 10^-2; and batch sizes 64 or 128.
  • Evaluation: Evaluation varies noise levels and tests out-of-distribution domains using validation RMSE from predictions on held-out data.The evaluation loop trains on noisy targets and records RMSE results.
  • Reproducibility: Experiments use fixed random seeds and support CPU, CUDA-enabled GPU, and Apple MPS execution.Source code, configurations, and reproduction scripts are provided publicly.
Loading 2609.01102v1…