PaperlayerView exact revision on arXiv

Source-linked AI summary

SMILE: Bridging Continuous Optimization and Discrete Symbolic Recovery

Mansooreh Montazerin, Antonio Ortega, Ajitesh Srivastava

arXiv:2609.04639v1cs.LG

TL;DR

Symbolic regression must search large combinatorial spaces while retaining interpretable closed-form expressions. SMILE combines structural analysis, continuous optimization in a fixed symbolic-activation network, and discrete recovery; it shows robust symbolic recovery under noise and favorable accuracy–complexity trade-offs across benchmark settings.

  • Problem

    Symbolic regression seeks interpretable closed-form expressions, but existing methods face slow convergence in combinatorial search spaces and limited use of compositional structure.

  • Method

    SMILE uses structural analysis, continuous optimization of a network with five fixed symbolic activations, and symbolic recovery through pruning, extraction, optimization, and coefficient rounding.

  • Results

    SMILE achieves the highest symbolic solution rate at the largest noise level, lies on the accuracy–complexity Pareto front across ground-truth and black-box datasets, and recovers simpler expressions in a fraction of leading baselines’ time.

  • Takeaways & Limitations

    SMILE provides a hybrid route to symbolic recovery that favors robustness, simpler expressions, and efficient discovery across noisy and black-box regression tasks.

Abstract

from arXiv · show

Symbolic regression (SR) discovers closed-form mathematical expressions from data, offering interpretability beyond black-box models. Existing methods suffer from slow convergence in combinatorial search spaces and lack mechanisms to exploit compositional structure in the data. We introduce SMILE (Sine, Multiplication, Identity, Logarithm, Exponential), a hybrid framework that unifies continuous gradient-based optimization with discrete symbolic recovery through three stages: structural analysis of the data to identify the compositional hierarchy of the target expression, continuous optimization to learn parameters of a network that encodes the target expression using interpretable activations, and symbolic recovery through structured pruning, coefficient optimization, and rounding. This final stage distills the learned network into a compact expression with exact symbolic constants. We evaluate SMILE on SRBench across ground-truth and black-box datasets, with ablation studies validating each component. SMILE achieves the highest symbolic solution rate at the largest noise levels, demonstrating strong robustness where competing methods degrade substantially. It consistently lies on the Pareto front of accuracy versus complexity, recovering significantly simpler expressions in a fraction of the time required by the competing methods.

1 Introduction

Symbolic regression seeks interpretable closed-form relationships but faces a vast combinatorial search space and slow convergence. SMILE addresses these challenges by combining data-driven structural analysis, continuous optimization, and discrete symbolic recovery.

  • Motivation: Symbolic regression searches jointly over equation structure and parameters to discover interpretable closed-form relationships from data.Its flexibility contrasts with predefined-model regression and opaque neural-network representations.
  • Motivation: Discrete symbolic methods can converge slowly because combinatorial expression landscapes prevent gradient-based guidance.Predefined operations and sequential generation can also produce overly complex models.
  • SMILE framework: SMILE formulates symbolic regression as structural optimization of a neural architecture, unifying continuous training with discrete symbolic recovery.Its pipeline uses structural analysis, continuous optimization, and symbolic recovery.
  • SMILE framework: SMILE uses five fixed symbolic activations—Sine, Multiplication, Identity, Logarithm, and Exponential—to represent expressions with interpretable network components.The activations are designed to reflect operators frequently encountered in physical and scientific laws.
  • SMILE framework: The end-to-end pipeline classifies problems through a log-linear R2 scan before training and recovering expressions through pruning, extraction, optimization, and coefficient rounding.The scan selects direct training, inverted-target training, or decomposition into simpler subproblems.

2 Related work

Related symbolic-regression methods navigate discrete, continuous, or hybrid search spaces with different trade-offs in efficiency, expressivity, complexity, and interpretability. SMILE distinguishes itself by analyzing compositional structure before fitting and recovering expressions from a compact fixed-activation network.

  • Overview: Symbolic regression methods differ primarily in how they explore the space of candidate expressions.The main paradigms include discrete evolutionary search, neural generation, reinforcement learning, and continuous optimization.
  • Discrete evolutionary search: Genetic programming evolves tree-structured expressions but often incurs high computational cost, hyperparameter sensitivity, and excessive expression complexity.Its heuristic exploration operates over a large combinatorial space.
  • Neural generation and reinforcement learning: Reinforcement-learning and neural-generation approaches guide symbolic construction while remaining discrete, which can cause slow convergence and sensitivity to reward design.These methods select symbolic operators and operands sequentially from predefined libraries.
  • Continuous and differentiable search: Continuous approaches improve optimization efficiency but may limit expressivity to a predefined dictionary or impose structural constraints on symbolic architectures.The cited approaches include sparse regression, neural fitting with decomposition, and fixed-activation networks.
  • SMILE’s distinction: SMILE analyzes compositional structure before fitting, uses fixed symbolic activations, and recovers expressions through structured pruning and rounding.This design targets high symbolic recovery rates while producing lower-complexity expressions across standard benchmarks.

3 Methodology

SMILE combines structural analysis, continuous optimization, and symbolic recovery in a compact neural architecture with interpretable activations. Its training and pruning mechanisms are designed to learn accurate expressions while simplifying the network into closed form.

  • Structural analysis: SMILE analyzes data to identify compositional structure and select a training configuration before optimizing the network.This reduces the effective degrees of freedom and simplifies symbolic recovery.
  • SMILE architecture: Each hidden layer uses sine, identity, logarithm, exponential, and multiplication neurons to represent symbolic operations.The multiplication neuron supports learned products and powers through trainable exponents.
  • Expressive power: SMILE networks are universal approximators because stacked blocks can emulate multilayer perceptrons with sigmoid activations.A single block constructs the sigmoid using identity, exponential, and logarithm neurons, while identity neurons reproduce affine transformations.
  • Gating and pruning: Differentiable edge and activation gates enable pruning of individual connections and entire neurons after continuous training.The gates provide a smooth relaxation of discrete selection and are driven toward binary decisions by a sparsity penalty.
  • Training and optimization: The training objective combines mean squared error with clamping and gating penalties to fit data, stabilize logarithmic and exponential activations, and promote sparsity.Input clamping protects against invalid logarithm inputs and rapidly growing exponential values, while penalties discourage reliance on clamped regions and intermediate gate values.
  • Symbolic recovery: Symbolic recovery distills the optimized network into a compact closed-form expression through pruning, parametric optimization, and coefficient rounding.The stated pipeline proceeds from structural analysis through continuous optimization to symbolic recovery.

4 Experiments and results

SMILE is evaluated on SRBench ground-truth and black-box datasets using symbolic recovery, accuracy, complexity, and training-time metrics. It is especially robust to noise, remains on favorable accuracy–complexity trade-offs, and benefits substantially from each pipeline component.

  • Benchmark and metrics: SRBench includes 119 Feynman equations, 14 Strogatz problems, and 57 filtered black-box regression problems evaluated against 15 original methods plus four additional baselines.Ground-truth tasks use symbolic solution rate and accuracy solution rate; black-box tasks use median R2 and median complexity.
  • Ground-truth results: At the highest evaluated noise level, SMILE achieves the top Feynman symbolic solution rate while its SSR decreases by at most 8%, versus more than 30% for ParFam and PySR.On noise-free data, SMILE ranks third in SSR behind PySR and ParFam.
  • Ground-truth results: SMILE recovers each Feynman expression in minutes rather than the several hours required by ParFam, PySR, and AI Feynman.Its accuracy solution rate is lower than the top baselines because the limited-depth pipeline favors exact symbolic recovery over regression accuracy.
  • Accuracy–complexity trade-off: SMILE lies on the Pareto front of accuracy versus expression complexity on both Feynman and black-box datasets, producing substantially simpler expressions at competitive accuracy.The Feynman comparison uses accuracy solution rate, whereas the black-box comparison uses median R2.
  • Ablation study: Removing greedy pruning or parametric optimization reduces SSR by approximately 35%, while removing variable analysis or gradient-based rounding reduces SSR by approximately 15% and increases complexity.These ablations are conducted on the Feynman dataset.

5 Discussion and conclusion

SMILE combines interpretable neural activations with structural analysis, continuous optimization, and discrete symbolic recovery. Across SRBench, it is robust at high noise and offers favorable accuracy–complexity–time trade-offs, while remaining limited by shallow architectures and assumptions about compositional structure.

  • Contribution: SMILE integrates structural analysis, continuous optimization, and discrete symbolic recovery in a framework built from fixed interpretable activations.The recovery stage uses pruning and rounding to remove small or unstable parameters.
  • Results: SMILE ranks among the top methods in SSR across noise levels and achieves the highest SSR at the largest noise level, where competing methods degrade substantially.The reported robustness is attributed to pruning, rounding, and structural analysis.
  • Results: SMILE remains on the Pareto front of accuracy versus complexity across ground-truth and black-box datasets while recovering simpler expressions in a fraction of leading-baseline training time.The conclusion states that ablation studies validate these strengths.
  • Limitations: SMILE favors shallow architectures because deeper networks increase degrees of freedom and expression complexity, making pruning harder without improving SSR.Increasing depth improves accuracy and R2 but does not improve symbolic solution rate.
  • Limitations: The data-driven analysis assumes compositional patterns common in physical and mathematical laws, which may not hold for arbitrary real-world relationships.The paper identifies adaptive depth selection and broader compositional analysis as future directions.

A Proof of universal approximation

SMILE networks are universal approximators because their interpretable activations can emulate sigmoid multilayer perceptrons and polynomial functions. The full pipeline then recovers compact symbolic expressions from trained networks through structured pruning and coefficient processing.

  • Sigmoid-based proof: SMILE networks are universal approximators by emulating multilayer perceptrons with sigmoid activations.The proof constructs a sigmoid from identity, exponential, logarithm, and exponential neurons, then stacks such blocks to recover arbitrary-width sigmoid MLPs.
  • Sigmoid-based proof: A Sigmoid block computes σ(a) = 1/(1 + exp(−a)) by sequentially combining linear, exponential, additive, logarithmic, and exponential operations.The identity neuron first computes a = w⊤x, after which the remaining neurons construct the sigmoid expression.
  • Polynomial-based proof: Multiplication and identity neurons represent monomials and their linear combinations, providing an independent polynomial-based universality argument.Polynomial density on compact sets is invoked through the Stone–Weierstrass theorem.
  • Symbolic recovery: The symbolic recovery stage distills a trained SMILE network into a closed-form expression using greedy pruning, parametric optimization, and gradient-based rounding.The complete discovery pipeline combines data analysis and decomposition with this recovery procedure.

B.1 Variable analysis: Window selection algorithm

SMILE analyzes variable contributions and selects defensible fixation windows before decomposing the target and recovering symbolic expressions. Its window criterion limits variation caused by a fixed variable, while recovery prunes gates, refits coefficients, and rounds parameters when the output change is negligible.

  • Variable analysis: SMILE begins variable analysis by fitting log-linear relationships for each input and comparing their R2 values.When all variables have comparably high R2, the pipeline compares polynomial fits to y and 1/y before training shallow networks.
  • Variable analysis: A variable with the lowest log-linear-fit R2 triggers decomposition by fixing that variable near two values and recovering the remaining relationship from windowed data.The method uses additive or multiplicative residuals to train a second network for the isolated variable.
  • Window selection: The algorithm tests six log-spaced relative window widths from 20% down to 2%, with each candidate centered at x∗ and having width W = 2p · x∗.Candidates are evaluated from largest to smallest, and the first passing window is selected.
  • Window selection: A window passes when C < τwindow = 0.05, meaning variation caused by xk is at most 5% of y’s standard deviation within the slab.The constancy score compares estimated drift across the window with natural y variability while other variables are restricted to a narrow band.
  • Symbolic recovery: Symbolic recovery greedily closes gates, extracts the active subgraph, refits numerical constants by least squares, and rounds coefficients when permitted by the rounding criterion.The recovery algorithm returns a closed-form expression after pruning and coefficient processing.
  • Window selection: If no candidate window passes, SMILE falls back to the smallest window, where treating the variable as constant is considered most defensible even if imperfect.This fallback supplies data for the subsequent SMILE fit.
  • Gradient-based rounding: Gradient-based rounding replaces ci with ri when the estimated output change is below threshold τ for every input sample, recovering exact constants without degrading expression quality.The bound is motivated by continuity and a mean-value-theorem argument.

C Datasets

SMILE is evaluated on ground-truth Feynman and Strogatz problems and on selected continuous-feature black-box regression problems from SRBench.

  • SRBench contains 252 regression problems across 119 Feynman equations, 14 Strogatz ODE problems, and 122 black-box problems.
  • The Feynman collection covers physics equations with 1–9 input variables and functions including polynomials, trigonometric functions, exponentials, and square roots.
  • The Strogatz collection contains 14 nonlinear-dynamics and chaos ODE problems requiring recovery of governing right-hand sides.
  • Black-box evaluation uses 57 continuous-feature problems with input dimension d ≤10, assessed by median R2 and expression complexity because ground-truth equations are unavailable.

D Baselines

The baselines span evolutionary, neural, continuous-optimization, and hybrid symbolic-regression methods, while SMILE uses fixed configurations across datasets.

  • SMILE is compared with 19 SRBench methods, including recent systems PySR, uDSR, E2E, and ParFam.
  • PySR evolves, simplifies, and gradient-optimizes expression trees using parallel multi-population genetic programming.
  • uDSR combines recursive simplification, neural-guided search, pretraining, genetic programming, and linear models, with substantial coordination cost.
  • E2E predicts complete expressions in one forward pass and refines constants with BFGS, but may struggle outside its pretraining distribution.
  • ParFam performs continuous global optimization over user-specified parametric symbolic families and applies sparsity-based coefficient pruning.
  • EQL÷ is excluded because it lacks logarithm, exponential, and square-root operations needed for many benchmark equations.
  • Hyperparameters are fixed across all datasets without per-problem tuning.

E.2 Ground-truth datasets

On ground-truth benchmarks, SMILE is compared across accuracy, symbolic solution rate, complexity, training time, recovered expressions, and noise robustness.

  • Figure 6 compares Feynman methods using median R2 test at multiple precision levels, expression complexity, and training time.
  • SMILE completes in minutes, whereas most competitive baselines require several hours.
  • At the highest noise level on Strogatz, SMILE achieves the best accuracy solution rate among all baselines.
  • SMILE’s Strogatz accuracy solution rate decreases by at most 20% from noise-free to highest noise, versus approximately 70% for ParFam and PySR.
  • SMILE recovers five of six visualized expressions exactly with R2 = 1, while the Gaussian PDF is approximated with R2 > 0.99.
  • Across Feynman and Strogatz, SMILE maintains competitive SSR as noise increases and declines more gradually than competing methods.

E.3 Black-box dataset

On 57 black-box problems, SMILE prioritizes speed and low expression complexity, but achieves lower median R2 than several baselines.

  • SMILE achieves the lowest training time among baselines on the 57 black-box problems.
  • SMILE produces very low-complexity expressions, ranking second only to DSR.
  • SMILE’s median R2 is lower than several baselines on the black-box dataset.
  • Figure 9 visualizes ground-truth and SMILE-recovered expressions for six problems, including three Jin et al. and three Feynman examples.

E.4 Ablation studies

The ablation study shows that every SMILE component supports symbolic recovery, with pruning and parametric optimization most important for keeping the search tractable.

  • Removing variable analysis or gradient-based rounding reduces SSR by approximately 15% and accuracy solution rate by roughly 30%, without timeout errors.Training time roughly doubles in both cases.
  • Variable analysis preserves compositional decomposition, whereas its removal produces longer formulas that fit the data but fail to recover the ground-truth symbolic form.
  • Gradient-based rounding is required to convert learned exponents and coefficients into exact symbolic constants for successful symbolic recovery.
  • Removing greedy pruning or parametric optimization drops SSR by approximately 35% and causes a significant fraction of formulas to incur timeout errors.Both removals increase the downstream combinatorial burden.
  • Expression complexity increases across all four ablations because compositional structure, redundant components, or coefficient simplification stages are lost.
  • Greedy pruning and parametric optimization are most critical for a manageable search space, while variable analysis and gradient-based rounding improve recovered-expression quality and interpretability.

NeurIPS Paper Checklist

The checklist responses report that the paper’s claims match its contributions and scope, while describing experimental, theoretical, and reproducibility details. It also records limitations related to shallow architecture and compositional-structure assumptions.

  • The abstract and introduction accurately state SMILE’s framework, data-driven decomposition pipeline, and SRBench evaluation.
  • The reported limitations are lower accuracy solution rates from the shallow architecture and reliance on compositional structure assumptions that may not hold for arbitrary black-box datasets.
  • The universal approximation theorem and gradient-based rounding theorem are supported by a proof sketch or complete proof in the paper’s appendices.
  • Training, testing, evaluation, and methodological details are described across the main paper, appendices, and experimental sections.
  • The paper reports three independent trials for ground-truth datasets and median R2 and median complexity for black-box datasets.
Loading 2609.04639v1…