PaperlayerView exact revision on arXiv

Source-linked AI summary

A Latent Space Optimization Approach for Symbolic Discovery of Dynamical Models

Tongjia Liu, Ilias Mitrai

arXiv:2610.11059v1eess.SYcs.SY

TL;DR

Symbolic regression can discover equations without fixing their form in advance, but global MINLP methods can be computationally expensive. The paper combines a VAE latent representation with BO and globally solved parameter fitting; it reports discovering governing equations with less computational effort than mathematical-programming methods and higher sample efficiency than evolutionary baselines under tight budgets.

  • Problem

    Symbolic regression searches expression forms and parameters without a priori fixing functional form, while MINLP approaches can be computationally expensive, particularly for large dynamical datasets.

    Source: §I · p. 1

  • Method

    A VAE maps expression trees to a continuous latent space, where BO searches candidate functions and global optimization solves their parameter-estimation problems.

    Source: §I · p. 1

  • Results

    The framework discovers correct governing equations with less computational effort than mathematical-programming methods and greater sample efficiency than evolutionary baselines under tight evaluation budgets.

    Source: §I · p. 2, §V · p. 6

  • Takeaways & Limitations

    The framework is evaluated on reaction-rate-law discovery and dynamics identification in a continuous stirred-tank reactor.

    Source: §I · p. 2

Abstract

from arXiv · show

In this paper, we propose a hybrid latent-space Bayesian Optimization (BO) and global optimization framework for solving symbolic regression tasks to discover dynamical models from data. Symbolic regression can discover equations from data without fixing the functional form of the expression a priori. The proposed framework uses the fact that if the functional form of the expression is fixed, the symbolic regression task reduces to a parameter estimation problem, which must be solved to global optimality. Motivated by this structure, first, we train a VAE to map the discrete space of expression trees into a continuous latent space. Then, we use BO to search this latent space while assessing a candidate expression's prediction error by solving the parameter estimation problem to global optimality. We evaluate our framework across two case studies: static reaction rate law discovery and dynamic concentration identification in a continuous stirred-tank reactor (CSTR). The results show that our framework identifies the true governing equations faster than MINLP formulations without solver timeouts and achieves higher sample efficiency than evolutionary baselines under tight evaluation budgets.

I. INTRODUCTION

Symbolic regression searches both expression forms and their parameters, offering interpretable models without fixing a functional form in advance. The paper combines latent-space Bayesian optimization with global parameter fitting to address the computational cost and limitations of existing approaches.

I. INTRODUCTION

  • Sparse-regression methods are restricted to predefined basis functions and cannot discover governing terms omitted from their candidate library.Other neural approaches can require large datasets, smooth functions, or domain-specific inputs such as physical units.Source: §I · p. 1
  • MINLP formulations offer deterministic global optimality but scale poorly as tree depth, allowed operators, and data points increase, especially for densely sampled dynamical systems.Local solvers are faster but can become trapped in local solutions and depend strongly on initialization.Source: §I · p. 1
  • The proposed framework uses BO to navigate function space while minimizing globally solved parameter-estimation evaluations once a candidate's functional form is fixed.This treats symbolic regression as black-box optimization, with the fitted expression assessed by prediction accuracy.Source: §I · p. 1
  • A VAE maps discrete expression trees into a continuous latent space, while prior latent-space approaches used local parameter optimization and targeted static functions rather than dynamical models.The paper evaluates its framework on reaction-rate laws and CSTR dynamics.Source: §I · pp. 1–2

II. SYMBOLIC DISCOVERY OF MODELS VIA MIXED INTEGER OPTIMIZATION

The paper formulates symbolic discovery as minimizing prediction error over expression trees and their continuous parameters. It specifies tree structure and evaluation constraints, yielding a nonconvex MINLP.

II. SYMBOLIC DISCOVERY OF MODELS VIA MIXED INTEGER OPTIMIZATION

  • The symbolic-regression formulation seeks a function minimizing mean absolute error on data from either a static system or a dynamical model.The dataset pairs inputs with outputs generated by an underlying function.Source: §II · p. 2

A. Symbolic Representation of Target Expressions

  • Each candidate function is a binary expression tree whose nodes take binary or unary operators, variables, or numerical constants.The representation specifies a tree depth and node count.Source: A. Symbolic Representation of Target… · p. 2

B. Tree- and value-defining constraints

  • Binary assignment variables select operators or operands at valid nodes, while constraints enforce valid assignments, child requirements, variable inclusion, and node-value computation.The constraints exclude constant-only expressions and evaluate the tree at each data point.Source: B. Tree- and value-defining constraints · p. 2
  • The resulting problem is a mixed-integer nonlinear program because value-defining constraints are nonconvex for operators such as multiplication and division.The formulation combines the tree constraints with its objective and value-defining constraints.Source: B. Tree- and value-defining constraints · p. 2

III. BLACK-BOX OPTIMIZATION FOR SYMBOLIC REGRESSION

The paper exploits the structure of symbolic regression: fixing expression structure leaves continuous parameter estimation. It learns a continuous representation of expression trees and uses BO to search it.

III. BLACK-BOX OPTIMIZATION FOR SYMBOLIC REGRESSION

  • Symbolic-regression complexity depends on tree depth, operator choices, and data size, while fixing binary structure variables leaves a continuous parameter-estimation problem.The remaining degrees of freedom are the continuous constants in the expression.Source: §III · p. 2
  • A variational autoencoder learns a continuous parameterization of the discrete space of symbolic differential equations.This representation supports navigating candidate expressions in latent space.Source: §III · p. 2
  • Figure 1 depicts the proposed symbolic-regression framework, with blue arrows marking the Bayesian-optimization loop.The figure identifies the loop within the broader framework.Source: Figure 1 · p. 3

A. Continuous Representation via Variational Autoencoders

The discrete expression-tree representation is converted to integer node labels, then encoded into a continuous latent distribution from which the decoder reconstructs trees. The VAE is trained on syntactically valid random trees using reconstruction and latent-distribution losses.

A. Continuous Representation via Variational Autoencoders

  • Each expression tree is converted from binary variables into an ordered integer vector of node labels.Each node label is an integer in {1, . . . , |O|}.Source: A. Continuous Representation via… · p. 3
  • The encoder maps the integer representation to Gaussian parameters, samples a latent vector by reparameterization, and the decoder maps it back to a reconstructed tree.The latent distribution has mean μ and variance σ²; the decoder uses learnable parameters θd.Source: A. Continuous Representation via… · p. 3
  • The VAE is trained on random syntactically valid trees by minimizing reconstruction cross-entropy plus a weighted KL divergence to a standard normal prior.β controls latent-space smoothness.Source: A. Continuous Representation via… · p. 3

B. Evaluating the Fitness of an Expression Tree

For a fixed expression-tree structure, evaluating prediction error requires fitting its constants. The resulting continuous nonlinear optimization is solved globally, with infeasible decoded expressions receiving a penalty.

B. Evaluating the Fitness of an Expression Tree

C. Latent-Space Bayesian Optimization

The method uses Bayesian optimization in a learned latent space to search for accurate expression trees, evaluating candidates through the expression-fitness procedure. The paper assesses search efficiency, structural recovery, and noise robustness in reaction-rate-law and CSTR case studies.

C. Latent-Space Bayesian Optimization

  • A Gaussian Process with a Matérn kernel models expression prediction error in latent space, and Log Expected Improvement selects candidates.The objective is the prediction error of the decoded expression tree.Source: C. Latent-Space Bayesian Optimization · p. 3
  • Each candidate latent coordinate is decoded to a tree, checked for arity validity, and evaluated; invalid trees receive a penalty, while valid trees undergo continuous parameter optimization before the surrogate is updated.The workflow begins with randomly sampled valid trees and repeats until a predefined termination criterion is met.Source: C. Latent-Space Bayesian Optimization · p. 3

IV. COMPUTATIONAL RESULTS

  • The computational evaluation covers nonlinear reaction-rate-law discovery and governing-equation discovery for a CSTR with varying flowrate and a first-order reaction.The stated evaluation aims include search efficiency, structural recovery, and noise robustness.Source: §IV · p. 4

A. VAE Training and Latent Dimension Sensitivity

The VAE is trained for expression trees shared by both case studies, and latent-dimension sensitivity motivates a 15-dimensional representation. The reaction-rate study generates noisy concentration-rate observations for a reversible reaction.

A. VAE Training and Latent Dimension Sensitivity

  • Both case studies use a VAE trained for 100 epochs on depth-three, 15-node trees with {+, −, ∗} operators and {x1, x2, cst} operands.The encoder uses a 10-dimensional embedding and 256-unit hidden layer; training uses Adam, learning rate 10^-3, batch size 512, and β = 0.05.Source: A. VAE Training and Latent Dimension… · p. 4
  • ≈97.5% reconstruction accuracy is reached at dz = 15, and accuracy remains above 95% across candidate dimensions; dz = 15 balances reconstruction accuracy with BO search efficiency.The supplied figure presents training losses and test-set tree reconstruction accuracy across latent dimensions.Source: Figure 2 · p. 4, A. VAE Training and Latent Dimension… · p. 4

1) Problem Definition and Data Generation:

  • The rate-law case study uses concentration-rate observations for the reversible reaction 2A ⇌ B, with reactant concentrations sampled uniformly over 0.1–2.5 M.The dataset contains 200 observations.Source: 1) Problem Definition and Data… · p. 4
  • The simulated net-rate dataset uses ground-truth kinetic rate constants k1 = 0.50 and k2 = 0.20, with zero-mean Gaussian noise (σ = 0.01) added to simulate experimental uncertainty.The dataset comprises a 200 × 2 feature matrix and a 200 × 1 noisy target vector.Source: 1) Problem Definition and Data… · p. 4

2) Proposed Framework Performance:

The approach recovered the reaction-rate law’s true kinetic form, with a small noise-attributed extra term. It also achieved a lower objective value in less time than the MINLP comparison.

2) Proposed Framework Performance:

  • The lowest-loss expression structure appeared on BO’s first iteration, at evaluation #31 after 30 random trees initialized the GP surrogate.The search allowed 50 total function evaluations and took 82.23 seconds.Source: 2) Proposed Framework Performance · p. 4
  • The proposed approach recovered the true kinetic functional form, while a small linear term with constant 0.0033 was attributed to noise.The recovered form is 0.50C_A^2 − 0.20C_B, with the extra linear term 0.0033C_A.Source: 2) Proposed Framework Performance · p. 4

3) Comparison with baselines:

C. Case Study B: Dynamic Isothermal CSTR System Identification

This case study identifies the mass-balance dynamics of an isothermal CSTR with an irreversible A → B reaction. The dataset combines noisy concentration trajectories from five initial conditions with numerically estimated time derivatives.

1) Problem Definition and Data Generation:

  • The isothermal CSTR case models an irreversible A → B reaction; concentration and feed flow are its state variables, with specified volume, feed concentration, and reaction constant.The setup uses V = 1 L, C_f = 2 M, and k = 2 s^-1.Source: 1) Problem Definition and Data… · p. 5
  • The dataset contains 1,005 observations from five initial concentrations over 0–2.0 s, with F(t) = 6 + sin(10t) and Gaussian concentration noise of σ = 0.01.Each trajectory has 201 time points at Δt = 0.01 s; the system was simulated with RK45.Source: 1) Problem Definition and Data… · p. 5
  • Time derivatives were estimated from the noisy concentration data using second-order finite differences via numpy.gradient.The resulting data are represented as state matrices and target derivative vectors.Source: 1) Problem Definition and Data… · p. 5

2) Proposed Framework Performance:

The proposed framework recovered the exact CSTR dynamics in most tested runs and outperformed EDHiE on recovery and test error under the evaluation budget. Its discovered model also tracked the true concentration profile under an unseen flow-rate profile.

2) Proposed Framework Performance:

  • The section reports finding the equation after 40 iterations (70 total evaluations), while its performance discussion says BO found it after 20 iterations (50 total evaluations).The reported expression identifies the bilinear state-input coupling and closely estimates the feed and decay coefficients; total computation time was 164.76 seconds.Source: 2) Proposed Framework Performance · p. 5

3) Comparison with baselines:

  • The proposed framework recovered the exact dynamic expression in 7 of 10 runs; EDHiE recovered none, and the proposed method had lower test RMSE in the other 3 runs.The RMSE comparison covers 10 random seeds and 100 evaluations per run.Source: Figure 4 · p. 5, 3) Comparison with baselines · p. 5
  • The comparison used 100 total evaluations per seed for both methods, with 10 random seeds.The proposed approach used 30 initial samples and 70 BO iterations; EDHiE used a population of 10 over 10 generations.Source: 3) Comparison with baselines · p. 5
  • Without a budget limit, EDHiE required about 1,500 candidate function evaluations to discover the ground-truth expression.Source: 3) Comparison with baselines · p. 5
  • Under unseen flow, the proposed model matched the true concentration profile throughout, while EDHiE’s prediction plateaued near 1.5 M after about 0.5 s.The EDHiE expression lacks the self-stabilizing −2x1 decay term.Source: Figure 5 · p. 6, 3) Comparison with baselines · p. 6

V. CONCLUSIONS

The paper presents a VAE-based continuous representation of expression trees searched with Bayesian Optimization for symbolic regression. Across reaction-rate and CSTR case studies, it reports less computation time than deterministic mathematical programming and greater sample efficiency than evolutionary baselines.

V. CONCLUSIONS

  • The framework represents candidate functions as expression trees, embeds them in a continuous space with a VAE, and searches that space using Bayesian Optimization.Source: §V · p. 6
  • The framework was evaluated on reaction-rate-law discovery and identification of an isothermal CSTR governing equation.Source: §V · p. 6
  • The framework identified governing equations in less computation time than deterministic mathematical programming and was more sample-efficient than evolutionary baselines.The conclusion states that it consistently uncovered the ground-truth function within a tight evaluation budget.Source: §V · p. 6
Loading 2610.11059v1…