Source-linked AI summary
End-to-end symbolic regression with transformers
Pierre-Alexandre Kamienny, Stéphane d'Ascoli, Guillaume Lample, François Charton
TL;DR
Symbolic regression must recover interpretable expressions from observed values, yet its usual skeleton-then-constant-fitting pipeline is slow or inaccurate. The paper trains a Transformer to predict full expressions directly, optionally refines constants with BFGS, and finds faster inference with a narrowed GP accuracy gap.
Problem
Symbolic regression commonly separates skeleton prediction from non-convex constant fitting, while GP methods are accurate but slow and prior neural methods are less accurate and limited to D ≤3.
Method
The paper pretrains Transformers on synthetic data to predict complete symbolic expressions with constants, using a hybrid symbolic-numeric vocabulary and optional BFGS refinement.
Results
On SRBench, the model narrows the accuracy gap with state-of-the-art GP and provides several orders of magnitude of inference-time speedup.
Takeaways & Limitations
End-to-end constant prediction improves performance over skeleton prediction and supplies informed initial conditions for refining constants.
Takeaways & Limitations
The evaluated dimensions remain moderate, D < 10; adapting to truly high-dimensional problems remains future work, and closing the GP–DL gap remains challenging.
Abstract
from arXiv · showhide
Symbolic regression, the task of predicting the mathematical expression of a function from the observation of its values, is a difficult task which usually involves a two-step procedure: predicting the "skeleton" of the expression up to the choice of numerical constants, then fitting the constants by optimizing a non-convex loss function. The dominant approach is genetic programming, which evolves candidates by iterating this subroutine a large number of times. Neural networks have recently been tasked to predict the correct skeleton in a single try, but remain much less powerful. In this paper, we challenge this two-step procedure, and task a Transformer to directly predict the full mathematical expression, constants included. One can subsequently refine the predicted constants by feeding them to the non-convex optimizer as an informed initialization. We present ablations to show that this end-to-end approach yields better results, sometimes even without the refinement step. We evaluate our model on problems from the SRBench benchmark and show that our model approaches the performance of state-of-the-art genetic programming with several orders of magnitude faster inference.
Introduction
Symbolic regression seeks interpretable mathematical laws from observations, but conventional GP-based and two-step neural approaches trade accuracy for costly or limited inference. This paper instead trains Transformers to predict complete expressions directly, narrowing the GP accuracy gap while substantially accelerating inference.
- Symbolic regression selects interpretable analytical expressions while retaining broader function families than parametric statistics.
- The standard pipeline predicts an expression skeleton first, then fits its numerical constants with BFGS.
- GP methods achieve good accuracy but are slow because they search vast function spaces, repeatedly invoke BFGS, and learn each problem from scratch.
- Earlier neural approaches accelerate skeleton prediction but remain less accurate than state-of-the-art GP and are limited to D ≤3.
- The proposed Transformer predicts complete expressions, including constants, using a hybrid symbolic-numeric vocabulary and optionally refines constants with informed BFGS initialization.
- On SRBench, the model narrows the accuracy gap with state-of-the-art GP while providing several orders of magnitude faster inference.
1 Data generation
The model is pretrained on synthetic input-output examples generated from random mathematical functions and diverse input distributions. Functions and numbers are serialized into token sequences so a Transformer can learn symbolic regression end to end.
- Each synthetic training example pairs N input points in R^D with outputs computed by a randomly generated function f.
- Functions are sampled as operator trees whose internal nodes contain binary or unary mathematical operators and whose leaves contain variables or constants.
- The generation procedure samples dimensions, operators, tree structure, variables, and unary operators, while ensuring the first D variables appear in ascending order.
- Out-of-domain inputs or outputs larger than 10^100 cause an example to be discarded and regenerated.
- Inputs are drawn from mixtures of Gaussian or uniform clusters with random centroids, variances, weights, rotations, and whitening.
- Numbers use four-significant-digit base-10 floating-point tokens, while functions use prefix-order sequences combining symbolic and numeric tokens.
2 Methods
The method uses a Transformer to predict symbolic-regression solutions, with embedding, permutation-aware encoding, inference-time scaling and bagging, and optional constant refinement.
- Model: An embedder maps each input point to one embedding before the Transformer, reducing long sequences that otherwise challenge quadratic attention complexity.Each point is projected through a 2-layer ReLU feedforward network from a padded input representation to dimension d_emb.
- Model: The sequence-to-sequence Transformer uses a deeper decoder and removes encoder positional embeddings to respect permutation invariance of input points.The architecture has 4 encoder layers, 16 decoder layers, 16 attention heads, and embedding dimension 512.
- Model: The embedder architecture did not improve in explored variants, while BFGS refinement uses analytical gradients and CPU-based candidate refinement.Inference combines GPU-parallelizable forward passes with CPU-based refinements; the explored embedder variants yielded no improvement.
- Inference tricks: End-to-end prediction supplies constants to BFGS as informed initial guesses, improving over skeleton prediction and avoiding the failure of random initialization.The E2E model predicts the function and constants simultaneously, after which BFGS can fine-tune the predicted constants.
- Inference tricks: At inference, scaling normalizes inputs and then unscales predicted variables, making the model insensitive to input-point scale.The procedure replaces x with (x − μ)/σ before prediction and recovers an approximation of f afterward.
- Inference tricks: For more than 200 input points, bagging splits the data into bags of 200, generates candidates per bag, and ranks candidates before refinement.The procedure uses B bags, C candidates per bag, removes redundant skeleton functions, and retains the best K candidates; experiments use B = 100, C = 10, K = 10.
3 Results
The model is evaluated on in-domain and out-of-domain symbolic-regression problems using R2 and tolerance-based accuracy, with ablations spanning difficulty, data quantity, extrapolation, and noise. End-to-end prediction improves over skeleton prediction, while refinement particularly benefits high-precision accuracy and overall benchmark tradeoffs.
- In-domain performance: Without refinement, the E2E model outperforms the matched skeleton model on R2 and Acc0.1, but not on Acc0.001.Small constant errors limit high-precision performance.
- In-domain performance: Three-fold higher Acc0.001 follows refinement, which also improves the other reported metrics.Refinement addresses errors in the constants predicted by the E2E model.
- In-domain performance: Random BFGS initialization degrades E2E performance, whereas initializing with E2E-estimated constants is crucial.The comparison isolates the benefit of an informed initialization.
- Ablations: Increasing unary operators, binary operators, or input dimension degrades performance, while more input points improve E2E performance more substantially than skeleton performance.The input-point ablation supports the importance of abundant observations for predicting constants.
- Extrapolation and robustness: Extrapolation performance remains decent at σ = 32 despite degrading as test-point scale increases.The ablation varies test-point variance relative to the normalized training inputs.
- Out-of-domain generalization: On SRBench, the model approaches state-of-the-art GP accuracy with a fraction of inference time and outputs lower-complexity formulas than the top two models.On Feynman problems, its robustness to target noise is similar to state-of-the-art GP models, while Figure 5 evaluates accuracy, speed, and complexity jointly.
Conclusion
The paper presents a competitive deep-learning model for symbolic regression using a numeric-symbolic approach. Ablations show that predicting constants improves performance and provides informed initialization for solver refinement, while current dimensional scalability remains limited.
- The model uses a novel numeric-symbolic approach for end-to-end symbolic regression.
- The evaluated dimensions remain moderate, with D < 10, and closing the gap with genetic programming remains future work.
- The methods may support symbolic-regression applications requiring real-time inference.
A Details on the training data
The training data are generated with a parameterized synthetic-data procedure, whose distributions vary with input dimension and unary-operator count. The generator intentionally retains more easy than hard examples to improve out-of-domain generalization.
- The data generator’s parameters are detailed in Table 3, with unary-operator probabilities matched to frequencies in the Feynman dataset.
- The number of generated expressions decreases as input dimension and unary-operator count increase because valid inputs become less likely.
- The generator keeps more easy examples than hard ones instead of enforcing a uniform distribution, because this empirically improves out-of-domain generalization.
- The multimodal procedure produces diverse input-distribution shapes.
B Attention maps
The Transformer’s attention heads show complementary patterns across input regions and distances. Their maps also reflect function-specific structure, including extrema, singularities, and periodicity.
- The attention maps use one-dimensional inputs sorted in ascending order and display the encoder’s 4 layers and 16 heads, with 8 heads shown.
- Attention heads are complementary: some focus on specific regions, while others spread broadly or emphasize neighboring and far-away points.
- For exponential, inverse, and sine functions, attention reflects extrema, the origin singularity, and periodicity, respectively.
- The displayed input distributions illustrate diversity of shapes produced by the multimodal generation procedure.
C Does memorization occur?
The paper examines whether extensive synthetic-data training leads to memorization by counting possible functions rather than only possible skeletons. For small expressions, repeated exposure cannot be excluded; larger expressions provide a broader comparison.
- Memorization cannot be assessed from skeleton counts alone because constants, inputs, and initial conditions produce different functions.
- For expressions with up to four operators, the number of possible expressions is lower or similar to the number encountered during training.
- The small-expression comparison means repeated exposure cannot be excluded, even when realizations differ through initial conditions.
D Additional in-domain results
The additional in-domain ablation reports R2 scores for varying function and input difficulty, using the same factors examined in the main-text ablation.
- Fig. 10 repeats the main-text ablation with R2 replacing accuracy as the evaluation metric.The analysis varies unary operators, binary operators, input dimension, input-point count, input variance, and multiplicative label noise.
E Additional out-of-domain results
Additional out-of-domain evaluations examine complexity–accuracy trade-offs, benchmark performance, decoding, attention, training coverage, and robustness across datasets and input conditions.
- Jin benchmark: The model gets all Jin benchmark functions correct except for one.
- Strogatz datasets: Strogatz datasets are trajectories from first-order ODEs with time-ordered inputs unlike the training distribution, where the model performs less well than GP-based methods.
- Ablation on input dimension: The out-of-domain analysis evaluates performance as input dimensionality changes on Feynman and black-box datasets.
- Ablation on decoding strategy: The decoding ablation compares random sampling with beam search, whose candidates can have low diversity because expressions differ only through small coefficient modifications.
- Attention analysis: Attention maps show distinctive patterns associated with exponential growth, inverse-function singularities, and sinusoidal periodicity.
- Training coverage: After 100 epochs, the models have seen only a fraction of possible expressions containing more than four operators.
- Complexity–accuracy: The evaluation includes a complexity–accuracy Pareto comparison across Feynman and black-box SRBench problems against deep-learning, GP, and classic ML families.