SymbolicGPT: A Generative Transformer Model for Symbolic Regression

1 Introduction

• Without structural constraints on the target equation, symbolic regression searches a much larger candidate-expression space than linear or multinomial regression.In the typical instance-based framework, a new dataset also requires the training procedure to begin again from scratch.Source: §1 · pp. 1–2
• SymbolicGPT treats symbolic mathematics as a language and frames regression as captioning a point cloud with its governing equation.The proposed model learns the relationship between point clouds and the mathematical expressions that define them.Source: §1 · p. 2
• A T-net provides an order-invariant embedding of the input point cloud, independent of dataset size and point order.The embedding represents the entire input dataset for a regression instance.Source: §1 · p. 2
• SymbolicGPT trains one language model once and uses it to solve new regression instances as captioning tasks, rather than learning an equation anew for each dataset.The introduction presents this as a route to faster solving and competent equation reconstruction.Source: §1 · p. 2

2 Related Work

• Genetic-algorithm methods search over symbolic expressions through mutation and evolution, but are computationally expensive, randomized, instance-based, and struggle with many variables and constants.Neural Equation Learner approaches also train from scratch for each new input dataset.Source: §2 · p. 2
• A language-based sequence-to-sequence LSTM avoids per-instance training, but its textual input format restricts it to one-dimensional data on a fixed-size mesh.The paper says its own approach removes these input-structure limitations for high-dimensional problems with many variables.Source: §2 · p. 3
• The cited reinforcement-learning method generates equation skeletons with deep RNNs, then uses numerical optimization for constants while retaining iterative search.The paper describes this prediction process as computationally intensive.Source: §2 · p. 3

3 Method

• A T-net [16] produces an order-invariant embedding of the input dataset.Source: §3 · p. 3
• A GPT language model obtains a skeleton equation from the dataset representation.Source: §3 · p. 3
• The method optimizes constant values to fill in the generated equation skeleton.Source: §3 · p. 3

3.1 Equation Generation

• SymbolicGPT training data pairs a numerical dataset with a string encoding the expression that governs it.Source: §3.1 · p. 3
• Training equations are sampled by decorating parse trees whose maximum depth controls potential function nesting and equation complexity.Unlike the cited tree-sampling approach, this method constrains tree levels rather than node count.Source: §3.1 · p. 3
• The tree template is a balanced binary tree with leaves assigned variables and interior nodes assigned allowed operators; unary identity operators and terminal nodes permit shallower or less complex expressions.The specified variables and operator set determine which expressions can be generated.Source: §3.1 · pp. 3–4
• Random constants can be inserted as multiplicative factors or additive biases, with their frequency controlled by the constant ratio.The method uses specified constant bounds and insertion probability.Source: §3.1 · p. 4
• Each generated equation is evaluated at randomly sampled inputs to create its dataset, and a wide, diverse set of training equations is needed to generalize to unseen equations without overfitting.Source: §3.1 · pp. 3–4

3.2 Order-Invariant Embedding

• The embedding must accommodate datasets with different numbers of points and remain invariant to their order.The point set is treated as a collection, not a sequence.Source: §3.2 · p. 4
• The T-net accepts n datapoints over a fixed number d of variables, padding instances with fewer variables with zeros.Its input matrix is X ∈ R^(n×(d+1)), and n can vary.Source: §3.2 · p. 4
• After three MLP stages encode each point in 4e dimensions, global max pooling reduces the n × 4e output to a 1 × 4e vector.The pooled vector is passed through two fully connected layers to produce the dataset embedding.Source: §3.2 · pp. 4–5
• Figure 1 depicts the T-net embedding the input dataset and the GPT model producing symbolic equation skeletons.The training set pairs input datasets with output equations for translation into equation strings.Source: §3.2 · p. 4, Figure 1 · p. 5

3.3 Generative Model Architecture

• GPT receives the T-net dataset embedding and a tokenized equation prefix, typically a Start-of-Sequence token when no output-expression information is given.The prefix may also be any desired beginning of the output equation.Source: §3.3 · p. 5
• The model sums the dataset representation, token embeddings, and positional embeddings before processing the combined input.The dataset representation is expanded to match the embedding matrix dimensions.Source: §3.3 · p. 5
• Eight GPT transformer blocks process the combined embedding, and a linear projection with a softmax produces token probabilities for equation generation.The output tokens are sampled using top-k sampling with k = 40.Source: §3.3 · p. 5
• During training, constants are replaced by ⟨C⟩ placeholders so GPT predicts a skeleton equation rather than exact constant values.This separates learning the equation structure from learning its constants.Source: §3.3 · p. 5

3.4 Learning Constants

• Fitting constants after skeleton prediction is described as a common strategy for string-based regression methods.Source: §3.4 · p. 6
• After the model predicts a skeleton, BFGS optimization learns the constant values and replaces the ⟨C⟩ placeholders to form the final expression.The implementation uses SciPy.Source: §3.4 · p. 6
• The optimized constants decorate the predicted skeleton, yielding a symbolic expression for the regression task.Source: §3.4 · p. 6

3.5 Evaluation Metric

• For data with large values, residuals can become large even when a predicted equation is close to the true underlying equation.Source: §3.5 · p. 6
• The method normalizes mean square error by ∥y + ϵ∥2 to address this scale-related issue.ϵ avoids division by zero, and ∥·∥2 denotes the Euclidean norm.Source: §3.5 · p. 6
• The resulting evaluation measure is called normalized mean square error, MSEN.Source: §3.5 · p. 6

3.6.1 One-Time Training

• One-time offline training lets a single trained model solve new regression instances without restarting training for each dataset.After training, solving an instance is inference, with time spent reading the dataset, obtaining its T-Net embedding, and optimizing constants.Source: §3.6.1 · p. 6

3.6.2 GPT Technology

• The GPT-based approach is expected to improve as language models evolve by replacing GPT with a newer, more powerful alternative.The authors state that this replacement would require no extra design or implementation effort.Source: §3.6.2 · p. 6

3.6.3 Scalability

• SymbolicGPT scales to multiple input variables and faster inference by inferring dimension from input points and performing constant optimization only once.The authors report that it produces superior results with significantly less computation time.Source: §3.6.3 · p. 6

4 Experiments and Results

• SymbolicGPT was trained for four epochs on 10,000 generated instances per dimensional setting and compared with DSR, GP, GP Max, and an MLP baseline.The input datasets and equation labels form the training pairs; validation and test sets each contain 1,000 pairs.Source: §4 · p. 7
• Each method was tested on 1,000 instances in each of four settings, with MSEN as the fitness metric; Figure 2 summarizes error-threshold performance.The settings use one, two, or three input variables, or a general setting with one to five variables.Source: §4 · p. 7
• SymbolicGPT finds better-fitting equations for more test cases than DSR and more equations with log MSEN < −10 than any other tested method.Figure 2 shows cumulative proportions of test cases below each log-MSEN threshold.Source: Figure 2 · p. 8
• SymbolicGPT solves an instance significantly faster than other methods, often by an order of magnitude or more.Table 1 reports mean running times and standard deviations; most computation is shifted to offline model setup.Source: §4 · p. 8, Table 1 · p. 8
• With 50-point datasets, SymbolicGPT achieves lower regression error than the other algorithms do with up to 500 points.All methods improve as more datapoints are provided; SymbolicGPT has lower error regardless of point count, although it was trained on datasets of 500 points.Source: §4 · p. 8, Figure 3 · p. 9

• The method is fast, scalable, and performs competently across several kinds of symbolic regression problems compared with existing approaches.Source: §5 · p. 10
• The approach combines language models with symbolic mathematics and order-invariant representations of point clouds.Source: §5 · p. 10
• The method eliminates per-instance computation expense found in most regression methods and resolves input restrictions of other language-based regression models.Source: §5 · p. 10