Source-linked AI summary
Discovering Symbolic Models from Deep Learning with Inductive Biases
Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, Shirley Ho
TL;DR
The paper asks how to obtain interpretable, generalizing symbolic relations from high-dimensional deep-learning models. It combines structured Graph Networks and compact latent representations with symbolic regression, recovering known physical laws and discovering a new cosmological formula. The approach extends existing symbolic-regression methods rather than replacing them and relies on assumptions such as decomposable pairwise and self energies in the flattened Hamiltonian model.
Problem
Symbolic regression is difficult on high-dimensional data, while deep-learning models are efficient but difficult to interpret and may extrapolate linearly despite inductive biases.
Method
The framework trains structured Graph Networks with compact latent representations, then fits symbolic expressions independently to internal functions and substitutes them into the model.
Results
The framework recovers injected Newtonian and Hamiltonian laws and discovers a new analytical expression for cosmological dark matter; the extracted formula achieves loss 0.0882 versus 0.121 for a hand-designed formula.
Takeaways & Limitations
Structured neural representations can support fine-grained interpretation, symbolic extraction of unknown force laws, and discovery of a new cosmological dark-matter law.
Takeaways & Limitations
The method is not itself a new symbolic-regression technique, and the flattened Hamiltonian assumes energy can be expressed as summed pairwise and self terms.
Abstract
from arXiv · showhide
We develop a general approach to distill symbolic representations of a learned deep model by introducing strong inductive biases. We focus on Graph Neural Networks (GNNs). The technique works as follows: we first encourage sparse latent representations when we train a GNN in a supervised setting, then we apply symbolic regression to components of the learned model to extract explicit physical relations. We find the correct known equations, including force laws and Hamiltonians, can be extracted from the neural network. We then apply our method to a non-trivial cosmology example-a detailed dark matter simulation-and discover a new analytic formula which can predict the concentration of dark matter from the mass distribution of nearby cosmic structures. The symbolic expressions extracted from the GNN using our technique also generalized to out-of-distribution data better than the GNN itself. Our approach offers alternative directions for interpreting neural networks and discovering novel physical principles from the representations they learn.
1 Introduction
The paper addresses the difficulty of interpreting deep-learning models and finding symbolic expressions in high-dimensional data. It proposes combining neural networks with inductive biases and symbolic regression to recover known physical laws and derive analytical expressions.
- Motivation: Symbolic expressions are compact, interpretable, and often generalize well, but traditional symbolic regression becomes difficult as inputs and operators increase.Genetic-algorithm approaches scale exponentially with the number of input variables and operators.
- Motivation: Deep-learning methods train efficiently on high-dimensional datasets, but their learned models are difficult to interpret.
- Motivation: Strong inductive biases can improve generalization, yet ReLU-based learned components typically extrapolate as linear piece-wise approximations.
- Approach: The proposed framework combines deep learning and symbolic regression by fitting symbolic expressions to internal components of a structured neural model.The expressions can be joined into an algebraic equation equivalent to the trained Graph Network.
- Results: Across force laws, Hamiltonians, and an astrophysical problem, the framework recovers known closed-form laws and derives a new interpretable analytical expression.The authors also report improved generalization and plausible analytical expressions.
2 Framework
The framework factorizes high-dimensional learning problems through structured Graph Networks, trains compact latent representations, and applies symbolic regression to the networks’ internal functions. This exposes explicit algebraic forms while retaining the neural model’s structured inductive biases.
- Framework: The workflow engineers a separable, data-matched deep model, trains it end-to-end, fits symbolic expressions to its internal functions, and replaces those functions symbolically.
- Graph Networks: Edge, node, and global models respectively compute messages, updated node properties, and global properties through distinct functions.Messages are summed at receiving nodes before being passed to the node model.
- Graph Networks: Graph Networks are used because they are permutation-equivariant, efficiently trainable, and composed of interpretable edge, node, and global functions.
- Factorization: Factorizing a high-dimensional problem into separate functions reduces a hypothetical search from 10^18 combined equations to 2×10^9 equations.
- Symbolic regression: The approach extends existing symbolic-regression methods to high-dimensional datasets rather than introducing a new symbolic-regression algorithm.Eureqa is used in the experiments, but other low-dimensional packages could replace it.
- Compact internal representations: Compact latent representations are encouraged with regularization, reducing hidden-vector dimensionality and filtering low-variance features without compromising accuracy.The paper investigates L1 and KL penalties with a fixed prior.
3 Case studies
The case studies apply physically motivated Graph Network variants to Newtonian forces, Hamiltonian energies, and cosmological halo properties. The structured models are designed so symbolic regression can recover force, energy, and environmental relations.
- Newtonian dynamics: Newtonian dynamics is modeled with pairwise messages, message aggregation, and a node function that predicts acceleration from net force and receiving-particle mass.
- Newtonian dynamics: The Newtonian experiment tests whether the learned message dimension matches the number of spatial dimensions and whether symbolic regression can recover force relationships.
- Hamiltonian dynamics: Hamiltonian dynamics represents total energy H(q, p) using canonical coordinates and momenta, with system updates obtained from derivatives of the learned Hamiltonian.
- Hamiltonian dynamics: The Flattened HGN expresses global energy as scalar pairwise interaction terms Hpair plus a per-particle term Hself.This decomposition is chosen because many physical systems can be described by summed interaction energies.
- Dark matter halos for cosmology: The cosmology case predicts halo overdensity from nearby halos’ positions, velocities, and masses in a static dark-matter simulation dataset.The exact analytical relationship is unknown, and the method seeks an interpretable model.
4 Experiments & results
Experiments show that constrained message representations let Graph Networks recover interpretable force and energy relations, while symbolic regression extracts known and novel formulas. In the cosmology case, the extracted expression generalized better out of distribution than the original GN.
- Newtonian dynamics: Graph Networks were trained on N-body simulations with known interaction laws to recover force laws from learned message representations.The datasets covered two- and three-dimensional systems with orbital, charged-particle, spring, damped, and discontinuous forces.
- Newtonian dynamics: L1 regularization produced the strongest prediction performance in most cases, while bottleneck models performed worse under limited training time.The comparison used a new dataset generated from a different random seed.
- Newtonian dynamics: Constrained message representations yielded components highly correlated with true forces, whereas the Standard model showed no strong correlations.The messages were interpretable as forces, including a rotation of the true force vector discovered during training.
- Newtonian dynamics: Symbolic regression recovered candidate force expressions for spring, inverse-square, and discontinuous-force systems without using prior knowledge of each force’s form.The fitting procedure selected compact expressions by balancing mean absolute error reduction against expression complexity.
- Hamiltonian dynamics: Hamiltonian models extracted scalar potential energies, including a charged-particle potential, and performed comparably to the L1-regularized model across datasets.The per-particle term also recovered the same kinetic-energy expression across systems.
- Dark matter halos for cosmology: The cosmology expression achieved 0.0892 out-of-distribution error versus 0.142 for the GN, despite training errors of 0.0811 and 0.0634, respectively.The experiment masked 20% of halos with δ_i > 1 before training and formula extraction.
5 Conclusion
The paper concludes that physically motivated inductive biases enable interpretable representations in Graph Networks and Hamiltonian Graph Networks. Symbolic regression extracts force and energy equations and discovers a new law for cosmological dark matter.
- Conclusion: Bottleneck or L1-regularized message passing, and flattened Hamiltonian architectures, learned representations equivalent to transformed force vectors or energies.These representations supported symbolic extraction of explicit equations from trained models.
- Conclusion: The framework recovered force laws and energies and discovered a new law for cosmological dark matter on a non-trivial dataset.Graph Network structure enabled finer-grained interpretation of learned computations than more homogeneous architectures.
A Model Implementation Details
The implementation uses Graph Networks with symbolic regression applied to their internal functions, and the models are implemented in PyTorch and PyTorch Geometric.
- Implementation: The models are implemented with PyTorch and PyTorch Geometric, trained using Adam with a decaying learning schedule.The paper directs readers to the appendix for further implementation details.
A.1 Basic Graph Representation
The graph representation maps node pairs to messages, sums messages at each receiving node, and uses the pooled result with node features to predict updated nodes. Models are trained with supervised loss, while KL regularization treats messages as Gaussian distributions and encourages sparse representations.
- Edge model: Each edge message is computed from the receiving and sending node features using an edge model φe.The edge model maps node-feature pairs to message features, with architecture and message dimensionality varying by model.
- Model variants: Standard and L1 models use 100 message features, the KL model uses 200, and bottleneck messages match the known force dimension.The edge network uses two hidden layers with 300 hidden nodes in the described implementation.
- Aggregation and node model: Messages are pooled by element-wise summation for each receiving node before the node model predicts its updated features.The node model takes the receiving node and its summed message as inputs.
- Training: The network jointly optimizes edge and node-model parameters with mini-batch gradient descent using Adam.The total loss includes prediction terms and regularization components for the selected model variant.
- KL model: The KL variant models each message component with a Gaussian mean and log variance, samples messages, and applies KL divergence to a zero-mean unit-variance prior.Uninformative components are encouraged toward μ = 0 and σ = 1 rather than merely small absolute values.
- KL model: KL regularization encourages sparse messages similarly to L1 regularization while assigning zero KL to components matching the Gaussian prior.The networks are trained with a decaying learning schedule.
A.3 Constraining Information in the Messages
Constraining message representations to the minimum useful dimension supports an interpretation of messages as linear transformations of underlying forces or energies. The same principle motivates flattened Hamiltonian networks that explicitly sum pairwise and self energies.
- Force representations: The graph-network hypothesis treats pooled messages as the summed interaction quantity that updates a body’s state.For Newtonian dynamics, the pooled messages correspond to summed force contributions under the perfect-prediction assumption.
- Force representations: When message dimensionality equals force dimensionality, message vectors can be linear transformations of the true forces under the stated invertibility condition.The argument uses invertibility of the node transformation with respect to the pooled messages.
- Force representations: L1 or KL regularization can encourage sparse messages when the minimum message dimension is unknown.The objective is for the messages to learn the minimal vector space required for computation automatically.
- Scope: The general graph-network setup assumes no edge-specific features or global properties, although edge features can be added to the edge-model inputs.The framework is stated to remain applicable to the general graph-network form.
- Hamiltonian representations: Energy is treated as another linear latent space because pairwise and particle energies are pooled into the system’s scalar Hamiltonian.For gravitational particles, the Hamiltonian combines kinetic and interaction-energy terms.
- Hamiltonian representations: A flattened Hamiltonian network explicitly sums pairwise interaction energies and per-particle self energies instead of producing energy solely as a global property.This design is intended to make the learned energy components simpler to extract and interpret.
B Simulations
The simulation benchmark contains two- and three-dimensional N-body systems with varied interaction laws, particle properties, and numerical trajectories. Test losses and message–force fits evaluate prediction and interpretability across these simulated settings.
- Simulation data: Figure 4 shows example trajectories for 4-node, two-dimensional simulations, with transparency encoding increasing time and point size encoding mass.The examples are presented as long exposures of the simulated particle paths.
- Simulation data: Each simulation is generated from random initial conditions and integrated for 1000 time steps with an adaptive RK4 integrator.The fixed nominal step size varies by interaction type and simulation scale.
- Simulation data: The simulations cover two and three dimensions with 4 or 8 bodies and multiple interaction laws, including orbital, inverse-square, charge, damping, discontinuous, and spring forces.Particle states include masses, charges, positions, velocities, and accelerations over time.
- Interpretability evaluation: Linear fits between significant message components and true force components are summarized in Table 1 and visualized for the two-dimensional spring simulation in Figure 5.The figure compares models’ message components with linear combinations of force components after applying an inductive bias.
- Evaluation: Table 3 reports test prediction losses for each model and dataset in both two and three dimensions under matched training schedules.The table compares models trained with the same batch size, schedule, and number of epochs.
C Symbolic Regression Details
The framework factorizes high-dimensional learned models into internal components, fits each component with symbolic regression, and composes the resulting expressions. It recovers several physical laws, while failures and raw-data experiments show that success depends on representation and factorization choices.
- Procedure: The method records internal message and node-function outputs, then fits symbolic expressions to these components over training examples.Inputs are transformed into relative positions and distances before symbolic fitting.
- Procedure: Symbolic regression minimizes mean absolute error while searching expressions built from arithmetic, comparisons, powers, exponentials, logarithms, conditionals, and real constants.Equation complexity counts operators, constants, and variables, with nonlinear operators weighted more heavily.
- Recovered laws: The method recovers expected forms for spring, inverse-square, and discontinuous force laws, including expressions involving distance powers and conditional terms.Examples include Spring 2D L1, 1/r^2 3D bottleneck, and discontinuous 2D L1 simulations.
- Limitations: Reconstruction does not always succeed, particularly for strategies other than L1 or bottleneck models lacking compact representations of the appropriate dimensionality.The authors do not claim general conditions determining when symbolic regression succeeds or fails.
- Limitations: Raw symbolic regression on the unreduced high-dimensional particle data produces meaningless, poorly fitting equations, whereas neural factorization makes the subproblems tractable.The paper frames factorization as the key step that reduces the search burden before symbolic regression.
- Hamiltonian reconstruction: For Hamiltonian data, symbolic regression recovers the correct potential form despite additive offsets depending on nondynamical parameters.Because only derivatives determine the dynamics, direct linear comparison with the true Hamiltonian is insufficient.
- Model selection: The largest fractional drop in mean absolute error over a complexity increase is used to select the model under the paper’s parametrization of Occam’s Razor.This criterion is illustrated for the 1/r simulation.
E Cosmological Experiments
In the cosmological experiment, a graph network trained with sparse message representations reduces the relevant interaction space to one dimension. Symbolic regression then extracts and refits functional forms for predicting dark-matter halo overdensity.
- Model setup: The cosmological graph network uses 500 hidden units, two hidden layers per node and message function, 100 message dimensions, and L1 regularization at scale 10^-2.These settings were selected through coarse hyperparameter tuning based on predictions of δ_i.
- Learned representation: Only one of 100 message components has standard deviation near 10^-2, while the other 99 are below 10^-8, indicating a one-dimensional discovered representation.This supports rewriting the prediction as a sum over a function of the central halo and each neighboring halo.
- Symbolic extraction: The extracted cosmological functional forms are obtained by symbolically approximating φe_1 and φv and analytically composing the two expressions.The resulting forms are listed in the upper portion of Table 6.
- Parameter fitting: The authors refit parameters of the extracted functional forms directly to the training data before evaluating symbolic-model accuracy.This avoids accumulated error from fitting the graph network and then fitting symbolic regression to the network.