Source-linked AI summary
Extrapolation and learning equations
Georg Martius, Christoph H. Lampert
TL;DR
Standard regression emphasizes generalization within the training distribution, but physical systems may require reliable predictions outside it and interpretable relationships rather than black-box functions. The paper proposes the differentiable Equation Learner network with sparse, equation-like computational structure, and experiments show that it can learn functional relations and extrapolate to unseen data domains. Its extrapolation depends on the underlying expression being representable by the network.
Problem
Regression is commonly evaluated through interpolation within the training distribution, while physical systems require interpretable functions and may produce future data outside that domain.
Method
EQL is a differentiable feed-forward network using equation-motivated computational units, sparse regularization, and extrapolation-oriented model selection.
Results
EQL learns functional relations and extrapolates well on synthetic and real data, including near and far extrapolation where MLP and SVR degrade or fail.
Takeaways & Limitations
The learned concise functional forms can provide insight into relationships within data, including physical measurements of x-ray transition energies.
Takeaways & Limitations
Good extrapolation cannot be achieved when the underlying expression lies outside the network's hypothesis class; extending the base-function class remains future work.
Abstract
from arXiv · showhide
In classical machine learning, regression is treated as a black box process of identifying a suitable function from a hypothesis set without attempting to gain insight into the mechanism connecting inputs and outputs. In the natural sciences, however, finding an interpretable function for a phenomenon is the prime goal as it allows to understand and generalize results. This paper proposes a novel type of function learning network, called equation learner (EQL), that can learn analytical expressions and is able to extrapolate to unseen domains. It is implemented as an end-to-end differentiable feed-forward network and allows for efficient gradient based training. Due to sparsity regularization concise interpretable expressions can be obtained. Often the true underlying source expression is identified.
Introduction
The paper targets regression beyond interpolation: models should make reliable predictions when future data lies outside the training domain. It introduces EQL, a network for learning analytical expressions and extrapolating to unseen domains.
- Introduction: Extrapolation generalization concerns predictions on future data outside the training domain, where robustness and safety require avoiding catastrophic failures.The motivating example is a robotic arm operated beyond its training specifications.
- Introduction: The paper focuses on regression for systems described by real-valued analytic expressions, including nonlinear mechanical systems such as pendulums and robotic arms.Their behavior may nevertheless be inferred on an extrapolation domain from behavior observed elsewhere.
- Introduction: The two main contributions are EQL, a network that learns analytical expressions and extrapolates to unseen domains, and a model-selection strategy tailored to extrapolation.
- Introduction: The paper proceeds from the regression and extrapolation setting to the method, experimental evaluation, and conclusions.
Regression and extrapolation
The paper models noisy multivariate observations as samples from an unknown analytic function and distinguishes interpolation from extrapolation by whether training and test domains overlap.
- Regression and extrapolation: The regression task is to learn ψ: R^n → R^m that approximates an unknown analytic function φ generating noisy observations y = φ(x) + ξ.The noise is additive, zero-mean, and the function may describe ordinary differential equations governing a mechanical system.
- Regression and extrapolation: Interpolation uses training and test data from the same distribution, whereas extrapolation tests predictions in domains not covered by the training data.Examples include predicting robotic-arm behavior at higher velocities than those represented during training.
- Regression and extrapolation: Training data may cover only a limited range of the data domain, with available training data split into model-training and validation or model-selection subsets.
Learning a network for function extrapolation
The Equation Learner combines analytic-function-oriented computational units with differentiable training and sparsity-based model selection to learn concise expressions suited to extrapolation.
- Architecture: The EQL is a multi-layer feed-forward network whose hidden layers apply linear mappings followed by specialized nonlinear transformations.Each layer can be designed independently provided input and output dimensions match.
- Architecture: Unary units can implement identity, sine, cosine, or sigmoid functions, while binary units multiply paired inputs.The nonlinear stage contains u unary units and v binary multiplication units.
- Architecture: The final layer computes regression values through a linear read-out, and the architecture is called the Equation Learner.The network defines the function ψ and is illustrated for three layers with one neuron per type in Figure 1.
- Architecture: Two-input multiplication units match common equation structure and let network depth control the maximum degree of learned polynomials.This avoids the high-order polynomial behavior associated with product units having large fan-in.
- Training and model selection: The EQL is trained end-to-end by back-propagation using a differentiable objective combining L2 loss with L1 regularization.Mini-batch stochastic optimization with Adam is used; experiments set α = 0.001 and the mini-batch size to 20.
- Training and model selection: Hybrid regularization first allows unrestricted parameter movement, then induces sparsity, and finally removes L1 shrinkage while enforcing the learned zero pattern.The final phase is intended to preserve sparse structure while fitting observed values closely.
Experimental evaluation
Experiments evaluate EQL on pendulum dynamics, double-pendulum and robotic-arm kinematics, synthetic formula learning, and X-ray transition energies. EQL generally extrapolates better than MLP and SVR while producing interpretable formulas, but performance degrades when the target lies outside its hypothesis class.
- Pendulum: 1000 noisy samples from a bounded pendulum domain are used for training, with separate interpolation, near-extrapolation, and far-extrapolation test sets.The targets include Gaussian noise with σ = 0.01, and the training region covers more than half a sine period.
- Pendulum: EQL learns pendulum dynamics and extrapolates well near and far beyond the training domain, unlike MLP and SVR.The authors attribute this to EQL finding the correct functional expression rather than merely interpolating training values.
- Double pendulum kinematics: EQL predicts unseen double-pendulum kinematics perfectly in the reported extrapolation test, whereas MLP and SVR fail near or beyond the training domain.The test trajectory covers several spins, while the training trajectory covers only part of the input and output domain.
- Formula learning: Across formula-learning tasks, EQL achieves good extrapolation except for F-3, where it reaches a local minimum in 9 out of 10 cases with the overcomplete base.Removing cosine from the base functions allows the correct F-3 formula to be found; F-1 is identified correctly, while F-2 extrapolates well with an approximation.
- Model scope: The cart-pendulum experiment is outside EQL’s hypothesis class because the target formulas contain divisions absent from the architecture.Near extrapolation remains acceptable for EQL and MLP, but both diverge considerably farther outside the training region.
Conclusions
The EQL architecture learns analytic expressions for physical systems and can extrapolate to unseen domains, while sparsity regularization yields concise forms. Its success is limited when optimization finds approximations or when the true expression lies outside the network’s hypothesis class.
- EQL learns analytic expressions governing physical, particularly mechanical, systems through a differentiable network.
- End-to-end backpropagation, sparsity regularization, and fixed L0 selection produce sparse representations with unbiased factor estimation.
- The method extrapolates functional relations to unseen data-space regions and learns concise forms from synthetic and real data.
- The approach may extrapolate poorly when optimization settles on an approximate equation rather than reliably finding the right one.
- Good extrapolation cannot be achieved when the underlying expression is not representable by the network’s hypothesis class.
Quantifying sparsity
Because formula complexity is difficult to measure directly, the method uses sparsity—the number of active hidden units—as a proxy for complexity.
- Sparsity s counts the number of active or used hidden units in a network.
- The complexity proxy uses a heaviside threshold, with 0.01 determining whether units or connections are counted as active.For multiplication units, the norms of incoming weights for both inputs are added.
Selection criteria
Model selection ranks networks by validation error and sparsity, then chooses the model with the smallest squared rank norm. In the kin2D-3 illustration, the best extrapolating networks are sparse and accurate on validation data.
- The selected network minimizes the squared norm of its ranks for validation error and sparsity.Ranking avoids directly combining quantities with different scales.
- In kin2D-3, the best-performing networks combine low validation error with high sparsity.
A2: Dependence on noise and number of data points
For kin-4-end, interpolation and extrapolation performance vary with data volume and additive noise. The method is generally robust to noise, and additional data can compensate for increased noise.
- The study scans interpolation and noise-free-test extrapolation across the number of data points and additive-noise size.The figure marks an arbitrary threshold for successful interpolation and extrapolation.
- More data generally compensates for more noise, indicating robustness across the scanned conditions.