Source-linked AI summary
Discovering physical concepts with neural networks
Raban Iten, Tony Metger, Henrik Wilming, Lidia del Rio, Renato Renner
TL;DR
The paper studies how latent representations can capture the variables and structure needed to model physical systems. SciNet recovers relevant parameters, conservation laws, quantum-system dimensions, and accurate planetary predictions from data without prior physical assumptions.
Problem
The paper asks how many latent variables are required to represent physical data and whether they can capture system structure.
Method
SciNet uses latent-variable encoding and decoding, with reparameterized Gaussian sampling to enable differentiable training.
Results
SciNet recovers physical parameters, stores total angular momentum, infers quantum-system dimension, and predicts Mars and Sun angles with root mean square error below 0.4% (with respect to 2π).
Takeaways & Limitations
The learned representations can expose physically meaningful quantities and support predictions from incomplete prior knowledge of the system.
Abstract
from arXiv · showhide
Despite the success of neural networks at solving concrete physics problems, their use as a general-purpose tool for scientific discovery is still in its infancy. Here, we approach this problem by modelling a neural network architecture after the human physical reasoning process, which has similarities to representation learning. This allows us to make progress towards the long-term goal of machine-assisted scientific discovery from experimental data without making prior assumptions about the system. We apply this method to toy examples and show that the network finds the physically relevant parameters, exploits conservation laws to make predictions, and can help to gain conceptual insights, e.g. Copernicus' conclusion that the solar system is heliocentric.
Appendix A: Implementation
The paper specifies SciNet by its input and output dimensions, latent-neuron count, and encoder and decoder sizes, with implementation and training details provided separately. The source code and pretrained networks are available, and the examples train within hours on a standard laptop.
- SciNet is specified by the question and observation input sizes, output size, number of latent neurons, and encoder and decoder sizes.Encoder and decoder widths are intended to be large enough for expressivity while remaining efficiently trainable.
- The network-structure parameters for the paper’s examples are reported in Tables I and II.
- The source code, network-structure and training details, and pretrained SciNets are publicly available.
- Training each example takes only a few hours on a standard laptop.
Appendix B: Detailed comparison with previous work
Previous work has used neural networks and related algorithms for physics prediction, physical-variable extraction, experimental design, wave-function representation, and dynamical-equation discovery. SciNet differs by targeting representations that support answers to questions about observations rather than only efficient state representation or model search.
- Neural networks have been applied to characterize phase transitions, learn local symmetries, and extract physical variables from physical data.The cited approaches include condensed-matter and many-body applications as well as unsupervised extraction from dynamical-system time series.
- Automated search and reinforcement learning have been used to generate experimental setups and discover versatile experimental building blocks.
- Neural networks have represented wave functions and measurement-outcome distributions for particular quantum systems, whereas SciNet focuses on extracting physical representations.
- Information-theoretic training of restricted Boltzmann machines has extracted relevant degrees of freedom under a Boltzmann-distribution assumption.
- Tables I and II summarize the network-structure and training-process parameters used in the paper’s examples.Table I distinguishes the structures used by the first four examples and the final example; Table II describes multi-phase training notation.
- Other work searches for dynamical equations or variables with approximately linear evolution, while SciNet uses a question-conditioned representation architecture.
Appendix C: Minimal representations
SciNet is trained to encode observations into a sufficient, minimal, and optionally uncorrelated latent representation that preserves the information needed to answer the question set. Smooth decoding gives the latent dimension a formal interpretation, while independence separates latent variables under the data distribution.
- SciNet’s encoder-decoder architecture is designed to learn simple representations from data rather than having a specific representation imposed in advance.
- A sufficient representation retains the information from each observation needed to answer all questions through a smooth decoder.
- An uncorrelated representation requires the latent variables to be mutually independent.This independence condition depends on the distribution from which observations are sampled.
- A minimal uncorrelated representation uses the fewest latent parameters, formalizing the paper’s notion of a simple physical-data representation.
- Smooth decoding is required because, without it, one real-valued latent variable could encode arbitrarily much information and the variable count would lose its intended meaning.
1. Interpretation of the number of latent variables
The paper formalizes the dimension of a question-answering dataset and proves that SciNet needs at least as many latent variables as the recoverable degrees of freedom. For autoencoders generated by a smooth nondegenerate map, the minimal sufficient representation has exactly the generator’s domain dimension.
- Dimension of a data set: A dataset has dimension at least n when selected questions map an n-dimensional observation submanifold diffeomorphically into answer space.
- Minimal representation for SciNet: Any sufficient SciNet representation for data of dimension at least n requires at least n latent variables.The proof assumes a smooth decoder and derives a contradiction when the latent dimension is smaller than n.
- Autoencoders: For an autoencoder, SciNet is obtained by asking the same question and requiring the output to reproduce the observation.
- Neural-network implementation: A three-layer feed-forward network maps input activations to output activations through successive neuron layers.
- Minimal representation for an autoencoder: The lower bound follows because a nondegenerate generator is locally a diffeomorphism, so fewer than d latent variables cannot represent the corresponding observation manifold.
- Minimal representation for an autoencoder: If a bounded domain G ⊂ R^d maps smoothly, nondegenerately, and surjectively to observations, the minimal sufficient autoencoder representation contains d latent variables.
Appendix D: Neural networks
Neural networks are functions built from layered neurons, with trainable weights and biases optimized on input-output examples. Training adjusts these parameters by gradient descent, often using mini-batches.
- Neural-network basics: A neuron maps multiple real inputs to an output using weighted summation, a bias, and an activation function.The paper uses the exponential linear unit for its examples.
- Neural-network basics: Feed-forward networks arrange neurons in layers, forwarding activations from each layer to the next as a function from R^n to R^m.
- Training: Training optimizes network parameters against known input-output pairs using a cost function that measures prediction error.
- Training: Gradient descent updates weights and biases in the negative gradient direction, while stochastic gradient descent estimates the gradient from mini-batches.
1. Variational autoencoders
Variational autoencoders represent inputs with probabilistic latent variables and decode sampled representations, while SciNet modifies this framework to answer questions about observations. Its objective combines reconstruction accuracy with pressure for disentangled, efficient latent variables.
- Core architecture: SciNet modifies variational autoencoders by adding question inputs and answering questions about observations instead of reconstructing inputs directly.
- Core architecture: Representation learning maps high-dimensional inputs to lower-dimensional latent vectors that retain relevant information, while standard autoencoders reconstruct the input.
- Probabilistic encoding and decoding: The encoder maps x to a Gaussian latent distribution, a latent vector is sampled, and the decoder maps that vector to a conditional output distribution.
- Probabilistic encoding and decoding: The reparameterization trick generates zi = µi + σiεi with εi sampled independently of trainable parameters, preserving differentiability for stochastic gradient descent.
- Training objective: The β-VAE objective rewards accurate input recovery while penalizing dependence among latent variables and, for large β, encourages compact representations.
- Damped-pendulum setup: For the damped-pendulum example, SciNet predicts future position at tpred, using three latent neurons and varying spring constant κ and damping factor b.
- Damped-pendulum results: Prediction error is below 2% of the fixed amplitude A0 = 1m, while κ and b occupy two latent neurons and the third stores no information.
Appendix E: Details about the physical examples
The paper introduces four physical examples to test whether SciNet can recover relevant variables and concepts from data. The supplied passage identifies this appendix as providing further details about those examples.
- Scope: The appendix provides additional information about the four physical-system examples used to apply SciNet.
1. Damped pendulum
For a weakly damped pendulum, SciNet receives position time series and predicts future positions while learning a compact latent representation. It recovers damping and spring constant as separate physical parameters and leaves an unnecessary neuron constant.
- Physical setup: The pendulum dynamics depend on spring constant κ and damping factor b, with mass held constant because it can be absorbed into rescaled parameters.
- Network task: SciNet receives a pendulum position time series, asks for a future position, and uses a three-neuron latent representation.
- Learned representation: SciNet extracts the two relevant parameters without being given physical concepts, assigning b and κ to separate latent neurons with nearly linear dependence.
- Learned representation: The third latent neuron is nearly constant and adds no information, indicating that two parameters suffice for this situation.
2. Conservation of angular momentum
SciNet is trained on collision data to predict one particle’s trajectory from the other using total angular momentum conservation. It learns to store that conserved quantity in its latent representation, even with noisy inputs.
- Conservation laws let physicists predict one interacting system from another without modelling the interaction details.For the collision scenario, initial angular momentum and one particle’s final trajectory determine the other particle’s trajectory.
- SciNet is trained on simulated two-body collision data with added Gaussian noise to test whether its encoding and decoding remain robust.
- SciNet stores the total angular momentum in its latent representation, matching the conservation-based reasoning used by a physicist.The authors interpret this as recovery of a conservation law through data compression and questions about joint properties of multiple systems.
3. Representation of qubits
The quantum-tomography example tests whether SciNet can learn compact state representations and predict measurement probabilities from operational data. With complete measurements, it recovers the parameter counts for one- and two-qubit states; with incomplete measurements, prediction remains imperfect and reveals missing information.
- Quantum state tomography seeks a faithful state representation that predicts outcome probabilities for arbitrary measurements, with complete measurements sufficient for reconstruction.
- SciNet predicts measurement probabilities for binary projective measurements using operational measurement data, without assumptions about quantum theory.
- Two latent neurons for one qubit and six for two qubits minimize prediction error with tomographically complete measurements, matching the required state parameters.
- SciNet can determine the minimal number of parameters needed to describe the quantum state and distinguish tomographically complete from incomplete measurement sets.
- With tomographically incomplete data, perfect prediction is impossible regardless of latent-neuron count, identifying the measurement set as incomplete.
- For two qubits, increasing the incomplete-data subspace from two to three dimensions improves prediction accuracy while increasing the required latent-neuron count.
4. Heliocentric model of the solar system
SciNet receives only Earth-based observations of the Sun and Mars and is trained to make their time evolution simple. It learns heliocentric variables—planetary angles as seen from the Sun—while predicting the observed angles accurately.
- The task is to predict Mars’s and the Sun’s Earth-observed angles from their initial angles using simulated weekly observations.The training set contains 3665 observations within Copernicus’ lifetime.
- The physical model assumes that Earth and Mars orbit the Sun with constant angular velocity on approximately circular orbits.
- SciNet uses a two-neuron representation with additive time updates to model the angle evolution.The representation is updated as r(t_i+1) = r(t_i) + b.
- SciNet predicts the Sun’s and Mars’s angles with root mean square error below 0.4% relative to 2π.
- The latent neurons store the angles of Earth and Mars as seen from the Sun, despite training data containing only Earth-based observations.
Appendix F: Representations of cyclic parameters
Neural-network encoders can struggle to represent cyclic physical parameters because globally simple coordinates on closed manifolds are discontinuous. The qubit example exhibits the same continuity problem when states are parameterized by spherical coordinates.
- Cyclic data on closed manifolds such as circles and spheres generally require several coordinate charts for representation.
- Spherical coordinates create a discontinuity at φ = 0, so a neural-network encoder restricted to continuous functions cannot implement the inverse coordinate map globally.
- Representations near the coordinate discontinuity are therefore especially problematic for the encoder.
- The qubit state can be parameterized by Bloch-sphere angles θ and φ, but continuously encoding measurement data into those coordinates leads to a contradiction.