Source-linked AI summary
Mutual Information, Neural Networks and the Renormalization Group
Maciej Koch-Janusz, Zohar Ringel
TL;DR
Identifying the degrees of freedom that matter for real-space renormalization is difficult, so the paper develops an information-theoretic neural network to learn them from configuration samples. The method recovers known Ising and dimer variables, reproduces 2D Ising RG flow, and estimates the critical exponent ν.
Problem
The relevant degrees of freedom governing long-distance behavior can be difficult to identify, while direct optimization over their probability distributions becomes exponentially impractical.
Method
The RSMI algorithm learns PΛ(H|V) by maximizing mutual information between hidden variables in a visible region and its environment, then iterates coarse-graining.
Results
ν ≈1.0 ± 0.15, consistent with the exact ν = 1, while the method recovers Ising block spins, dimer electric fields, and the expected 2D Ising RG flow.
Takeaways & Limitations
Machine-learning representations optimized through real-space mutual information can expose physically relevant degrees of freedom and support quantitative real-space RG analysis.
Takeaways & Limitations
The dimer-model filters disagree with those from KL-divergence optimization, indicating that standard RBMs minimizing KL divergence do not generally perform RG.
Abstract
from arXiv · showhide
Physical systems differring in their microscopic details often display strikingly similar behaviour when probed at macroscopic scales. Those universal properties, largely determining their physical characteristics, are revealed by the powerful renormalization group (RG) procedure, which systematically retains "slow" degrees of freedom and integrates out the rest. However, the important degrees of freedom may be difficult to identify. Here we demonstrate a machine learning algorithm capable of identifying the relevant degrees of freedom and executing RG steps iteratively without any prior knowledge about the system. We introduce an artificial neural network based on a model-independent, information-theoretic characterization of a real-space RG procedure, performing this task. We apply the algorithm to classical statistical physics problems in one and two dimensions. We demonstrate RG flow and extract the Ising critical exponent. Our results demonstrate that machine learning techniques can extract abstract physical concepts and consequently become an integral part of theory- and model-building.
THE REAL SPACE MUTUAL INFORMATION ALGORITHM
The RSMI algorithm identifies coarse-grained degrees of freedom by maximizing mutual information between hidden variables in a visible region and its environment, then iterates this procedure as an RG transformation.
- Defining relevant degrees of freedom: Relevance is defined by selecting composite hidden variables H from a visible region V that maximize mutual information with the environment E.This targets degrees of freedom carrying long-distance information rather than local fluctuations.
- Defining relevant degrees of freedom: The central object is the conditional distribution PΛ(H|V), which specifies how coarse-grained variables depend on configurations in V.The associated joint and marginal distributions are constructed from P(V,E) and PΛ(H|V).
- Motivation for the architecture: The formal optimization is well posed but becomes impractical as the number of local degrees of freedom grows exponentially, motivating the neural-network approximation.RBMs provide dimensionality reduction while approximating the required probability distributions.
- Neural-network implementation: Two contrastive-divergence-trained RBMs approximate P(V,E) and P(V), providing inputs to the RSMI network that learns PΛ(H|V).The RSMI network has a distinct objective: maximizing the mutual information IΛ(H:E).
- Iterative RG procedure: The RSMI weights are optimized iteratively using Monte Carlo estimates of mutual information and its gradients followed by stochastic gradient descent.The learned conditional distribution generates coarse-grained samples for the next RG iteration.
VALIDATION
The algorithm recovers known and nontrivial relevant variables in Ising and dimer systems, rejects irrelevant noise, and reproduces the 2D Ising RG flow and critical exponent.
- Ising model: For a 2×2 Ising block, one hidden unit couples uniformly to four spins, recovering the Kadanoff majority-rule block spin.With four hidden units, each hidden unit instead tracks a different visible spin.
- Dimer model: The dimer model’s relevant coarse-grained variables are electric-field components rather than the microscopic dimers themselves.The analytical mapping identifies low-momentum components of Ex and Ey as the relevant variables.
- Dimer model: Without access to the electric-field mapping, RSMI extracts filters corresponding to Ey, translated Ey patterns, and linear combinations Ex + Ey or Ex − Ey.These filters match the expected dimer-model relevant degrees of freedom.
- Dimer model: Vanishing weights on added paired-spin sites show that RSMI discards their short-range fluctuations as irrelevant despite their pronounced regular pattern.The additional spins are decoupled from the dimers and from one another.
- Validation of the objective: The dimer filters match analytical expectations but are orthogonal to filters obtained by minimizing KL divergence, challenging standard RBMs as general RG procedures.The paper uses this contrast to distinguish mutual-information optimization from ordinary data-distribution approximation.
- Iterated 2D Ising RG: Systems initialized below Tc flow toward decreasing temperature and order, whereas systems above Tc flow toward a paramagnet under iterative 2D Ising RG.The critical point is estimated with 1% accuracy, and ν ≈ 1.0 ± 0.15 agrees with the exact ν = 1.
FUTURE DIRECTIONS
The authors present RSMI as a way for machine-learning representations to expose physically meaningful structure while supporting quantitative RG analysis.
- FUTURE DIRECTIONS: RSMI-based neural networks extract physically relevant degrees of freedom and use them to perform real-space RG.The approach is applied to identifying critical points, RG flow, correlation functions, and critical exponents.
- FUTURE DIRECTIONS: The paper argues that learned internal representations can reflect underlying physical structure rather than serving only as technical intermediates.This motivates collaboration between machine computation and human scientific reasoning.
ESTIMATING MUTUAL INFORMATION
Mutual information is estimated from Monte Carlo samples and RBM approximations, yielding a numerically tractable objective for training the RSMI network.
- Probability distributions: The system is partitioned into a visible region V, environment E, and hidden variables H representing coarse-grained degrees of freedom.A buffer B can separate V and E to exclude immediate local correlations.
- Probability-distribution estimation: Monte Carlo samples provide the equilibrium data distribution, while two contrastive-divergence-trained RBMs approximate P(V,E) and P(V).This avoids explicitly performing computationally costly marginalizations.
- Mutual-information objective: A separate Λ-RBM defines PΛ(H|V) and is optimized to maximize mutual information between H and E rather than to reproduce the data distribution.The objective is expressed using the joint distribution PΛ(V,E,H) and its marginal PΛ(E,H).
- Numerical evaluation: The mutual-information expression is rewritten using Boltzmann-form energies and marginalizations so it can be evaluated numerically.The resulting quantity AΛ is an expectation over PΛ(E,H) involving the energy difference ΔEΛ,Θ.
- Numerical evaluation: Monte Carlo averages replace sums over configurations, including sampling hidden states from PΛ(H|V) and evaluating expectations under the corresponding Boltzmann distribution.These estimates provide the gradients used to train the RSMI parameters.
A. One step of RSMI
The one-step RSMI procedure trains a Λ-RBM by stochastic optimization of a mutual-information proxy, using Monte Carlo samples and explicit gradients. For the reported Ising and dimer calculations, the implementation used specified training schedules and sampling settings, while gradient computation requires care because of stochastic parameter dependence.
- Training procedure: The Λ-RBM parameters are optimized by stochastic gradient descent using mini-batch estimates of the proxy AΛ gradients.Each pass through all mini-batches constitutes one training epoch.
- Convergence: The proxy AΛ saturates across Λ-RBM training epochs, indicating algorithmic convergence for the Ising data.The figure also shows evolving weight matrices for one hidden neuron over 0, 10, 100, and 190 epochs.
- Training procedure: 2000 epochs, mini-batches of 800, and learning rates η = 0.01−0.05 were used for the final RSMI data.The internal Monte Carlo estimate used two samples after a burn-in period of 126.
- Sampling and setup: The environment-buffer-visible setup was restricted to a window three times the visible area's linear extent.This restriction was adopted for numerical efficiency.
- Sampling and setup: The initial dimer data used a 64×64 lattice, while the Ising system used a 128×128 lattice with cluster-update Monte Carlo.The dimer simulations used loop updates and tuned sampling intervals to the noise level.
- Gradient caveat: Automated gradients can fail because Monte Carlo acceptance thresholds introduce stochastic, piecewise-constant parameter dependence, so AΛ gradients should be computed explicitly.Numerical gradients would otherwise be zero in most cases.
B. Multiple steps
Multiple RSMI steps reuse trained filters to generate coarse-grained configurations, retrain the networks, and produce physical observables across length scales. The resulting data support quantitative characterization of RG flow, while multi-hidden coarse-graining requires controlling relative filter phases.
- B. Multiple steps: Trained Λ-RBM filters drive a Monte Carlo sampler that maps configurations of size L to coarse-grained configurations of size L/2.For the 2D Ising model, one coarse-grained variable represents each 2×2 visible area.
- B. Multiple steps: Successive Monte Carlo configurations, Θ-RBM filters, and mutual-information estimates provide data for correlations, expectation values, effective temperatures, and related observables.Θ-RBMs can also generate samples for computing system properties without coarse-grained Monte Carlo samples.
- B. Multiple steps: The iterative data flow trains Θ-RBMs, feeds their parameters to Λ-RBMs, and uses newly sampled configurations for subsequent RG iterations.The process repeats after each coarse-graining step.
- B. Multiple steps: The generated data can characterize the RG flow, including the critical-point position, stable or unstable flow, and critical exponents.The paper demonstrates these quantitative predictions using the 2D Ising model.
- B. Multiple steps: Using more than one hidden degree of freedom per visible area requires care because independent hidden variables can choose aligned or anti-aligned phases.This local Z2 symmetry follows from mutual-information maximization and RBM conditional independence.
- B. Multiple steps: For multiple hiddens, relative filter phases can be fixed by comparing coarse-grained and original composite-variable correlators.With one hidden degree of freedom, a single filter is used across the system.
RG FLOW
The RSMI procedure characterizes iterative RG flow in the 2D Ising model, locating the critical point and recovering its unstable behavior and critical exponent.
- The 2D Ising model has paramagnetic and ferromagnetic phases separated by a critical point that is an unstable RG fixed point.
- Effective β decreases below βc, increases above βc, and remains constant at β = βc, revealing divergent flow around the unstable point.
- The direction of RG flow identifies the critical point and whether it is stable or unstable, within 1% accuracy for the simulated system.
- The finite-size data-collapse slope is consistent with the exact Ising critical exponent ν = 1.
- Increasing Monte Carlo samples and especially the initial system size can reduce the reported errors.
RELATION OF MUTUAL INFORMATION RG PROCEDURE TO CONVENTIONAL RG SCHEMES
The mutual-information prescription gives a precise, neural-network-compatible formulation of real-space RG by selecting coarse-grained variables that retain environmental information.
- Conventional real-space RG introduces localized coarse-grained variables H through conditional probabilities PΛ(hi|X).
- Normalization preserves the partition function and therefore the free energy of the original system.
- Conditional independence allows certain coarse-grained expectation values to be calculated exactly from local functions of the original variables.
- A useful RG scheme chooses PΛ so the effective Hamiltonian remains short-ranged and involves few-body terms, enabling recursive coarse-graining.
- Mutual-information maximization is argued to be equivalent to retaining combinations most strongly correlated with the environment, and is invariant under reparameterizations of degrees of freedom.
1. Ideal filters for the 1D Ising model
For the 1D Ising model, maximizing mutual information favors decimation filters over boundary and uniform alternatives. At mutual-information saturation, the resulting effective Hamiltonian excludes next-nearest-neighbor interactions.
- The 1D Ising model remains short-range correlated because transfer-matrix contributions decay to zero at large separation.
- The decimation filter couples a hidden variable to a single visible spin.It is defined by P(h|v1, vLV) = 1 exactly when h = v1.
- The boundary filter determines the coarse-grained variable by majority rule over the boundary spins.
- The uniform filter applies equal coupling across all visible spins and uses majority rule over the entire region.
- Decimation filters yield twice the mutual information of boundary filters, while both outperform uniform filters.The result also agrees with the analytically optimal 1D Ising decimation scheme.
- When mutual information has saturated, no next-nearest-neighbor interactions appear in the effective Hamiltonian.The argument assumes a 1D or quasi-1D local system and that adding hidden variables no longer increases mutual information.
A COMMENT ON UNIFORM VS. BOUNDARY-CENTERED FILTERS FOR 2D ISING MODEL
For the 2D Ising model, RSMI weights increasingly favor the boundary of larger visible regions rather than uniform modulation. The dimer model appears different because constrained degrees of freedom make uniform gradient coupling equivalent to boundary coupling.
- A COMMENT ON UNIFORM VS. BOUNDARY-CENTERED FILTERS FOR 2D ISING MODEL: RSMI weights couple to the boundaries of increasingly large 2D Ising visible areas to maximize mutual information.This behavior is reported as consistent with the expected real-space RG scheme.
- A COMMENT ON UNIFORM VS. BOUNDARY-CENTERED FILTERS FOR 2D ISING MODEL: The boundary-coupling insight holds exactly for standard real-space RG when the number of coarse-grained variables equals the boundary size.
- A COMMENT ON UNIFORM VS. BOUNDARY-CENTERED FILTERS FOR 2D ISING MODEL: Dimer-model weights remain uniformly textured because dimers are constrained degrees of freedom and the height field obeys a conservation equation.
- A COMMENT ON UNIFORM VS. BOUNDARY-CENTERED FILTERS FOR 2D ISING MODEL: Uniform coupling to the dimer model’s height gradient is equivalent to coupling to boundary terms of the height field.The height field itself is not microscopically accessible, whereas its gradient is.
- A COMMENT ON UNIFORM VS. BOUNDARY-CENTERED FILTERS FOR 2D ISING MODEL: The scaling of weights with visible-region size may indicate whether the underlying degrees of freedom are constrained.
- COMPARISON WITH CONTRASTIVE DIVERGENCE TRAINED RBMS: Maximizing mutual information, rather than fitting the input distribution with a divergence-based cost, yields a network identifying physically relevant degrees of freedom.