PaperlayerView exact revision on arXiv

Source-linked AI summary

A high-bias, low-variance introduction to Machine Learning for physicists

Pankaj Mehta, Marin Bukov, Ching-Hao Wang, Alexandre G. R. Day, Clint Richardson, Charles K. Fisher, David J. Schwab

arXiv:1803.08823v3physics.comp-phcond-mat.stat-mechcs.LGstat.ML

TL;DR

The review addresses the need for an accessible, physics-oriented introduction to modern machine learning and data science. It develops core and advanced methods through statistical-physics connections, practical notebooks, and physics-inspired datasets. The review shows that predictive performance depends on balancing model complexity, noise, and training-data quantity, while noting that ML methods are not universally superior to simpler feature-based approaches.

  • Problem

    Physicists need accessible conceptual and practical foundations for using machine learning with increasingly large scientific datasets.

  • Method

    The review presents supervised and unsupervised ML methods through statistical-physics connections, physics-inspired examples, Jupyter notebooks, and a standard train-test evaluation workflow.

  • Results

    Predictive performance depends on the interaction between model complexity, noise, and training-data quantity: simpler models can outperform complex ones with limited noisy data, whereas more data can improve complex-model predictions within the training range.

  • Takeaways & Limitations

    The review provides physicists with background and practical tools for applying ML while highlighting connections that make many ML concepts familiar from statistical physics.

  • Takeaways & Limitations

    Deep neural networks are not a universal solution, and comparable or better performance may come from hand-engineered or random features when data or Monte-Carlo samples are scarce.

Abstract

from arXiv · show

Machine Learning (ML) is one of the most exciting and dynamic areas of modern research and application. The purpose of this review is to provide an introduction to the core concepts and tools of machine learning in a manner easily understood and intuitive to physicists. The review begins by covering fundamental concepts in ML and modern statistics such as the bias-variance tradeoff, overfitting, regularization, generalization, and gradient descent before moving on to more advanced topics in both supervised and unsupervised learning. Topics covered in the review include ensemble models, deep learning and neural networks, clustering and data visualization, energy-based models (including MaxEnt models and Restricted Boltzmann Machines), and variational methods. Throughout, we emphasize the many natural connections between ML and statistical physics. A notable aspect of the review is the use of Python Jupyter notebooks to introduce modern ML/statistical packages to readers using physics-inspired datasets (the Ising Model and Monte-Carlo simulations of supersymmetric decays of proton-proton collisions). We conclude with an extended outlook discussing possible uses of machine learning for furthering our understanding of the physical world as well as open problems in ML where physicists may be able to contribute. (Notebooks are available at https://physics.bu.edu/~pankajm/MLnotebooks.html )

I. INTRODUCTION

This review introduces foundational and advanced machine-learning concepts for physicists by connecting them to statistical physics and practical data-analysis workflows. It emphasizes both theoretical understanding and hands-on application, while illustrating how model complexity, data quantity, and noise affect predictive performance.

  • Motivation: Machine learning is increasingly relevant to the physical sciences because modern physics generates and analyzes large datasets across disciplines.The review also situates ML within broader technological applications, from biotechnology to autonomous systems.
  • Purpose and audience: The review provides an introduction to foundational and state-of-the-art ML and data-science techniques using language and intuition familiar to physicists.It builds on statistical-physics knowledge and uses simple examples before advancing to more specialized topics.
  • Physics connections: Physics offers direct conceptual connections to ML through Monte-Carlo methods, simulated annealing, variational methods, and energy-based models.These shared ideas make substantial parts of modern ML familiar to physicists and motivate their participation in the field.
  • Scope: The review narrows its scope to supervised and unsupervised learning while omitting reinforcement learning to preserve cohesiveness and limit length.The omission is explicitly not presented as a judgment about reinforcement learning’s utility for physical problems.
  • Practical workflow: The review combines theoretical foundations with practical workflows, including train-test splitting, cost minimization, and evaluation on held-out data.The dataset must be partitioned before analysis to avoid incorrect conclusions from data-dependent preprocessing.
  • Polynomial regression: With noiseless data, the model class that generated the data gives the best fit and most accurate out-of-sample predictions.For Ntrain = 10 and σ = 0, the linear model performs best for linear data, while the tenth-order model performs best for tenth-order data.
  • Polynomial regression: With noisy data, the tenth-order polynomial can achieve the lowest in-sample error yet the worst out-of-sample predictions, even when it generated the data.Increasing the training set to Ntrain = 10^4 improves tenth-order predictions over the training range, but performance still degrades beyond that range.

IV. GRADIENT DESCENT AND ITS GENERALIZATIONS

Gradient descent minimizes a cost function by iteratively updating model parameters, but its behavior depends strongly on the learning rate and the geometry of the objective. The review develops gradient descent alongside Newton’s method and surveys practical limitations motivating improved algorithms.

  • Gradient descent: Gradient descent fits a model by iteratively adjusting parameters in the direction of the negative cost-function gradient.The learning rate controls the size of each update toward a local minimum.
  • Limitations: Gradient descent is limited by rugged non-convex landscapes, local minima, costly gradient calculations, and sensitivity to initialization and learning-rate choices.These limitations can lead to poor performance, long training times, instability, or divergence.
  • Gradient descent and Newton’s method: Newton’s method adapts parameter-specific step sizes using curvature information, but Hessian computation and storage make it impractical for models with millions of parameters.The review presents Newton’s method mainly as a source of intuition for modifying gradient-descent algorithms.
  • Learning-rate regimes: The learning rate creates distinct convergence regimes: small values converge slowly, intermediate values oscillate before convergence, and sufficiently large values diverge.For a quadratic potential, η < ηopt gives multiple steps, ηopt reaches the minimum in one step, ηopt < η < 2ηopt oscillates, and η > 2ηopt diverges.
  • Limitations: Because gradient descent uses the same learning rate in every direction, steep directions constrain progress in flatter directions.Adaptive methods would ideally take smaller steps in steep directions and larger steps in flat directions, but tracking curvature can require expensive second derivatives.
  • Limitations: Even with random initialization, gradient descent can require exponential time to escape saddle points prevalent in high-dimensional spaces.The review notes that modified gradient-descent methods have been developed to accelerate saddle-point escape.

E. Methods that use the second moment of the gradient

Second-moment methods adapt learning rates using gradient-history statistics, allowing larger steps in flat directions without computing Hessians. RMSprop and ADAM can therefore navigate complex landscapes faster than simpler first-order methods, although trajectories may still miss the global minimum.

  • Motivation: Adaptive optimizers track gradient moments to adjust parameter-specific learning rates while avoiding the computational cost of Hessian calculations.The methods discussed include RMSprop and ADAM, which use second-moment information to adapt step sizes.
  • RMSprop: RMSprop reduces the learning rate in directions with consistently large gradients, enabling larger learning rates in flatter directions and speeding convergence.Its second-moment average uses β typically near 0.9, η_t typically near 10^-3, and ϵ near 10^-8.
  • ADAM: ADAM combines running averages of the first and second gradient moments with bias correction to adapt learning rates across parameters.The first- and second-moment memory parameters β1 and β2 are typically 0.9 and 0.99.
  • ADAM: ADAM suppresses persistent large gradients and scales fluctuating gradients according to their signal-to-noise ratio.This limits steps in steep directions while adapting updates when gradients fluctuate substantially.
  • Comparison of methods: On Beale’s function, RMSprop and ADAM use η = 10^-3 rather than η = 10^-6 and generally navigate the landscape faster than GD, GDM, and NAG.The comparison uses 10^4 steps and three initial conditions; some trajectories follow a narrow ravine instead of reaching the global minimum.
  • Practical guidance: Mini-batch gradient estimates introduce stochasticity, while randomizing the data prevents fitting correlations caused by presentation order.These practices are presented as practical guidance for gradient-descent-based algorithms, especially in deep neural networks.

B. Ridge-Regression

Ridge regression adds an L2 penalty that constrains parameter magnitudes and shrinks least-squares components, while LASSO uses an L1 penalty that promotes sparse solutions. In the Ising example, regularization strength affects predictive performance and the learned interaction representation.

  • Ridge regression: Ridge regression adds an L2 penalty to least-squares loss, equivalently constraining the magnitude of the learned parameter vector.The penalized and constrained formulations are equivalent for Ridge and LASSO, but not generally for best subset selection.
  • LASSO and sparse regression: LASSO applies an L1 penalty and, because its objective is nondifferentiable at zero, uses subgradient optimality to obtain its solution.The review presents the analytic solution under the simplifying assumption that X is orthogonal.
  • LASSO and sparse regression: LASSO performs soft-thresholding, whereas Ridge scales the least-squares solution without the same thresholding behavior.The comparison is illustrated in Fig. 12 through the corresponding estimator curves.
  • LASSO and sparse regression: LASSO tends to produce sparse solutions because the vertices of its L1 feasible region favor intersections where parameters are zero.Ridge and LASSO are both convex, but only Ridge is strictly convex for λ > 0, guaranteeing a unique solution.
  • Ising-model application: For the Ising model, the LASSO test curve has an optimum near λ ≈ 10−2, while different regularizers can learn equivalent interaction models in different gauges.OLS and Ridge learn nearly symmetric weights, whereas LASSO tends to break that symmetry.

G. Recap and a general perspective on regularizers

Regularization improves generalization by restricting model complexity, while Bayesian priors provide intuition for this constraint. The review then connects these ideas to classification and physics applications, including Ising phases and particle-physics event selection.

  • Regularization: Regularization typically improves generalization in high-dimensional regression by shrinking the allowed parameter space and reducing overfitting.The constrained and penalized formulations are equivalent for LASSO and Ridge, but not generally for L0 penalization.
  • Regularization: Regularization functions can be interpreted as priors in Bayesian inference, explaining why they favor models less likely to overfit.The review uses a pendulum experiment to illustrate how prior beliefs suppress attention to irrelevant features.
  • Regularization: The regularization hyperparameter λ strongly affects learning performance, making its selection essential.The review identifies an SGD sweet spot near λ ∼10^-1 in the Ising phase-recognition example.
  • Classification: Logistic regression extends the framework to discrete outcomes, while SoftMax regression handles multiple categories through cross-entropy optimization.The review derives these models using Bayesian reasoning and statistical-mechanics intuition.
  • Classification: In Ising phase recognition, training and test accuracies remain close, but accuracy on near-critical states is about 7% lower when those states are excluded from training.Liblinear generally outperforms SGD, although SGD can perform better on near-critical data for some λ values.
  • Classification: Logistic regression provides clearer signal-background discrimination than a cut-based strategy, with higher-order variables contributing noticeable additional performance.The comparison uses ROC curves for simple variables, the full variable set, and a leading-lepton pT requirement.

D. Softmax Regression

SoftMax regression represents multiclass labels with one-hot vectors and assigns probabilities across categories. The section also motivates ensemble methods through bias-variance analysis, emphasizing decorrelation as the route to variance reduction.

  • D. Softmax Regression: SoftMax regression generalizes logistic regression from binary labels to M classes represented by one-hot target vectors.For M = 1, the cross-entropy cost reduces to the logistic-regression form.
  • D. Softmax Regression: MNIST provides a ten-class SoftMax task using 28 × 28-pixel handwritten-digit images with 256 grayscale values per pixel.The review visualizes the learned weight vectors for the ten digit classes.
  • D. Softmax Regression: The SoftMax function gives the probability that an input belongs to each class, with the likelihood and cost defined accordingly.The formulation uses the components y_im′ of the one-hot label vector.
  • Ensemble methods: Ensemble methods combine multiple models to improve predictive performance, and their effectiveness depends strongly on model correlation.The review introduces bagging, boosting, random forests, and gradient-boosted trees within this bias-variance framework.
  • Ensemble methods: Averaging many uncorrelated models can substantially reduce variance without increasing bias, making decorrelation central to random-ensemble performance.As ensemble size M approaches infinity and ρ(x) = 0, variance suppression is maximized.
  • Ensemble methods: Bagging reduces variance for unstable predictors but can increase bias because bootstrap samples approximate the empirical training distribution rather than the true data distribution.The bias increase may be negligible compared with the variance reduction in many cases.

C. Boosting

This section introduces ensemble methods that combine weak, high-variance classifiers into stronger predictors. It explains bagging, boosting, random forests, and their applications to Iris, Ising, and SUSY datasets.

  • C. Boosting: Bagging combines predictors with equal weight, whereas boosting assigns weights that emphasize stronger classifiers.Both methods build a strong predictor from many weaker classifiers.
  • C. Boosting: AdaBoost iteratively reweights poorly classified data points so later classifiers focus more on correcting those errors.The procedure initializes uniform weights, selects a low-error hypothesis, updates α_t, and renormalizes data weights.
  • D. Random Forests: Random forests reduce correlations between randomized decision trees through bootstrap datasets, feature subsets, or randomized splits.Feature bagging is the defining randomization procedure of random forests, while extreme random forests also randomize splitting.
  • D. Random Forests: On Iris, decision trees, random forests, and AdaBoost produce decision surfaces across feature pairs, with columns representing the three aggregation methods.The classifiers use two of four flower measurements, and the plots use 10-fold cross-validation; forests and AdaBoost use 30 components.
  • F. Applications to the Ising model and Supersymmetry Datasets: Random forests achieve over 99% training and test accuracy on Ising phases, while fine trees reach nearly 85% accuracy in the untrained critical region.Coarse trees perform poorly in the critical region, and increasing ensemble size improves performance but increases training time.
  • F. Applications to the Ising model and Supersymmetry Datasets: XGBoost classifies SUSY collisions with about 79% accuracy using default parameters and nearly 80% after fine-tuning, while ranking feature importance scores.The result uses 100,000 of 5,000,000 Monte-Carlo samples and combines detector measurements with physics-informed features.

1. The basic building block: neurons

Neural networks layer neurons that apply weighted linear transformations followed by nonlinear activations. Their hidden layers increase representational power, while activation choices and architecture affect trainability and performance.

  • 1. The basic building block: neurons: A neuron maps d input features to a scalar by applying a weighted linear operation and a nonlinear transformation.Neural networks stack these neurons into input, hidden, and output layers.
  • 1. The basic building block: neurons: Modern deep networks commonly use ReLUs, leaky ReLUs, or ELUs rather than saturating activations such as sigmoids and tanh.Saturating functions can produce vanishing gradients when inputs become large, hindering gradient-based training.
  • 1. The basic building block: neurons: Feed-forward neural networks pass each layer’s outputs to the next layer, creating a hierarchical network architecture.The architecture repeats neuron transformations through hidden layers until reaching the output layer.
  • 1. The basic building block: neurons: Hidden layers increase neural-network expressivity, and a single hidden layer can approximate any continuous multi-input/multi-output function arbitrarily accurately.This universal-approximation statement concerns representational capacity and does not by itself specify a practical architecture.
  • 1. The basic building block: neurons: Choosing a neural-network architecture remains problem-specific, depending on the task, available data, and computational resources.The number of hidden layers and neurons affects performance, with no single architecture guaranteed to work best.
  • 1. The basic building block: neurons: Neural networks are trained by minimizing a loss with gradient descent, using backpropagation to compute gradients through multiple hidden layers.Common losses include mean squared or mean absolute error for continuous data and cross-entropy for categorical data, with optional regularization.

D. Regularizing neural networks and other practical considerations

Practical neural-network performance depends on controlling overfitting, tuning training choices, and matching architectures to structure in the data. The review connects these practices to physics through regularization, critical Ising behavior, and spatially structured CNNs.

  • Practical training choices: DNN training requires random weight initialization and learning-rate searches over logarithmic grid points, with input centering or whitening as a common preprocessing step.If the optimum lies at a grid edge, the search is repeated with a shifted grid.
  • Regularization: Early Stopping halts training when validation error begins rising, using the validation set as a proxy for out-of-sample performance.Training error can continue decreasing while validation error increases because of overfitting.
  • Regularization: Regularization methods help DNNs generalize, with Dropout reducing spurious neuron correlations through randomized removal of neurons and connections during training.Dropout approximates ensemble-style averaging while avoiding the cost of training multiple full networks.
  • Regularization: Batch Normalization accelerates learning by preventing vanishing gradients and also appears to regularize training through mini-batch-dependent randomness.The regularizing mechanism is presented as plausible rather than fully understood.
  • Physics-inspired examples: About 10 hidden-layer neurons at learning rate 0.1 achieve very high Ising test accuracy, but more neurons are required to learn complex correlations near criticality.The result comes from a grid search over learning rate and hidden-layer width.
  • Structure-aware architectures: CNNs exploit locality and translational invariance that all-to-all networks discard, while shared filters reduce parameters by a factor of H×W at each layer.For D = 102 and H = W = 102, the review reports a parameter reduction of nearly 10^6.
  • Physics-inspired examples: A simple Ising CNN reached 100% test accuracy across tested architectures and 80%–90% accuracy on near-critical samples.The network used one convolutional layer followed by a soft-max layer and was trained on far-paramagnetic and ordered phases.

C. Pre-trained CNNs and transfer learning

Pre-trained CNNs can be repurposed for new supervised tasks by reusing learned convolutional features, replacing classifiers, or fine-tuning weights. Their advantage grows with large datasets, while limited, labeled, mixed-type, or physics-oriented data can favor other approaches.

  • C. Pre-trained CNNs and transfer learning: Transfer learning reuses convolutional filters learned on one image-recognition task for related tasks with limited modification and fine-tuning.The review describes fixed feature detectors at the top or intermediate layers and full-network fine-tuning as alternative strategies.
  • C. Pre-trained CNNs and transfer learning: Pre-trained CNNs can serve as fixed feature detectors by replacing the original soft-max classifier with a task-specific classifier.For small, similar datasets, the review recommends retraining a linear SVM or soft-max layer while keeping the CNN fixed.
  • C. Pre-trained CNNs and transfer learning: Deep networks learn relevant representations with minimal hand-crafted features, especially when large datasets are available.Their performance can be comparable to or worse than other methods on small datasets when those methods use hand-engineered features.
  • C. Pre-trained CNNs and transfer learning: Large DNNs can exploit additional data more effectively than SVMs and ensemble methods, whereas they offer no substantial benefit and may perform worse on small datasets.The review states that sufficiently large DNNs can generalize well in data-rich settings, while deep learning has been less successful where data are limited.
  • C. Pre-trained CNNs and transfer learning: DNNs require labeled data, are highly data intensive, and handle mixed data types less naturally than random forests or gradient-boosted trees.Their utility is described as extremely limited for small datasets, where hand-engineered features can outperform them.
  • C. Pre-trained CNNs and transfer learning: Prediction performance alone may not reveal the underlying distribution that generates physical data.The review cautions that a model can make good predictions while remaining unhelpful for understanding the physics.

D. t-SNE

t-SNE constructs nonlinear low-dimensional embeddings that preserve local structure by matching neighborhood probabilities, using a long-tailed latent distribution to separate distant points. Its visualizations are useful but stochastic, distort scale, cannot directly embed new points, and can be computationally expensive.

  • D. t-SNE: t-SNE maps high-dimensional points to low-dimensional coordinates optimized to preserve local structure in the data.It is a nonparametric method that does not explicitly parametrize feature extraction for new points.
  • D. t-SNE: The review presents t-SNE as useful for revealing hidden structure and preserving locality in physics datasets, including Ising and Fermi-Hubbard spin configurations.It also notes applications to clustering transitions in glass-like systems.
  • D. t-SNE: t-SNE uses a long-tailed low-dimensional distribution to preserve nearby relationships while strongly repelling points far apart in the original space.The embedding minimizes the Kullback-Leibler divergence between high- and low-dimensional neighborhood distributions.
  • D. t-SNE: The t-SNE map places a nearby point at a short distance and a distant point at a relatively large distance from a reference point.The figure contrasts Gaussian short-tail neighborhoods in the original space with Cauchy fat-tail neighborhoods in the embedding.
  • D. t-SNE: t-SNE results vary with the random initialization used by gradient descent.Different seeds can produce slightly different maps, while rotations of the latent space are considered equivalent.
  • D. t-SNE: t-SNE generally preserves neighborhood ordering rather than actual distances, so nearby embedded points are expected to be close in the original space.The review advises against interpreting latent-space cluster sizes as meaningful because scales are deformed.
  • D. t-SNE: Direct t-SNE has O(N^2) complexity, with O(N log N) approximations available through Barnes-Hut methods.The direct implementation is described as applicable only to small-to-medium datasets.

B. Clustering and Latent Variables via the Gaussian Mixture Models

Gaussian mixture models formulate clustering with latent component assignments and component-specific Gaussian distributions, while EM alternates responsibility estimation with parameter updates. This provides a probabilistic clustering framework, but its results depend on generative assumptions and local optimization.

  • B. Clustering and Latent Variables via the Gaussian Mixture Models: Clustering can be viewed as inferring an unobserved latent variable representing each data point’s cluster identity.The interpretation requires assumptions about the probability distribution that generated the dataset.
  • B. Clustering and Latent Variables via the Gaussian Mixture Models: A Gaussian mixture model assumes that points arise from one of K Gaussians, each with mean µ_k, covariance Σ_k, and mixing probability π_k.The parameter set is θ = {µ_k, Σ_k, π_k}.
  • B. Clustering and Latent Variables via the Gaussian Mixture Models: GMM clustering assigns each point responsibilities γ(z_k), the probabilities that mixture k explains the point.A hard assignment selects the cluster with the largest responsibility.
  • B. Clustering and Latent Variables via the Gaussian Mixture Models: Maximum-likelihood GMM clustering seeks parameters that maximize the likelihood of the observed dataset before computing cluster assignments.The global likelihood maximum is generally difficult to obtain, so practical procedures may settle for a local maximum.
  • B. Clustering and Latent Variables via the Gaussian Mixture Models: Expectation Maximization alternates calculating latent-variable conditional probabilities with maximizing the expected complete-data log likelihood over model parameters.The parameter updates weight each data point according to its current probability of belonging to each cluster.
  • B. Clustering and Latent Variables via the Gaussian Mixture Models: Variational methods generalize the EM perspective to latent-variable models by approximating complex probabilistic distributions.The review presents EM for GMM clustering as a variational procedure applicable to broader hidden-variable models.
  • B. Clustering and Latent Variables via the Gaussian Mixture Models: The variational free energy is at least as large as the true free energy, and optimizing it is equivalent to minimizing the KL divergence between q and p.Equality holds only when q = p.
  • B. Clustering and Latent Variables via the Gaussian Mixture Models: Mean-field theory can yield closed Ising equations, but an independent-spin variational distribution misses correlations and can predict an incorrect critical temperature and spurious one-dimensional phase transition.The review therefore describes variational mean-field theory as powerful but requiring careful application and interpretation.

B. Expectation Maximization (EM)

Expectation Maximization estimates parameters in latent-variable models by alternating inference of hidden-variable distributions with parameter updates. Its variational-free-energy formulation links EM to statistical-physics variational methods and guarantees nondecreasing true log-likelihood.

  • B. Expectation Maximization (EM): EM provides a physics-based interpretation of latent-variable maximum likelihood through its correspondence with variational mean-field theory.The review also notes a one-to-one dictionary between EM for latent-variable models and mean-field theory for spin systems with quenched disorder.
  • B. Expectation Maximization (EM): EM estimates parameters when observed data depend on unobserved latent variables by iteratively optimizing an alternative objective based on the hidden-variable distribution.The E-step estimates q(z|x), while the M-step re-estimates parameters using that distribution.
  • B. Expectation Maximization (EM): Each EM iteration increases the true log-likelihood or leaves it unchanged, and the procedure usually converges to a local maximum.This monotonicity is represented by successive E- and M-step updates in the convergence figure.
  • B. Expectation Maximization (EM): The variational free-energy Fq is an upper bound on the true free-energy Fp because the KL-divergence is nonnegative.With the physics sign convention, −Fq is therefore a lower bound on the log-likelihood-related quantity −Fp.
  • B. Expectation Maximization (EM): The E-step constructs q(z) and the variational objective −Fq(θ), after which the M-step maximizes this objective with respect to θ.The alternating updates produce θ(t+1), which is used in the next E-step.

2. From statistical mechanics to machine learning

Statistical-mechanics ideas provide a foundation for generative machine-learning models, while data-based constraints create practical estimation and model-selection challenges. Latent variables and regularization extend expressive power while addressing correlations and overfitting.

  • 2. From statistical mechanics to machine learning: Maximum Entropy models become statistical inference procedures by replacing exact observed averages with empirical averages estimated from data.This substitution makes training sensitive to sampling error and the choice of constrained functions.
  • 2. From statistical mechanics to machine learning: Constraining first and second moments yields a multidimensional Gaussian for continuous variables and a generalized Ising model with all-to-all couplings for binary variables.The corresponding parameters are Lagrange multipliers inferred from the data.
  • 2. From statistical mechanics to machine learning: Partition functions are often intractable for energy-based models, motivating special cases and approximate training procedures.For binary variables, the resulting Ising-model partition function cannot generally be computed in closed form.
  • 2. From statistical mechanics to machine learning: Generative-model training must balance similarity to the training data with generalization beyond spurious details specific to that dataset.The review identifies this as a central difficulty distinct from simply predicting labels.
  • 2. From statistical mechanics to machine learning: Restricted Boltzmann models use latent variables to represent sophisticated correlations and can learn which interaction orders matter directly from data.Multiple hidden units can encode complex interactions at all orders without specifying those orders beforehand.

B. Restricted Boltzmann Machines (RBMs)

Restricted Boltzmann Machines are bipartite energy-based generative models whose latent units encode complex visible-variable correlations. Their structure supports Gibbs-based sampling and trainability, while approximate sampling introduces limitations.

  • B. Restricted Boltzmann Machines (RBMs): An RBM contains visible and hidden units that interact across layers but not within either layer.This bipartite structure makes visible and hidden units conditionally independent.
  • B. Restricted Boltzmann Machines (RBMs): The bipartite structure enables block Gibbs sampling, which supports estimating model expectations and makes RBMs easier to train.Training typically minimizes negative log-likelihood with stochastic gradient descent, using data and model phases of the gradient.
  • B. Restricted Boltzmann Machines (RBMs): Marginalizing hidden units produces all orders of visible-unit interactions, weighted by the hidden-unit cumulants.For a Hopfield-model hidden distribution, only the first cumulant remains and the familiar form is recovered.
  • B. Restricted Boltzmann Machines (RBMs): Bernoulli hidden units give RBMs substantial representational power because each hidden unit can encode arbitrarily high-order visible correlations.Combining hidden units allows complex interactions to be learned directly from data.
  • B. Restricted Boltzmann Machines (RBMs): Contrastive Divergence accelerates training by truncating Gibbs sampling, but its samples are not drawn from the true model distribution.Persistent Contrastive Divergence instead continues chains from fantasy particles generated in the previous update.
  • B. Restricted Boltzmann Machines (RBMs): A DBM with Nhidden = 800 worked well in disordered and critical Ising regions but used a nonoptimal architecture in the ordered phase at T/J = 1.75.The review attributes the ordered-phase issue presumably to effects related to symmetry breaking.

XVII. VARIATIONAL AUTOENCODERS (VAES) AND GENERATIVE ADVERSARIAL NETWORKS (GANS)

The review contrasts likelihood-based generative modeling with GANs and VAEs, emphasizing how divergence choices shape model behavior. GANs use adversarial discrimination to address likelihood-based tendencies to place probability in low-density regions, though training is difficult.

  • XVII. VARIATIONAL AUTOENCODERS (VAES) AND GENERATIVE ADVERSARIAL NETWORKS (GANS): GANs and VAEs extend generative modeling with differentiable neural networks beyond the energy-based models discussed earlier.The section motivates GANs through limitations of maximum likelihood and connects VAEs to variational methods.
  • XVII. VARIATIONAL AUTOENCODERS (VAES) AND GENERATIVE ADVERSARIAL NETWORKS (GANS): The Jensen-Shannon objective underlying original GANs is sensitive both to missing data regions and to model probability placed where no data are observed.This contrasts with the asymmetric sensitivities of the two KL-divergences.
  • XVII. VARIATIONAL AUTOENCODERS (VAES) AND GENERATIVE ADVERSARIAL NETWORKS (GANS): KL-divergence direction changes which distributional mismatches a likelihood-based model penalizes.DKL(pdata||pθ) favors probability where training data occur, whereas DKL(pθ||pdata) penalizes probability where no data occur.
  • XVII. VARIATIONAL AUTOENCODERS (VAES) AND GENERATIVE ADVERSARIAL NETWORKS (GANS): Likelihood-based training may improperly fill low-probability regions between peaks in the data distribution.The review presents this as a likely failure mode of likelihood-based methods.
  • XVII. VARIATIONAL AUTOENCODERS (VAES) AND GENERATIVE ADVERSARIAL NETWORKS (GANS): GANs are difficult to train, and the review limits its treatment to a high-level overview.Readers seeking implementation guidance are directed to a separate practical discussion.
  • XVII. VARIATIONAL AUTOENCODERS (VAES) AND GENERATIVE ADVERSARIAL NETWORKS (GANS): GANs train a generator and discriminator adversarially, with the generator mapping latent samples to model outputs and the discriminator distinguishing data from generated samples.The discriminator penalizes generated points that are easily distinguished from the data.

3. Connection to the information bottleneck

The review connects variational autoencoders to the information bottleneck through variational bounds, while illustrating latent representations, generation, and their limitations on physics-inspired data.

  • Connection to the information bottleneck: The information bottleneck compresses x into z while retaining information about a relevance variable y, with β controlling compression versus accuracy.
  • Connection to the information bottleneck: Variational approximations replace the intractable decoder and encoding prior with tractable distributions, producing an upper bound with the VAE objective's form.
  • Latent representations: VAE latent embeddings generally place similar digits near one another, although some points remain poorly organized in the low-dimensional space.
  • Generative modeling: VAEs generate new examples by sampling latent variables and decoding them, distinguishing them from non-generative embeddings such as t-SNE.
  • Generative modeling: Ising-model samples generated by the VAE lack critical-region patchiness because the model has no spatial structure, only two latent dimensions, and a correlation-insensitive cost.
  • Review perspective: The review presents ML as prediction-oriented statistical learning, connected to statistical physics through concepts including Monte Carlo, gradient descent, variational methods, and mean-field theory.

Appendix A: Overview of the Datasets used in the Review

The review uses Ising, supersymmetric-collision, and MNIST datasets to demonstrate machine-learning methods across physics and standard image-recognition settings.

  • Ising dataset: The Ising dataset contains 160,000 samples of 40×40 spin configurations generated by Metropolis sampling at 16 temperatures from 0.25 to 4.0.
  • Ising dataset: The Ising configurations represent the Boltzmann distribution of a two-dimensional ferromagnetic model on a 40×40 periodic square lattice.
  • SUSY dataset: The SUSY dataset uses Monte Carlo simulations of events containing two leptons to study deep-learning classification of collision events.
  • SUSY dataset: In the SUSY example, logistic regression predicts the relative probability that an event is signal or background using final-state kinematic information.
  • MNIST dataset: MNIST contains 60,000 training and 10,000 test examples of handwritten numerical characters from 0 to 9.
  • MNIST dataset: MNIST images are centered 28×28 grayscale representations produced by size normalization, anti-aliasing, and center-of-mass translation.
Loading 1803.08823v3…