Source-linked AI summary
Sampling the XY model with an attention-based neural network
Piotr Białas, Christian D. Morales, Tomasz Stebel, Dawid Zapolski
TL;DR
Sampling continuous lattice systems near phase transitions remains challenging, and existing neural samplers for the XY model degrade with system size. This paper introduces a physics-informed transformer autoregressive sampler and reports strong sampling performance, including agreement with Wolff-algorithm observables on the largest studied lattice.
Problem
Existing neural samplers for the continuous-degree-of-freedom XY model degrade quickly with system size, while MCMC faces critical slowing down near phase transitions.
Source: §1 · pp. 1–2Method
The authors sample the XY model spin by spin with a transformer autoregressive network combining von Mises distributions, locality-informed attention, and analytically combined lowest-order approximate probabilities.
Source: §1 · p. 2Results
At L = 16, tVAN reaches ESS close to one versus order 10^-2 or lower for CAN and a normalizing flow; at L = 96, its energy, magnetization, and helicity modulus agree with Wolff results within statistical uncertainties.
Source: §4 · pp. 12–13Takeaways & Limitations
At L = 96, averaging over four lattice rotations raises ESS from 0.07 to 0.38 at a 20% generation-time cost, without retraining.
Source: §4 · p. 13A large ESS alone cannot rule out mode collapse, especially with finite statistics, because reweighting and NMCMC cannot recover configurations never generated.
Source: §2.3 · p. 4
Abstract
from arXiv · showhide
We present a transformer-based variational autoregressive network for sampling lattice systems with continuous degrees of freedom and apply it to the two-dimensional XY model near the Berezinskii-Kosterlitz-Thouless transition. The architecture is combined with physics-informed ingredients: locality-informed attention and approximate probabilities. The resulting sampler significantly outperforms existing neural samplers. We train the model on lattices up to $96\times96$, verify with a mode-dropping estimator that no parts of the configuration space are missed, and reproduce the energy, magnetization and helicity modulus obtained with the traditional Markov Chain Monte Carlo algorithm.
1. Introduction
The introduction frames neural samplers as a route around limitations of MCMC and motivates the two-dimensional XY model as a demanding continuous-spin benchmark. It presents a transformer-based autoregressive sampler with physics-informed components.
1. Introduction
- MCMC suffers critical slowing down near phase transitions, while cluster algorithms are model-specific and MCMC does not directly provide the partition function.Free energy and entropy therefore require dedicated techniques such as thermodynamic integration or Wang–Landau.Source: §1 · p. 1
- Neural samplers generate independent configurations with exact normalized probabilities, enabling asymptotically exact bias correction and direct free-energy estimates.The introduction identifies variational autoregressive networks and normalizing flows as the two dominant architecture families.Source: §1 · p. 1
- The XY model combines continuous O(2) symmetry, a BKT transition driven by vortex unbinding, and compact U(1)-valued variables shared with lattice gauge theories.Existing XY neural samplers include CAN, normalizing flows, and adversarially trained flows, whose sampling quality degrades quickly with system size.Source: §1 · p. 2
- The proposed autoregressive sampler uses a transformer and von Mises functions, plus locality-informed attention, analytically combined lowest-order approximate probabilities, and post-training lattice-symmetry averaging.The paper positions these ingredients against CAN's convolutional network and beta distributions.Source: §1 · p. 2
- The paper reviews neural-sampler methods and diagnostics, then evaluates sampler comparisons, modifications, scaling, observables, symmetrization, and use as a normalizing-flow prior.Additional derivations, a symmetrization proof, and hyperparameter studies appear in the appendices.Source: §1 · p. 2
2. Method
The method trains a normalized autoregressive sampler to approximate the XY model’s Boltzmann distribution, then uses importance sampling or an independent-proposal Markov chain to correct residual bias. Its continuous-spin transformer uses lattice-local attention and a two-neighbor approximate probability prior.
2.1. Neural Samplers
- The neural sampler generates configurations from a normalized qθ approximating the Boltzmann target, and training minimizes the reverse Kullback–Leibler divergence through the variational free energy.The variational free energy is an upper bound on the true free energy.Source: §2.1 · pp. 2–3
2.2. Autoregressive representation
- Autoregressive factorization writes the joint density as conditional densities, sampled sequentially so each spin depends only on previously generated spins.Generation proceeds by ancestral sampling from the first spin through the full configuration.Source: §2.2 · p. 3
2.3. Asymptotically unbiased observables
- Neural importance sampling and neural Markov chain Monte Carlo provide asymptotically exact observables, while the variational estimate alone does not guarantee correct sampling.The importance-sampling guarantee requires proposal support wherever the target distribution has support.Source: §2.3 · pp. 3–4
- A large effective sample size is necessary but insufficient for correctness: finite-sample mode collapse can leave unseen regions unrecoverable by reweighting.The mode-dropping estimator signals missing modes when its value is below one.Source: §2.3 · pp. 4–5
2.4. XY model
- The model uses unit-length XY spins with nearest-neighbor interactions, J = 1, and periodic boundaries; the thermodynamic-limit BKT transition occurs around β = 1.12.Global rotational symmetry permits fixing the first spin to zero during generation while retaining its uniform-spin probability contribution.Source: §2.4 · p. 5
2.5. Transformer architecture
- The transformer represents each angle as (cos si, sin si) and outputs a mixture of von Mises conditional densities, sampled with inverse-CDF sampling.The von Mises choice resembles the XY Boltzmann distribution and supports efficient representation for small K; inverse-CDF sampling avoids rejections.Source: §2.5 · p. 6
2.6. Locality-informed attention
- Locality-informed attention restricts each conditional to lattice-relevant earlier spins; the context comprises the first row and most recently generated L spins.The restriction follows nearest-neighbor interactions and is justified in the passage by the Hammersley–Clifford theorem.Source: §2.6 · pp. 6–7
2.7. Approximate probabilities
- The approximate-probability prior uses the lowest-order t1 approximation based on two nearest neighbors, leaving the transformer to learn the remaining difference.The passage notes that the higher-order t2 approximation no longer belongs to the von Mises family.Source: §2.7 · p. 7
3. Results
The transformer sampler outperforms the tested continuous-spin neural baselines, while approximate probabilities improve training and global gauge fixing does not visibly help ESS. Across larger lattices it retains mode coverage, reproduces MCMC observables, and benefits from post-training symmetrization, whereas adding a normalizing flow prior yields little improvement.
3.1. Comparison with existing neural samplers
- At L = 16 and β = 1.12, tVAN’s ESS rapidly approaches 1, versus 10^-3–10^-2 for CAN and 10^-2 for NF; CAN and NF fail to train on larger systems.At L = 32, tVAN’s average acceptance rate is 80.48% at β = 1.12, compared with AdvNF’s best 7.2% averaged over 10 temperatures.Source: §3.1 · p. 8
3.2. Physics-informed improvements
- After 20k optimizer steps, approximate probabilities increase ESS by a factor of 1.5–1.8, while global gauge fixing yields no visible ESS advantage.The comparison used four combinations of approximate probabilities and gauge fixing at L = 32 and β = 1.12.Source: §3.2 · p. 9
3.3. Results for different lattice sizes
- Across lattice sizes, tVAN’s ESS falls from 0.976 at L = 16 to 0.074 at L = 96, while W remains consistent with 1.The passage says the ESS drop is not related to a problem with missing modes and that NIS/NMCMC can compensate for the difference between qθ and p.Source: §3.3 · p. 9
3.4. Physical observables
- At L = 96 and β ∈ {1.0, 1.12, 1.2}, tVAN reproduces MCMC results for the three observables up to statistical fluctuations.The comparison also includes the mode-dropping estimator W and uses 10^6 tVAN and 10^6 Wolff configurations.Source: §3.4 · p. 10, Figure 4 · p. 10
3.5. Model symmetrization
- For group sizes 2–4, symmetrization improves ESS by factors of 3–6 at a 10–20% cost increase; a 64-element group reaches ESS 0.81 at four times the cost.For example, C4 rotations raise ESS from 0.066 to 0.38; symmetrized helicity-modulus results remain consistent with Wolff MCMC.Source: §3.5 · p. 11, Figure 5 · p. 12
3.6. tVAN as NF prior
- Adding a normalizing flow to a tVAN prior increases ESS only marginally at L = 32, and the difference is smaller or sampling quality deteriorates at larger sizes.The authors attribute this behavior to the low performance of the normalizing-flow architecture.Source: §3.6 · p. 12
4. Summary
The paper extends tVAN to continuous XY spins using von Mises mixtures and physics-informed modifications. It reports substantially stronger sampling than existing neural samplers, validates observables at L = 96, and finds that symmetry averaging can further improve ESS.
4. Summary
- The continuous-spin tVAN uses (cos s_i, sin s_i) inputs and von Mises mixture conditionals, with locality-informed attention and analytically combinable t1 approximate probabilities.The von Mises choice gives normalized, easily sampled circle densities resembling the local Boltzmann factor; K = 4 components and one transformer layer suffice.Source: §4 · p. 12
- At L = 16, tVAN reaches ESS close to one while CAN and normalizing flows reach 10^-2 or lower; at L = 32, its acceptance rate is about 80% versus 7.2% for AdvNF.The ESS decreases from 0.98 at L = 16 to 0.07 at L = 96, while the mode-dropping estimator remains consistent with one across studied sizes.Source: §4 · pp. 12–13
- Energy, magnetization, and helicity modulus at L = 96 agree with Wolff-algorithm results within statistical uncertainties at all three studied temperatures.The mode-dropping estimator remains consistent with one at every studied lattice size, and the ESS decline can be compensated by NIS or NMCMC.Source: §4 · p. 13
- Symmetry averaging provably does not decrease ESS; at L = 96, four rotations raise ESS from 0.07 to 0.38 for 20% more generation time, while 64 transformations yield 0.81.Small groups provide the best ESS gain per computational time; using tVAN as a normalizing-flow prior improves ESS only marginally, attributed to limited flow expressivity.Source: §4 · p. 13
- Applying the method to lattice gauge theories is proposed as a natural next step, with a tVAN prior combined with normalizing-flow architectures described as promising.Source: §4 · p. 13
Appendix A. t1 approximate probabilities for the XY model
The t1 approximation simplifies the XY model's conditional probability by retaining interactions with already generated nearest neighbors. The resulting approximation remains within the von Mises family and combines with the network mixture.
Appendix A. t1 approximate probabilities for the XY model
- Because future spins are integrated out, terms independent of s_i become normalization constants, leaving a conditional density determined by the current-spin contribution.The passage states that the denominator does not depend on s_i and that the integral over future spins is constant when its integrand is independent of them.Source: Appendix A · pp. 13–14
- The t1 approximation replaces the remaining energy contribution E(s≥i) with E(s_i), retaining interactions between the current spin and its already generated closest neighbors.Those neighbors are the left and top spins in the generation ordering.Source: Appendix A · p. 13
- The retained neighbor interactions reduce to a linear combination of cos(s_i) and sin(s_i), which can be rewritten in von Mises form.The t1 approximation is itself a von Mises distribution after normalization.Source: Appendix A · p. 14
- The network conditional is a mixture of von Mises distributions, and applying t1 preserves that mixture family after its component terms are combined and normalized.Source: Appendix A · pp. 14–15
Appendix B. Proof that ESS ¯q ≥ESSq
Averaging a proposal over energy-preserving symmetries produces a normalized, symmetry-invariant distribution. The proof uses convexity and invariance to show that this symmetrization does not reduce ESS.
Appendix B. Proof that ESS q̄ ≥ ESS q
- The symmetrized probability is normalized and invariant under the symmetry group, provided the target and integration measure are invariant under its transformations.The group acts on configurations without changing their energy, so the target probability is also invariant.Source: Appendix B · p. 15
- Convexity and target invariance yield I_q ≥ I_q̄, establishing that the symmetrized proposal's ESS is at least that of the original proposal.The proof applies Jensen's inequality and uses invariance of the measure under group transformations.Source: Appendix B · p. 16
Appendix C. Hyperparameters
Appendix tests vary mixture size and transformer depth while holding the rest of the setup fixed. They find no statistically significant ESS differences across mixture sizes, while a second layer trades longer training time for higher ESS at 20k steps.
Appendix C. Hyperparameters
- At L = 96 and β = 1.12, mixture sizes K = 4, 8, and 12 show no statistically significant ESS differences; two layers reach higher ESS at 20k steps but take longer to train.The authors conclude that K = 4 and one transformer layer are a reasonable choice.Source: Appendix C · p. 16, Figure C.6 · p. 17