FUND: Density Flow for Sampling Unnormalised Distributions

1 Introduction

• MCMC and molecular dynamics use local perturbations that can converge slowly, cost much, and mix modes poorly across high-dimensional energy landscapes.The cited passage attributes poor mode mixing to high energy barriers between metastable states.Source: §1 · p. 1
• Deep generative models remain data-intensive, often relying on MCMC or molecular dynamics to generate their training data.Source: §1 · p. 2
• Normalizing flows can train from energies without target samples, but reverse-KL training can concentrate on a few low-energy regions and collapse modes.Source: §1 · p. 2
• FUND models the evolving density rather than individual sample paths, enabling model-density computation without true target samples.Source: §1 · p. 2
• FUND advances through intermediate targets, reusing and reweighting model samples to learn each next distribution and capture modes as they separate.The method also uses a sample-free objective on each intermediate target.Source: §1 · pp. 2–3
• FUND is evaluated on three families of unnormalized distributions and achieves competitive deep-learning-based sample generation without true samples requiring MCMC or molecular-dynamics simulations.Source: §1 · p. 3

2 Proposed Method

• FUND transforms samples from a simple prior through an invertible flow and models the evolving density qt directly rather than individual sample trajectories.The flow is trained to match the intermediate target density using a distance measure.Source: §2 · p. 3
• The loss combines mode-covering likelihood or score objectives with reverse KL, whose mode-seeking behavior is intended to balance exploration and concentration.Source: §2 · p. 3

2.1 Designing Intermediate Distribution pt

• The target path pt(x) ∝ p(x)^t smooths modes at small t, then sharpens and separates them as t increases toward the target.The authors describe this progressive construction as reducing complexity and helping avoid missing modes.Source: §2.1 · p. 4, §2 · p. 4

2.2 Distance Measure with Importance Weighting

• Because direct samples from pt are unavailable, FUND estimates forward-KL or Fisher-divergence objectives using importance-weighted samples from an earlier qt′.Small t increments keep qt′ close to pt, supporting overlap and stable importance weights.Source: §2 · p. 3, §2.2 · p. 4

2.3 Sequential Modeling Strategy

• An alternative that appends and freezes a new coupling block at each stage is memory-inefficient and fails to learn all target modes.Source: §2.3 · p. 4
• A single flow block is refined across time steps, saving memory and facilitating incremental learning; Figure 1 depicts successive distributions learned with sample reuse.The caption describes replay-buffer samples and latent samples contributing to separate loss components.Source: §2.3 · p. 4, Figure 1 · p. 5
• A persistent buffer stores transformed samples and likelihoods for later stages; the method uses uniform replay to reuse earlier samples and maintain mode coverage.Source: §2.3 · p. 5
• Training anneals the weights on the sample-driven loss and reverse-KL term, moving from propagating modal structure toward mode sharpening.Source: §2.3 · pp. 5–6

3 Results

• The evaluation uses NLL, RNLL, and ESS: NLL reflects likelihood fit, RNLL with NLL informs mode behavior, and higher ESS indicates greater sample diversity and independence.High NLL with low RNLL can indicate mode collapse, whereas low values on both NLL and RNLL indicate alignment with the target.Source: §3 · pp. 6–7

3.1 Mixture of Gaussian (MOG-40)

• On MOG-40, FUND(SCORE) has the lowest NLL and second-best ESS, while FUND(FKL) is slightly worse in NLL and has higher ESS than FUND(SCORE), yet both outperform baselines.FUND variants also achieve better RNLL than FKL and FAB; Figure 3 presents samples from the methods alongside ground truth for comparing mode coverage.Source: §3.1 · pp. 7–8, Figure 3 · p. 7, Table 1 · p. 8
• Figure 4 shows MOG-40 intermediate distributions sharpening and separating from t1 = 0.01 to t2 = 0.05 as the flow refines and preserves the target structure.The figure also shows qtN at tN = 1.0 and the target distribution p.Source: §3.1 · p. 8, Figure 4 · p. 8

3.2 Many-Well Potential

• On Many-Well distributions, FUND(SCORE) consistently attains the best well-rounded NLL, RNLL, and ESS performance; FUND(FKL) performs at par with it.The evaluation covers 8- and 16-dimensional Many-Well distributions with metastable wells and energy barriers.Source: §3.2 · p. 8, Table 1 · p. 8

3.3 Scalar ϕ4 Theory distribution

• Across scalar ϕ4 lattice sizes, FUND(SCORE) achieves the best NLL and ESS, while FUND(FKL) also outperforms all baselines and baselines degrade as dimensionality increases.At 12 × 12 (144 dimensions), RKL retains moderate ESS and good RNLL but has poor likelihood fit and learns only a few dominant modes.Source: Table 2 · p. 9, §3.3 · p. 9

4 Discussion

• Across MOG-40, Many-Well, and scalar ϕ4, FUND consistently surpasses existing state-of-the-art approaches, with FUND(FKL) favoring cost–quality balance and FUND(SCORE) higher performance at greater computational cost.The sequential annealing strategy also preserves fast parallel inference.Source: §4 · pp. 9–10
• Both FUND variants improve test NLL monotonically across intermediate densities, with the largest gains early and smaller gains near t = 1.Figure 5 shows trajectories for scalar ϕ4 and MW-16.Source: §4 · p. 10, Figure 5 · p. 11
• FUND maintains strong performance across scalar ϕ4 lattice sizes as dimensionality increases, while several baselines degrade.The reported comparison is in Table 2.Source: §4 · p. 10
• Progressive annealing adds sequential optimization stages but is associated with improved stability, mode coverage, NLL, and ESS, followed by parallel single-pass sampling.FUND produces samples through one forward pass after training.Source: §4 · p. 10
• FUND(FKL) offers a favorable cost–quality trade-off, whereas FUND(SCORE) provides stronger sampling performance and accuracy at higher computational cost.Both retain fast parallel inference through sequential annealing.Source: §4 · p. 10
• The sequential procedure requires progressing through intermediate distributions, and overly coarse discretization may disrupt smooth evolution and hinder accurate learning.Fewer intermediates can reduce computational cost, but this must be balanced against smooth structural evolution.Source: §4 · p. 11

5 Conclusion

• FUND learns unnormalised target distributions without true target samples, using importance-weighted samples from intermediate distributions for stable learning, mode coverage, and tractable likelihood evaluation.Across MOG-40, Many-Well, and scalar ϕ4, the authors report consistent state-of-the-art performance and variant-specific cost–quality trade-offs.Source: §5 · pp. 11–12

A Distributions

• The appendix introduces the distributions modeled in the paper.It then specifies the MOG-40 and Many-Well benchmarks.Source: A Distributions · p. 15

A.1 Mixture of Gaussian (MOG-40)

• MOG-40 is a 2D mixture of 40 equally weighted Gaussian components, with means sampled uniformly from [−40, 40] and identity covariances.Its evaluation uses 5,000 test samples and separate training and validation samples for the FKL baseline.Source: §A.1 · p. 15

A.2 Many-Well Distribution

• Many-Well is constructed from independent copies of a 2D double-well distribution, and the experiments evaluate MW-8 and MW-16.For each, 10,000 test samples and additional training and validation samples are generated for the FKL baseline.Source: §A.2 · p. 15

A.3 Scalar ϕ4 theory distribution

• Scalar ϕ4 theory is a lattice field model used to study interacting scalar fields, renormalisation, symmetry breaking, and critical phenomena.Source: §A.3 · p. 15
• The experiments use 8 × 8, 10 × 10, and 12 × 12 lattices with λ = 4 and m2 = −4.HMC sampling uses step size 0.05 and 20 leapfrog steps per trajectory.Source: §A.3 · p. 16
• The setup discards 1,000 initial samples for thermalization and generates 10,000 test samples, plus training and validation samples for the FKL baseline.Source: §A.3 · p. 16

B Model details

• The authors adopt one coupling-block model updated at each time step after incremental modeling caused missed modes, higher NLL, and memory inefficiency.The single-block implementation is used throughout the paper.Source: B Model details · p. 16
• MOG-40 and Many-Well use dense affine coupling networks, while scalar ϕ4 uses convolutional layers with periodic padding.The scalar ϕ4 architecture uses 3×3 filters and separate scale and translation outputs.Source: B Model details · p. 16, Table 4 · p. 17
• Batch sizes are 512 for MOG-40 and Many-Well and 256 for scalar ϕ4; training uses Adam with a learning rate of 1 × 10−5, reduced near the final time steps.Source: B Model details · pp. 16–17
• IDEM follows its official implementation with limited ϕ4 tuning, while FKL and RKL share FUND’s RealNVP architecture and methods use comparable budgets and hardware.FAB and FUND flow architectures are matched where possible; FAB uses annealing distributions and MCMC transitions across benchmarks.Source: B Model details · p. 17

C Hyperparameter Selection and Tuning

• Increasing N consistently improves performance across metrics on the 8 × 8 scalar ϕ4 distribution, though larger N increases computational cost.Larger N makes the interpolation smoother; smaller values are more efficient but reduce performance.Source: C Hyperparameter Selection and Tuning · p. 18, Table 5 · p. 18
• In high-dimensional settings, a moderately larger initial t1 can maintain reliable distributional learning while lowering computational requirements.The authors tune t1 by grid search across distributions.Source: C Hyperparameter Selection and Tuning · p. 18, Table 6 · p. 18
• Replay buffers use 10,000 samples for MOG-40, MW-8, and MW-16, and 10,000, 25,000, or 25,000 for the 8 × 8, 10 × 10, or 12 × 12 ϕ4 cases, respectively.A larger buffer is used for the larger ϕ4 lattices to preserve sample diversity.Source: C Hyperparameter Selection and Tuning · p. 18
• At t1, FUND uses only the reverse KL term; at later stages it starts with larger λ1, then lowers λ1 and raises λ2 when reverse NLL plateaus.The later transition shifts optimization toward reverse-KL refinement.Source: C Hyperparameter Selection and Tuning · pp. 18–19

D Sensitivity Analysis of the Loss Annealing Schedule

• At the first stage, replay-buffer samples are unavailable, so optimization uses only the reverse KL term; later stages use prior-stage samples and shift weights as reverse NLL plateaus.λ1 begins relatively large to preserve learned mode structure, then decreases as λ2 increases.Source: D Sensitivity Analysis of the Loss… · p. 19

Setting 1

• Setting 1 anneals λ1 from 1 to 0.01 while keeping λ2 fixed at 1.Source: Setting 1 · p. 19

Setting 2

• Setting 2 keeps λ1 fixed at 1 while annealing λ2 upward from 0.1 to 1.Source: Setting 2 · p. 19

Setting 3

• Setting 3 anneals λ1 downward and λ2 upward over four stages.Source: Setting 3 · p. 19

Setting 4

• FUND remained reasonably stable when the supervision objective dominated early optimization.Source: Setting 4 · p. 19
• Gradually annealing both coefficients over eight stages yielded the best empirical performance across schedules, but increased computational cost.The comparison is on MW-8 using FUND(FKL).Source: Table 7 · p. 20, Setting 4 · p. 20
• Setting 3 achieved results close to Setting 4 while requiring fewer annealing stages, offering a favorable trade-off between efficiency and performance.Source: Setting 4 · p. 20

E Sensitivity Analysis of Initial Annealing Parameter t1

• At t1 = 0.01, FUND preserved all major MOG-40 modes, whereas t1 = 0.1 produced noticeable mode collapse.Figure 6 compares target and generated samples across initial t1 values and illustrates improved support coverage at smaller values.Source: E Sensitivity Analysis of Initial… · p. 20, Figure 6 · p. 21

F.1 Choice of Initial Annealing Parameter t1

• Highly multimodal targets with sharp energy barriers benefit from smaller t1, while simpler or weakly multimodal high-dimensional targets can tolerate larger values.A sufficiently small t1 promotes overlap across modes in the first annealed distribution.Source: §F.1 · p. 21

F.2 Number of Annealing Stages N

• More complex or highly multimodal targets generally require larger N, while simpler distributions can often be modeled with fewer stages.Increasing N improves performance across metrics on the 8 × 8 scalar ϕ4 distribution.Source: §F.2 · p. 21

F.3 Replay Buffer Size

• Larger replay buffers can stabilize importance-weight estimation and preserve modes, but require more memory; insufficient coverage can bias supervision toward dominant modes.The buffer should represent all modes adequately when their approximate number is known.Source: §F.3 · p. 22

F.4 Annealing Schedule for (λ1, λ2)

• Smooth transitions between the two objectives improve optimization stability and mode coverage, while abrupt changes can degrade performance; gradual loss-weight annealing is more stable than abrupt transitions.Larger initial λ1 preserves previously learned mode structure early, while progressively increasing λ2 improves local refinement and density correction later.Source: §F.4 · p. 22

• MCMC inference uses expensive sequential sampling, and its cost increases with the number of generated samples.Source: G Computational Cost and Runtime… · p. 22
• IDEM has comparatively low training cost, but generation is substantially slower because inference simulates a reverse-time SDE.Source: G Computational Cost and Runtime… · p. 22
• FUND and FAB incur higher training costs; FUND’s overhead comes from progressive annealing, while FAB repeatedly uses AIS to generate training samples.Source: G Computational Cost and Runtime… · p. 23
• Flow-based methods generate samples in parallel through a single forward pass after training, while MCMC sampling is sequential.Source: G Computational Cost and Runtime… · pp. 22–23
• Compared with FAB, FUND(FKL) has lower training cost while maintaining similarly fast inference-time generation.Source: G Computational Cost and Runtime… · p. 23