PaperlayerView exact revision on arXiv

Source-linked AI summary

Particle Guidance: non-I.I.D. Diverse Sampling with Diffusion Models

Gabriele Corso, Yilun Xu, Valentin de Bortoli, Regina Barzilay, Tommi Jaakkola

arXiv:2310.13102v2cs.LGcs.AI

TL;DR

Finite independent samples may represent a target distribution inefficiently, motivating a method that improves diversity without requiring independent generation. Particle guidance adds a time-evolving joint potential to diffusion sampling and analyzes how to control the resulting joint distribution. Across conditional image and molecular conformer generation, it improves diversity or accuracy while preserving quality or improving precision and coverage.

  • Problem

    Finite I.I.D. samples may not efficiently represent a distribution, creating a need for more sample-efficient diverse generation.

  • Method

    Particle guidance jointly samples diffusion-model particles with a time-evolving potential, using fixed or learned potentials to control diversity and joint-distribution properties.

  • Results

    Particle guidance increases Stable Diffusion diversity at comparable quality and reduces Torsional Diffusion median precision and coverage errors by 19% and 8%, respectively.

  • Takeaways & Limitations

    The framework provides diffusion-based finite-sample generation with controllable diversity and theoretical options for optimality and marginal preservation.

  • Takeaways & Limitations

    The maximum-entropy technique does not generally preserve individual-particle marginals, although learned potentials can address this property when needed.

Abstract

from arXiv · show

In light of the widespread success of generative models, a significant amount of research has gone into speeding up their sampling time. However, generative models are often sampled multiple times to obtain a diverse set incurring a cost that is orthogonal to sampling time. We tackle the question of how to improve diversity and sample efficiency by moving beyond the common assumption of independent samples. We propose particle guidance, an extension of diffusion-based generative sampling where a joint-particle time-evolving potential enforces diversity. We analyze theoretically the joint distribution that particle guidance generates, how to learn a potential that achieves optimal diversity, and the connections with methods in other disciplines. Empirically, we test the framework both in the setting of conditional image generation, where we are able to increase diversity without affecting quality, and molecular conformer generation, where we reduce the state-of-the-art median error by 13% on average.

1 INTRODUCTION

The paper addresses the inefficiency of using independent samples to represent a distribution with a finite set by introducing particle guidance for diffusion models. It develops theoretical guarantees and evaluates diversity improvements in image and molecular generation.

  • Particle guidance uses a joint, time-evolving potential to produce finite sample sets that better represent target distributions than I.I.D. sampling.The framework extends diffusion sampling beyond independent particles by enforcing diversity across the sampled set.
  • The framework includes fixed potentials with little inference overhead and learned potentials that provide greater control and theoretical guarantees.Fixed potentials require no additional training, whereas learned potentials require training but can target desired joint distributions.
  • The theory derives the joint marginal distribution for arbitrary guidance potentials and an objective for learning potentials that achieve optimal diversity and can preserve marginals.The framework also connects particle guidance to non-I.I.D. sampling methods from other fields.
  • Particle guidance increases Stable Diffusion sample diversity while maintaining quality comparable to I.I.D. sampling.The image-generation evaluation uses Stable Diffusion and compares diversity and quality under particle guidance and independent sampling.
  • For molecular conformer generation, particle guidance reduces median precision error by 19% and coverage error by 8% on Torsional Diffusion.The method simultaneously improves both precision and coverage for this application.

2 BACKGROUND

Diffusion models reverse a learned forward noising process to generate samples efficiently across complex distributions. The paper builds on conditional guidance while targeting a separate challenge: improving finite-sample diversity without sacrificing diffusion models’ finite-time sampling advantages.

  • Diffusion models learn a reverse stochastic process whose score estimates the evolving data density during denoising.The forward process adds noise, while the reverse SDE uses the learned score to recover samples.
  • Their finite-time sampling property helps overcome energy barriers and reach different modes more efficiently than Langevin dynamics under comparable discretization error.The paper distinguishes this per-sample efficiency from the separate cost of generating enough samples to cover a distribution.
  • Diffusion models have been extended to Riemannian manifolds, supporting applications where scientific data lie near predefined submanifolds.This formulation has been used in several scientific domains.
  • Classifier guidance adds a classifier-derived gradient to the reverse diffusion process for conditional generation.The guidance uses a model of pθ(y|x_t,t) to steer samples toward a desired class.
  • Increasing classifier-guidance strength beyond its theoretical value can collapse generation onto a few modes and reduce diversity.The paper presents this as a motivation for developing a different guidance mechanism.

3 PARTICLE GUIDANCE

Particle guidance modifies reverse diffusion for a whole set of particles by adding a permutation-invariant joint potential. This enables properties such as diversity without retraining a score model over sets, through fixed or learned potential variants.

  • Particle guidance adds the gradient of a permutation-invariant potential defined over all particles to the reverse diffusion process.The particles are sampled jointly rather than independently from the learned score dynamics.
  • The method can impose set-level properties such as diversity without retraining a score model that operates directly on particle sets.Particles are initially sampled I.I.D. from a prior distribution before joint guidance is applied.
  • Fixed Potential PG uses handcrafted time-evolving potentials for efficient sampling without additional training.This variant is designed for practical use with little extra sampling cost.
  • Learned Potential PG trains the time-evolving joint potential to obtain provably optimal joint distributions and control properties such as marginal preservation.This variant provides stronger theoretical control than the fixed-potential approach.

4 CONNECTIONS WITH EXISTING METHODS

Particle guidance is related to existing non-I.I.D. sampling methods through shared use of interactions or potentials, while retaining diffusion models’ finite-time sampling property. The connections span coupled replicas and metadynamics, SVGD, and electrostatic generative models.

  • 4.1 COUPLED REPLICAS AND METADYNAMICS: Coupled replicas create dependencies among parallel samples, whereas metadynamics sequentially biases new samples using previously sampled structures.Both approaches improve coverage by departing from independent sampling, but they differ in whether samples are generated in parallel or sequentially.
  • 4.1 COUPLED REPLICAS AND METADYNAMICS: Coupled replicas and metadynamics control joint sampling through added potentials, but their energy-based MCMC procedures can struggle with complex distributions and require many steps.Particle guidance aims to combine controllable diversity with finite-time diffusion sampling using steps independent of data dimension.
  • 4.2 SVGD: Particle guidance and SVGD share particle-based updates involving target-distribution scores and interactions among particles, but they smooth scores differently.Particle guidance uses the diffused score at each particle, while SVGD averages nearby-particle scores with a similarity kernel.
  • 4.2 SVGD: SVGD lacks diffusion’s finite-time guarantee, can collapse near initialization without discovering isolated modes, and may produce blurry image samples.The paper notes proposed remedies including annealing schedules and noise-conditioned kernels.
  • 4.3 ELECTROSTATICS: Particle guidance can be interpreted as combining Poisson Flow Generative Models with repelling point charges in a data-defined electric field.Under this connection, the similarity kernel extends the Green’s function in N+1 dimensions as k(x,y) ∝ 1/||x−y||^(N−1).

5 FIXED POTENTIAL PARTICLE GUIDANCE

Fixed potential particle guidance uses pairwise similarity potentials to diversify diffusion samples with little inference overhead, while its theory characterizes the resulting joint distribution and invariant constructions.

  • 5 FIXED POTENTIAL PARTICLE GUIDANCE: Pairwise similarity kernels make samples dissimilar while retaining the diffusion score's distribution-matching role, without requiring retraining or an additional classifier.The kernel cost is typically small relative to the score network, and probability-flow ODEs can replace reverse-time SDEs for faster generation.
  • 5.1 THEORETICAL ANALYSIS: For arbitrary time-evolving potentials, the marginal sequence may not define a diffusion process, so exact re-equilibration with Langevin steps would substantially increase runtime and is exact only in the infinite-step limit.This limitation motivates the paper's analytical reweighting treatment.
  • 5.1 THEORETICAL ANALYSIS: Theoretical analysis shows particle guidance reweights the joint distribution of I.I.D. samples according to the time-evolving potential, rather than generally sampling the intended marginal sequence exactly.The result is derived with the Feynman-Kac theorem under integrability assumptions.
  • 5.1 THEORETICAL ANALYSIS: The framework's theoretical insights extend to Riemannian manifolds because the Feynman-Kac theorem also extends to manifold settings.This is stated as a direct consequence of the corresponding manifold extension.
  • 5.2 PRESERVING INVARIANCES: Particle guidance preserves group invariance when its time-evolving potential is invariant to transformations of each set element.This condition applies to groups such as rotations and permutations.

6 EXPERIMENTS

Experiments apply particle guidance to text-to-image and molecular conformer generation to improve the diversity–quality or coverage–precision trade-off without retraining the underlying diffusion models.

  • 6.1 TEXT-TO-IMAGE GENERATION: The experiments use Stable Diffusion v1.5 at 512×512 resolution, evaluating text-image alignment, visual quality, and within-batch diversity with CLIP, Aesthetic, and DINO-based metrics.The diversity metric is average pairwise cosine similarity of DINO features within each generated image batch.
  • 6.1 TEXT-TO-IMAGE GENERATION: At the same CLIP or Aesthetic score, particle guidance generally achieves lower in-batch similarity than I.I.D. sampling in most cases across classifier-free guidance scales from 6 to 10.With a DINO feature-space similarity kernel, particle guidance attains lower in-batch similarity than I.I.D. sampling and the original downsampled pixel-space approach.
  • 6.1 TEXT-TO-IMAGE GENERATION: Particle guidance improves visual diversity in four-image text-to-image batches and can avoid repeated training-data replicas that appear under I.I.D. sampling.The comparison fixes the random seeds and classifier-free guidance scale at 9; SVGD instead produces blurry images in the reported extended samples.
  • 6.2 MOLECULAR CONFORMER GENERATION: Molecular conformer generation requires broad coverage and precision because different low-energy conformers can produce different behaviors, but existing methods commonly generate over 200 I.I.D. samples.Particle guidance is applied to torsional diffusion on the torsion-angle manifold, using an RBF kernel over dihedral-angle differences and permutation-invariant construction.
  • 6.2 MOLECULAR CONFORMER GENERATION: 8% and 19% simultaneous reductions in recall and precision median AMR are obtained on GEOM-DRUGS by applying particle guidance to torsional diffusion without retraining.Table 1 reports improved results on both coverage and precision-related metrics.

7 LEARNED POTENTIAL PARTICLE GUIDANCE

Learned potential particle guidance targets desired joint distributions by training a time-evolving potential, with extensions that can preserve individual-particle marginals. The framework is motivated by the intractability of fixed-potential distributions and the need for controlled sampling in scientific applications.

  • 7 LEARNED POTENTIAL PARTICLE GUIDANCE: Learned potentials address the intractability of fixed-potential joint and marginal distributions while enabling control over the sampled distribution.These controls matter for applications requiring correct importance weights or free-energy differences.
  • 7 LEARNED POTENTIAL PARTICLE GUIDANCE: Maximum-entropy particle guidance learns a time-evolving potential whose joint distribution satisfies a diversity constraint while minimizing divergence from independent sampling.The potential is trained to generate the target joint distribution, with β determined by the diversity-bound value.
  • 7.1 TRAINING PROCEDURE: The training procedure uses diffusion perturbation kernels and independently sampled data points, allowing potential learning to run in parallel with score-model training.The procedure enforces the desired sequence of marginals under the particle-guided generation process.
  • 7.2 PRESERVING MARGINAL DISTRIBUTIONS: For arbitrary diversity potentials, maximum-entropy learning does not preserve each particle’s marginal distribution, so marginal recovery may require a separate construction.This limitation is important when unbiased estimates or other marginal-dependent quantities are needed.
  • 7.2 PRESERVING MARGINAL DISTRIBUTIONS: A learned scalar function γθ can counterbalance a joint diversity potential, preserving marginals while retaining its effect on sample diversity.The approach first learns γθ under a marginal-preserving objective, then uses it in the subsequent potential-training procedure.

8 CONCLUSION

The paper frames particle guidance as diffusion sampling with a time-evolving joint potential that improves finite-sample diversity and can preserve marginal distributions when required. Its theory characterizes the resulting process and its learned-potential variants, while the conclusion reports improved diversity–quality trade-offs across image and molecular applications.

  • 8 CONCLUSION: Particle guidance steers diffusion sampling toward diverse sample sets by defining a time-evolving joint potential, with experiments showing improved diversity–quality trade-offs.The reported applications are text-to-image generation and molecular conformer generation.
  • A.1 JOINT DISTRIBUTION UNDER PARTICLE GUIDANCE: The guided diffusion dynamics add the gradient of the joint potential to the ordinary reverse-diffusion score, changing the drift of the particle process.The unguided and guided stochastic differential equations differ by this potential-gradient term.
  • A.2 SAMPLING A PREDEFINED JOINT DISTRIBUTION: Doob h-transform analysis provides a route to sampling a predefined joint distribution by constructing a time-dependent potential satisfying a backward Kolmogorov equation.The resulting process has drift involving both the original score and the potential gradient, but computing the potential at every time is difficult.
  • A.2 SAMPLING A PREDEFINED JOINT DISTRIBUTION: The potential-learning objective can be parameterized and trained against the original generative process, although the direct loss requires sampling and integration over that process.This requirement motivates alternative training implementations.
  • A.3 PRESERVING MARGINAL DISTRIBUTIONS: Marginal-preserving potentials are difficult to choose for nontrivial potentials and distributions, motivating a learned scalar correction that counterbalances the diversity potential.The correction is designed to preserve individual marginals while maintaining the potential’s effect on joint diversity.
  • A.3 PRESERVING MARGINAL DISTRIBUTIONS: The marginal-preserving update uses costly Monte Carlo expectations over n −1 independent samples, so the implementation adopts a greedy single-sample update related to iterative proportional fitting.The stochastic-gradient update is described as an unbiased estimator of the corresponding objective gradient.

A.3.1 EMPIRICAL SYNTHETIC EXPERIMENTS

Synthetic and theoretical analyses show that particle guidance can increase finite-sample diversity, preserve symmetries, and characterize when independent sampling misses modes, while introducing computational and tuning trade-offs.

  • A.3.1 EMPIRICAL SYNTHETIC EXPERIMENTS: Learning a marginal-preserving potential retains closely matching marginals while providing more diverse particle sets than I.I.D. sampling, though less diversity than an unconstrained potential.Without marginal preservation, diversity increases but the central mode falls from 40% to 13% and outer-mode shapes change.
  • A.4 INVARIANCE OF PARTICLE GUIDANCE: Particle guidance preserves rotation and permutation invariance when the score and drift are equivariant and the potential is invariant to transformations of particle inputs.The proposition is stated for rotations or permutations and extends in principle to broader Lie-group settings.
  • A.5 PARTICLE GUIDANCE AS SVGD: Particle guidance connects to SVGD through a kernel-based potential whose gradient produces particle interactions, while retaining diffusion-model sampling structure.The derivation uses symmetric distance-based kernels and an approximation of average particle distances.
  • A.6 PARTICLE GUIDANCE IN POISSON FLOW GENERATIVE MODELS: In PFGM++, particle guidance adds mutual repulsive forces among particles to the pretrained model’s attractive data force in the augmented space.The guidance direction corresponds to gradients of an N + D-dimensional Green’s function.
  • A.7 COMBINATORIAL ANALYSIS OF SYNTHETIC EXPERIMENTS: 10 samples recover about 6.5 equiprobable modes under I.I.D. sampling, while discovering all modes requires more than 29 samples on average.The expected discovered fraction is 1 − (N−1/N)^N, tending to 1 − 1/e; the expected full-coverage cost is N H_N = Θ(N log N).
  • B.2 RUNTIME AND MEMORY OVERHEAD: Kernel computation scales quadratically with particle count, and excessive guidance weights can alter marginals or create aliasing artifacts.GPU execution may require transfers between RAM and GPU memory when the particle set exceeds the score-model batch capacity.

C SYNTHETIC EXPERIMENTS

A two-dimensional Gaussian-mixture experiment demonstrates that particle guidance improves mode coverage over I.I.D. sampling, especially when the diversity kernel matches the distribution’s radial structure.

  • C SYNTHETIC EXPERIMENTS: The experiment uses 10 equally spaced Gaussian modes on the unit circle, with an oracle score and approximately disjoint equal-probability bins.Each Gaussian has variance 0.005, making the setup a direct test of mode coverage.
  • C SYNTHETIC EXPERIMENTS: Nearly 9 of 10 modes are discovered on average with particle guidance, compared with 6.5 modes for I.I.D. sampling.The radial kernel consistently discovers all modes, while increasing Euclidean-kernel potential weight can cause instability.
  • C SYNTHETIC EXPERIMENTS: A lower-dimensional angular RBF kernel improves coverage by encoding the radial degrees of freedom where diversity lies.This kernel design is motivated by the distribution’s geometry and parallels kernels defined on collective variables in metadynamics.

D.1 DATASET, METRICS AND BASELINES

The conformer-generation evaluation uses GEOM-DRUGS benchmarks and applies particle guidance to Torsional Diffusion with a kernel defined on the torsional manifold and invariant to molecular symmetries.

  • D.1 DATASET, METRICS AND BASELINES: GEOM-DRUGS contains 304k drug-like molecules, with conformers generated by CREST metadynamics at an average cost of 90 core-hours per molecule.Evaluation follows established splits and fine-tunes inference parameters on 200 validation molecules without retraining the score model.
  • D.1 DATASET, METRICS AND BASELINES: Conformer quality is evaluated using Coverage for recall and Average Minimum RMSD for precision, with methods generating twice the reference ensemble size.The coverage threshold for GEOM-DRUGS is 0.75 Å.
  • D.2 PARTICLE GUIDANCE SETUP: Particle guidance is applied to state-of-the-art Torsional Diffusion using a kernel on the implicit hypertorus manifold, preserving the diffusion process’s lower-dimensional structure.Inference uses batches of up to 128 conformers when a molecule requires more samples.
  • D.2 PARTICLE GUIDANCE SETUP: The conformer kernel is invariant to SE(3) transformations and is made invariant to graph-preserving atom permutations by minimizing across isomorphic torsion assignments.When more than 32 relevant isomorphisms exist, the implementation subsamples them to limit overhead.
  • D.3 FULL RESULTS: The full-results evaluation compares invariant and non-invariant particle guidance with low-temperature sampling and established cheminformatics and machine-learning baselines.Low-temperature sampling concentrates on high-likelihood modes while trading sample diversity for quality.

E.1 SETUP

The image-generation setup evaluates particle guidance on COCO prompts using classifier-free guidance, feature- or pixel-space RBF kernels, and an in-batch similarity metric for diversity.

  • E.1 SETUP: Table 3 evaluates conformer ensembles using Coverage (%) and Average Minimum RMSD (Å), with separate validation choices for minimizing recall- and precision-oriented AMR.The table’s metrics distinguish set coverage from pose accuracy.
  • E.1 SETUP: Classifier-free guidance replaces the original score with a weighted combination of conditional and unconditional scores before adding particle guidance.The experiments use an Euler solver with 30 function evaluations.
  • E.1 SETUP: RBF kernels are computed either in downsampled pixel latent space or DINO feature space using the predicted clean image x0.The feature-space formulation is motivated as a more semantically meaningful distance than pixel-space Euclidean distance.
  • E.1 SETUP: Particle guidance achieves lower in-batch similarity than I.I.D. sampling, indicating better diversity under the evaluated Stable Diffusion setup.The comparison uses 500 COCO validation prompts with batches of four images and adjusts I.I.D. guidance weights to favor diversity.
  • E.2 IN-BATCH SIMILARITY SCORE: The in-batch similarity score averages pairwise cosine similarity between DINO features of images generated for the same prompt.Lower similarity corresponds to greater within-batch diversity.

F EXTENDED IMAGE SAMPLES

The extended samples compare I.I.D. sampling, particle guidance in pixel and DINO feature spaces, and SVGD-guidance across several text prompts. The examples illustrate differences in sample diversity and show that increasing SVGD guidance can produce blurry images.

  • The visual comparison includes I.I.D. sampling, particle guidance in pixel space, particle guidance in DINO feature space, and SVGD-guidance.The figures cover prompts including Captain Marvel, a tiger portrait, a Van Gogh cafe terrace, a glass duck, and a unicorn in a snowy forest.
  • The displayed examples use four prompts, including prompts previously associated with Stable Diffusion reproducing content directly from the LAION dataset.The comparison therefore includes cases selected from the prompts studied by Somepalli et al. [2023].
  • SVGD-guidance generally yields blurrier images as the guidance scale αt increases.Its guidance term averages scores from nearby samples, steering samples toward their mean.
  • Particle guidance produces visibly varied generations across prompts, with examples spanning different depictions of the same requested content.For a COCO prompt about a baby eating cake, I.I.D. sampling tends toward a brown-haired white child, whereas particle guidance produces babies with varying hair and skin colors.
Loading 2310.13102v2…