Variational Flow Matching for Graph Generation

1 Introduction

• Diffusion models use constrained sampling paths, whereas continuous normalizing flows can learn more general paths but require an ODE solve during each training step.Flow matching offers a simulation-free approach to training continuous normalizing flows.Source: §1 · p. 1
• VFM parameterizes the learned vector field as an expectation under a variational distribution over possible end points.Its objective minimizes the KL divergence between the posterior probability path and the variational approximation.Source: §1 · pp. 1–2
• With a Gaussian variational approximation and a conditional vector field linear in the data, VFM recovers the original flow matching objective.Under linearity, matching the posterior path’s marginals makes the VFM solution exact, allowing a fully factorized approximation without loss of generality.Source: §1 · p. 2
• CatFlow matches or substantially exceeds existing methods across abstract graph and molecular generation tasks.The method trains a classifier over end points on a per-component basis.Source: §1 · p. 2

2.1 Transport Framework for Generative Modeling and CNFs

• Continuous normalizing flows parameterize transformations from a tractable initial density to the target density, but training requires solving the induced ODE to obtain loss gradients.The transformation must balance expressiveness against tractability of its determinant term.Source: §2.1 · p. 2

2.2 Flow Matching

• Flow matching regresses a CNF’s underlying vector field over t ∈ [0, 1] using per-example conditional flows toward specific data points.The conditional vector field and trajectories can be constructed without access to the marginal vector field or probability path.Source: §2.2 · p. 2
• Straight-line continuation toward a data point lets flow matching sample conditional trajectories by interpolating initial and target samples.The conditional flow matching objective is equivalent in expectation, up to a constant, to the flow matching objective.Source: §2.2 · pp. 2–3
• Choosing tractable conditional trajectories, including straight lines, makes flow matching simulation-free and more scalable than likelihood-based CNF training.Diffusion models can also be instantiated as flow matching with a specific conditional trajectory.Source: §2.2 · p. 3

3.1 Flow Matching using Variational Inference

• VFM represents the learned vector field as an expectation over endpoint continuations and minimizes the KL divergence between the posterior probability path and a variational approximation.The posterior probability path is the distribution over possible endpoints of paths passing through the current state.Source: §3.1 · p. 3
• Direct VFM can require approximating a high-dimensional, correlated endpoint posterior and evaluating an expectation-based vector field.Under typical assumptions on the conditional velocity field, the objective simplifies to the computational demands of standard flow matching.Source: §3.1 · pp. 3–4

3.2 Mean-Field Variational Flow Matching

• When the conditional field is linear in endpoints, matching the posterior’s marginal expectations is sufficient, so a fully factorized approximation can recover the same field without modeling covariance.Straight-line interpolations are given as a common case; this mean-field formulation reduces a high-dimensional problem to lower-dimensional ones.Source: §3.2 · p. 4

3.3 CatFlow: Mean-Field Variational Flow Matching for Categorical Data

• For categorical data, CatFlow uses endpoint-class distributions to parameterize trajectories toward simplex points and trains with a standard cross-entropy loss.The passage describes the objective as no less efficient to train than flow matching.Source: §3.3 · pp. 5–6, Figure 1 · p. 5
• CatFlow’s simplex-directed paths provide an inductive bias, while cross-entropy offers different gradient behavior from mean-squared error; the paper evaluates their learning-dynamics and convergence effects.The authors state that both aspects enhance learning dynamics and speed convergence.Source: §3.3 · p. 6
• CatFlow generates exchangeable graph distributions, assigning equal probability to all permutations of a graph under the stated natural conditions.The model’s probability-vector predictions also express uncertainty over variables, which can decrease as molecular structure becomes established.Source: §3.3 · p. 6

Relationship to Flow Matching

• Under a Gaussian variational approximation and a conditional vector field linear in the endpoint, VFM reduces to standard flow matching.Under linearity, the solution is exact when the approximation matches the posterior-path marginals, allowing a fully factorized approximation.Source: §4 · p. 6

Relationship to Score-Based Models

• The score function is an expectation of the conditional score under the posterior endpoint distribution, and VFM parameterizes this expectation with a variational approximation.The paper relates this parameterization to continuous diffusion for categorical data.Source: §4 · pp. 6–7
• A reweighted VFM objective bounds model log-likelihood, while the variational approximation can define both deterministic and stochastic dynamics.The stochastic dynamics are expressed through an SDE with scalar noise function g_t and Wiener process w.Source: §4 · p. 7

Diffusion Models for Discrete Data

• Graph diffusion methods include progressive graph edits trained to reverse noise [58] and an SDE-based score model [20]; these non-autoregressive approaches perform on par with autoregressive methods [31] [26].Other non-autoregressive approaches include [28] [34] [33].Source: §5 · p. 7
• Continuous diffusion for categorical sequences [11] uses an expected conditional score and cross-entropy, but differs from CatFlow in its embedding-space flow and time-warped objective [22].The time warping is optimized so prediction entropy decreases linearly from noise to data in uniform time.Source: §5 · pp. 7–8

Flow-based methods for Discrete Data

• Dirichlet flows [56] represent conditional paths with simplex-constrained Dirichlet distributions, while Discrete Flow Models [6] generate through sequential sampling from time-dependent categorical distributions.Neither approach had been evaluated on graph generation; the authors report preliminary experiments with released Dirichlet Flow code.Source: §5 · p. 8
• CatFlow remains a continuous model that integrates during generation rather than iteratively sampling, despite framing dynamics as endpoint classification [58].Source: §5 · p. 8

6 Experiments

• CatFlow is evaluated on an abstract graph-generation task for topological properties and on QM9 and ZINC250k for molecular semantics, treating graphs as categorical/discrete rather than geometric objects embedded in Euclidean space.For QM9 and ZINC250k, molecules are kekulized using RDKit and hydrogen atoms are removed; 10,000 molecules are evaluated for validity, uniqueness, and Fréchet ChemNet Distance, with validity computed without valency correction or edge resampling.Source: §6 · p. 8, §6.2 · p. 9

6.1 Abstract graph generation

• On abstract graph generation, CatFlow captures graph topology and performs well on Ego-small and Community-small, evaluated with MMDs for degree, clustering, and four-node orbit counts.The reported results are summarized in Table 1.Source: §6.1 · p. 9

6.2 Molecular Generation: QM9 & ZINC250k

• CatFlow obtains state-of-the-art, virtually perfect performance on QM9 and ZINC250k, converges faster than flow matching, and is not more computationally expensive than the baselines.The molecular results are summarized in Table 2.Source: Table 2 · p. 9, §6.2 · p. 10

6.3 CatFlow Ablations

• CatFlow outperforms standard flow matching in the large-model, full-data setting and is more robust as model size or training data decreases.The ablation score is the percentage of generated molecules that are valid and unique; the authors also report faster convergence, with curves not shown.Source: §6.3 · p. 10, Figure 3 · p. 10

Conclusion

• VFM is presented as a general formulation, while CatFlow achieves strong performance on graph generation tasks.The authors identify possible future applications of VFM to text, source code, and mixed discrete-continuous modalities, and see its score-based-model connections as a path toward deterministic and stochastic dynamics.Source: §7 · p. 10

Limitations

• VFM has so far been applied only to categorical graph generation, and CatFlow’s quadratic edge reasoning does not scale well to proteins with 10^4 or more atoms.Source: §7 · p. 10

Ethics Statement

• The authors note potential misuse in creating harmful substances and foresee no direct benefits or risks at the work’s current technology-readiness level.Source: §7 · p. 10

A.1 Derivation of the Variational Flow Matching Objective

• VFM optimization is equivalent to minimizing the KL divergence between the true endpoint distribution and its variational approximation.The derivation drops the parameter-independent entropy term, leaving negative cross-entropy as the optimization target.Source: §A.1 · p. 15

A.2 Flow Matching as a Special Case of Variational Flow Matching

• Under the stated linearity, invertibility, and covariance assumptions, VFM reduces to flow matching.Source: §A.2 · p. 15

A.3 Decomposition of the Flow

• When the conditional vector field is linear in the endpoint, matching endpoint marginals preserves the expected conditional vector field.Source: §A.3 · pp. 15–16

A.4 CatFlow

• For exchangeable priors and permutation-equivariant μ_t, CatFlow generates all permutations of a graph with equal probability.The appendix states that permutation equivariance of μ_t implies the corresponding property for v_t.Source: §A.4 · p. 16

A.5 Derivation of the Score Function

• The appendix sets out to express the score function ∇x log p_t(x) in terms of the posterior probability path p_t(x | x_1).Source: §A.5 · p. 17

A.6 Stochastic Dynamics with Variational Flow Matching

• VFM approximates stochastic dynamics by replacing the endpoint distribution in the drift with its variational approximation; deterministic dynamics arise when g_t ≡ 0.When the endpoint distributions match, the processes coincide for any g_t.Source: §A.6 · p. 18

A.7 Variational Flow Matching as a Variational Bound on the Log-likelihood

• A reweighted integral of the point-wise VFM objective provides an upper bound on model likelihood.The derivation separates the parameter-dependent likelihood from a parameter-independent entropy term and uses a non-negative weighting function.Source: §A.7 · pp. 18–19

A.8 Stochastic Dynamics under Linearity Conditions

• Under linearity of the conditional vector field and score, endpoint distributions with matching marginals preserve the drift and the connections to stochastic dynamics and model likelihood.Source: §A.8 · p. 19

B.1 General Variational Flow Matching

• General VFM training samples data and prior endpoints, interpolates between them, and minimizes negative log probability under the variational endpoint distribution.Source: §B.1 · p. 20

B.2 Categorical Variational Flow Matching (CatFlow)

• CatFlow trains its learnable categorical component distributions with a cross-entropy loss on the sampled endpoint components.Source: §B.2 · p. 20

B.3 Gaussian Variational Flow Matching

• Gaussian VFM trains its learnable mean using a mean-squared-error objective on sampled interpolated data.Source: §B.3 · p. 20

• CatFlow results include standard deviations computed across multiple seeds.Source: §C.1 · p. 21
• The listed abstract graph tasks are Ego-small and Community-small.Source: §C.1 · p. 21
• The listed metrics include degree, clustering, and orbit, plus validity, uniqueness, and FCD.Source: §C.1 · p. 21

• CatFlow achieves state-of-the-art performance on all Stochastic Block Model and Planar Graph tasks and metrics.Source: §C.2 · p. 21
• CatFlow trains significantly faster than Dirichlet flow matching because its forward process is computationally cheaper.Source: §C.2 · p. 21

D.1 Model

• The graph transformer takes a graph as input and predicts a distribution over clean graphs, using structural and spectral features to improve expressivity.Its layers combine transformed node and edge features, FiLM modulation, self-attention, and PNA pooling with global features.Source: §D.1 · p. 22

D.2 Hyperparameters and Computational Costs

• Models were trained until convergence using the same data splits as [20] and hidden dimensions as [58], with experiments run on one NVIDIA RTX 6000 for about a day.The hyperparameter setup is summarized in Table 3.Source: §D.2 · p. 22, Table 3 · p. 23

E Detail CNFs

• Directly optimizing the empirical divergence for a CNF requires solving the ODE at each training step to obtain gradient samples, making it computationally prohibitive.The appendix describes the continuity equation as a way to assess whether a vector field generates a probability path.Source: E Detail CNFs · p. 23

NeurIPS Paper Checklist

• The checklist reports that model descriptions and hyperparameters are provided in appendix D, code and public data will be provided, and error bars are documented in appendix C.The authors also direct readers to appendix D for compute-resource information.Source: NeurIPS Paper Checklist · pp. 25–27
• The authors report no new assets, use previously existing code and assets for experiments, and state that relevant assets are cited and their licenses respected.They mark new-asset documentation as not applicable because the paper provides a new objective and uses previous code/assets.Source: NeurIPS Paper Checklist · pp. 28–29