Is There a Better Source Distribution than Gaussian? Exploring Source Distributions for Image Flow Matching

1 Introduction

• Although flow matching permits flexible source distributions, the introduction notes that target-aware designs have rarely been applied to complex high-dimensional data and that naive cross-modal replacements can be unstable.This motivates asking what makes departures from Gaussian sources difficult and what properties make a source advantageous.Source: §1 · p. 2
• The paper redesigns 2D experiments to reflect high-dimensional geometry by separating direction and norm, since naive low-dimensional examples do not capture essential high-dimensional learning dynamics.The redesigned simulation supports visualization of vector-field behavior in 2D.Source: §1 · pp. 2–3, Figure 1 · p. 2
• The study compares density approximation, directional alignment, and norm alignment, testing how these source strategies interact with independent and in-batch optimal-transport pairings.The strategies examine target density, target directions, and expected source-target norms, respectively.Source: §1 · p. 3
• The analysis finds that density approximation can worsen performance through mode discrepancy, concentrated directional alignment can entangle paths, and Gaussian sources provide robust omnidirectional coverage.It also identifies sparse-data regions as a source of generation failures and norm mismatch as a substantial learning cost.Source: §1 · p. 3
• The proposed hybrid combines Gaussian-source training with Norm Alignment and inference-time Pruned Sampling, which can also improve pretrained models without retraining.It retains Gaussian training's omnidirectional learning while excluding initializations in data-sparse regions at inference.Source: §1 · p. 3
• The paper proposes 2D simulations intended to reveal high-dimensional flow-matching dynamics, derive source-design guidelines, and motivate a practical sampling technique.The stated goal is to improve existing models without retraining.Source: §1 · p. 3

2.1 Flow Matching

• Flow matching trains a time-varying vector field for an ODE transporting a source distribution to a target, while conditional flow matching makes training tractable through conditional paths and samples.With independent pairing, each source sample is paired with a randomly chosen target sample; training need only sample from the source and target distributions.Source: §2.1 · pp. 3–4

2.2 Optimal Transport CFM (OT-CFM)

• Independent coupling pairs source and target samples randomly and generally costs more than optimal transport, while mini-batch OT selects a lower-cost permutation for each batch.The batchwise alignment is recomputed each iteration, so it may not persist long enough to yield long-term convergence benefits.Source: §2.2 · p. 4

2.3 Alternative Source Distributions

• Prior alternatives include domain-specific target-approximating sources and cross-modal text or token sources, but these approaches remain constrained by low-dimensional domains or substantial alignment engineering.The cited examples include time-series and biomolecular sources, as well as text-to-image embedding and token-space approaches.Source: §2.3 · pp. 4–5

2.4 Von Mises-Fisher (vMF) Distribution

• The von Mises–Fisher distribution models directions on a unit hypersphere, with concentration κ controlling angular concentration and κ = 0 giving a uniform distribution.The passage describes it as suitable for directional data and sampling unit vectors with controlled concentration.Source: §2.4 · p. 5

3 Revisiting Gaussian Source Distributions for Flow Matching

• The standard isotropic Gaussian is widely used for its simplicity and mathematical properties, but its optimality for complex image generation remains an open question.The section uses 2D simulations to examine geometric properties relevant to source-distribution design.Source: §3 · p. 5

3.1 High-Dimensional Geometry and Directional Decomposition

• χ-Sphere decomposition factors Gaussian samples into independent radii and uniform directions while preserving Gaussian statistics; χ-Sphere and Gaussian sampling are interchangeable in training and inference without significant FID differences.The paper therefore treats the two sampling methods as functionally equivalent.Source: §3.1 · p. 5
• Normalized data occupies a bounded norm region, allowing each point to be viewed by its norm and direction and its angular similarity to Gaussian samples measured by cosine similarity.From this perspective, the data distribution occupies a subset of directions on the full hypersphere.Source: §3.1 · p. 6

3.2 Proposed Simulation for High-Dimensional Flow Analysis

• The proposed simulation represents high-dimensional geometry in 2D using random directions, χ(d)-distributed norms, and clustered targets, enabling visual analysis of flow trajectories.Source norms are normalized across source types; successful paths end within one L2 unit of the nearest data point.Source: §3.2 · p. 6
• Normalized Wasserstein, averaged over 10 runs, provides a relative measure of structural similarity within each source-distribution group, with lower values indicating better alignment.The authors caution that this controlled-simulation metric should not be interpreted in absolute terms.Source: §3.2 · p. 6

4.1 Density Approximation Strategy

• Progressively better target-density approximation worsens generation because sparse modes are lost, creating source–target pairing mismatch and inefficient trajectories called mode discrepancy.The approximated source can still miss a sparse mode after 10,000 training iterations.Source: §4.1 · pp. 7–8

4.2 Directional Alignment Strategy

• Directional sources cover data modes, but mini-batch OT falls short of global OT; tighter alignment concentrates support, entangles paths, and destabilizes learning.As γ increases, trajectories become nearly coincident at initialization, sharply increasing the required local Lipschitz constant.Source: §4.2 · pp. 8–9

4.3 Pairing Method Analysis

• I-CFM’s omnidirectional Gaussian coverage supports robust learning, while OT-CFM learns straighter localized paths but undertrains broad directions, leaving some samples stranded.Low-density directions receive less training and supervision, making failures more likely for samples initialized there.Source: §4.3 · pp. 9–10
• On CIFAR10, DCT-based mild refinement improves performance over Gaussian, while stronger target approximations degrade quality in DCT > GMM > CNF order.Source: Table 2 · p. 11
• Training I-CFM with the full χ-Sphere source and pruning low-density initializations at sampling combines broad training coverage with fewer failed trajectories and improved performance.Source: §4.3 · pp. 10–11

5.1 Density Approximation Strategy

• Mild DCT refinement improves on the Gaussian baseline, but stronger density approximations progressively worsen quality, with DCT > GMM > CNF.The authors attribute the trend to information loss in low-density regions, leaving some data modes without suitable source samples and producing inefficient, entangled trajectories.Source: §5.1 · pp. 11–12

5.2 Directional Alignment Strategy

• Preserving target directions while ignoring norms is proposed to avoid the sparse-region information loss associated with matching the full target density.The authors report an FID of 0.29 between normalized and original data and use χ-Sphere decomposition with vMF sampling for directional pairing.Source: §5.2 · p. 12

5.2.1 Oracle Approach

• At κ values of 1500 and 3000, oracle directional pairing generates structurally similar images with varying fine details, while κ approaching infinity trivially reproduces training data with norm scaling.The oracle setup aligns each vMF mean direction with a normalized data point; it requires storing the entire training set, limiting practicality at large scale.Source: §5.2.1 · p. 12

5.2.2 Clustering-based Approach

• Clustering-based pairing assigns each spherical K-means cluster a shared vMF source, approximating oracle pairing with lower storage and computational overhead.With K = 3, κ ∈ {50, 100} improves performance, whereas κ ≥ 300 falls below the Gaussian baseline as concentrated sources cause path entanglement and training instability.Source: §5.2.2 · p. 13

6.1 Pruned Sampling

• Gaussian→Pruned consistently outperforms the other evaluated configurations, while fully pruned training has inconsistent effects and can hurt OT-CFM.The comparison evaluates Gaussian→Gaussian, Pruned→Pruned, and Gaussian→Pruned training and sampling configurations.Source: Table 4 · p. 14, §6.1 · p. 14
• Pruned Sampling uses PCA directions and their opposites to identify remote directions, then rejection-samples Gaussian initializations using a conservative threshold.Low maximum cosine similarity to normalized data marks remote basis directions; the rejection threshold is set above the pruning threshold to reduce support deficiency.Source: §6.1 · pp. 13–14
• Training on the full Gaussian supports a comprehensive vector field, while inference-time pruning avoids data-sparse or poorly learned initializations.The authors connect full angular coverage during training with generalization, and note that sparse-region initializations receive fewer updates and can yield erroneous outputs.Source: §6.1 · p. 14
• Pruned Sampling improves pretrained Gaussian-source models without retraining, with reported gains on ImageNet64 and across OT-CFM and I-CFM at different NFEs.The authors describe inference-time filtering as having minimal computational overhead.Source: §6.1 · pp. 14–15, Table 5 · p. 15

6.2 Norm Alignment Strategy

• FID falls by 0.67 for OT-CFM and 0.72 for I-CFM when Norm Alignment and Pruned Sampling are combined at NFE 100 on CIFAR10.The reported comparison is for the combined method.Source: Table 6 · p. 15, §6.2 · p. 16
• Norm Alignment scales target samples to match the source’s average norm, improving performance overall but potentially hurting at low NFE as paths become more curved.Without alignment, trajectories can move more directly toward lower-norm regions near the data manifold at low NFE.Source: §6.2 · pp. 15–16

7 Conclusion

• The proposed 2D simulation visualizes high-dimensional flow-matching behavior, and its findings show strong agreement with high-dimensional results.The authors describe it as a way to analyze and gain insight into flow matching.Source: §7 · p. 16
• Density approximations can omit low-density modes and cause inefficient pathways, while overly concentrated directional sources can entangle paths and destabilize optimization.The authors attribute Gaussian’s success to omnidirectional coverage that supports vector-field learning around all data modes.Source: §7 · p. 16
• The hybrid framework combines Norm Alignment during training with Pruned Sampling at inference, preserving Gaussian-training robustness while removing problematic initializations.The authors state that Pruned Sampling can be applied to pretrained models without retraining.Source: §7 · p. 16

B Implementation Details and Results for 2D Simulations

• The vector field is an MLP trained with Adam for 20,000 iterations at batch size 16 using TorchCFM.The simulations use PyTorch, time-varying inputs, and fully connected layers.Source: B Implementation Details and Results… · p. 20
• The source uses χ(3072) radii and uniform 2D directions, while three target clusters vary in density and use χ(625) radii.The target clusters have density ratios 0.05, 0.3, and 0.65, with angular perturbations around their base directions.Source: B Implementation Details and Results… · pp. 20–21
• Trajectories are visualized every 5,000 iterations and classified by nearest-target distance, with a 1.0-unit threshold separating successes from failures.The ODE solver uses adaptive Dormand–Prince integration with absolute and relative tolerances of 1 × 10^-4.Source: B Implementation Details and Results… · p. 21
• The simulations assess accuracy (AMD), failure frequency, mode-distribution divergence (MDD), and normalized Wasserstein distance; lower values indicate better performance.Results in the comparison table are averaged over 10 independent runs with standard deviations.Source: Table 7 · p. 20, B Implementation Details and Results… · p. 21

C Derivation of Path-Entanglement Bound

• As γ approaches 1, the local Lipschitz estimate diverges, capturing the path-entanglement effect.The bound identifies the (1 − γ)^1/2 term as dominating the denominator.Source: C Derivation of Path-Entanglement Bound · p. 22

D DCT Refinement

• The DCT masks filter unnecessary high-frequency details from both source and target distributions.Source: D DCT Refinement · p. 22

D.1 DCT Block

• The DCT converts zero-centered 8 × 8 image blocks into frequency coefficients independently across Y, Cb, and Cr channels.The channels represent luminance and two chrominance components.Source: §D.1 · p. 22

D.2 Data dependent DCT mask

• The masks use CIFAR-10 DCT-coefficient statistics across YCbCr channels, with luminance left unmasked to preserve perceptually important brightness information.The weak and strong masks use different standard-deviation thresholds under a shared absolute-mean threshold.Source: §D.2 · p. 22
• Weak DCT masking slightly improves FID over naïve Gaussian, whereas strong masking performs worse than the Gaussian baseline.The results are attributed to suppressing unnecessary high-frequency details, while stronger symmetric masking can reduce target fidelity.Source: §D.2 · pp. 22–23, Table 8 · p. 23

E Explanation on why Norm Alignment Hurts at Low NFE

• Norm alignment places source and target on similar-radius hyperspheres, so finite-batch pairing bias can bend trajectories laterally toward high-density regions.Without norm alignment, transport from larger to smaller radii makes this pairing bias less disruptive to path straightness.Source: E Explanation on why Norm Alignment… · pp. 23–24
• At very low NFE, ODE solvers poorly approximate these curved paths, accumulating integration error and degrading sample quality.Figure 7 visualizes the norm-aligned trajectories and the Gaussian-source comparison.Source: E Explanation on why Norm Alignment… · p. 24, Figure 7 · p. 24

• Experiments evaluate models on CIFAR-10 and ImageNet64 using an improved UNet with attention following the ADM framework.Source: F Experimental Setup of… · p. 24
• CIFAR-10 uses two residual blocks per resolution and attention at 16 × 16, while ImageNet64 uses three blocks and attention at 32 × 32 and 16 × 16.Both models use 128 base channels, four attention heads, Group Normalization, GELU, 0.1 dropout, and convolutional resampling.Source: F Experimental Setup of… · p. 24
• Training uses Adam at 2×10−4; CIFAR-10 runs 200K steps with batch size 512, and ImageNet64 runs 350K steps with batch size 384.The experiments use linear warmup and run on a single NVIDIA RTX A6000 GPU.Source: F Experimental Setup of… · p. 24

G Source Pruning Settings

• The CIFAR-10 pruning thresholds are τ = 0.01 and τr = 0.048; ImageNet64 uses τ = 0.005 and τr = 0.026.Source: G Source Pruning Settings · p. 24
• Pruned sampling consistently outperforms the standard Gaussian baseline in FID, while increasing τr reduces generation time but slightly worsens sample quality.The threshold ablation reports FID, time to generate 512 images, and rejection rates.Source: G Source Pruning Settings · p. 24, Table 9 · p. 25

H Norm Scaling of 2D Simulations

• The 2D simulations scale approximated sources to match the χ-Sphere’s average radius, making transport patterns comparable across source settings.The normalization may reduce apparent source-to-data convergence; without it, the dynamics are harder to interpret, as illustrated by Figure 8.Source: H Norm Scaling of 2D Simulations · p. 25, Figure 8 · p. 25

I Limitations

• The findings primarily derive from CIFAR-10 and ImageNet64 experiments, leaving their generalizability to text, audio, molecular data, and varied image-tokenizer latent spaces open.The authors note that distribution geometry and semantics can differ across these domains.Source: I Limitations · p. 25
• The analysis of source distributions does not establish a comprehensive theoretical foundation for optimal high-dimensional source design.The authors call for rigorous guarantees and theoretical bounds across application domains.Source: I Limitations · pp. 25–26
• Pruning and norm alignment add tunable hyperparameters and rejection-sampling overhead, while their effectiveness depends on NFEs.Norm alignment may degrade performance in very low-NFE regimes, presumably because of increased trajectory curvature.Source: I Limitations · p. 26
• The study focuses on unconditional generation, so the applicability of its findings to class-conditional and text-to-image generation remains unexplored.The authors note that conditioning mechanisms may interact with source-distribution design.Source: I Limitations · p. 26

J Qualitative examples

• With vMF initialization at κ = 1500, generated images remain diverse and visually distinct while resembling their target samples.The figure contrasts each target image with five generated images from vMF(µ(x1), κ = 1500).Source: Figure 9 · p. 26
• The qualitative examples compare source and generated samples for discrete mixtures and for vMF distributions with κ increasing from 10 to 1500.Figure 10 covers DCT-Weak, DCT-Strong, three GMM settings, and FFJORD; Figure 11 compares empirical data with model outputs.Source: Figure 10 · p. 27, Figure 11 · p. 28