PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable Velocity: A Variance Perspective on Flow Matching

Donglin Yang, Yongxing Zhang, Xin Yu, Liang Hou, Xin Tao, Pengfei Wan, Xiaojuan Qi, Renjie Liao

arXiv:2602.05435v2cs.CV

TL;DR

Single-sample conditional velocity targets in flow matching can be high variance, slowing and destabilizing optimization. The paper characterizes two variance regimes and introduces StableVM, VA-REPA, and StableVS to improve training and sampling. Experiments report consistent training improvements and more than 2× faster low-variance-regime sampling without degrading sample quality.

  • Problem

    Single-sample conditional velocity targets can have high variance near the prior, slowing convergence and mismatching population optimization dynamics.

  • Method

    Stable Velocity combines StableVM for unbiased multi-sample variance reduction, VA-REPA for adaptive auxiliary supervision, and StableVS for finetuning-free low-variance-regime acceleration.

  • Results

    StableVM and VA-REPA consistently outperform REPA across training settings, while StableVS provides more than 2× inference acceleration without perceptible sample-quality degradation.

  • Takeaways & Limitations

    The two-regime variance perspective supports stable supervision near the data distribution and large-step sampling for recent flow-based models.

  • Takeaways & Limitations

    The work acknowledges potential misuse risks associated with image and video generation technologies and emphasizes responsible deployment and safeguards.

Abstract

from arXiv · show

While flow matching is elegant, its reliance on single-sample conditional velocities leads to high-variance training targets that destabilize optimization and slow convergence. By explicitly characterizing this variance, we identify 1) a high-variance regime near the prior, where optimization is challenging, and 2) a low-variance regime near the data distribution, where conditional and marginal velocities nearly coincide. Leveraging this insight, we propose Stable Velocity, a unified framework that improves both training and sampling. For training, we introduce Stable Velocity Matching (StableVM), an unbiased variance-reduction objective, along with Variance-Aware Representation Alignment (VA-REPA), which adaptively strengthen auxiliary supervision in the low-variance regime. For inference, we show that dynamics in the low-variance regime admit closed-form simplifications, enabling Stable Velocity Sampling (StableVS), a finetuning-free acceleration. Extensive experiments on ImageNet $256\times256$ and large pretrained text-to-image and text-to-video models, including SD3.5, Flux, Qwen-Image, and Wan2.2, demonstrate consistent improvements in training efficiency and more than $2\times$ faster sampling within the low-variance regime without degrading sample quality. Our code is available at https://github.com/linYDTHU/StableVelocity.

1. Introduction

Flow matching offers scalable probability-flow learning, but single-sample conditional velocity targets can be high variance and destabilize optimization. Stable Velocity addresses this with variance-reduced training, adaptive auxiliary supervision, and finetuning-free sampling acceleration.

  • Conditional Flow Matching learns probability-flow fields without explicitly simulating the forward SDE or PF-ODE, combining theoretical guarantees with practical scalability.
  • High-variance conditional velocity targets arise particularly near the prior, slowing convergence and creating a mismatch with population optimization dynamics.
  • Stable Velocity Matching provides unbiased variance reduction, while VA-REPA adaptively strengthens auxiliary supervision in the low-variance regime.
  • Stable Velocity Sampling uses low-variance dynamics to accelerate inference without finetuning.
  • StableVM and VA-REPA consistently outperform REPA-based alternatives, while StableVS achieves more than 2× acceleration without perceptible sample-quality degradation.

2. Variance Analysis of Flow Matching

The variance analysis characterizes conditional flow-matching targets across distributions and dimensionalities, revealing low- and high-variance regimes. These regimes motivate variance reduction during training and stronger supervision or faster sampling where conditional and marginal velocities align.

  • Conditional Flow Matching trains a neural velocity field using conditional velocity targets, whose objective has the true marginal velocity field as its minimizer.
  • VCFM(t) stays near zero at small t and rises rapidly, separating low variance for 0 ≤ t < ξ from high variance for ξ ≤ t ≤ 1.
  • As dimensionality increases, ξ shifts toward 1, enlarging the low-variance regime while increasing overall variance magnitude.
  • In the low-variance regime, the posterior concentrates on one reference sample; in the high-variance regime, it spreads across multiple samples.
  • The analysis motivates reducing high-regime training variance without changing the global minimizer and exploiting the low-variance regime for supervision and faster sampling.

3. Variance-Driven Optimization of Training and Sampling

The section develops variance-driven methods for training and sampling flow-matching models. StableVM reduces training-target variance while preserving unbiasedness, VA-REPA activates representation alignment where supervision is informative, and StableVS exploits low-variance dynamics for larger-step sampling.

  • Stable Velocity Matching: StableVM replaces the single-sample conditional velocity with a multi-sample self-normalized aggregation, reducing training variance while preserving CFM’s global minimizer.The construction uses a composite conditional path over reference samples and yields an unbiased target.
  • Stable Velocity Matching: StableVM’s target variance is strictly lower than CFM’s under the stated affine-field assumption and decays at rate O(1/n) with the number of reference samples.The strict reduction requires that the reference samples are not all identical.
  • Variance-Aware Representation Alignment: Representation alignment is informative in the low-variance regime because the corrupted input retains semantic information, but becomes ill-posed near pure noise.The alignment loss remains low at early timesteps and high in the high-variance regime.
  • Variance-Aware Representation Alignment: VA-REPA applies semantic alignment selectively in the low-variance regime, using a weighting function rather than enforcing alignment uniformly across timesteps.The framework supports hard-threshold, sigmoid, and SNR-based weighting schemes, with normalization to preserve effective gradient scale.
  • Stable Velocity Sampling: StableVS uses negligible conditional variance to approximate marginal dynamics with conditional velocity, enabling finetuning-free stable large-step integration.In this regime, the probability-flow trajectory becomes a straight line with constant velocity, so Euler steps can integrate it exactly at arbitrary size.

4. Experiments

The experiments evaluate StableVM, VA-REPA, and StableVS across training settings, model variants, and pretrained image/video generators. Results show consistent training gains and reduced sampling cost without degrading quality.

  • Training evaluation: StableVM and VA-REPA consistently improve FID, IS, precision, and recall over vanilla REPA across model scales and training checkpoints.The comparison is conducted without classifier-free guidance across multiple SiT architectures and checkpoints.
  • Training evaluation: StableVM and VA-REPA provide reliable performance gains when integrated with vanilla REPA, REG, and iREPA at 100k iterations.The methods are presented as drop-in additions that do not modify the underlying REPA-style pipelines.
  • Ablations: Soft VA-REPA weighting outperforms hard thresholding, while K = 256 is sufficient for stable variance reduction and K = 1024 yields only marginal gains.The weighting and memory-bank ablation is evaluated at 100k iterations against the REPA baseline.
  • Sampling evaluation: StableVS with 9 low-variance steps consistently matches or exceeds 30-step baselines across SD3.5, Flux, Qwen-Image-2512, and Wan2.2.The comparison spans image and video generation and multiple solvers.
  • Sampling evaluation: A larger StableVS split point, such as ξ = 0.85, enables more aggressive step reduction with minimal quality degradation, whereas ξ = 0.7 better matches the original 30-step baseline.The split point used for StableVS differs from the one used for VA-REPA.

5. Related Works

Prior work accelerates diffusion and flow models through representation learning, variance reduction, and improved numerical solvers. Stable Velocity differs by using the variance regime along the trajectory to guide both training and sampling without retraining.

  • Training acceleration: Training acceleration research reduces costs through latent representations and auxiliary regularization for high-resolution diffusion and flow models.Latent diffusion compresses data into lower-dimensional representations, while other methods improve autoencoders or use localized modeling.
  • Sampling acceleration: StableVS comparisons use 30- and 20-step Euler sampling under identical random seeds, with StableVS outputs more closely resembling the 30-step results.The qualitative comparison is conducted on SD3.5 across prompts.
  • Variance reduction: Variance-reduction methods include timestep importance sampling, control variates, and self-normalized importance sampling, with SNIS introducing bias.StableVM instead provides an unbiased variance-reducing objective.
  • Sampling acceleration: Sampling acceleration methods either require additional training or improve integration accuracy while treating the diffusion trajectory uniformly.StableVS exploits the low-variance regime to reduce steps without retraining and remains compatible with existing solvers.

6. Conclusion

The paper frames variance structure as governing training and sampling dynamics, then builds StableVM, VA-REPA, and StableVS around that structure. Experiments report improved training stability and sampling speedups without sacrificing sample quality.

  • Conclusion: The Stable Velocity framework combines StableVM for unbiased variance-reduced training, VA-REPA for selective auxiliary supervision, and StableVS for finetuning-free inference acceleration.The framework is based on a two-regime structure separating high-variance and low-variance dynamics.
  • Conclusion: Explicitly modeling variance structure provides a basis for designing more efficient training objectives and sampling algorithms in diffusion and flow-based generative models.The conclusion presents this as a broader implication beyond the specific methods introduced.

Impact Statement

The impact statement situates the work within generative-model research and acknowledges potential misuse risks. It reports no human subjects or personal data and uses public benchmarks.

  • Impact Statement: The work aims to advance understanding of variance structure in deep generative models.The stated scope concerns diffusion and flow-based generative modeling research.
  • Impact Statement: The authors identify potential misuse of image and video generation technologies as a risk requiring responsible deployment and safeguards.They emphasize transparency, reproducibility, and responsible AI research practices.
  • Impact Statement: The experiments do not involve human subjects or personal data and use widely available public benchmarks.This is stated as part of the authors’ responsible research practices.

C. Comparison with Stable Target Field

The comparison with Stable Target Field shows that StableVM targets variance reduction without sacrificing unbiasedness, while STF’s practical gains depend on an implementation that deviates from its theoretical objective.

  • Stable Target Field: STF reduces target covariance by a factor of n but introduces bias that vanishes as n approaches infinity.Its finite-sample minimizer deviates from the true score.
  • StableVM: StableVM extends variance reduction to flow matching and stochastic interpolants while preserving an unbiased objective.The framework also includes a class-conditional extension compatible with classifier-free guidance.
  • Target construction: StableVM samples from a Gaussian mixture over the reference batch, producing unbiased targets with reduced variance.This differs from direct STF, which conditions the noisy input on a single reference sample.
  • Empirical comparison: StableVM consistently accelerates convergence and improves sample quality, while direct STF underperforms standard CFM and the original STF implementation still trails StableVM.The CIFAR-10 comparison uses n = 2048 and 50K generated samples.
  • Implementation cost: StableVM’s class-conditional implementation uses per-class and unconditional memory banks, with computation and memory overhead scaling linearly with bank capacity.For ImageNet, storing 256 fp16 latents for each of 1,000 classes requires approximately 2 GB.

E.3. Proofs of the Variance Bounds (Theorem 3.2 and Theorem 3.3)

The variance-bound proofs establish that StableVM’s estimator has strictly lower variance than CFM under the stated assumptions, while the low-variance regime supports explicit backward dynamics for sampling.

  • Variance bounds: StableVM has strictly lower variance than CFM when the conditional velocity field is affine and the reference samples are not almost surely identical.The strict inequality follows from the equality condition of Jensen’s inequality.
  • Estimator interpretation: The StableVM estimator can be interpreted through a posterior mixture in which one reference sample is drawn conditionally and the others independently follow the data distribution.This construction yields the joint distribution used by the Gaussian-mixture posterior.
  • Proof strategy: The estimator’s covariance is decomposed using the law of total covariance, with conditional independence simplifying the analysis.Conditioned on xt and the sampled index, nonselected reference samples follow the data distribution independently.
  • Asymptotic analysis: As the reference batch size grows, the self-normalized importance-sampling estimator converges through law-of-large-numbers and Slutsky arguments.The proof identifies the proposal as q(x0), the nominal distribution as pt(x0 | xt), and the estimator as self-normalized importance sampling.
  • Low-variance sampling: In the low-variance regime, conditional and marginal velocities approximately coincide, allowing the target x0 and backward PF-ODE solution to be written explicitly.The resulting backward solution depends only on xt and the velocity field vt(xt).
  • Controlled validation: StableVM converges faster and reaches lower final error than CFM across evaluated timesteps in a controlled synthetic setting.The experiment estimates the second-order discrepancy between predicted and true velocity fields.

F.3. Ablation Studies of Hyperparameters in StableVS

StableVS remains robust to moderate hyperparameter changes, but excessive stochasticity or stepping beyond the low-variance boundary degrades generation quality.

  • Hyperparameter robustness: StableVS is robust to moderate variations in the variance factor, low-variance step count, and split point ξ.The ablation uses SD3.5-Large at 1024 × 1024 resolution on GenEval.
  • Variance factor: Increasing the variance factor fβ adds stochasticity and degrades reference metrics, consistent with nearly deterministic low-variance dynamics.The ablation compares the effects of fβ, the number of low-variance steps, and ξ.
  • Regime allocation: Too few low-variance steps reduce fidelity, while choosing ξ too large and reducing steps beyond the regime boundary degrades quality.These results identify the low-variance boundary as a practical constraint on step reduction.

G. Experimental Details

The experiments use standardized ImageNet preprocessing and AdamW optimization, with mixed precision, gradient clipping, and EMA scheduling; StableVS is implemented with diffusers.

  • Data and optimization: Images are center-cropped and resized to 256 × 256 following the ADM preprocessing protocol.Optimization uses AdamW with a constant learning rate of 1 × 10^-4 and global batch size 256.
  • Training setup: Training uses fp16 mixed precision, gradient clipping, and an adaptive exponential moving average decay schedule.These measures are reported as enhancing training efficiency and numerical stability.
  • Sampling implementation: StableVS is built on the Hugging Face diffusers library to support pretrained models and flexible sampling pipelines.

H. More Qualitative Results

Additional qualitative results compare StableVS with standard solvers across SD3.5, Flux, Qwen-Image, and Wan2.2. Across the shown settings, StableVS outputs more closely resemble higher-step results while replacing the base solver only in the low-variance regime.

  • Quantitative evaluation: The GenEval evaluation keeps the high-variance regime unchanged and applies StableVS in the low-variance regime [0, ξ] with ξ = 0.85.The table reports overall and per-category GenEval scores at 1024 × 1024 resolution, with step counts indicated in parentheses.
  • Visual comparisons: StableVS outputs for SD3.5 and Flux more closely resemble 30-step results than standard 20-step Euler outputs.The comparisons use the same random seed, with StableVS replacing Euler in the low-variance regime.
  • Visual comparisons: StableVS outputs for Qwen-Image-2512 more closely resemble 30-step results than standard 17-step Euler outputs.StableVS replaces Euler in the low-variance regime under the same random seed.
  • Visual comparisons: StableVS outputs for Wan2.2 more closely resemble 30-step results than standard 20-step UniPC outputs.The Wan2.2 comparisons use the same random seed and replace UniPC with StableVS in the low-variance regime.
Loading 2602.05435v2…