UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANs

Introduction

• Diffusion models’ iterative denoising slows generation, while advanced PF-ODE solvers still need 10 to 20 steps and distillation struggles at extremely few steps.Solver methods face a step-size/accuracy trade-off; progressive distillation [29] [39] [50] and consistency distillation [37] also struggle in the very-small-step regime.Source: §1 · p. 1
• UFOGen combines GANs with diffusion models, using adversarial loss rather than KL minimization to learn the denoising distribution.Earlier diffusion-GAN hybrids [59] [60] [63] [69] generated in as few as four steps on small-scale datasets, but one-step text-to-image generation remained challenging.Source: §1 · p. 2
• UFOGen generates high-quality text-conditioned images in one inference step and, when initialized from Stable Diffusion, largely preserves generated-content quality.The authors describe it as among the pioneering text-to-image diffusion models to reduce sampling to one step.Source: §1 · p. 2
• Its simplified training process fine-tunes pre-existing large-scale diffusion models, such as Stable Diffusion, into one-step generators.The paper presents this as an efficient way to adapt pre-trained models.Source: §1 · p. 2
• UFOGen also extends one-step inference to image-to-image and controllable generation.The authors present these as applications of the model’s versatility beyond text-to-image generation.Source: §1 · p. 2

2. Related Works

• Denoising diffusion models reconstruct data from corrupted inputs, while latent models such as Stable Diffusion [42] [47] are popular for their simplicity and efficiency over pixel-space counterparts.Large-scale text-to-image diffusion models have been trained on image-text datasets.Source: §2 · p. 3
• Diffusion acceleration research uses improved numerical solvers [2] [24] [35] [36] [55] or distillation [15], but has not reduced text-to-image sampling to a truly minimal number of steps.Distillation condenses a numerical solver’s sampling trajectory into fewer steps [3] [29] [37] [39] [50].Source: §2 · p. 3
• Earlier text-to-image GANs used small datasets, while later models trained on large datasets showed promise; UFOGen contrasts with GigaGAN [20] through simpler, more robust training and flexible downstream applications.The paper attributes this to its diffusion component, pre-trained diffusion initialization, and downstream flexibility; it says GAN training otherwise requires auxiliary losses and complex regularization.Source: §2 · p. 3

Recent Progress on Few-step Text-to-image Generation

• Latent Consistency Model [37] enables reasonable-quality 4-step sampling but degrades at fewer steps, while InstaFlow [33] generates in one step and UFOGen outperforms it quantitatively and visually.InstaFlow extends Rectified Flow models [31] [32]; UFOGen uses one fine-tuning stage with few iterations rather than multiple fine-tuning stages followed by distillation.Source: Recent Progress on Few-step… · p. 3

Diffusion Models

• Diffusion models add scheduled Gaussian noise over T forward steps and learn Gaussian reverse denoising transitions, trained with a negative ELBO [16] [54] [16] [25].The Gaussian reverse parameterization is motivated by the Gaussian denoising distribution when steps are sufficiently small.Source: §3 · p. 3

Diffusion-GAN Hybrids

• DDGAN [60] replaces Gaussian conditional denoising with a conditional GAN because large steps break the Gaussian assumption, enabling more aggressive denoising and four-step sampling.Source: §3 · pp. 3–4
• SIDDMs [63] address adversarial instability by matching joint distributions via marginal adversarial and conditional KL/reconstruction terms, stabilizing training and improving results on intricate datasets.Source: §3 · p. 4

4.1. Enabling One-step Sampling for UFOGen

• Changing the generator parameterization and replacing the reconstruction term with clean-sample reconstruction enables one-step sampling while retaining training with several denoising steps.Prior diffusion-GAN models could not sample in one step because training with a single denoising step reduces to conventional GAN training; the new parameterization supports matching at x0.Source: §4.1 · pp. 4–5, Figure 2 · p. 5
• UFOGen’s objective combines adversarial matching of noisy samples at time step t −1 with reconstruction at time step 0, interpretable as diffusion training with adversarial refinement.The reconstruction term is essentially the training objective of diffusion models [16] [56].Source: §4.1 · p. 5

4.2. Leverage Pre-trained Diffusion Models

• Scaling diffusion-GAN text-to-image training is challenging because the discriminator must judge texture and semantics for text-image alignment, while adding parameters and cost.The alignment challenge is particularly pronounced during the initial stage of training.Source: §4.2 · p. 5
• UFOGen uses the same UNet structure for its generator and discriminator, enabling initialization from Stable Diffusion [47], which yields stable training dynamics and fast convergence.The authors propose that Stable Diffusion’s internal features encode information about the interplay between textual and visual data.Source: §4.2 · p. 5

5.1. Text-to-image Generation

• UFOGen’s single-step results outperform Progressive Distillation at one, two, and four steps and CFG-Aware distillation at eight steps on FID and CLIP; it also beats single-step InstaFlow-0.9B on both scores.It remains competitive with InstaFlow-1.7B, which uses stacked UNets and roughly twice the parameters.Source: §5.1 · p. 6, Table 1 · p. 6
• Because MS-COCO zero-shot FID may not reliably reflect visual quality [26] [42], as discussed by [4], qualitative comparisons show sharper textures and finer details than four-step LCM samples.UFOGen also surpasses InstaFlow in image quality and produces less-blurry samples than two-step LCM; the authors report that single-step LCM fails to generate textures.Source: §5.1 · p. 6, Table 2 · p. 7

5.2. Ablation Studies

• The ablation study finds that the proposed UFOGen objective’s modifications enable one-step sampling compared with the SIDDM objective in diffusion-GAN hybrids.The authors also provide qualitative samples and an additional ablation on training denoising-step size in the appendix.Source: §5.2 · p. 7, Table 4 · p. 8

5.1. Text-to-image Generation

• FID is 12.78 for UFOGen and 13.10 for InstaFlow on MSCOCO 2014 with 30k images, but the broader model comparisons are not directly comparable across architectures, parameter counts, and training data.The authors describe UFOGen as competitive while offering one-step generation speed relative to autoregressive and diffusion models.Source: §5.1 · p. 6, §5.3 · p. 8, Table 3 · p. 8

5.3. Applications

• With SDEdit-style input noise, UFOGen performs single-step image-to-image generation that follows both the prompt and input image.For controllable generation, fine-tuning with an adapter that takes depth maps or Canny edges as control signals yields samples aligned with both prompt and signal.Source: §5.3 · p. 8, Table 5 · p. 8, Table 6 · p. 8
• UFOGen handles zero-shot image-to-image and controllable generation, capabilities the authors distinguish from prior GAN-based text-to-image models [20] [52].They attribute this flexibility to denoising-based generation and describe it as distinctive among one-step text-to-image models.Source: §5.3 · pp. 7–8

• UFOGen combines diffusion models with a GAN objective to generate high-quality, text-conditioned images in one step.Source: §6 · p. 8
• The comprehensive evaluations affirm UFOGen’s superiority over existing accelerated diffusion-based methods.Source: §6 · p. 8
• Its one-step text-to-image capability and proficiency in downstream tasks underscore UFOGen’s versatility.Source: §6 · p. 9

A.1. Deriving the KL objective in Equation 4

• The forward conditional q(x_t|x_{t−1}) is Gaussian, with mean √(1−β_t)x_{t−1} and covariance β_tI.Source: §A.1 · p. 13
• Although p_θ(x_t|x′_{t−1}) depends on the generator, at a given x′_{t−1} it follows the forward-diffusion distribution.Its covariance is β_tI.Source: §A.1 · p. 13
• With these conditional distributions, Equation 7 is a KL divergence between two Gaussian distributions that can be computed in closed form.Source: §A.1 · p. 13
• Since the two covariance matrices are both β_tI, substituting the means and variances simplifies the KL expression to a form containing a constant C.Source: §A.1 · p. 13
• Minimizing Equation 4’s KL objective is equivalent to minimizing a weighted reconstruction loss between x′_{t−1} and x_{t−1}.The target x_{t−1} is sampled by first drawing x_0 ∼ q(x_0), then x_{t−1} ∼ q(x_{t−1}|x_0).Source: §A.1 · p. 13
• Unlike this derivation, [63] decomposes the KL divergence into entropy and cross-entropy, then empirically estimates their expectations.Source: §A.1 · p. 13
• The authors of [63] add an auxiliary parametric distribution for entropy estimation, yielding an adversarial objective; this appendix says the extra term is dispensable without practical challenges.Source: §A.1 · p. 13

A.2. Analysis of the distribution matching at x0

• Aligning x_{t-1} and x'_{t-1} implicitly encourages matching x_0 and x'_0 because each noisy sample comes from an independent Gaussian corruption of its respective base image.The section presents this as the rationale for why the objective in Equation 4 results in matching x_0 and x'_0.Source: §A.2 · p. 13

Adversarial term

• Convolving two distributions with the same positive-density kernel preserves whether they are equal under a divergence that is zero only for identical distributions.The proof uses the convolution of densities and the invertibility of the Fourier transform.Source: §A.2.1 · pp. 13–14
• When JSD(q(x_{t−1})||pθ(x_{t−1})) is zero, q(x0) and pθ(x0) are equal almost everywhere.This establishes why the adversarial objective on x_{t−1} corresponds to matching the original distributions.Source: §A.2.1 · pp. 13–14
• Direct GAN training on x0 is avoided because additive Gaussian noise smooths distributions and stabilizes GAN training, an essential component of diffusion-GAN hybrids [60].The forward diffusion process expresses q(x_{t−1}) as a convolution of q(x0) with a Gaussian kernel; the generator distribution is convolved with the same kernel.Source: §A.2.1 · pp. 13–14

A.2.2 KL term

• Under Gaussian forward diffusion [16], the reconstruction objective over q(x_t) reduces to matching clean x_0 with x′_0, up to a model-independent constant.The derivation rewrites the expectation using Gaussian noise and independent noise samples.Source: §A.2.2 · p. 14
• Averaging this match over Gaussian noise when sampling x_{t−1} and x′_{t−1} can produce high variance and a less robust clean-data alignment signal.The paper identifies this as a limitation of the equivalence-based objective.Source: §A.2.2 · p. 14
• The proposed refinement directly enforces reconstruction between x_0 and x′_0, adding explicit clean-data distribution matching to improve robustness and effectiveness.The passage points to Section 4.1 for the refinement.Source: §A.2.2 · p. 14

A.3. Experimental Details

• Experiments initialize latent-space generator and discriminator from Stable Diffusion 1.5, retaining its VAE and frozen CLIP ViT-L/14 text encoder.Training uses denoising step size 250 and λKL = 1.0; AdamW, batch size 1024, peak learning rate 1e-4, and generator EMA 0.999, with convergence typically in <50k steps.Source: §A.3 · p. 15

A.4. Additional Results of Ablation Studies

• Appendix A.4 supplements the main-text ablations with qualitative results and an additional experiment on training denoising step size.The qualitative comparison is presented in A.4.1, and the step-size ablation in A.4.2.Source: §A.4 · p. 15

A.4.1 Qualitative Results for Table 4

• In one-step samples, SIDDM appears blurry and over-smoothed, whereas UFOGen produces sharp images.The appendix presents qualitative comparisons associated with the main-text ablation study.Source: §A.4.1 · p. 15, Table 7 · p. 16, Table 8 · p. 17

A.4.2 Ablation on Denoising Step-size

• A denoising step size of 250 is the reported sweet spot, while 1000 fails; the results support using adversarial loss on noisy data to stabilize training.The authors conjecture that reducing the step size degrades performance because the discriminator lacks capacity for many distinct noise levels.Source: §A.4.2 · p. 15, Table 9 · p. 17

A.5.1 Failure of Single-step LCM

• Single-step LCM sampling fails to produce reasonable textures, so the paper limits its qualitative comparison with LCM to two and four steps.The authors attribute the difficulty to the complexity of the probability-flow ODE trajectory and the need for iterative refinement.Source: §A.5.1 · p. 16, Figure 4 · p. 18

A.6. Additional Qualitative Samples from UFOGen

• UFOGen produces high-quality, prompt-aligned single-step images across styles and content, but also exhibits missing objects, attribute leakage, and counting errors.The paper describes these failure modes as common issues in Stable Diffusion-based models.Source: §A.6 · p. 18

A.7. Additional Results of UFOGen’s Applications

• The appendix extends UFOGen’s image-to-image and controllable-generation results.Figure 5 shows image-to-image edits using added noise and slightly modified prompts; Figure 6 shows controllable generation with canny edges and depth maps.Source: §A.7 · p. 18, Figure 5 · p. 33, Figure 6 · p. 34

A.5.2 Extended Results of Table 2

• UFOGen remains highly competitive with updated LCM at one step, outperforming its two-step version and approaching its four-step version qualitatively.The comparison uses updated LCM models sharing the Stable Diffusion 1.5 backbone; the authors note that these models improve on the initially released LCM.Source: §A.5.2 · p. 18, Table 18 · p. 27, Table 19 · p. 28