PaperlayerView exact revision on arXiv

Source-linked AI summary

InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation

Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, Qiang Liu

arXiv:2309.06380v2cs.LGcs.CV

TL;DR

Diffusion-based text-to-image models are costly to sample because they typically require multiple inference steps. InstaFlow uses text-conditioned Rectified Flow and reflow to produce high-quality images in one step, achieving generation in 0.09 second after 199 A100 GPU days of training.

  • Problem

    Reducing inference cost while retaining accurate simulation requires learning straighter probability-flow trajectories that can be simulated with fewer steps.

  • Method

    InstaFlow applies text-conditioned reflow to straighten probability-flow trajectories and regularize the mapping before distilling Stable Diffusion into a one-step model.

  • Results

    0.09 second and 199 A100 GPU days yield InstaFlow, a one-step Stable Diffusion model capable of generating high-quality images with intricate details.

  • Takeaways & Limitations

    Straightening probability-flow trajectories enables accurate low-step simulation, supporting one-step text-to-image generation with substantially reduced inference cost.

  • Takeaways & Limitations

    InstaFlow may struggle with complex compositions in text prompts, though longer training and larger datasets may mitigate this issue.

Abstract

from arXiv · show

Diffusion models have revolutionized text-to-image generation with its exceptional quality and creativity. However, its multi-step sampling process is known to be slow, often requiring tens of inference steps to obtain satisfactory results. Previous attempts to improve its sampling speed and reduce computational costs through distillation have been unsuccessful in achieving a functional one-step model. In this paper, we explore a recent method called Rectified Flow, which, thus far, has only been applied to small datasets. The core of Rectified Flow lies in its \emph{reflow} procedure, which straightens the trajectories of probability flows, refines the coupling between noises and images, and facilitates the distillation process with student models. We propose a novel text-conditioned pipeline to turn Stable Diffusion (SD) into an ultra-fast one-step model, in which we find reflow plays a critical role in improving the assignment between noise and images. Leveraging our new pipeline, we create, to the best of our knowledge, the first one-step diffusion-based text-to-image generator with SD-level image quality, achieving an FID (Frechet Inception Distance) of $23.3$ on MS COCO 2017-5k, surpassing the previous state-of-the-art technique, progressive distillation, by a significant margin ($37.2$ $\rightarrow$ $23.3$ in FID). By utilizing an expanded network with 1.7B parameters, we further improve the FID to $22.4$. We call our one-step models \emph{InstaFlow}. On MS COCO 2014-30k, InstaFlow yields an FID of $13.1$ in just $0.09$ second, the best in $\leq 0.1$ second regime, outperforming the recent StyleGAN-T ($13.9$ in $0.1$ second). Notably, the training of InstaFlow only costs 199 A100 GPU days. Codes and pre-trained models are available at \url{github.com/gnobitab/InstaFlow}.

1 INTRODUCTION

Modern text-to-image models generate realistic, artistic, and detailed images from textual descriptions, but their quality comes with substantial inference-time and computational costs. Stable Diffusion typically needs more than 20 sampling steps to produce acceptable images, motivating faster generation approaches.

  • Generation quality: Modern T2I models synthesize realistic, artistic, and detailed images from textual descriptions using large-scale datasets and models.Examples include DALL-E, Imagen, Stable Diffusion, StyleGAN-T, and GigaGAN.
  • One-step generation: 0.09s is the inference time for one-step InstaFlow-0.9B generating 512 × 512 images on an NVIDIA A100 GPU.The generated images can be further enhanced by SDXL-Refiner for higher resolution and finer details.
  • Inference cost: T2I models often incur excessive inference time and computational consumption because many are autoregressive or diffusion models.The passage identifies architectural sampling requirements as the source of this burden.
  • Inference cost: More than 20 steps are typically required for Stable Diffusion to generate acceptable images, even with a state-of-the-art sampler.This illustrates the sampling bottleneck motivating faster text-to-image generation.

2 METHODS

InstaFlow combines text-conditioned Rectified Flow reflow with distillation to straighten probability-flow trajectories, preserve image distributions, improve noise–image coupling, and enable accurate one-step generation. A classifier-free guidance velocity field further trades sample diversity against generation quality.

  • Rectified Flow: Rectified Flow learns an ODE transport mapping from a Gaussian noise distribution π0 to an image distribution π1 through a learned velocity field.The flow is initialized from Z0 ∼ π0 and evolves toward Z1 ∼ π1.
  • Straight Flows Yield Fast Generation: Straight trajectories reduce inference cost because Euler simulation with a single step, N = 1, can produce perfect simulation.Larger N improves ODE approximation but increases computational cost, creating a cost–accuracy trade-off.
  • Straightening Text-Conditioned Probability Flows via Text-Conditioned Reflow: Text-conditioned reflow iteratively preserves the terminal image distribution while straightening trajectories and producing lower-cost, easier-to-learn noise–image couplings.Each new velocity field uses linear interpolations between noise and images generated by the preceding text-conditioned ODE.
  • Text-Conditioned Distillation: Distillation learns a single Euler step x + v(x | T) that approximates the reflowed ODE mapping using a differentiable image-similarity loss.Reflow regularizes the mapping before distillation, while the two procedures remain orthogonal techniques.
  • Classifier-Free Guidance Velocity Field for Text-Conditioned Rectified Flow: Classifier-free guidance defines vα(Zt, t | T) = αv(Zt, t | T) + (1 − α)v(Zt, t | NULL), with α controlling diversity versus quality.When α = 1, the guided field equals the original text-conditioned velocity field.

3 PRELIMINARY RESULTS: REFLOW IS THE KEY TO IMPROVE DISTILLATION

Experiments with Stable Diffusion 1.4 show that direct distillation struggles to imitate the multi-step teacher, whereas reflow straightens trajectories, improves noise–image coupling, and enables stronger one-step distillation. The resulting 2-Rectified Flow+Distill model reaches FID 31.0 within 0.09s at approximately 24.65 A100 GPU days.

  • Straightening Effects of Reflow: Reflow refines the coupling between noises and images, providing a better teacher for distillation and enabling more meaningful one-step generation.Figure 5 compares models using the same random noise and text prompt.
  • Observation and Analysis: Direct distillation converges but produces a large FID gap from the 25-step Stable Diffusion teacher and substantially different images under identical noise and prompts.All models were trained for 100,000 steps on 3,200,000 generated training pairs.
  • Training Cost: ≈ 24.65 A100 GPU days: fine-tuning 2-Rectified Flow+Distill from publicly available Stable Diffusion requires substantially less training than the listed large-scale models and Progressive Distillation lower bound.Reference costs include 6250 days for SD 1.4 from scratch, 1792 for StyleGAN-T, 4783 for GigaGAN, and 108.8 as the Progressive Distillation lower bound.
  • Comparison on MS COCO: 22.1 ↔ 22.8 FID: (Pre) 2-Rectified Flow matches Stable Diffusion 1.4 with 25-step DPMSolver on MS COCO 2017-5k.The comparison concerns realistic image generation before one-step distillation.
  • Comparison on MS COCO: 31.0 FID: (Pre) 2-Rectified Flow+Distill surpasses the previous one-step Stable Diffusion model’s 37.2 FID within 0.09s.The previous model was distilled using Progressive Distillation.
  • Straightening Effects of Reflow: Reflow decreases trajectory straightness S(Z), making 2-Rectified Flow trajectories substantially straighter than Stable Diffusion trajectories.A smaller S(Z) indicates straighter trajectories, with S(Z) = 0 for totally straight ODE trajectories.

4 INSTAFLOW: SCALING UP FOR BETTER ONE-STEP GENERATION

Scaling training to 199 A100 GPU days yields InstaFlow, a high-quality one-step Stable Diffusion model. InstaFlow-0.9B achieves 23.4 FID-5k at 0.09s inference, while the 1.7B variant expands model capacity for further generation improvements.

  • Scaling Up: 199 A100 GPU days of scaled training produces InstaFlow, a high-quality one-step Stable Diffusion model.The scaling uses a larger batch size and extended training duration compared with the preliminary 24.65 A100 GPU days.
  • InstaFlow-1.7B: 1.7B parameters and 0.12 second inference define InstaFlow-1.7B, built with a Stacked U-Net for larger-scale generation.The model stacks two U-Nets in series and uses 39.6 A100 GPU days for distillation.
  • MS COCO Comparison: 23.4 FID-5k at 0.09s makes InstaFlow-0.9B substantially better and similarly costly than Progressive Distillation-SD’s 37.2 FID in one step.The reported distillation costs are 108 and 108.8 A100 GPU days, respectively.
  • Few-Step Inference: 2-Rectified Flow shows a clear advantage over standard few-step inference when the number of inference steps is ≤4.The comparison evaluates inference steps of 1, 2, 4, and 8.
  • Guidance Scale α: Increasing guidance scale α from 1.0 to 4.0 raises both FID-5k and CLIP score on MS COCO 2017, trading image quality for semantic alignment.Higher FID indicates degraded image quality, whereas higher CLIP score indicates enhanced semantic alignment.
  • Fast Preview with One-Step InstaFlow: One-step InstaFlow generates 512 × 512 images in ∼0.1s, enabling fast previews that can be refined by SDXL-Refiner.The proposed workflow uses rapid low-resolution filtering followed by quality improvement and resolution enhancement.

5 LIMITATIONS AND CONCLUSIONS

The paper introduces InstaFlow as a state-of-the-art one-step text-to-image generator based on a text-conditioned Rectified Flow pipeline with pure supervised learning. It may struggle with complex prompt compositions, but longer training and larger datasets could mitigate these challenges.

  • InstaFlow is introduced as a state-of-the-art one-step text-to-image generator derived from a text-conditioned Rectified Flow pipeline with pure supervised learning.
  • InstaFlow may encounter challenges with complex compositions in text prompts.The paper refers to Figure 11 as an example of a failure case.
  • Longer training and larger datasets are suggested as likely ways to mitigate these composition challenges.

SOCIETAL IMPACT … 4. Remove ‘Downsample Blocks 2 (the green blocks on the right)’

The paper frames one-step text-to-image generation as both an efficiency opportunity and a potential misuse risk, situating its approach within diffusion-model research and describing the pipeline and architectural ablations used for acceleration.

  • SOCIETAL IMPACT: One-step generators may conserve energy and benefit the environment, but they can also accelerate harmful information and fake-news creation when misused.The societal-impact discussion identifies both benefits and risks of faster generation.
  • A RELATED WORKS: Diffusion and flow-based models have achieved strong results across image, video, audio, point-cloud, and biological generation.The related-work passage surveys applications across several generative-modeling domains.
  • A RELATED WORKS: Text-to-image research progressed from small datasets such as flowers and birds toward complex scenarios represented by MS COCO.The passage describes the field’s shift toward larger, more challenging text-to-image settings.
  • A RELATED WORKS: Diffusion models are slow because inference requires multiple iterations, while fast post-hoc samplers typically reduce pre-trained models to 20-50 steps.The related work distinguishes fast samplers from other acceleration approaches.
  • B NEURAL NETWORK STRUCTURE: The generation pipeline combines a CLIP ViT-L/14 text encoder, a latent-space generative model, and a decoder inherited from Stable Diffusion.The text encoder and latent decoder are frozen during training; encoding takes 0.01s and decoding takes 0.04s per image on an NVIDIA A100.
  • B NEURAL NETWORK STRUCTURE: The default latent generative model is a U-Net; reflow fine-tunes it unchanged, while distillation evaluates original and concatenated shared-parameter U-Net structures.The concatenated structure significantly decreases distillation loss and improves generation quality.
  • 1. Remove ‘Downsample Blocks 1 (the green blocks on the left)’: Ablations delete different blocks from the concatenated network to assess their importance, remove unnecessary components, and reduce inference time.Figure 12 compares distillation structures and their inference times, with same-colored blocks able to share weights.
  • 2. Remove ‘Upsample Blocks 1 (the yellow blocks on the left)’: The ablation sequence also considers removing the middle ‘In+Out Block,’ corresponding to the blue and purple blocks in the middle.This block-removal experiment is listed as part of the network-structure studies.

5. Remove ‘Upsample blocks 2 (the yellow blocks on the right)’

Among the evaluated alternatives, only Structure 3 avoids hurting performance, reducing inference time by 7.7%; this Stacked U-Net is shown in Figure 12(c).

  • Remove ‘Upsample blocks 2 (the yellow blocks on the right)’: 7.7% reduction in inference time is achieved by Structure 3, the only structure reported not to hurt performance.The reduction is calculated from 0.13 to 0.12.
  • Remove ‘Upsample blocks 2 (the yellow blocks on the right)’: Stacked U-Net is the third structure and is illustrated in Figure 12(c).

C ADDITIONAL DETAILS AND RESULTS ON THE PRELIMINARY EXPERIMENTS

The preliminary experiments initialize from Stable Diffusion 1.4 and use SD-aligned reflow and distillation settings. Evaluation measures batch-one inference time on an NVIDIA A100, including text encoding and latent decoding but excluding NSFW detection.

  • General Experiment Settings: Stable Diffusion 1.4 initializes the model weights for the experiments.
  • General Experiment Settings: Reflow and distillation use the SD U-Net, while distillation similarity is measured with LPIPS loss.Training uses text prompts from laion2B-en filtered like Stable Diffusion, with batch size 32 on 8 A100 GPUs and AdamW.
  • General Experiment Settings: Inference time is measured with batch size 1 on an NVIDIA A100 using PyTorch 2.0.1 and Hugging Face Diffusers 0.19.3.Timing includes the text encoder and latent decoder, but excludes the NSFW detector.

C.1 ADDITIONAL DETAILS AND RESULTS OF DIRECT DISTILLATION · C.2 ADDITIONAL QUANTITATIVE COMPARISON

The appendix details a nine-model direct-distillation grid search for Stable Diffusion 1.4, identifying learning-rate regimes that improve performance or cause instability. It also reports an additional multiple-reflow experiment using 3-Rectified Flow and parameter-sharing Stacked U-Nets.

  • C.1 ADDITIONAL DETAILS AND RESULTS OF DIRECT DISTILLATION: 9 models were trained for 100,000 steps across a grid of three learning rates and three weight-decay coefficients.The learning rates were {10^-5, 10^-6, 10^-7}, while weight decay coefficients were {10^-1, 10^-2, 10^-3}.
  • C.1 ADDITIONAL DETAILS AND RESULTS OF DIRECT DISTILLATION: 3,200,000 training pairs were generated for distillation from 32 × 100,000 pairs of (X0, ODE[vSD](X0)).These pairs formed the training set for the direct-distillation experiments.
  • C.1 ADDITIONAL DETAILS AND RESULTS OF DIRECT DISTILLATION: ≥10^-4 learning rates caused unstable training and NaN errors despite boosting performance at lower values.The appendix reports that increasing the learning rate improved performance until the instability threshold was reached.
  • C.1 ADDITIONAL DETAILS AND RESULTS OF DIRECT DISTILLATION: 10^-6 and 10^-7 learning rates produced slow convergence and blurry generations after 100,000 training steps.These results were reported for direct distillation of Stable Diffusion 1.4.
  • C.2 ADDITIONAL QUANTITATIVE COMPARISON: Additional quantitative results evaluated parameter-sharing Stacked U-Nets and multiple reflow.The results are reported in Tables 4 and 5.
  • C.2 ADDITIONAL QUANTITATIVE COMPARISON: 3-Rectified Flow (v3) was obtained by repeating the reflow procedure once more from 2-Rectified Flow (v2).The procedure can be repeated multiple times according to equation 5.
  • C.2 ADDITIONAL QUANTITATIVE COMPARISON: 50,000 training steps were used to train 3-Rectified Flow by minimizing equation 5.The resulting model was then distilled using newly generated pairs.

C.3 ESTIMATION OF TRAINING COST

The paper estimates 24.65 A100 GPU days for (Pre) 2-Rectified Flow+Distill with a U-Net and contrasts its training cost with Progressive Distillation.

  • C.3 ESTIMATION OF TRAINING COST: 100,000 iterations per A100 GPU day are achievable with L2 loss, versus 86,000 with LPIPS loss, using batch size 4 and a U-Net.With batch size 16, one A100 GPU day generates 200,000 data pairs.
  • C.3 ESTIMATION OF TRAINING COST: 24.65 A100 GPU days is the estimated training cost for (Pre) 2-Rectified Flow+Distill with a U-Net.The estimate combines data generation, reflow, and distillation costs.
  • C.3 ESTIMATION OF TRAINING COST: Progressive Distillation is estimated from its 512-step starting point and progressive distillation to 1 step using batch size 512.The estimate uses a quoted lower-bound gradient-update schedule from Appendix C.2.1 of [58].
  • C.3 ESTIMATION OF TRAINING COST: 2-Rectified Flow+Distill outperforms Progressive Distillation at the same inference time while using much less training cost.The comparison evaluates FID on MS COCO 2017 under the setup described in Table 4.

D ADDITIONAL TRAINING DETAILS ON INSTAFLOW

InstaFlow-0.9B is trained through two reflow and two distillation stages, progressively increasing batch size and switching from L2 to LPIPS similarity loss. InstaFlow-1.7B instead uses a 1.7B-parameter Stacked U-Net, while training observations highlight incomplete reflow convergence, faster distillation, and LPIPS-driven visual improvements.

  • InstaFlow-0.9B: 70,000 iterations at batch size 64 initialize reflow from pre-trained SD 1.5 weights, costing 11.2 A100 GPU days.This is Reflow (Stage 1) using the reflow objective.
  • InstaFlow-0.9B: 25,000 iterations at batch size 1024 complete Reflow (Stage 2), producing 2-Rectified Flow at a cost of 64 A100 GPU days.The model continues training with the reflow objective.
  • InstaFlow-0.9B: 21,500 iterations of L2-based distillation at batch size 1024 fine-tune 2-Rectified Flow with t = 0 fixed, costing 54.4 A100 GPU days.The teacher guidance scale α is 1.5.
  • InstaFlow-0.9B: 18,000 further iterations switch similarity loss from L2 to LPIPS, yielding InstaFlow-0.9B after 53.6 A100 GPU days.The final model is 2-Rectified Flow+Distill (U-Net).
  • InstaFlow-1.7B: 1.7B parameters define the expanded Stacked U-Net without parameter sharing; its distillation uses 110,000 L2 iterations followed by 2,500 LPIPS iterations.The two stages cost 35.2 and 4.4 A100 GPU days, respectively, and produce InstaFlow-1.7B.
  • Experiment Observations: LPIPS immediately enhanced visual quality, while distillation converged faster than reflow and 2-Rectified Flow had not fully converged during training.The authors suggest longer training and additional computation could improve performance.

E ADDITIONAL QUALITATIVE RESULTS

Additional qualitative results show that InstaFlow’s one-step models align with the continuous 2-Rectified Flow and retain useful latent-space structure. Visualizations also demonstrate fast generation, prompt-consistent pose and lighting, and clear low-step outputs under suitable guidance.

  • Latent-space properties: 0.09s generation with InstaFlow-0.9B saves ∼90% of computational time versus the 25-step SD-1.5 teacher model.This result is shown for latent-space interpolation.
  • Prompt and noise consistency: 0.12s generation with InstaFlow-1.7B preserves pose and lighting across different text prompts when random noise is held fixed.The comparison uses the same random noise across prompts.
  • Inference settings: N ≤4 inference steps produce clear images with 2-Rectified Flow, whereas SD 1.5-DPM Solver cannot under the same random seed.The comparison varies the number of inference steps while keeping the random seed fixed.
  • Inference settings: At α = 1.0, 2-Rectified Flow produces blurry edges and twisted details, while α ≥2.0 gradually causes over-saturation.The figure compares image quality across guidance-scale settings.
  • Latent-space properties: With the same random noise and text prompts, one-step models generate similar images to continuous 2-Rectified Flow, indicating aligned latent spaces.The alignment supports using one-step models as surrogates for analyzing the continuous flow’s latent space.
Loading 2309.06380v2…