PaperlayerView exact revision on arXiv

Source-linked AI summary

Glance: Accelerating Diffusion Models with 1 Sample

Zhuobai Dong, Rui Zhao, Songjie Wu, Junchao Yi, Linjie Li, Zhengyuan Yang, Lijuan Wang, Alex Jinpeng Wang

arXiv:2512.02899v2cs.CV

TL;DR

Diffusion models’ iterative denoising is costly, and customizing distilled models can require computationally expensive retraining. Glance attaches phase-specialized Slow-LoRA and Fast-LoRA adapters to the unchanged base model, applying smaller speedups early and larger ones late. Across six benchmarks, it reports 5× speed-up while retaining teacher-level performance, with experts trained from one sample within one hour.

  • Problem

    Diffusion generation has high inference cost, while customizing distilled models remains unclear and repeating distillation can require substantial computation.

    Source: §1 · p. 1

  • Method

    Glance attaches Slow-LoRA and Fast-LoRA adapters to the pretrained model, assigning smaller speedups to early semantic steps and larger speedups to later redundant steps.

    Source: §1 · p. 2

  • Results

    5× speed-up over FLUX.1 and Qwen-Image retains teacher-level performance across six benchmarks; Glance’s performance curves closely track the base models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Glance offers data- and compute-efficient acceleration with comparable visual fidelity and is presented as a candidate for data-scarce applications.

    Source: §5 · pp. 8–9

    Text rendering is weaker, especially for extremely dense or very small text, where strokes may blur or local artifacts may appear.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Diffusion models have achieved remarkable success in image generation, yet their deployment remains constrained by the heavy computational cost and the need for numerous inference steps. Previous efforts on fewer-step distillation attempt to skip redundant steps by training compact student models, yet they often suffer from heavy retraining costs and degraded generalization. In this work, we take a different perspective: we accelerate smartly, not evenly, applying smaller speedups to early semantic stages and larger ones to later redundant phases. We instantiate this phase-aware strategy with two experts that specialize in slow and fast denoising phases. Surprisingly, instead of investing massive effort in retraining student models, we find that simply equipping the base model with lightweight LoRA adapters achieves both efficient acceleration and strong generalization. We refer to these two adapters as Slow-LoRA and Fast-LoRA. Through extensive experiments, our method achieves up to 5 acceleration over the base model while maintaining comparable visual quality across diverse benchmarks. Remarkably, the LoRA experts are trained with only 1 samples on a single V100 within one hour, yet the resulting models generalize strongly on unseen prompts.

1. Introduction

Diffusion generation is costly, and adapting distilled models can require repeating expensive distillation. Glance instead uses phase-specialized LoRA adapters to accelerate denoising, reporting 5× speed-up with teacher-level performance across six benchmarks.

1. Introduction

  • Prior fewer-step distillation methods can generate high-fidelity images in one to four steps, while Glance requires only one training sample for its two adapters.Figure 2 contrasts large training sets and costly retraining in prior distillation pipelines with Glance’s plug-and-play acceleration.Source: §1 · p. 1, Figure 2 · p. 2
  • Customizing distilled models remains unclear, and repeating distillation after customization is less feasible because of its large computational cost.The passage cites 3,840 A100 GPU-hours for SDXL-DMD2 and 3,072 A100 GPU-hours for Qwen-Image-Lightning.Source: §1 · p. 1
  • The trajectory has an early semantic phase that determines global structure and a late redundant phase that refines texture, motivating smaller early and larger late speedups.Semantic steps are described as more sensitive to perturbation than redundant steps.Source: §1 · p. 2
  • Glance attaches Slow-LoRA and Fast-LoRA to the pretrained model, preserving its base parameters without training a new student network.The experts are trained from one sample on a single V100 GPU within one hour.Source: Figure 1 · p. 1, §1 · p. 2
  • 5× speed-up over FLUX.1 and Qwen-Image retains teacher-level performance across six benchmarks; Glance reaches 92.60%, 99.67%, and 96.71% of teacher performance on OneIG-Bench, HPSv2, and GenEval, respectively.The reported performance curves closely track those of the base models.Source: §1 · p. 2

2. Related Work

Related work spans methods for reducing diffusion inference cost and efficient LoRA adaptation. Glance distinguishes its approach by assigning LoRA experts to different denoising phases rather than treating timesteps uniformly.

2. Related Work

  • Diffusion models require many denoising steps for high-fidelity synthesis, creating substantial inference cost that limits real-time or resource-constrained use.The cited work discusses diffusion distillation methods that reduce the number of inference steps.Source: §2 · p. 2
  • Distillation methods trade off fidelity, stability, and computation: regression can blur results, progressive training risks error accumulation, and consistency methods may require costly JVPs.Distribution-matching methods can achieve high perceptual quality but are prone to mode collapse and instability.Source: §2 · pp. 2–3
  • The Slow-Fast visualization samples five timesteps from the first 20, then five uniformly from the remaining 40, executing the slow-stage timesteps first.The figure presents the slow–fast timestep schedule.Source: Figure 3 · p. 3
  • Recent compact FLUX and Qwen-Image students use trajectory-level supervision and extensive teacher sampling, whereas Glance reports comparable or superior quality with minimal training cost.The comparison notes that these recent models include 4-NFE or 8-NFE versions.Source: §2 · p. 3
  • LoRA has been used for efficient diffusion adaptation and in distillation, but generation quality in recent LoRA-based distilled models remains limited.Glance addresses this by allocating slow and fast adapters according to denoising phases.Source: §2 · p. 3

3. Method

Glance accelerates diffusion through phase-specific LoRA experts, assigning slower processing to early semantic reconstruction and faster processing to later refinement. The method uses SNR-guided phase partitioning and flow-matching supervision while retaining the pretrained model.

3.1. Preliminary

  • Flow matching describes denoising as transport from Gaussian noise to clean data, predicting a guidance-conditioned velocity at each intermediate state.Its objective is mean-squared error between predicted and target velocities.Source: §3.1 · p. 3
  • LoRA augments frozen pretrained parameters with trainable low-rank matrices, avoiding updates to the full model parameters.The added matrices A and B form the update BA, with rank much smaller than the pretrained dimensions.Source: §3.1 · p. 3

3.2. Phase-aware LoRA Experts for Phase-wise Denoising

  • An SNR boundary assigns high-noise coarse semantic reconstruction to a slow expert and low-noise texture refinement to a fast expert.The boundary corresponds to an SNR threshold, such as half the initial SNR.Source: §3.2 · p. 4
  • Even with one training sample, phase-wise LoRAs produced stable acceleration behavior.The authors attribute this data efficiency to direct target-velocity prediction through flow matching.Source: §3.2 · p. 5
  • Separating the trajectory into dedicated slow and fast regions yielded the most robust specialization among the tested configurations.Alternative adapter assignments lacked capacity for high-noise complexity or failed to exploit temporal locality in low-noise refinement.Source: §3.2 · p. 5
  • Each phase-specific expert receives flow-matching supervision through weighted velocity error and trains on samples from its assigned denoising phase.The timestep-dependent weighting function is optional.Source: §3.2 · p. 5

4. Experiments

Glance evaluates phase-aware Slow–Fast LoRA acceleration on text-to-image benchmarks and ablates its timestep allocation and one-sample training behavior. It reaches nearly the quality of 50-step base models at 5× speed, while tests also identify text-rendering weaknesses.

4.1. Experimental Setup

  • Evaluation spans six prompt benchmarks and compares Glance with few-step students distilled from the same teachers, plus one-sample uniform-timestep LoRA.The evaluation includes COCO, HPSv2, OneIG-Bench, GenEval, DPG-Bench, and LongText-Bench.Source: §4.1 · p. 5, §4.2 · p. 6

4.2. Main Results

  • 5× faster than 50-step base models, Glance achieves nearly the same generation quality across benchmarks, with competitive FID, CLIP, and HPSv2 scores.Visual comparisons show preserved global semantics and color composition at eight steps; adding steps restores fine textures and small structures.Source: §4.2 · pp. 6–7

4.3. Ablation Study

  • Phase-aware Slow3–Fast5 ranks highest across metrics, while uniform single-LoRA performs worst; early Slow-LoRA contributes more to final image quality.The ablation compares phase-aware assignment with base substitutions and single-LoRA variants.Source: Table 3 · p. 7, §4.3 · p. 7
  • Increasing training pairs from 1 to 10 and then 100 brings no notable performance gains, while the style score slightly declines.The authors note that data quality and phase alignment may matter more than simply increasing dataset size.Source: Table 4 · p. 7, §4.3 · p. 7
  • More LoRA-equipped timesteps steadily improve overall performance, with a clear gain on the text metric.The comparison holds data selection and scale fixed at one text–image pair.Source: Table 5 · p. 7, §4.3 · p. 7

4.4. Discussion

  • Across single-image settings, performance differences are minor; Fox is strongest, Bookstore weakest, real-world OOD remains competitive, and Gaussian-noise-only training fails.The Fox-trained LoRA improves across evaluation metrics, but its gap over other coherent-image settings is modest.Source: §4.4 · pp. 7–8, Table 6 · p. 8
  • Text rendering is weakest for extremely dense or tiny text, while shorter text occupying more image area can be reproduced faithfully.The reported deficits appear on Text-Render and LongText-Bench; fine-grained textual details are harder to capture under few-step constraints.Source: §4.4 · p. 8, Figure 7 · p. 8

5. Conclusion

Glance uses phase-aware Slow–Fast adapters to accelerate diffusion inference. The conclusion reports 5× speedup with comparable visual fidelity and generalization to unseen prompts despite training with as few as one image.

5. Conclusion

  • Glance achieves an 5× speed-up over the base model while maintaining comparable visual fidelity and generalizing to unseen prompts after training with as few as one image.The conclusion describes the design as lightweight and reports training with a few GPU hours.Source: §5 · pp. 8–9

A. Qwen-Image-Edit task

The paper asks whether its phase-aware distillation strategy transfers beyond text-to-image generation and evaluates it on image editing.

A. Qwen-Image-Edit task

A.1. Training Setup

For the Qwen-Image-Edit task, the authors train a new pair of Slow-LoRA and Fast-LoRA adapters on the base model using one image.

A.1. Training Setup

  • On the base Qwen-Image-Edit model, new Slow-LoRA and Fast-LoRA adapters are trained on a single image with instruction “Put a hat on the woman’s head.” and learning rate 2e-4.Source: §A.1 · p. 12

A.2. Results and Analysis

The section reports one-shot editing behavior and tests whether phase-aware adaptation generalizes to remote-sensing imagery. A single remote-sensing example supplies domain-specific aerial viewpoints on unseen prompts.

A.2. Results and Analysis

  • On unseen editing instructions, Glance inserts realistic glasses while preserving the original subject’s appearance and identity.For hat edits, it also preserves non-edited regions and generates varied, context-appropriate hats rather than copying the training hat.Source: §A.2 · p. 12
  • The authors identify controllable generation, inpainting, and video editing as domains for future exploration of the framework.They conclude that phase-aware LoRA acceleration extends beyond text-to-image synthesis with minimal adaptation.Source: §A.2 · p. 12

B.1. Zero-Shot Behavior

  • Applying Glance trained on natural images to remote sensing yields plausible structures, but its perspectives resemble natural images rather than overhead views.The authors conclude that zero-shot adaptation does not inject domain-specific geometric priors.Source: §B.1 · p. 13

B.2. One-Shot Slow-Fast LoRA Adaptation

  • After adaptation on one remote-sensing sample, Glance produces proper aerial viewpoints and characteristic structural layouts on unseen prompts.The adaptation retains the acceleration benefits of the phase-aware LoRA design.Source: §B.2 · p. 13, Figure 9 · p. 13

C. More Implementation details

The implementation uses model-specific LoRA placements and settings for Qwen-Image and FLUX. Both models use the same 60-step, single-sample training setup.

C.1. Glance (Qwen-Image)

  • For Qwen-Image, LoRA is placed in MM-DiT attention projections, multimodal MLPs, and modulation pathways, with rank 32 and scaling 128.The listed modules include to q, to k, to v, add q proj, add k proj, add v proj, and to out.0, as well as multimodal MLP and modulation modules.Source: §C.1 · p. 13
  • Qwen-Image training uses one sample for 60 steps, equivalent to 60 epochs, with batch size 1 and bf16 mixed precision.The setup also uses AdamW, gradient clipping, 8-bit Adam, quantized weight loading, and precomputed image and text embeddings.Source: §C.1 · p. 13

C.2. Glance (FLUX)

  • For FLUX, LoRA is applied to MM-DiT attention projections with rank 16 and scaling 64.The listed modules are to q, to k, to v, and to out.0; its training setup also uses one sample for 60 steps.Source: §C.2 · pp. 13–14

D. More Qualitative Results

The supplementary results include uncurated prompt batches for Qwen-Image and FLUX, plus a visual comparison of FLUX sampling configurations. The future-work discussion proposes prompt-aware switching and zero-shot distillation.

D.1. Glance (Qwen-Image)

E. Future Work

  • Future work proposes switching experts according to prompt difficulty and exploring zero-shot distillation using diffusion priors, self-consistency, or synthetic trajectories.The proposed switching would keep challenging prompts longer in the slow phase and move simpler prompts earlier to the fast phase.Source: E. Future Work · p. 14

D.2. Glance (FLUX)

  • Figure 12 compares FLUX configurations from the same initial noise; Slow-Fast preserves semantic fidelity under acceleration, while more steps progressively enhance fine details.Figures 13 and 14 provide additional uncurated FLUX results for OneIG-Bench and DPG prompts.Source: §D.2 · p. 14, Figure 12 · p. 16, Figure 13 · p. 17
Loading 2512.02899v2…