PaperlayerView exact revision on arXiv

Source-linked AI summary

Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflection

Lichen Bai, Shitong Shao, Zikai Zhou, Zipeng Qi, Zhiqiang Xu, Haoyi Xiong, Zeke Xie

arXiv:2412.10891v2cs.CVcs.LG

TL;DR

Text-to-image diffusion models can struggle to preserve image quality while aligning to challenging prompts. The paper alternates denoising and inversion to exploit their guidance gap, and reports improvements across settings, including reaching standard sampling’s best performance with 36% less computational time.

  • Problem

    For challenging prompts, existing text-to-image diffusion models can struggle to maintain both image quality and prompt-image alignment.

    Source: §1 · p. 2

  • Method

    Z-Sampling repeatedly alternates denoising and inversion to exploit their guidance gap and accumulate prompt-related semantic information along the sampling path.

    Source: §1 · pp. 2–3

  • Results

    Across varied settings, models self-improve significantly with Z-Sampling; with 36% less computational time, it reaches the best performance of standard sampling.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The paper reports that Z-Sampling can further improve orthogonal methods, including Diffusion-DPO, while using limited computational cost.

    Source: §1 · p. 3

    Z-Sampling relies on deterministic inversion and was studied only for text-to-image models.

    Source: §5 · p. 10

Abstract

from arXiv · show

Diffusion models, the most popular generative paradigm so far, can inject conditional information into the generation path to guide the latent towards desired directions. However, existing text-to-image diffusion models often fail to maintain high image quality and high prompt-image alignment for those challenging prompts. To mitigate this issue and enhance existing pretrained diffusion models, we mainly made three contributions in this paper. First, we propose diffusion self-reflection that alternately performs denoising and inversion and demonstrate that such diffusion self-reflection can leverage the guidance gap between denoising and inversion to capture prompt-related semantic information with theoretical and empirical evidence. Second, motivated by theoretical analysis, we derive Zigzag Diffusion Sampling (Z-Sampling), a novel self-reflection-based diffusion sampling method that leverages the guidance gap between denosing and inversion to accumulate semantic information step by step along the sampling path, leading to improved sampling results. Moreover, as a plug-and-play method, Z-Sampling can be generally applied to various diffusion models (e.g., accelerated ones and Transformer-based ones) with very limited coding and computational costs. Third, our extensive experiments demonstrate that Z-Sampling can generally and significantly enhance generation quality across various benchmark datasets, diffusion models, and performance evaluation metrics. For example, DreamShaper with Z-Sampling can self-improve with the HPSv2 winning rate up to 94% over the original results. Moreover, Z-Sampling can further enhance existing diffusion models combined with other orthogonal methods, including Diffusion-DPO.

1 INTRODUCTION

The introduction frames prompt alignment and image fidelity on challenging prompts as a persistent sampling challenge. It proposes alternating denoising and inversion to use their guidance gap to accumulate semantic information and improve generation.

1 INTRODUCTION

  • Strong guidance can improve alignment on challenging prompts but may reduce fidelity, cause mode collapse, and accumulate sampling errors.Existing manifold constraints may compromise diversity, while varying guidance scales can produce unnatural outputs.Source: §1 · p. 2
  • Latents that generate a concept unconditionally tend to produce higher-quality results for prompts about that concept.The introduction illustrates this with flower-related prompts and seed 21, and describes semantic-rich latents as useful even without conditional guidance.Source: §1 · p. 2, Figure 2 · p. 2, Figure 3 · p. 2
  • Strong denoising guidance paired with weak inversion guidance creates a gap that can inject prompt-related semantic information into the latent.The introduction presents diffusion self-reflection as an unreported mechanism for self-improving diffusion sampling.Source: §1 · p. 2
  • Z-Sampling alternates denoising and inversion along the sampling path, using repeated self-reflection to capture and accumulate semantic information.The paper says the guidance gap can capture latent semantics that matter to generation quality and prompt-image alignment; the method offers flexible control and applies across diffusion architectures with limited coding costs.Source: §1 · pp. 2–3
  • 36% less computational time lets Z-Sampling reach the best performance of standard sampling, alongside gains across benchmarks, models, and metrics.The paper also reports particular improvements for challenging prompts.Source: §1 · p. 3

2 PRELIMINARIES

The preliminaries describe diffusion generation and inversion as mappings between Gaussian noise and target data. They also introduce classifier-free guidance and express the forward and reverse processes as sequences of single-step mappings.

2 PRELIMINARIES

  • Figure 4 contrasts weak-guidance sampling, strong-guidance sampling, and Z-Sampling with diffusion self-reflection.Source: Figure 4 · p. 3
  • A diffusion model maps an initial Gaussian latent to target data under a prompt and guidance scale, with inversion mapping data back to noise.The text denotes these processes by Φ and Ψ, respectively.Source: §2 · p. 3
  • The paper decomposes the denoising mapping Φ into T single-step mappings along the sampling process.The inversion process Ψ is likewise decomposed into timestep-specific mappings.Source: §2 · p. 4
  • The inversion-score approximation can make Φ and Ψ inverse functions when its error is negligible.The text attributes this result to Mokady et al. (2023).Source: §2 · p. 4
  • Classifier-free guidance interpolates conditional and unconditional score predictions, allowing the guidance scale to be adjusted during inference.The denoising and inversion processes use guidance scales γ1 and γ2, respectively.Source: §2 · p. 4

3 METHODOLOGY

The methodology treats latent variables as carrying prompt-relevant semantics, then uses the guidance difference between denoising and inversion to inject and accumulate that information through alternating steps.

3.1 LATENTS WITH RELEVANT SEMANTIC INFORMATION

  • Latents that generate a concept unconditionally tend to produce higher-quality images for related prompts, supporting the view that latents carry relevant semantic information.The paper illustrates this with a latent that generates flowers unconditionally and then performs better with flower-related prompts.Source: §3.1 · p. 4

3.2 CAPTURE SEMANTIC INFORMATION FROM THE GUIDANCE GAP

  • Using stronger guidance for denoising than inversion yields an inverted latent that the authors argue contains more prompt-related semantics than the original latent.They compare a typical denoising guidance scale of 5.5 with weak or unconditional inversion guidance of 0.Source: §3.2 · p. 5

3.3 ZIGZAG DIFFUSION SAMPLING

  • Vanilla inversion injects information end-to-end, but its summed per-step semantic gains may cancel and leave little net gain.The method measures vanilla injection through the difference between the original and inverted initial latents.Source: §3.3 · p. 5
  • Z-Sampling alternates denoising with inversion at each timestep and replaces the denoised latent with its inverted counterpart to inject semantic information.This repeated denoising-and-inversion operation is termed diffusion self-reflection.Source: §3.3 · p. 5
  • The cumulative semantic gain in Z-Sampling can exceed end-to-end gain because the stepwise gains accumulate rather than cancel.The theoretical analysis also states that the gain depends on the guidance gap and is approximately standard sampling when that gap is zero.Source: §3.3 · p. 6
  • At early generation stages, Z-Sampling produces more semantically focused cross-attention regions for entity tokens such as “dog.”The figure’s zigzag path shows sharper attention with relatively clear boundaries.Source: §3.3 · p. 6, Figure 5 · p. 6

4 EMPIRICAL ANALYSIS

The experiments evaluate Z-Sampling across datasets, metrics, and diffusion architectures. They report stronger results than standard sampling, including at matched generation time, and test its robustness and compatibility with other methods.

4.1 EXPERIMENTS SETTING

  • Experiments use Pick-a-Pic, DrawBench, and GenEval, multiple human-preference and image-quality metrics, and SD2.1, SDXL, Hunyuan-DiT, and DreamShaper-xl-v2-turbo.The baselines include standard sampling and Resampling; the study also discusses other baselines in the appendix.Source: §4.1 · pp. 6–7

4.2 MAIN EXPERIMENTS

  • Z-Sampling improves performance across nearly all metrics, exceeds 88% winning rates on HPS v2, and improves GenEval counting, two-object relations, and color attribution.The experiments include Pick-a-Pic, DrawBench, and GenEval, and evaluate multiple diffusion architectures.Source: Figure 6 · p. 7, Table 1 · p. 7, Table 2 · p. 8
  • Under extremely high classifier-free guidance, Z-Sampling can balance image quality and prompt adherence while suppressing artifacts and oversaturation.Source: §4.2 · p. 8
  • Z-Sampling further enhances AYS-Sampling and Diffusion-DPO, including the training-based Diffusion-DPO method.AYS-Sampling is a training-free scheduler strategy, while Diffusion-DPO is a training-based method.Source: §4.2 · pp. 8–9, Table 4 · p. 8, Table 5 · p. 8
  • As the denoising–inversion guidance gap narrows, Z-Sampling’s gains diminish; a zero gap approximately recovers standard sampling, while a negative gap can produce negative gain.The robustness analysis varies inversion guidance while fixing denoising guidance at 5.5.Source: §4.2 · p. 9, Figure 7 · p. 9, Figure 8 · p. 9
  • On SDXL, increasing zigzag steps from 0 to 25 raises Pick-a-Pic winning rate from 50% to 75%, while increasing them from 25 to 50 raises it to 80%.The authors conclude that zigzag operations are more effective during early denoising.Source: §4.2 · p. 9, Figure 9 · p. 10
  • With 36% less computational time, Z-Sampling surpasses the best standard-sampling performance on average HPS v2 for Pick-a-Pic.Figure 10 compares average HPS v2 against average time per image.Source: Figure 10 · p. 10, §4.2 · p. 10

5 DISCUSSION AND CONCLUSION

The discussion identifies scope and efficiency limits, while the conclusion presents Z-Sampling as a flexible method that improves generation and can enhance orthogonal methods. The section also points to the experimental details covering benchmarks and metrics.

5 DISCUSSION AND CONCLUSION

  • Z-Sampling relies on semantic information gain through deterministic inversion, limiting it to deterministic samplers such as DDIM; extending it to SDE-based diffusion remains future work.The study is limited to text-to-image models, and zigzag steps add computational time.Source: §5 · p. 10
  • The conclusion characterizes Z-Sampling as accumulating semantic information through zigzag self-reflection and improving generation across models and settings.It also reports that Z-Sampling can enhance orthogonal methods and describes the approach as flexible, additive, and low in coding and computation costs.Source: §5 · p. 10

A EXPERIMENTAL DETAILS

A.1 DATASETS

The appendix describes the evaluation datasets and metrics, then situates Z-Sampling among prior work on latent semantics and diffusion sampling strategies.

A.1 DATASETS

  • Pick-a-Pic uses user preferences over paired generated images; DrawBench covers 11 categories with about 200 prompts, GenEval tests compositional properties with 553 prompts, and PartiPrompts contributes a random 100-prompt sample.The Pick-a-Pic test set uses the first 100 prompts.Source: §A.1 · p. 14

A.2 METRICS

  • AES scores visual quality, while PickScore, ImageReward, and HPS v2 are described as human-preference scoring models.PickScore predicts preferences; ImageReward is trained on expert comparisons, and HPS v2 uses a large human-preference dataset.Source: §A.2 · p. 15

A.3 BASELINES

  • The compared baselines include adaptive semantic guidance, Diffusion-DPO fine-tuning, optimized sampler timesteps, energy guidance, and CFG++ manifold constraints.The appendix also describes SEG and Smoothed Energy Guidance as using energy-landscape and intermediate self-attention information.Source: §A.3 · p. 15

B RELATED WORKS

  • Prior work studies semantic information in noise latents and explicit priors; this work instead investigates guidance-based semantic integration without reference data.Source: B RELATED WORKS · pp. 15–16
  • Earlier sampling strategies include repeated resampling, inversion-based editing, adaptive guidance, and manifold constraints, with limitations noted for prompt adherence, adaptability, or reference-data requirements.The discussion also identifies transfer of Z-Sampling to masked generative models and IV-Mixed Sampler as possible extensions.Source: B RELATED WORKS · p. 16

C.1 LATENTS WITH SEMANTIC INFORMATION

The supplementary examples support the claim that latents can carry concept-related semantic information, including latents obtained through inversion. They also report stronger prompt-matched generation from the corresponding inverted latents.

C.2 INVERSION MAKES GOOD LATENTS

  • With DDIM inversion at guidance scale 0, the authors obtained latents from cat and spider images and hypothesized that each carried the corresponding concept information.Source: §C.2 · p. 16
  • The cat-derived inverted latent performed better with cat prompts, while the spider-derived latent performed better with spider prompts.The authors present this as empirical support that inverted latents matter.Source: §C.2 · p. 16

D SUPPLEMENTARY EXPERIMENTAL RESULTS

D.1 SUPPLEMENTARY QUANTITATIVE RESULTS

Supplementary comparisons report stronger results for Z-Sampling on additional image benchmarks and video generation. The reported backtracking-step comparison favors the one-step zigzag setting.

D.1 SUPPLEMENTARY QUANTITATIVE RESULTS

  • Z-Sampling shows higher performance on 100 PartiPrompts prompts with SDXL and higher generation quality on MS-COCO 2014 with DreamShaper-xl-v2-turbo.The MS-COCO evaluation uses 30K prompts and a four-step distilled model.Source: §D.1 · pp. 16–17, Table 6 · p. 18, Table 7 · p. 18
  • On GenEval, Z-Sampling has higher average scores than Resampling and Diffusion-DPO.Source: §D.1 · p. 18, Table 8 · p. 18
  • The SEG comparison is reported for SDXL; the CFG++ comparison uses ω = 0.5 to correspond to Z-Sampling's CFG scale of 5.5.The authors state that a fair comparison is otherwise not possible because CFG++ restricts its CFG scale to 0.0–1.0.Source: §D.1 · p. 18, Table 9 · p. 18, Table 10 · p. 18
  • On Chronomagic-Bench-150, Z-Sampling outperforms AnimateDiff and FreeInit on UMT-FVD, UMT-SCORE, and GPT4o-MTSCORE.The reported setup uses AnimateDiff with γ1 = 7.5 and γ2 = 0.Source: §D.1 · p. 18, Table 11 · p. 19
  • The best reported performance occurs at backtracking stepsize k=1; performance deteriorates as k increases.The authors relate larger k to a more end-to-end approach and an increased error term τ2.Source: §D.1 · p. 19, Table 12 · p. 19

D.2 SUPPLEMENTARY QUALITATIVE RESULTS

Supplementary results show Z-Sampling improving prompt interpretation across qualitative categories and winning-rate comparisons. Under stronger guidance, it continues to improve results while standard sampling develops artifacts and oversaturation.

D.2 SUPPLEMENTARY QUALITATIVE RESULTS

  • Z-Sampling better interprets stylistic descriptions, object positions, and color attributes, aligning images more closely with prompts.Examples include more realistic Mario characters and relationships such as underneath, on top of, and on the right of.Source: §D.2 · p. 19, Figure 14 · p. 19, Figure 15 · p. 20
  • Z-Sampling handles counting and object co-occurrence, including specified quantities, multi-object composites, and counterfactual scenes.Examples include three suitcases, four buses, a mouse with a bowl, and an elephant in the sea.Source: §D.2 · p. 19, Figure 17 · p. 20, Figure 18 · p. 21

D.3 WINNING RATES COMPARISON

  • Z-Sampling's winning rates exceed 50% across the reported metrics and settings, with HPS v2 especially favorable.The comparison covers different models and denoising-step counts; the blue bars denote Z-Sampling and orange bars standard sampling.Source: §D.3 · p. 22, Figure 19 · p. 22, Figure 20 · p. 22

D.4 PERFORMANCE OF Z-SAMPLING UNDER HIGH CFG SCALE

  • At high guidance scales, Z-Sampling continues to yield positive gains, while standard sampling develops oversaturation and artifacts.For DreamShaper-xl-turbo-v2, standard sampling performs best at the recommended γ1 = 3.5; the comparison also includes SDXL.Source: §D.4 · p. 23, Table 13 · p. 23, Figure 23 · p. 24

D.5 ADDITIONAL EXPERIMENTS ON VARIOUS GUIDANCE SCALES

Additional experiments report improved performance across three models when guidance scales are grid-searched, and show that Z-Sampling mitigates high-guidance visual degradation. The accompanying analysis examines how inversion approximation error affects performance.

D.5 ADDITIONAL EXPERIMENTS ON VARIOUS GUIDANCE SCALES

  • As guidance increases, standard sampling shows more artifacts and oversaturation, while Z-Sampling mitigates these issues.The qualitative observation is reported for SDXL and similarly for DreamShaper.Source: §D.5 · p. 25

E ANALYSIS OF THE APPROXIMATION ERROR TERM

D.5 ADDITIONAL EXPERIMENTS ON VARIOUS GUIDANCE SCALES

  • Z-Sampling outperforms standard sampling on Pick-a-Pic across SDXL, SD2.1, and DreamShaper-xl-v2-turbo with grid-searched guidance scales.The reported comparison searches guidance ranges separately by model; DreamShaper with Z-Sampling can outperform SDXL, unlike DreamShaper with standard sampling.Source: §D.5 · p. 25, Table 14 · p. 26

E.1 UNCERTAINTY AND STOCHASTIC SAMPLERS

The experiments show that inversion accuracy matters: stochastic-sampler errors can reduce Z-Sampling's effectiveness, while added approximation error degrades generation quality. With the semantic term present, the step-by-step method performs better than the end-to-end method.

E.1 UNCERTAINTY AND STOCHASTIC SAMPLERS

  • Inversion inaccuracies from stochastic Euler(a) reduce Z-Sampling's effectiveness, whereas deterministic Euler yields gains over its corresponding baseline.With deterministic Euler, PickScore increases from 20.3643 to 20.9639; the authors attribute Euler(a)'s declines on most metrics to increased approximation error and departure from the data manifold.Source: §E.1 · p. 26, Table 15 · p. 26

E.2 THE INCREASE IN APPROXIMATION ERROR RESULTS IN NEGATIVE GAINS

  • With semantic term τ1 present, both methods yield positive gains, and the step-by-step method performs better.The comparison reports PickScore for SDXL on Pick-a-Pic because τ1 and τ2 are mixed and exact approximation error cannot be reported.Source: §E.2 · p. 27, Table 17 · p. 27

E.3 ARTIFICIALLY INTRODUCING GAUSSIAN ERROR

  • Increasing the magnitude of the artificially introduced Gaussian error worsens Z-Sampling's generation performance.The table reports generation quality as the error coefficient increases.Source: §E.3 · p. 28, Table 18 · p. 28

F PROOFS

The proofs relate Z-Sampling to end-to-end semantic injection and formalize semantic injection through the guidance gap. They also contrast the end-to-end algorithm with the step-by-step approach.

F PROOFS

  • The proof analysis states that Z-Sampling accumulates semantic gain across steps, avoiding the cancellation risk of end-to-end injection.The visual comparison describes timely semantic injection at each step and a more pronounced effect.Source: Figure 27 · p. 30, F PROOFS · p. 30
  • The guidance gap δγ = γ1 − γ2 can regulate optimization magnitude and intensity, while the prompt-conditioned and unconditional network-output difference also affects Z-Sampling.Source: F PROOFS · p. 31

G THE END-TO-END SEMANTIC INJECTION ALGORITHM

Related papers

Loading 2412.10891v2…