PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncovering the Generative Landscape of Diffusion Models: A Spectral-Guided Quality-Diversity Diffusion Framework

Hannah Janmohamed, Joao Correia, Penousal Machado, Antoine Cully

arXiv:2610.15724v1cs.LGcs.AI

TL;DR

Standard diffusion sampling favors high-likelihood regions, limiting access to the diversity needed for portfolios of strong outputs. The paper combines Quality-Diversity optimisation with diffusion models and introduces spectral-guided mutation. In text-to-image evaluation, it reports the lowest FID across 12 baselines and two architectures.

  • Problem

    Standard diffusion sampling is biased toward high-likelihood regions, while design and creative tasks can require diverse portfolios rather than a single high-likelihood output.

    Source: §1 · p. 2

  • Method

    Spectral-Guided Quality-Diversity Diffusion optimises latent noise and conditioning embeddings, measures diversity with pretrained foundation-model embeddings, and mutates along principal Jacobian directions.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    The method consistently achieves the lowest FID across 12 baselines in text-to-image generation, across two architectures.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    In text-to-image generation, the framework uncovers diverse, high-fidelity portfolios of solutions.

    Source: §6 · p. 10

    The approach assumes access to pretrained embedding models that serve as proxies for semantic distinctiveness.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Denoising Diffusion Probabilistic Models have emerged as powerful generative priors across domains ranging from molecular design to high-resolution image synthesis. However, standard diffusion sampling is inherently biased toward high-likelihood regions, often limiting the diversity of outputs. In many domains, a single, high-likelihood sample is insufficient. Instead, diverse portfolios of high-performing solutions are required, motivating methods that uncover a model's full expressive range. Here we show that integrating Quality-Diversity (QD) optimisation with diffusion models enables the discovery of diverse, high-performing outputs. We first propose Quality-Diversity Diffusion, a framework that adapts QD to diffusion models by directly optimising the model's latent noise and conditioning embeddings and leveraging embedding models to quantify diversity. Second, we introduce a novel h-space Spectral Mutation operator, which guides search along directions of maximal semantic sensitivity by analysing principal singular vectors of the model's input-to-activation Jacobian. We evaluate our combined approach, Spectral-Guided Quality-Diversity Diffusion, on text-to-image generation. Against 12 baselines, our method consistently achieves the lowest FID score, demonstrating an excellent balance of image quality and diversity.

1 INTRODUCTION

Standard diffusion sampling favors high-likelihood regions, while many applications need diverse portfolios rather than a single strong sample. The paper adapts Quality-Diversity optimisation to diffusion models and evaluates a spectral-guided approach for text-to-image generation.

1 INTRODUCTION

  • Standard diffusion sampling favors high-likelihood modes, while Quality-Diversity optimisation seeks archives of high-performing, behaviourally diverse solutions.Diverse portfolios are motivated by the need for structurally distinct candidates in design and varied visual interpretations in text-to-image generation.Source: §1 · p. 2
  • Semantic or functional diversity is task-dependent and difficult to specify manually, while diffusion models induce high-dimensional search spaces that require efficient exploration.Pretrained foundation-model embeddings are proposed as proxies for artefact distinctiveness.Source: §1 · p. 2
  • Quality-Diversity Diffusion optimises diffusion models’ latent noise and conditioning embeddings, using pretrained foundation-model embeddings to measure diversity.This avoids the hand-defined, low-dimensional features required by previous QD approaches, supporting domains where explicit diversity axes are difficult to specify.Source: §1 · p. 2
  • The h-space Spectral Mutation operator perturbs along principal singular vectors of the model’s input-to-activation Jacobian to explore directions of maximal semantic sensitivity.The authors motivate this operator by the inefficiency of traditional evolutionary operators in high-dimensional latent spaces.Source: §1 · p. 2
  • Across 12 baselines, the method achieves the lowest FID score in text-to-image generation.The evaluation uses a representative subset of Parti Prompts.Source: §1 · p. 2

2 BACKGROUND

The background describes diffusion generation as iterative denoising and introduces Quality-Diversity algorithms as a way to preserve both performance and behavioural variety. It outlines grid-based MAP-Elites and an unstructured alternative for high-dimensional feature spaces.

2.1 DIFFUSION MODELS

  • The diffusion framework combines a fixed forward chain that adds Gaussian noise with a parameterized reverse chain that learns to denoise data.At each reverse step, the model estimates the mean and variance for transitioning to a less noisy state.Source: §2.1 · pp. 2–3
  • A diffusion model generates an image by sampling Gaussian noise and iteratively denoising it over T timesteps; Figure 2 visualizes this reverse process.Each reverse transition is approximated by a Gaussian when forward noise increments are sufficiently small, and conditioning embeddings can guide outputs toward desired attributes.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 QUALITY-DIVERSITY ALGORITHMS

  • Quality-Diversity algorithms seek high-performing solutions across distinct feature-space regions, and MAP-Elites maintains an archive by retaining the best candidate in each grid cell (Cully and Demiris, 2018; Mouret and Clune, 2015).The feature function maps candidates into the space where diversity is sought.Source: §2.2 · p. 3
  • Dominated Novelty Search uses an unstructured archive and ranks solutions by fitness and novelty, avoiding grid discretisation in high-dimensional feature spaces.Its score averages distances to higher-fitness nearest neighbours, preserving solutions that are distinctive or high-performing.Source: §2.2 · p. 3

3 METHODOLOGY

Quality-Diversity Diffusion searches diffusion-model inputs with an evolutionary archive to find collections that are both high-performing and diverse. Its spectral mutation operator uses hidden-space sensitivity directions to guide latent changes, then maps them back to initial noise for archiving.

3.1 FRAMEWORK

  • The framework varies diffusion-model genotypes, generates offspring, evaluates fitness, embeds artifacts, and adds qualifying candidates to a Dominated Novelty Search archive.The genotype is the tuple (xT, c): initial Gaussian noise and a conditioning embedding.Source: §3.1 · p. 4, Figure 3 · p. 4
  • High-dimensional pretrained embeddings proxy semantic distinctiveness, while fitness thresholding prevents highly novel but invalid candidates from entering the archive.A candidate is added only when its fitness exceeds a minimum threshold.Source: §3.1 · p. 4
  • The operator finds latent perturbations that maximize changes in hidden activations by analyzing the Jacobian from timestep latents to the model’s internal h-space.The leading right singular vector gives the direction producing the largest internal-feature change for a fixed step size.Source: §3.2 · p. 5
  • It perturbs the latent along a principal semantic direction, using the model’s hidden-space mapping in either U-Net or DiT architectures.The step size is sampled uniformly, and the mapping can correspond to an intermediate U-Net convolutional block or DiT transformer layer.Source: §3.2 · p. 5
  • The same spectral analysis can mutate conditioning embeddings as well as latent noise, allowing joint optimization of both genotype components.Source: §3.1 · p. 4, §3.2 · p. 5
  • Mutations occur at an intermediate timestep because semantic edits are more effective there, but the changed latent is deterministically inverted to initial noise for compatible archiving.The text identifies typical effective edit timesteps as t* in [0.5T, 0.7T]; the caption depicts propagation, perturbation, and inversion.Source: §3.2 · p. 5, Figure 4 · p. 5
  • Because exact Jacobians are intractable, power iteration with finite differences and automatic differentiation approximates top singular vectors for offspring perturbations.The sensitivity directions are computed once per parent and reused across a batch of offspring.Source: §3.2 · p. 6

4 EXPERIMENTAL SETUP

The evaluation uses text-to-image tasks with prompt alignment as fitness and DreamSim embeddings as features. It compares framework and mutation alternatives under specified evaluation budgets, reporting FID alongside fitness, diversity, and archive size.

4.1 TASK

  • SDXL-Turbo is evaluated on 36 Parti Prompts tasks, with Flux tested on 12 prompts; CLIP score measures prompt-image fidelity and DreamSim supplies 768-dimensional semantic features.Genotypes include all available model inputs, with batch size split equally among them.Source: §4.1 · p. 6

4.2 BASELINES

  • Framework baselines compare archive designs and fitness-only optimizers, while mutation baselines include random, diversity-oriented, evolutionary, and intermediate-layer Gaussian operators.CVT-ME and QDHF use the proposed spectral mutation to isolate archive effects; the intermediate-layer Gaussian baseline tests spectral mutation against unspecialized intermediate perturbation.Source: §4.2 · pp. 6–7

4.3 EVALUATION METRICS

  • SDXL-Turbo algorithms receive 2,000 evaluations and Flux algorithms 400; evaluation reports FID, total fitness, mean pairwise embedding distance, and number of solutions.FID combines quality and diversity relative to real images and is omitted for algorithms that fail to discover a significant number of solutions.Source: §4.2 · p. 7, §4.3 · p. 8

5 RESULTS

The method achieves the lowest FID scores on both SDXL-Turbo and Flux in the framework and variation-operator evaluations. Comparisons with other methods reveal trade-offs in fitness, diversity, and image quality.

5 RESULTS

  • FID is 81.90 on SDXL-Turbo and 137.58 on Flux, the lowest scores across both models.The comparison is reported for the Spectral-Guided approach in the framework evaluation.Source: Table 1 · p. 8, §5.1 · p. 8
  • CMA-MAE records higher Diversity than the proposed method, but the differences are not significant and its images can be corrupted or prompt-inconsistent.The authors report 5.92 solutions on Flux and interpret the diversity as arising from corrupted or irrelevant outputs rather than genuine semantic variation.Source: §5.1 · p. 8
  • Adam has the highest Total Fitness (23.71) but low Diversity (0.15), while Sep-CMA-ES has comparable Total Fitness and significantly lower Diversity on both models.Sep-CMA-ES Diversity is 0.20 vs 0.47 on SDXL-Turbo and 0.095 vs 0.472 on Flux; the authors also report visual artefacts for both fitness-maximising methods.Source: §5.1 · pp. 8–9
  • On SDXL-Turbo, CVT-ME has lower Total Fitness (2.45 vs 20.8) and Diversity (0.31 vs 0.47) than the proposed method.On Flux, CVT-ME has marginally higher Diversity (0.497 vs 0.472; p = 1.0) but lower Total Fitness (19.5 vs 1.97). It averages 6.9 solutions per prompt.Source: §5.1 · p. 8, Figure 5 · p. 9
  • Against QDHF on SDXL-Turbo, the method has higher fitness (16.1 vs 20.8) and diversity (0.28 vs 0.47).The authors suggest that projecting high-dimensional embeddings into lower-dimensional spaces may lose information and limit retention of diverse, high-performing solutions.Source: §5.1 · p. 9

5.2 VARIATION OPERATOR RESULTS

  • The method also achieves the lowest FID among all tested variation operators on both SDXL-Turbo and Flux.The authors present this as evidence that h-space Spectral Mutation balances quality and diversity.Source: Table 3 · p. 9, §5.2 · p. 9
  • On SDXL-Turbo, the method exceeds Intermediate-layer Gaussian in fitness (20.78 vs 19.59) and diversity (0.473 vs 0.341); on Flux, neither metric differs significantly.The Flux diversity scores are 0.498 vs 0.489 and Total Fitness scores are 19.46 vs 19.85; qualitative review reports heavily corrupted images from this baseline on both models.Source: §5.2 · pp. 9–10
  • Figure 6 shows Pareto dominance over all tested operators on both models except Intermediate-Gaussian on Flux.For Flux, differences on both metrics are significant for Random Seeds and Imagebreeder; for other baselines, fitness differences are not significant while diversity differences are.Source: §5.2 · p. 9, Figure 6 · p. 10

6 CONCLUSION AND FUTURE WORKS

The paper presents Spectral-Guided Quality-Diversity Diffusion as a way to use pretrained diffusion models as searchable landscapes for evolutionary optimisation. Its text-to-image evaluation reports performance across two architectures, while cross-domain generalisability remains unvalidated.

6 CONCLUSION AND FUTURE WORKS

  • The framework repurposes pretrained diffusion models as searchable landscapes for evolutionary optimisation.Source: §6 · p. 10
  • On text-to-image generation, the method outperforms 12 baselines across two architectures in FID, with a reported balance of image quality and diversity.Source: §6 · p. 10
  • Cross-domain generalisability remains to be validated empirically; future work will extend the framework to scientific discovery, including biological sequence design.Source: §6 · p. 10

AI ASSISTANCE STATEMENT

The authors describe using generative AI for writing and research support, but not for synthetic datasets or mathematical proofs. They report reviewing and taking responsibility for AI-assisted work.

AI ASSISTANCE STATEMENT

  • Generative AI tools supported writing, literature retrieval and discovery, research ideation and execution, and drafting paper sections.Source: AI ASSISTANCE STATEMENT · p. 10
  • The authors state that generative AI was not used to generate synthetic datasets or prove mathematical claims.Source: AI ASSISTANCE STATEMENT · p. 10
  • The authors reviewed AI-assisted work, verified suggested literature, proofread AI-assisted text, and accept responsibility for the final content.Source: AI ASSISTANCE STATEMENT · p. 10

SUPPLEMENTARY MATERIALS

The supplementary material outlines standard QD archive procedures and related approaches, then details how the framework adapts high-dimensional search and spectral mutation to diffusion models. It also records the experimental settings used for the SDXL-Turbo and Flux implementations.

A.1 MAP-ELITES PSEUDOCODE

  • MAP-Elites selects archive parents, mutates offspring, evaluates fitness and features, and replaces a cell occupant only when the cell is empty or the offspring has higher fitness.The pseudocode begins from randomly sampled solutions and an archive with empty cells for discretised features.Source: §A.1 · p. 14

A.2 DNS PSEUDOCODE

  • DNS ranks candidates by mean feature-space distance to fitter neighbours, then selects the top-scoring solutions from the combined parent and offspring population.Its pseudocode initializes a population of N and uses k-nearest neighbours for scoring.Source: §A.2 · p. 14

B.1 EVOLUTIONARY ALGORITHMS IN GENERATIVE DESIGN; B.2 QUALITY-DIVERSITY FOR GENERATIVE DESIGN

  • Earlier evolutionary approaches include single-objective latent optimisation, hand-designed image operators, and prompt evolution, while the framework aims to remain domain-agnostic rather than evolve text prompts.Prior QD applications to generative models often relied on hand-defined low-dimensional features; the paper instead uses high-dimensional embeddings and spectral properties of the generative model.Source: §B.1 · p. 15, §B.2 · p. 15

B.3 QUALITY-DIVERSITY FOR HIGH-DIMENSIONAL FEATURE SPACES

  • The framework uses DNS with fitness thresholding to reduce retention of invalid, out-of-distribution generative artifacts, while operating directly in high-dimensional embedding space avoids projection-related information loss.The paper contrasts this with DNS’s tendency to reward extremely novel solutions regardless of validity and reports that direct embedding-space optimisation reduces information loss associated with projection.Source: §B.3 · p. 15

B.4 IMAGE EDITING IN DIFFUSION MODELS

  • The method applies spectral analysis of the diffusion model’s Jacobian as a mutation operator, extending geometric insights from single-image editing to population search over noise and conditioning inputs.The paper notes deterministic inversion is needed to map intermediate-layer modifications back to initial noise as an evolvable genotype.Source: §B.4 · p. 16

B.5 IMAGE DIVERSITY IN DIFFUSION MODELS

  • Unlike methods targeting immediate batch variety, the archive-based approach accumulates semantic diversity over iterations and can extend diversity optimisation to one-step models such as SDXL-Turbo.The related-work discussion describes Diverse Diffusion, Particle Guidance, and CADS as inference-time approaches with distinct sampling or model applicability constraints.Source: §B.5 · p. 16

C.1 GLOBAL HYPERPARAMETERS

  • The SDXL-Turbo experiments used 2,000 evaluations, a 64-solution archive capacity, and mixed mutations of latent noise and prompt embeddings; Flux used 400 evaluations across 12 prompts.Flux also used 8-bit transformer quantisation, latent-only genotypes, and an untuned mutation strength α.Source: §C.1 · pp. 16–17

C.2 SPECTRAL-GUIDED DIFFUSION IMPLEMENTATION

  • The spectral mutation implementation estimates three principal Jacobian directions with at most five Power Iteration steps, then perturbs genotypes along a sampled direction using α from [−10, 10].For Flux, the genotype is restricted to latent embeddings, and the intermediate timestep is sampled from [0.2, 0.8].Source: §C.2 · p. 17, §C.1 · p. 17

C.3 BASELINE IMPLEMENTATIONS

The supplement describes baseline configurations, prompt sampling, and FID estimation, then reports that removing fitness thresholding increases diversity while reducing fitness and semantic validity.

C.3 BASELINE IMPLEMENTATIONS

  • Baseline configurations include tuned evolutionary operators and CMA-ES variants, while Adam uses 2,000 gradient steps and was omitted on Flux because of memory limits.The excerpt also describes QDHF’s projection-based embedding space, sampling-based diversity baselines, and ImageBreeder’s latent-only operators.Source: §C.3 · pp. 17–18

C.4 BASELINES EVALUATION

  • Algorithms were seeded with the same starting image for each prompt, and passive DNS archives collected non-QD outputs solely for comparable metric reporting.The passive archives did not affect the search process of Adam, CMA-ES, or random-seed methods.Source: §C.4 · p. 19

C.5 PARTI PROMPTS DISTRIBUTION

  • The Parti Prompts sample contains 36 prompts, with three sampled from each category; “Basic” and “Fine-grained Detail” are the most common challenge levels.Figure 7 depicts the category and challenge distributions.Source: §C.5 · p. 19, Figure 7 · p. 19

C.6.1 FID EXTRAPOLATION

  • FID∞ is estimated as the intercept of a linear fit of raw FID against 1/N, using subsamples from N = 200 to 2,000; methods with fewer than 200 samples are excluded.The extrapolation is used because archive sizes vary and FID is sensitive to sample size.Source: §C.6.1 · p. 19

D ABLATION EXPERIMENTS

D.1.1 NO FITNESS THRESHOLD

  • p < 2×10−10 for higher diversity and p < 1×10−2 for lower Total Fitness in the no-threshold ablation; its images also include semantically invalid examples.Figure 8 compares the ablation with the full method for “a knight holding a long sword”; the ablation can render a sword as a walking stick, while the full method maintains fitness and validity.Source: Figure 8 · p. 20, §D.1.1 · p. 20

D.1.2 FITNESS THRESHOLD ROBUSTNESS

The threshold robustness study finds smooth performance variation across tested values, with 0.3 serving as a stable balance between semantic integrity and diversity.

D.1.2 FITNESS THRESHOLD ROBUSTNESS

  • Lowering the threshold below 0.3 slightly increases diversity, while raising it above 0.3 monotonically reduces archive coverage and Total Fitness.Performance metrics vary smoothly across the tested range, and the authors identify 0.3 as an effective balance between semantic integrity and diversity.Source: §D.1.2 · pp. 20–21, Table 6 · p. 21

D.2 GENOTYPE ABLATION

The ablations compare genotype components and spectral-mutation settings. The mixed genotype combines strong fitness and diversity, while performance is generally stable across mutation strengths, spectral direction counts above three, and hidden-layer choices.

D.2 GENOTYPE ABLATION

  • Mixed Genotype and Latent Noise perform best on Total Fitness, while Mixed Genotype and Prompt Embedding achieve the highest Diversity; the mixed strategy ranks strongly on both axes.The results average across 36 prompts; Number of Solutions does not differ significantly among genotype strategies.Source: Table 7 · p. 21, §D.2 · p. 21

D.3 MUTATION STRENGTH (α) ABLATION

  • Halving or doubling mutation strength α has negligible effect on overall performance, with all tested configurations achieving comparable fitness and diversity.Table 8 compares the default magnitude with half and twice that magnitude.Source: §D.3 · p. 21, Table 8 · p. 22

D.4 SPECTRAL DIRECTION (k) ITERATION ABLATION

  • Performance plateaus at k = 3 spectral directions; only reducing k to 1 meaningfully lowers archive stability and fitness.The authors report diminishing returns for larger k values.Source: §D.4 · p. 22, Table 9 · p. 22

D.5 TARGETED HIDDEN LAYER ABLATION

  • Hidden-layer choice has minimal overall impact, though targeting decoder-side up blocks yields slightly lower stability and fitness than bottleneck or encoder-side layers.Table 10 compares spectral mutation across U-Net layers.Source: §D.5 · p. 22, Table 10 · p. 22

E.1 FRAMEWORK EXPERIMENT IMAGES

  • The supplementary figures show generated images from the framework baselines and their corresponding seed images for sample prompts on SDXL-Turbo and Flux.Source: §E.1 · p. 22

E.2 VARIATION OPERATOR EXPERIMENT IMAGES

The appendix presents sample outputs from variation-operator experiments, lists prompt categories and challenge levels, and describes the seed selection for wall-clock experiments.

E.2 VARIATION OPERATOR EXPERIMENT IMAGES

  • Figures 9–12 show top-16 outputs for sample prompts from SDXL-Turbo and Flux variation-operator experiments.Figures 11 and 12 include the seed image for SDXL-Turbo and Flux, respectively.Source: §E.2 · p. 22, Figure 9 · p. 23, Figure 10 · p. 24

F FULL LIST OF PROMPTS

G WALL-CLOCK TIME EXPERIMENTS

  • The wall-clock comparison uses 12 seeds, selecting one seed from each prompt category, and excludes CVT-ME and QDHF because they use Spectral Mutation.Source: G WALL-CLOCK TIME EXPERIMENTS · p. 27

G.1 PER-IMAGE GENERATION TIME

Spectral Mutation adds substantial per-image computation relative to unstructured noise-based mutation, with its practical cost depending on the application. The impact discussion also notes potential safety risks when exploration moves beyond high-likelihood modes.

G.1 PER-IMAGE GENERATION TIME

  • On Flux.1, Spectral Mutation costs 14,564s per image, versus 18,885s for Particle Guidance and 5,865s for Intermediate-layer Gaussian.The paper describes the Particle Guidance cost as the same order and the Intermediate-layer Gaussian comparison as roughly 2.5× lower.Source: §G.1 · pp. 27–28, Table 12 · p. 28
  • Jacobian-based spectral estimation is substantially more costly than unstructured noise-based mutation, making overhead a key limitation.The analysis describes meaningful overhead relative to a base-model forward pass.Source: §G.1 · pp. 27–28
  • The cost may matter less when downstream scientific validation dominates generation, but the paper does not evaluate those applications.The authors identify biological and materials design as examples where synthesis, assays, or physical testing may dominate resource budgets.Source: §G.1 · p. 28

H IMPACT STATEMENT

  • The impact statement presents diverse candidate generation as potentially useful in scientific discovery, including protein, material, and drug design.Source: H IMPACT STATEMENT · p. 28
  • Exploring beyond high-likelihood modes could circumvent safety alignment, so the authors encourage robust post-generation filtering in open-ended or user-facing applications.Source: H IMPACT STATEMENT · pp. 28–29

Related papers

Loading 2610.15724v1…