PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Pairwise Repulsion: Collective Diversity via Determinant Maximization in Diffusion Sampling

Kwanyoung Lee, SeungJu Cha, Yebin Ahn, Sungho Koh, Dong-Jin Kim

arXiv:2610.22267v1cs.CV

TL;DR

Strong guidance in text-to-image diffusion narrows generated samples to high-density modes, while pairwise diversity objectives can miss batch-level redundancy. DetNO optimizes initial noise using log-determinant RKHS volume with an anchor-based fidelity term. Across three backbones it achieves leading intra-batch diversity, alongside statistically significant few-shot accuracy gains in the low-shot regime.

  • Problem

    Strong guidance concentrates text-to-image samples on narrow, repeated modes, limiting synthetic data augmentation, while pairwise objectives respond weakly to higher-order batch redundancy.

    Source: §1 · p. 2

  • Method

    DetNO optimizes initial noise by maximizing the log-determinant RKHS volume of Tweedie predictions alongside an anchor-based fidelity term.

    Source: §1 · p. 3, §4.1 · p. 6

  • Results

    Across SD1.5, SDXL, and SD3, DetNO achieves leading intra-batch diversity in VendiScore and Mean Self-Similarity, with statistically significant few-shot accuracy gains in the low-shot regime.

    Source: §1 · p. 3

  • Takeaways & Limitations

    DetNO’s broader mode coverage provides downstream accuracy gains for synthetic data augmentation in the low-data regime.

    Source: §1 · p. 3, §6 · p. 14

Abstract

from arXiv · show

Strong guidance in text-to-image (T2I) diffusion models concentrates sampling toward high-density modes of the conditional distribution, collapsing independently sampled noise vectors onto nearly identical visual prototypes. This limits utility for tasks that demand distributional breadth, most critically synthetic data augmentation. Diversity, however, is a collective property of the whole batch: a set can preserve every pairwise distance yet still collapse onto a low-dimensional subspace. Existing sum-of-pairwise noise-optimization and repulsion objectives respond only weakly to this higher-order failure. We propose DetNO (Determinant Noise Optimization), which formulates diverse generation as RKHS volume maximization on the initial noise, maximizing the log-determinant of an RBF Gram matrix over one-step Tweedie predictions. This set-level functional equals the squared batch volume; its repulsion is strongest when the batch is most collapsed and self-attenuates as diversity is achieved. An anchor-based fidelity term keeps the optimized noises near plausible regions, and Lagrangian relaxation gives a two-term objective with a single hyperparameter $\lambda$ governing the diversity-quality trade-off, without training or sampler modification. Across three backbones (SD1.5, SDXL, SD3), DetNO achieves leading intra-batch diversity in VendiScore and Mean Self-Similarity, and a downstream few-shot classification experiment confirms that broader mode coverage brings statistically significant accuracy gains for synthetic data augmentation in the low-shot regime. Fixing the pipeline and varying only the batch objective, both RBF-based aggregators outperform InfoNCE, and the log-determinant attains the best downstream accuracy on both benchmarks. Its diversity advantage over InfoNCE widens as the batch grows. Code is available at https://github.com/mobled37/DetNO.

1 Introduction

Strong guidance concentrates text-to-image sampling on narrow, repeated visual modes, limiting synthetic data augmentation. DetNO instead optimizes batch-level diversity through RKHS volume maximization while retaining fidelity to unoptimized samples.

1 Introduction

  • Strong CFG concentrates sampling on high-density modes, producing near-identical images that limit synthetic data augmentation.Under deterministic samplers such as DDIM, the initial noise determines the final image.Source: §1 · p. 2
  • Pairwise objectives can miss higher-order redundancy because a batch may preserve pairwise distances yet collapse onto a low-dimensional subspace.Their gradients remain confined to pairwise directions and respond weakly to this failure mode.Source: §1 · pp. 2–3
  • DetNO maximizes the log-determinant of an RBF Gram matrix over Tweedie predictions to measure the batch’s collective RKHS volume.The objective amplifies repulsion when redundancy is high and relaxes it as diversity is achieved.Source: §1 · pp. 3–4
  • Across SD1.5, SDXL, and SD3, DetNO achieves leading intra-batch diversity in VendiScore and Mean Self-Similarity, with statistically significant few-shot accuracy gains in the low-shot regime.Its advantage over CNO widens with batch size.Source: §1 · pp. 3–4
  • An anchor-based fidelity term and a single hyperparameter λ balance diversity and quality without training or sampler modification.Fig. 2 illustrates the diversity and fidelity terms.Source: §1 · p. 3, Figure 2 · p. 3

2 Related Work

Prior diffusion-diversity methods apply repulsion during denoising, alter text embeddings, or optimize initial noise, but their diversity signals rely on pairwise interactions. DetNO retains initial-noise optimization while replacing that signal with set-level log-determinant maximization.

2.1 Recovering Mode Diversity in Diffusion Models.

  • SPELL, Particle Guidance, and DiversityPrompt intervene during denoising or optimize text embeddings, while CNO optimizes initial noise with InfoNCE; these approaches use pairwise diversity signals.The denoising and embedding methods incur per-step overhead that scales with trajectory length.Source: §2.1 · p. 4

2.2 Initial Noise Optimization in Diffusion Models.

  • DetNO inherits CNO’s initial-noise pipeline but uses log-determinant maximization to measure squared RKHS volume and penalize redundancy at every order.It applies the objective at inference time to frozen-model Tweedie predictions, without training or DPP sampling.Source: §2.2 · p. 4

3 Preliminaries

The preliminaries describe diffusion sampling and Tweedie predictions, then explain how classifier-free guidance sharpens conditional sampling. Tweedie estimates expose semantic structure at the initial timestep, while strong guidance suppresses low-density modes.

3.1 Tweedie’s Formula and Clean Data Estimation

  • Diffusion models iteratively denoise a Gaussian noise sample through a learned reverse process, with a network predicting noise from the noisy latent, timestep, and text condition.Tweedie’s formula estimates the posterior mean clean latent from a noisy observation; at the initial timestep its prediction still captures semantic structure, which DetNO optimizes before denoising.Source: §3.1 · p. 5

3.2 Classifier-Free Guidance

  • Classifier-free guidance interpolates conditional and unconditional noise predictions; stronger guidance sharpens sampling toward dominant conditional modes and suppresses low-density modes.This sharpness improves prompt fidelity while reducing diversity among images generated from the same prompt.Source: §3.2 · p. 5

4 Proposed Method

DetNO optimizes the initial noise batch using semantic Tweedie predictions, maximizing collective RKHS volume while constraining displacement from plausible reference predictions. Its determinant objective targets collapsed directions, and an anchor-based fidelity term balances diversity against quality.

4.1 Constrained Diversity Maximization on Initial Noise

  • Optimizing initial noise diversifies one-step Tweedie predictions, which carry semantic structure and propagate through deterministic sampling to the final images.The method combines RKHS volume maximization with an anchor-based displacement constraint through Lagrangian relaxation.Source: §4.1 · pp. 5–6

4.2 Diversity: RKHS Volume Maximization

  • Pooled, normalized Tweedie predictions feed an RBF Gram matrix, with pooling intended to emphasize semantic structure over high-frequency variation.The RBF kernel yields a positive-semidefinite Gram matrix, making log det(S + ϵI) well-defined.Source: §4.2 · pp. 6–7
  • The log-determinant maximizes squared RKHS batch volume, approaching zero when an embedding lies in the span of the others.Its repulsion concentrates along collapsed directions and self-attenuates as eigenvalues grow; the analysis reports Pearson r=0.999 between Gram-matrix and noise-space gradient amplification.Source: §4.2 · pp. 7–8

4.3 Fidelity: Anchor-Based Displacement Constraint

  • An anchor-based displacement constraint keeps each optimized prediction near its unoptimized reference, while λ controls the diversity–fidelity trade-off.The final objective applies fidelity to full-resolution predictions and diversity to pooled, normalized representations, without requiring sampler modification.Source: §4.3 · p. 8
  • In the 2D ring mixture, DetNO covers all 16 modes across the 8 inner and 8 outer modes while preserving their structure.CFG misses the outer modes, while CNO recovers them but distorts the inner ring, retaining only 4 inner modes.Source: Figure 3 · p. 9

5 Experiments

Across three backbones, DetNO improves intra-batch diversity while trading off some quality; few-shot classification results show downstream accuracy gains. Ablations examine the fidelity term, pooling window, and batch-size effects.

5 Experiments

  • The experiments compare methods under identical prompts, seeds, CFG scale, and hardware, using Precision, Density, CLIPScore, PickScore, and ImageReward for quality and four metrics for diversity.VendiScore and MSS are the primary diversity metrics because they measure intra-batch coverage without an external reference distribution.Source: §5 · p. 9

5.1 Results

  • DetNO achieves the best MSS and VendiScore on SD1.5, SDXL, and SD3, while PickScore decreases marginally and ImageReward matches or modestly trails the baseline.The text attributes decreases in reference-dependent metrics to their reward for overlap with the narrow MS-COCO reference set.Source: §5.1 · p. 10
  • On SD1.5 examples, DetNO varies object identity, viewpoint, and scene composition while remaining prompt-faithful, unlike the near-identical DDIM and CNO outputs.The experiments use 2,000 MS-COCO prompts, generating five images per prompt for SD1.5 and SD3 and three for SDXL.Source: §5 · p. 9, §5.1 · p. 11, Figure 4 · p. 11
  • DetNO improves few-shot accuracy over DDIM by +2.0pp at 1-shot on Pets and +1.7pp on CUB-200, with gains most pronounced in the low-data regime.CNO remains within noise of DDIM on both datasets; the evaluation uses synthetic images with a pretrained ResNet-18.Source: Table 2 · p. 10, §5.1 · p. 12
  • With the pipeline held fixed, both RBF-based aggregators outperform InfoNCE on intra-batch diversity, and the log-determinant performs best.The ablation varies only the batch aggregator; its downstream comparison uses the protocol of Table 2.Source: §5.1 · p. 12, Table 3 · p. 12

5.2 Ablation Studies

  • Across SD3 pooling-window sweeps, all tested configurations improve over the baseline on CLIPScore, PickScore, and ImageReward; smaller windows widen the diversity range but steepen the quality trade-off.The study selects w=32 as a balance across the three quality metrics.Source: Figure 5 · p. 12, §5.2 · p. 13
  • The full objective combines the diversity term’s stronger diversity but substantial quality cost with fidelity’s quality preservation and lack of diversity improvement.The full objective recovers most diversity gains while maintaining prompt-faithful outputs; without fidelity, unconstrained volume maximization can produce artifacts.Source: §5.2 · pp. 13–14, Figure 6 · p. 13
  • DetNO’s advantage over CNO is marginal at small batch sizes and widens monotonically as batch size grows, though the comparison also varies kernel and fidelity design.All-order and pairwise objectives coincide at B=2; the added matrix factorization costs O(B^3) per step, negligible relative to B denoiser evaluations.Source: §5.2 · p. 14, Figure 7 · p. 14

6 Conclusion

DetNO maximizes RKHS volume among Tweedie predictions by combining log-determinant diversity and anchor-based fidelity terms. Across three backbones, it improves diversity and yields statistically significant few-shot accuracy gains in the low-data regime.

6 Conclusion

  • DetNO maximizes RKHS volume among Tweedie predictions using a log-determinant diversity term and an anchor-based fidelity term.The conclusion describes an initial-noise optimization framework with a λ-controlled trade-off and no sampler modification.Source: §6 · p. 14
  • Across SD1.5, SDXL, and SD3, DetNO produces consistent diversity gains, while λ smoothly interpolates between diversity and fidelity objectives.A toy experiment and spectral analysis attribute the mechanism to repulsion that intensifies under collapse and relaxes as diversity is restored.Source: §6 · p. 14
  • In downstream few-shot classification, broader mode coverage accompanies statistically significant accuracy gains in the low-data regime.The conclusion presents batch-level diversity as producing image sets that are measurably more useful.Source: §6 · p. 14

A.1 Proof of Proposition 1

The proof distinguishes conditions ensuring that the regularized log-determinant is defined from the higher-dimensional assumption used to make eigenvalue bounds non-vacuous. Its gradient analysis relates small Gram-matrix eigenvalues to stronger determinant repulsion.

A.1 Proof of Proposition 1

  • Strict positive definiteness of the RBF kernel on distinct inputs makes S invertible and log det(S + ϵI) well-defined in any dimension.The separate assumption d ≫ B is used for eigenvalue bounds, ensuring generic points are in general position and bounds are non-vacuous.Source: §A.1 · p. 18
  • The determinant-gradient norm depends on the Gram eigenvalues through the sum of 1/(µ_k + ϵ)^2, whereas the pairwise-gradient norm scales as 1/2√(B(B−1)).The proof derives these quantities from the inverse of S + ϵI and the off-diagonal entries of the pairwise gradient matrix.Source: §A.1 · p. 18
  • The determinant-to-pairwise gradient ratio decreases as eigenvalues grow, concentrating gradient energy where redundancy is highest and relaxing repulsion as diversity is achieved.The measured noise-gradient ratio is strongly coupled to the Gram-matrix-level amplification (Pearson r = 0.999; R² = 0.999).Source: §A.1 · p. 19

A.2 Derivation of the Two-Term Objective via Lagrangian Relaxation

The section derives a two-term objective by relaxing per-sample fidelity constraints on an RKHS batch-diversity objective. A shared λ balances log-determinant diversity against displacement from reference anchors.

A.2 Derivation of the Two-Term Objective via Lagrangian Relaxation

  • DetNO maximizes batch diversity in the RKHS of Tweedie prediction space by minimizing −log det(S+ϵI), subject to bounded displacement from each reference anchor.The constraint bounds each optimized Tweedie prediction’s displacement from its reference prediction.Source: §A.2 · p. 19
  • Symmetric per-sample constraints motivate using a shared Lagrange multiplier λ across all samples.The samples share the text condition, backbone, and noise prior.Source: §A.2 · p. 19
  • Dropping a constant independent of ZT yields a two-term objective, with λ ≥0 governing the diversity–fidelity trade-off.The terms are log-determinant diversity and summed squared displacement from reference predictions.Source: §A.2 · p. 19

A.3 Empirical Validation of Spectral Gradient Amplification.

The SD1.5 optimization analysis tests the predicted spectral-gradient amplification during DetNO. The gradient ratio exceeds 1 significantly at most steps, while declining later as effective rank increases.

A.3 Empirical Validation of Spectral Gradient Amplification.

  • The analysis used SD1.5 with B=5, 20 MS-COCO prompts, 3 seeds, and 10 optimization steps, extending beyond deployed Nopt=3.The longer run traces the predicted rise and decay of the gradient ratio across steps.Source: §A.3 · pp. 19–20
  • Rz exceeds 1 at 8/10 optimization steps with statistical significance.The two nonsignificant steps are Steps 1 and 2.Source: §A.3 · p. 20, Table S1 · p. 20
  • From Step 5 onward, Rz decreases monotonically as effective rank increases, consistent with the predicted self-attenuation.The first two nonsignificant steps coincide with transient rises in µmin and effective rank before both stabilize.Source: §A.3 · p. 20

A.4 Hyperparameter Configuration

These appendix experiments specify per-backbone settings and examine diversity propagation, kernel bandwidth, compute, gradient strategy, and noise scale. DetNO’s initial diversity persists through denoising, while the ablations identify practical trade-offs and boundaries.

A.4 Hyperparameter Configuration

  • Batch size is B=5 for SD1.5 and SD3 and B=3 for SDXL; NFE is 50 DDIM steps for SD1.5/SDXL and 28 Euler steps for SD3.The pooling window is larger for SD3; λ can be adjusted independently to control the diversity–quality trade-off.Source: §A.4 · p. 20

A.5 Diversity Propagation Across Timesteps

  • At t≈0, DetNO’s cosine similarity is 0.026 versus DDIM’s 0.120 and CNO’s 0.101; at t=T it is 0.403 versus DDIM’s 0.548.Lower cosine similarity indicates greater diversity; the initial advantage is preserved and widens through denoising.Source: §A.5 · p. 21, Table S3 · p. 21

A.6 Effect of Kernel Bandwidth σ

  • The fixed median bandwidth achieves the best MSS and VendiScore and the highest Precision, Density, and Coverage among tested bandwidths.The heuristic is computed once from reference predictions; small fixed bandwidths give weak diversity signals, while σ = 1.0 and dynamic median incur moderate quality cost.Source: §A.6 · pp. 21–22, Table S4 · p. 21

A.7 Computational Overhead

  • DetNO matches CNO’s 28.3 GB GPU memory and Nopt+1 extra forward passes; marginal time is 17.8 s versus 17.7 s for CNO.The reported time difference comes from Gram-matrix computation and inversion, negligible relative to UNet forward passes.Source: §A.7 · p. 22, Table S5 · p. 22

A.8 Effect of Gradient Computation Strategy.

  • Stop-gradient yields clear diversity gains at marginal quality cost; full gradient gives moderately stronger gains, especially in Recall and VendiScore, but requires storing intermediate UNet activations.Stop-gradient closes most of the gap while reducing peak memory to standard-inference levels, making it the practical default when memory is constrained.Source: §A.8 · p. 22, Table S6 · p. 22

A.9 Ablation on Noise Scale

  • Any tested noise-scale deviation from s = 1 severely degrades quality; at s = 0.5, Recall is 0.0002 and ImageReward is −2.28.For s ≥1.2, Recall falls below 0.04, CLIPScore drops by more than 8 points, and ImageReward becomes strongly negative.Source: §A.9 · p. 23, Table S7 · p. 23

Related papers

Loading 2610.22267v1…